Electronic device and imaging device

Abstract

[Problem] To obtain a high-quality captured image even when the amount of incident light is small. [Solution] This electronic apparatus is provided with: a display unit; a first imaging unit disposed on a side of the display unit opposite a display surface thereof and capable of imaging an infrared wavelength band of light that has passed through the display unit; a second imaging unit disposed on a side of the display unit opposite the display surface and capable of imaging a visible wavelength band of light that has passed through the display unit; and a correction unit which, on the basis of image data obtained by the imaging performed by the first imaging unit, corrects image data obtained by the imaging performed by the second imaging unit.

Description

Technical Field

[0001] The present disclosure relates to an electronic device and an imaging device.

Background Art

[0002] In recent electronic devices such as smartphones, mobile phones, and PCs (Personal Computers), various sensors such as cameras are mounted on the bezel of the display panel. On the other hand, there is a demand to make the external size of the electronic device as compact as possible without affecting the screen size, and the bezel width tends to narrow. Against this background, a technique has been proposed in which a camera module is disposed directly below the display panel and subject light that has passed through the display panel is captured by the camera module.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, the display panel is composed of multiple layers, and some of the layers have a low visible light transmittance. Therefore, when subject light that has passed through the display panel is captured by the camera module, the captured image becomes dark or the entire image becomes blurred. In addition, when the subject light passes through the display panel, it may be affected by flare and diffraction, resulting in a degradation of the image quality of the captured image.

[0005] In addition, when a camera module is disposed on the surface of a small electronic device such as a smartphone, the lens cannot be made thin and the lens aperture cannot be made large. Therefore, in a situation where the surroundings are dark, the captured image becomes dark and an unclear image is likely to be obtained.

[0006] The present disclosure provides an electronic device and an imaging device capable of obtaining high-quality captured images even when the amount of incident light is small.

Means for Solving the Problems

[0007] In order to solve the above problems, according to the present disclosure, there is provided a display unit, a first imaging unit disposed on the side opposite to the display surface of the display unit and capable of imaging light in the infrared wavelength band that has passed through the display unit, a second imaging unit disposed on the side opposite to the display surface of the display unit and capable of imaging light in the visible wavelength band that has passed through the display unit, and a correction unit that corrects the image data captured by the second imaging unit based on the image data captured by the first imaging unit. An electronic device is provided.

[0008] The correction unit may correct the sensitivity of the image data captured by the second imaging unit based on the image data captured by the first imaging unit.

[0009] A learning unit that learns the correlation between the sensitivity of the image data captured by the first imaging unit and the sensitivity of the image data captured by the second imaging unit is provided, and the correction unit may correct the sensitivity of the image data captured by the second imaging unit based on the image data captured by the first imaging unit with reference to the learning result of the learning unit.

[0010] The correction unit may correct the resolution of the image data captured by the second imaging unit based on the image data captured by the first imaging unit.

[0011] A learning unit that learns the correlation between the resolution of the image data captured by the first imaging unit and the resolution of the image data captured by the second imaging unit is provided, and the correction unit may correct the resolution of the image data captured by the second imaging unit based on the image data captured by the first imaging unit with reference to the learning result of the learning unit.

[0012] The correction unit may correct at least one of a flare component and a diffracted light component included in the image data captured by the second imaging unit based on the image data captured by the first imaging unit.

[0013] A learning unit that learns a correlation relationship between at least one of a flare component and a diffracted light component included in the image data captured by the first imaging unit and at least one of a flare component and a diffracted light component included in the image data captured by the second imaging unit is provided. The correction unit may correct at least one of a flare component and a diffracted light component included in the image data captured by the second imaging unit based on the image data captured by the first imaging unit with reference to the learning result of the learning unit.

[0014] A reference determination unit that determines whether at least one of the sensitivity, resolution, flare component, and diffracted light component of the image data captured by the second imaging unit satisfies a predetermined first reference. An imaging start instruction unit that starts imaging by the first imaging unit when it is determined by the reference determination unit that the first reference is not satisfied. Based on the result of comparing at least one of sensitivity, resolution, flare component, and diffracted light component between the image data captured by the first imaging unit and the image data captured by the second imaging unit, a correction procedure determination unit that determines whether to perform correction by the correction unit and the type of image data serving as a reference for correction when performing correction by the correction unit may be provided.

[0015] When the type of image data serving as a reference for correction is determined by the correction procedure determination unit, the learning unit may learn a correlation relationship between at least one of the sensitivity, resolution, flare component, and diffracted light component of the determined image data and at least one of the sensitivity, resolution, flare component, and diffracted light component of the image data captured by the second imaging unit.

[0016] A sensor that detects at least one of the shape and color of an object. A reliability estimation unit that estimates the reliability of the learning by the learning unit. When the reliability estimated by the reliability estimation unit is equal to or lower than a predetermined second criterion, an object identification determination unit that determines whether the object can be identified based on the detection data of the sensor; When it is determined by the object identification determination unit that the object can be identified, a color identification determination unit that determines whether the color of the object identified by the sensor can be specified; When it is determined by the color identification determination unit that the color of the object can be specified, the correction unit may correct the image data captured by the second imaging unit so as to approach the specified color.

[0017] The correction unit may increase the degree of noise removal of a pixel region in the image data captured by the second imaging unit where the luminance change is equal to or lower than a predetermined reference value compared to the degree of noise removal of a pixel region in the image data where the luminance change is greater than the reference value.

[0018] A light emitting unit that emits light in an infrared light wavelength band; A light emission control unit that controls the light emission timing of the light emitting unit so that the subject is illuminated by the light emitted by the light emitting unit when the first imaging unit captures image data.

[0019] The light emitting unit has a plurality of light sources that emit light in different light emission wavelength bands within the infrared light wavelength band; The light emission control unit sequentially switches and controls the light emission by the plurality of light sources while the first imaging unit is capturing an image; The first imaging unit outputs a plurality of image data captured in different light emission wavelength bands; The correction unit may correct the image data captured by the second imaging unit based on the plurality of image data.

[0020] The light emitting unit may be disposed on the display surface side of the display unit.

[0021] At least one of the first imaging unit and the second imaging unit may include pixels that capture light in an infrared light wavelength band and pixels that capture light in a visible light wavelength band.

[0022] The first imaging unit may have sensitivity to light of 550 nm or more.

[0023] The correction unit may increase the degree of correction as the wavelength becomes shorter with respect to the image data captured by the second imaging unit.

[0024] The first imaging unit may have a photoelectric conversion unit that is arranged longer in the normal direction of the light incident surface than the second imaging unit.

[0025] The area in the light incident surface direction per pixel of the first imaging unit is larger than the area in the light incident surface direction per pixel of the second imaging unit, and The area in the light incident surface direction for all the pixels of the first imaging unit may be smaller than the area in the light incident surface direction for all the pixels of the second imaging unit.

[0026] In another aspect of the present disclosure, a first imaging unit that is arranged on the side opposite to the display surface of the display unit and can capture light in the infrared wavelength band, and a second imaging unit that is arranged on the side opposite to the display surface of the display unit and can capture light in the visible light wavelength band, and a correction unit that corrects the image data captured by the second imaging unit based on the image data captured by the first imaging unit, are provided.

[0027] In another aspect of the present disclosure, a first imaging unit that can capture light in the infrared wavelength band, and a second imaging unit that can capture light in the visible light wavelength band, and a correction unit that corrects the image data captured by the second imaging unit based on the image data captured by the first imaging unit, are provided.

Brief Description of the Drawings

[0028]

Figure 1

Figure 2

Figure 3A

Figure 3B

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10A

Figure 10B

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16A

Figure 16B

Figure 17A

Figure 17B

Figure 18A

Figure 18B

Figure 19

Figure 20

Embodiments for Carrying Out the Invention

[0029] Hereinafter, embodiments of an electronic device and an imaging device will be described with reference to the drawings. Hereinafter, the description will focus on the main components of the electronic device and the imaging device, but there may be components and functions that are not shown or described in the electronic device and the imaging device. The following description does not exclude components and functions that are not shown or described.

[0030] (First Embodiment) FIG. 1 is an external view and a cross-sectional view taken along line A-A of an electronic device 2 equipped with an imaging device 1 according to the first embodiment, and FIG. 2 is a cross-sectional view taken along line B-B of FIG. 1. The electronic device 2 according to the present embodiment is an arbitrary electronic device 2 having both a display function and a photographing function, such as a smartphone, a mobile phone, a tablet, or a PC. The electronic device 2 in FIG. 1 includes camera modules (first and second imaging units) 4 and 5 disposed on the side opposite to the display surface 3a of the display unit 3. Thus, the electronic device 2 in FIG. 1 is provided with the camera modules 4 and 5 on the back side of the display surface 3a of the display unit 3. Therefore, the camera modules 4 and 5 will perform photographing through the display unit 3.

[0031] The imaging device 1 according to this embodiment includes a plurality of camera modules 4 and 5. In this specification, an example in which two camera modules 4 and 5 are provided will be mainly described, but three or more camera modules may be provided. Hereinafter, an example in which the imaging device 1 includes two camera modules 4 and 5 will be mainly described. As will be described later, the camera module 4 has a first imaging unit 6, and the camera module 5 has a second imaging unit 7. The first imaging unit 6 can image light in the infrared wavelength band that has passed through the display unit 3. The infrared wavelength band is, for example, a wavelength band within the range of 780 nm to 1000 nm. The second imaging unit 7 can image light in the visible light wavelength band that has passed through the display unit 3. The visible light wavelength band is, for example, a wavelength band within the range of 380 nm to 750 nm.

[0032] The electronic device 2 according to this embodiment has a display surface 3a that extends up to the vicinity of the outer size of the electronic device 2, and the width of the bezel 3b around the display surface 3a is set to several millimeters or less. Usually, a front camera is often mounted on the bezel 3b. However, in FIG. 1, as shown by the dashed circle, a plurality of camera modules 4 and 5 that function as a front camera are arranged on the back side of the substantially central portion of the display surface 3a. In this way, by providing the front camera on the back side of the display surface 3a, it is not necessary to arrange the front camera on the bezel 3b, and the width of the bezel 3b can be narrowed.

[0033] In FIG. 1, the camera modules 4 and 5 are arranged on the back side of the substantially central portion of the display surface 3a. However, the camera modules 4 and 5 may be arranged at any location on the back side of the display surface 3a. For example, the camera modules 4 and 5 may be arranged on the back side near the peripheral edge of the display surface 3a. In this way, the plurality of camera modules 4 and 5 in this embodiment are arranged at any position on the back side that overlaps the display surface 3a.

[0034] As shown in FIGS. 1 and 2, the display unit 3 is a laminate in which a protective film 3c, a polyimide substrate 3d, a display layer 3e, a barrier layer 3f, a touch sensor layer 3g, an adhesive layer 3h, a circular polarizing plate 3i, an optical adhesive sheet (OCA: Optical Clear Adhesive) 3j, and a cover glass 3k are laminated in this order. The display layer 3e may be, for example, an OLED (Organic Light Emitting Device) display layer, a liquid crystal display layer, a MicroLED, or a display layer based on other display principles. The display layer 3e may also be composed of a plurality of layers. For example, the display layer 3e may include a color filter layer, a backlight layer, and the like. The display unit 3 performs display using light in the visible light wavelength range, but the light displayed on the display unit 3 may contain an infrared light component.

[0035] The barrier layer 3f is a layer that prevents oxygen and moisture from entering the display layer 3e. A touch sensor is incorporated in the touch sensor layer 3g. There are various types of touch sensors, such as a capacitance type and a resistive film type, and any type may be adopted. Also, the touch sensor layer 3g and the display layer 3e may be integrated.

[0036] The adhesive layer 3h is provided to bond the circular polarizing plate 3i and the touch sensor layer 3g. A material with a high visible light transmittance is used for the adhesive layer 3h. The circular polarizing plate 3i is provided to reduce glare and enhance the visibility of the display surface 3a even in a bright environment. The optical adhesive sheet 3j is provided to enhance the adhesion between the circular polarizing plate 3i and the cover glass 3k. A material with a high visible light transmittance is used for the optical adhesive sheet 3j. The cover glass 3k is provided to protect the display layer 3e and the like. Note that the layer configuration of the display unit 3 is not necessarily limited to that shown in FIGS. 1 and 2.

[0037] As shown in FIG. 2, the camera module 4 has a first optical system 8 associated with the first imaging unit 6. Similarly, the camera modules 4 and 5 have a second optical system 9 associated with the second imaging unit 7. The first and second optical systems 8 and 9 are disposed on the light incident surface side of the first and second imaging units 6 and 7, that is, on the side closer to the display unit 3, and condense the light that has passed through the display unit 3 onto the first and second imaging units 6 and 7. The optical system 5 is usually composed of a plurality of lenses, but the specific optical configuration of the optical system is not limited.

[0038] As will be described later, the first imaging unit 6 has a first photoelectric conversion unit, and the second imaging unit 7 has a second photoelectric conversion unit. The first photoelectric conversion unit and the second photoelectric conversion unit photoelectrically convert the light incident through the display unit 3. The first photoelectric conversion unit and the second photoelectric conversion unit have different light wavelength bands capable of photoelectric conversion. The first photoelectric conversion unit mainly photoelectrically converts light in the infrared light wavelength band, and the second photoelectric conversion unit mainly photoelectrically converts light in the visible light wavelength band. The first photoelectric conversion unit and the second photoelectric conversion unit may be CMOS (Complementary Metal Oxide Semiconductor) sensors or CCD (Charge Coupled Device) sensors. Further, the photoelectric conversion unit may be a photodiode or an organic photoelectric conversion film.

[0039] The first and second photoelectric conversion units have a photoelectric conversion element such as a CMOS sensor for each pixel. Each pixel can be arranged in an arbitrary manner. Specifically, the arrangement method of each pixel may be a Bayer arrangement, an interline arrangement, a checkerboard arrangement, a stripe arrangement, or other arrangements.

[0040] As shown in FIGS. 1 and 2, in the electronic device 2 of the present embodiment, a plurality of camera modules 4 and 5 image the subject light that has passed through the display unit 3. As shown in FIG. 1, the display unit 3 is formed of a plurality of layers. Each layer has no problem as long as the transmittance of light in the wavelength band in which the first imaging unit 6 and the second imaging unit 7 have sufficient sensitivity. However, in practice, the transmittance of some layers may be low. For example, the polyimide substrate 3d has a relatively low visible light transmittance, while the infrared light transmittance is higher than the visible light transmittance.

[0041] Therefore, in the present embodiment, a correction unit described later is provided to correct the imaging image data of the second imaging unit 7 that images light in the visible light wavelength band based on the imaging image data of the first imaging unit 6 that images light in the infrared light wavelength band.

[0042] FIG. 3A is a cross-sectional view showing an example of the cross-sectional structure of the first imaging unit 6, and FIG. 3B is a cross-sectional view showing an example of the cross-sectional structure of the second imaging unit 7. The first imaging unit 6 shown in FIG. 3A includes a first photoelectric conversion unit 12a formed in a semiconductor substrate 11, and the first photoelectric conversion unit 12a is separated by an element isolation layer 13 for each pixel. A planarization layer 14 is disposed on the first photoelectric conversion unit 12a, and an on-chip lens 15a is disposed thereon. Light is incident through the on-chip lens 15a. Therefore, the on-chip lens 15a becomes the light incident surface. In this specification, the side on which the on-chip lens 15a is disposed is referred to as the back surface side of the first imaging unit 6.

[0043] On the other hand, the second imaging unit 7 shown in FIG. 3B includes a second photoelectric conversion unit 12b formed in a semiconductor substrate, and the second photoelectric conversion unit 12b is separated by an element isolation layer 13 for each pixel. A planarization layer 14 is disposed on the second photoelectric conversion unit 12b, and a color filter layer 16 is disposed thereon. The color filter layer 16 may have three-color filter layers of RGB, or may have filter layers of cyan, magenta, and yellow, which are its complementary colors. An on-chip lens 15b is disposed on the color filter layer 16.

[0044] As can be seen by comparing the cross-sectional views of FIGS. 3A and 3B, the first photoelectric conversion unit 12a of the first imaging unit 6 is longer in the length in the normal direction of the light incident surface than the second photoelectric conversion unit 12b of the second imaging unit 7. This is because the photoelectric conversion efficiency of light in the infrared light wavelength band is worse than that of light in the visible light wavelength band, and by making the length in the normal direction of the light incident surface of the first photoelectric conversion unit 12a longer, the photoelectric conversion efficiency is improved.

[0045] In addition, since infrared light is less likely to be refracted than visible light, as shown in FIGS. 3A and 3B, the curvature of the on-chip lens 15a of the first imaging unit 6 may be made larger than the curvature of the on-chip lens 15b of the second imaging unit 7.

[0046] The area in the light incident surface direction per pixel of the first imaging unit 6 may be made larger than the area in the light incident surface direction per pixel of the second imaging unit 7. The first imaging unit 6 is provided to correct the sensitivity and the like of the captured image of the second imaging unit 7, and it is desirable that the area be larger because the sensitivity can be improved.

[0047] On the other hand, the area in the light incident surface direction (effective pixel area) obtained by combining all the pixels of the second imaging unit 7 may be larger than the area in the light incident surface direction (effective pixel area) obtained by combining all the pixels of the first imaging unit 6. The first imaging unit 6 is for correcting the captured image of the second imaging unit 7, and for example, it may be provided at a ratio of one pixel to a plurality of pixels of the second imaging unit 7. Note that the sizes and the number of pixels of the first imaging unit 6 and the second imaging unit 7 are not limited to those described above. Specific layout arrangement examples of the first imaging unit 6 and the second imaging unit 7 will be described later.

[0048] On the surface side (the side opposite to the on-chip lenses 15a and 15b) of the first imaging unit 6 and the second imaging unit 7, a readout circuit 17 is formed on the semiconductor substrate 11, and the periphery of the readout circuit 17 is covered with an interlayer insulating film 18. The readout circuit 17 includes a transfer transistor, a reset transistor, an amplification transistor, a selection transistor, and the like. Note that the cross-sectional structures of the first imaging unit 6 and the second imaging unit 7 are not limited to those shown in FIG. 3.

[0049] FIG. 4 is a block diagram showing the internal configuration of the electronic device 2 according to the first embodiment. As shown in FIG. 4, the electronic device 2 includes an imaging device 1, an application processor 21, a video signal generation unit 22, an A / D conversion unit 23, a display control unit 24, and a display unit 3.

[0050] The imaging device 1 can be composed of one or more semiconductor devices. More specifically, the imaging device 1 includes first and second camera modules 4 and 5, a first optical system 8 corresponding to the first imaging unit 6, a second optical system 9 corresponding to the second imaging unit 7 and an IR (Infrared Ray) cut filter 10, a first A / D conversion unit 31 corresponding to the first imaging unit 6, a second A / D conversion unit 32 corresponding to the second imaging unit 7, a signal processing unit 33, an imaging control unit 34, and an output unit 35. The first and second camera modules 4 and 5 may be mounted on a common substrate.

[0051] The first A / D conversion unit 31 converts the analog pixel signal photoelectrically converted by the first photoelectric conversion unit 12a into digital pixel data. The second A / D conversion unit 32 converts the analog pixel signal photoelectrically converted by the second photoelectric conversion unit 12b into digital pixel data.

[0052] As will be described later, the signal processing unit 33 generates image data corresponding to the captured images by the first imaging unit 6 and the second imaging unit 7. The signal processing unit 33 functions as a correction unit that corrects the sensitivity of the image data captured by the second imaging unit 7 based on the image data captured by the first imaging unit 6. The imaging control unit 34 controls whether to perform imaging by the first imaging unit 6 according to an instruction from the signal processing unit 33.

[0053] The application processor 21 is a semiconductor device separate from the first and second camera modules 4 and 5, and is mounted on the same or a different substrate as the first and second camera modules 4 and 5. The application processor 21 has a CPU (Central Processing Unit) etc. inside, and executes programs such as an operating system and various application softwares. The application processor 21 may be equipped with functions for performing image processing and signal processing such as a GPU (Graphics Processing Unit) and a baseband processor. The application processor 21 executes various processes as necessary on the input image data and calculation results, controls the display of an image on the display unit 3 of the electronic device 2, or transmits it to an external cloud server via a predetermined network.

[0054] The video signal generation unit 22 generates a video signal for display on the display unit 3. The A / D conversion unit 23 converts the video signal into digital pixel data. The display control unit 24 controls the display of the digital pixel data on the display unit 3.

[0055] FIG. 5 is a flowchart showing a first example of the processing operation of the signal processing unit 33 according to the first embodiment. The flowchart of FIG. 5 corrects the sensitivity of the image data captured by the second imaging unit 7 based on the image data captured by the first imaging unit 6.

[0056] First, it is determined whether or not the pixel value of the image data captured by the second imaging unit 7 is equal to or less than a predetermined threshold (step S1). Here, the average pixel value of the image data captured by the second imaging unit 7 may be compared with the predetermined threshold, or the pixel values of a partial pixel region of the image data captured by the second imaging unit 7 may be compared with the predetermined threshold.

[0057] If it is determined in step S1 that the value is not less than a predetermined threshold, it can be estimated that the image data captured by the second imaging unit 7 has sufficient sensitivity. Therefore, based on the image data captured by the second imaging unit 7, predetermined signal processing is performed to generate and output RGB data (step S2).

[0058] On the other hand, if it is determined in step S1 that the value is less than or equal to the predetermined threshold, it can be estimated that the image data captured by the second imaging unit 7 has insufficient sensitivity. Therefore, via the imaging control unit, the first imaging unit 6 is instructed to start imaging (step S3). The first imaging unit 6 may have a plurality of imaging sensors capable of separately imaging light in a plurality of narrow wavelength bands (for example, 750 nm, 800 nm, 850 nm, etc.) within the infrared light wavelength band.

[0059] Thereafter, the pixel values of the image data captured by the first imaging unit 6 and the pixel values of the image data captured by the second imaging unit 7 are compared (step S4). In this step S4, for example, it is determined whether the pixel value of the image data captured by the first imaging unit 6 is equal to or greater than X times (X is a predetermined reference value greater than 1) the pixel value of the image data captured by the second imaging unit 7. The value of X may be set and changed by the user.

[0060] If it is determined as NO in step S4, it can be estimated that the image data captured by the first imaging unit 6 does not have very high sensitivity. Therefore, the process of step S2 described above is performed without performing correction processing. If it is determined as YES in step S4, it can be estimated that the sensitivity of the image data captured by the first imaging unit 6 is sufficiently high. Therefore, it is determined whether there are pixels with pixel values greater than or equal to a predetermined threshold in the image data captured by the second imaging unit 7 (step S5). The reason for providing the determination process of step S5 is that if there is pixel data with a large pixel value in the captured image data of the second imaging unit 7 that captures light in the visible light wavelength band, the pixel data is regarded as highly sensitive and valid data.

[0061] When it is determined as YES in step S5, among the image data captured by the second imaging unit 7, pixel data with a pixel value equal to or greater than a predetermined threshold value and the image data captured by the first imaging unit 6 are selected as valid data (step S6).

[0062] When it is determined as NO in step S5, since it can be estimated that the image data captured by the second imaging unit 7 is not reliable, the image data captured by the first imaging unit 6 is selected as valid data (step S7).

[0063] Next, based on the valid data selected in step S6 or S7, the image data captured by the second imaging unit 7 is corrected (step S8). Here, for example, a learning model that has learned in advance the correlation between the sensitivity of the image data captured by the first imaging unit 6 and the sensitivity of the image data captured by the second imaging unit 7 is generated. For example, using a CNN (Convolution Neural Network), features of an image for appropriately performing sensitivity correction are extracted, and a learning model is generated that can automatically correct the sensitivity of the image data captured by the second imaging unit 7 using the image data captured by the first imaging unit 6. A series of processes for generating the learning model is called machine learning. By inputting the valid data selected in step S6 or S7 to the learning model for which sufficient learning has been performed and performing arithmetic processing, corrected image data is output from the learning model. The output image data is data for which sensitivity correction has been appropriately performed for each RGB color component.

[0064] In this way, when it is estimated that the pixel value of the image data captured by the second imaging unit 7 is large and highly reliable, the signal processing unit 33 outputs it without correcting the sensitivity, and when it is estimated that the pixel value is small and highly reliable, based on the image data captured by the first imaging unit 6, the sensitivity of the image data captured by the second imaging unit 7 is corrected. When performing the correction, the correlation between the sensitivity of the image data captured by the first imaging unit 6 and the sensitivity of the image data captured by the second imaging unit 7 is learned in advance, and the correction is performed using the learning result.

[0065] In FIG. 5, an example of correcting the sensitivity of the image data captured by the second imaging unit 7 was described. However, the same processing procedure can also be applied when correcting the resolution of the image data captured by the second imaging unit 7. The decrease in resolution occurs because part of the light that should be incident on the second camera module 5 is lost as it passes through the display unit 3. Therefore, by performing the same processing as in FIG. 5, the resolution of the image data captured by the second imaging unit 7 can be improved by correcting the image data captured by the second imaging unit 7 based on the image data captured by the first imaging unit 6. At that time, the correction process in step S8 needs to perform a process different from the sensitivity correction. That is, in the case of sensitivity correction, a learning model for appropriately correcting the sensitivity was generated. In the case of resolution correction, a learning model for appropriately correcting the resolution is generated, and when sufficient learning of the learning model has been performed, the effective data selected in step S6 or S7 is input to the learning model to obtain resolution-corrected image data. More specifically, the learning model for correcting the resolution learns the correlation between the resolution of the image data captured by the first imaging unit 6 and the resolution of the image data captured by the second imaging unit 7. The signal processing unit 33 that functions as a correction unit refers to the learning result in the learning unit and corrects the resolution of the image data captured by the second imaging unit 7 based on the image data captured by the first imaging unit 6.

[0066] The signal processing unit 33 that functions as a correction unit may increase the degree of correction toward the short-wavelength side for the image data captured by the second imaging unit 7. This is because the subject light passing through the display unit 3 is more likely to be absorbed by the display unit 3 in the blue component.

[0067] In the above, an example of performing sensitivity correction or resolution correction on the image data captured by the second imaging unit 7 was described. However, reflection or diffraction may occur while the subject light passes through the display unit 3, and there is a possibility that the subject light affected by flare due to reflection or diffraction is incident on the first and second camera modules 4 and 5. Therefore, the signal processing unit 33 may perform a process of correcting the influence of flare or diffracted light.

[0068] FIG. 6 is a flowchart showing a second example of the processing operation of the signal processing unit 33. The flowchart of FIG. 6 corrects the influence of flare or diffraction on the image data captured by the second imaging unit 7 based on the image data captured by the first imaging unit 6.

[0069] First, it is determined whether the pixel value of the image data captured by the second imaging unit 7 is equal to or greater than a predetermined threshold (step S11). When the light incident on the second imaging unit 7 is affected by flare or diffraction, generally the pixel value becomes large. Therefore, in step S11, it is determined whether the pixel value of the image data captured by the second imaging unit 7 is equal to or greater than a predetermined threshold. If it is less than the threshold, it is determined that it is not affected by flare or diffraction, and predetermined signal processing is performed based on the image data captured by the second imaging unit 7 to generate and output RGB data (step S12).

[0070] On the other hand, when it is determined in step S11 that the value is equal to or greater than the predetermined threshold, since it can be estimated that the image data captured by the second imaging unit 7 is affected by flare or diffraction, an imaging start is instructed to the first imaging unit 6 via the imaging control unit (step S13).

[0071] Thereafter, the pixel value of the image data captured by the first imaging unit 6 is compared with the pixel value of the image data captured by the second imaging unit 7 (step S14). Here, similar to step S4 in FIG. 5, for example, it is determined whether the pixel value of the image data captured by the first imaging unit 6 is equal to or less than X times (X is a predetermined reference value greater than 1) the pixel value of the image data captured by the second imaging unit 7.

[0072] When it is determined as NO in step S14, it can be estimated that the influence of flare or diffraction on the image data captured by the second imaging unit 7 cannot be corrected even by using the image data captured by the first imaging unit 6. Therefore, the process of step S12 is performed without performing the correction process. When it is determined as YES in step S14, since it can be estimated that the image data captured by the first imaging unit 6 is not affected by flare or diffraction, it is determined whether there are pixels with pixel values equal to or less than a predetermined threshold in the image data captured by the second imaging unit 7 (step S15).

[0073] When it is determined as YES in step S15, among the image data captured by the second imaging unit 7, pixel data with a pixel value equal to or less than a predetermined threshold value and the image data captured by the first imaging unit 6 are selected as valid data (step S16).

[0074] When it is determined as NO in step S15, since the image data captured by the second imaging unit 7 can be estimated as unreliable, the image data captured by the first imaging unit 6 is selected as valid data (step S17).

[0075] Next, based on the valid data selected in step S16 or S17, the image data captured by the second imaging unit 7 is corrected (step S18). Here, for example, a learning model is generated in advance by learning the correlation between at least one of the flare component and the diffracted light component included in the image data captured by the first imaging unit 6 and at least one of the flare component and the diffracted light component included in the image data captured by the second imaging unit 7. By inputting the valid data selected in step S16 or S17 to the learning model for which sufficient learning has been performed, corrected image data is output from the learning model. The output image data is data in which the influence of flare or diffraction is appropriately corrected for each RGB color component.

[0076] Summarizing the processes of the flowcharts in FIGS. 5 and 6, the signal processing unit 33 performs the processes of a reference determination unit (steps S1, S11), an imaging start instruction unit (steps S2, S12), and a correction procedure determination unit (steps S3 to S8, S13 to S18). The reference determination unit determines whether at least one of the sensitivity, resolution, flare component, and diffracted light component of the image data captured by the second imaging unit 7 satisfies a predetermined first reference. When the imaging start designation unit determines that the first reference is not satisfied by the reference determination unit, it causes the first imaging unit 6 to start imaging. The correction procedure determination unit determines whether to perform correction by the correction unit and the type of image data serving as the reference for correction when performing correction by the correction unit based on the result of comparing at least one of the sensitivity, resolution, flare component, and diffracted light component between the image data captured by the first imaging unit 6 and the image data captured by the second imaging unit 7. When the type of image data serving as the reference for correction is determined by the correction procedure determination unit, the signal processing unit 33 (learning unit) learns the correlation between at least one of the sensitivity, resolution, flare component, and diffracted light component of the determined image data and at least one of the sensitivity, resolution, flare component, and diffracted light component of the image data captured by the second imaging unit 7.

[0077] Note that the processes in FIG. 5 and the processes in FIG. 6 may be implemented in combination. That is, the signal processing unit 33 may correct two or more of the sensitivity, resolution, the influence of the flare component, and the influence of the diffracted light component of the image data captured by the second imaging unit 7 based on the image data captured by the first imaging unit 6.

[0078] As described above, in the first embodiment, based on the image data captured by the first imaging unit 6 that captures light in the infrared wavelength band, the image data captured by the second imaging unit 7 that captures light in the visible light wavelength band is corrected as necessary. Therefore, even when subject light is incident on the first and second camera modules 4 and 5 through the display unit 3, there is no possibility of a decrease in sensitivity or resolution, and there is no influence of flare or diffraction.

[0079] (Second Embodiment) The second embodiment takes measures when sufficient learning is not performed in the machine learning carried out in step S8 of FIG. 5 or step S18 of FIG. 6.

[0080] FIG. 7 is a block diagram showing the internal configuration of the electronic device 2a according to the second embodiment. The electronic device 2a in FIG. 7 includes a depth sensor 36 in addition to the configuration in FIG. 4. The depth sensor 36 is a sensor that detects distance information to an object. The depth sensor 36 may detect distance information, for example, by an indirect ToF (Time of Flight) method. The depth sensor 36 is arranged, for example, in the bezel 3b portion on the display surface side of the electronic device 2 in FIG. 1. The distance information detected by the depth sensor 36 is sent to the signal processing unit 33.

[0081] FIG. 8 is a flowchart showing a first example of the processing operation performed by the signal processing unit 33 in the electronic device 2a according to the second embodiment. The signal processing unit 33 executes the processing in FIG. 8 after performing the processing in FIG. 5 or FIG. 6.

[0082] First, it is determined whether or not machine learning has been sufficiently performed (step S21). When input data is given to the learning model generated by machine learning, arithmetic processing is performed using the learning model, and output data is generated. If the input data is within the range of data assumed in advance, appropriate output data reflecting the result of machine learning can be obtained by performing interpolation processing. However, when the input data is outside the range of data assumed in advance, it is necessary to perform extrapolation processing, and the accuracy of the output data will decrease. Therefore, in step S21, for example, it is determined whether or not machine learning has been sufficiently performed based on whether or not extrapolation processing is required for the input data. Note that other methods may be used to determine whether or not machine learning has been sufficiently performed.

[0083] If it is determined in step S21 that machine learning has been sufficiently performed, the data obtained by arithmetic processing using the learning model is output without performing color correction (step S22).

[0084] If it is determined in step S21 that machine learning has not been sufficiently performed, distance information is detected by the depth sensor 36 (step S23). Next, based on the distance information detected by the depth sensor 36, the shape of the object copied into the corrected imaging data in the process of FIG. 5 or FIG. 6 is grasped, and the object is identified (step S24). When the signal processing unit 33 alone cannot identify the object, the corrected imaging data in the process of FIG. 5 or FIG. 6 and the distance information detected by the depth sensor 36 are transmitted to a cloud server or the like via a network, and the huge data (big data) managed by the cloud server or the like is used to perform shape analysis of the object to identify the object, and the information of the identified object may be received by the signal processing unit 33 via the network. Alternatively, the corrected imaging data in the process of FIG. 5 or FIG. 6 and the distance information detected by the depth sensor 36 may be sent to the application processor, and the object may be identified by the application processor.

[0085] Next, it is determined whether the color of the object identified in step S24 is known (step S25). If the color of the object is not known, color correction is abandoned and the process of step S22 is performed. If the color of the object is known, color correction is performed so as to approach the known color (step S26). Here, a database for managing the color information of the identified object may be provided, and this database may be accessed to obtain the color information of the identified object.

[0086] As described above, in the process of FIG. 8, when sufficient learning has not been performed by machine learning, the shape of the object is detected using the depth sensor 36 to identify the object, and color correction is performed based on the known color information about the identified object.

[0087] In FIGS. 7 and 8, an example of detecting the shape of an object by the depth sensor 36 is shown, but information of the object may be detected using other sensors. For example, instead of or in addition to the depth sensor 36, a multispectral sensor may be provided.

[0088] FIG. 9 is a flowchart showing a second example of the processing operation performed by the signal processing unit 33 when using a multi-spectral sensor. The processing of steps S31 to S32 is the same as the processing of steps S21 to S22 in FIG. 8. When it is determined in step S31 that machine learning has not been sufficiently performed, color information is detected by the multi-spectral sensor (step S33). Since the multi-spectral sensor can separately detect light in a plurality of wavelength bands, the color information of the object included in the image data captured by the second imaging unit 7 can be detected in detail. Next, the object is identified from the color information detected in step S33 (step S34). At this time, as described above, the object may be identified by the cloud server or the application processor.

[0089] Next, it is determined whether the color of the object identified in step S34 is known (step S35). If the color of the object is not known, color correction is performed based on the color information detected by the multi-spectral sensor (step S36). If the color of the object is known, color correction is performed so as to approach the known color (step S37).

[0090] Summarizing the processing of the flowcharts in FIGS. 8 and 9, the signal processing unit 33 performs the processing of a reliability estimation unit (steps S21, S31), an object identification determination unit (steps S23, S33), and a color specification determination unit (steps S24, S34). The reliability estimation unit estimates the reliability of the learning by the learning unit. The object identification determination unit determines whether the object can be identified based on the detection data of the sensor when the reliability estimated by the reliability estimation unit is equal to or lower than a predetermined second criterion. The color specification determination unit determines whether the color of the object identified by the sensor can be specified when it is determined by the object identification determination unit that the object can be identified. The signal processing unit 33 functioning as a correction unit corrects the image data captured by the second imaging unit 7 so as to approach the specified color when it is determined by the color specification determination unit that the color of the object can be specified.

[0091] Note that the processes of FIG. 8 and FIG. 9 may be combined and implemented. If the object is identified based on the results of detecting the shape information and color information of the object by the depth sensor 36, the multi-spectral sensor, etc., the object can be identified more accurately, and color correction of the identified object can be performed with high accuracy.

[0092] Thus, in the second embodiment, since it is not preferable to correct the image data captured by the second imaging data in a state where the learning by machine learning is insufficient, when the learning by machine learning is insufficient, other sensors such as the depth sensor 36 and the multi-spectral sensor are used to perform color correction on the image data captured by the second imaging unit 7. Thereby, it is possible to avoid the risk of performing unreliable correction in a state where the learning by machine learning is insufficient.

[0093] (Third Embodiment) The third embodiment is characterized by the pixel arrangement of the first imaging unit 6 and the second imaging unit 7. The first imaging unit 6 that receives light in the infrared wavelength band and performs photoelectric conversion is composed of a plurality of pixels. Similarly, the second imaging unit 7 that receives light in the visible light wavelength band and performs photoelectric conversion is also composed of a plurality of pixels. A part of the plurality of pixels constituting the first imaging unit 6 may include pixels capable of receiving light in the visible light wavelength band.

[0094] FIG. 10A is a diagram showing a first example of pixel arrangement. In the first example shown in FIG. 10A, the pixels that receive light in the infrared wavelength band and perform photoelectric conversion are denoted as IR, and the pixels that receive light in the wavelength band corresponding to red and perform photoelectric conversion are denoted as R. In the example of FIG. 10A, the IR pixels and the R pixels are arranged in a staggered pattern. Note that the arrangement order of the IR pixels and the R pixels and the ratio of the number of each pixel are arbitrary.

[0095] The red light has wavelength components close to infrared light (e.g., 550 nm or more). By arranging R pixels between the IR pixels, the light wavelength band that can be received by the first imaging unit 6 can be made wider. When the signal processing unit 33 corrects the image data captured by the second imaging unit 7 based on the image data captured by the first imaging unit 6 as described above, by including a red component in the image data captured by the first imaging unit 6, it becomes easier to adjust with the red component included in the image data captured by the second imaging unit 7, and highly reliable correction processing can be performed.

[0096] FIG. 10B is a diagram showing a second example of pixel arrangement. In the second example shown in FIG. 10B, among a plurality of pixels constituting the second imaging unit 7, pixels capable of receiving light in the infrared light wavelength band are included. In FIG. 10B, pixels that receive light in the visible light wavelength band and perform photoelectric conversion are denoted as R, G, or G, and pixels that receive light in the infrared light band and perform photoelectric conversion are denoted as IR. One IR pixel is arranged for every three RGB pixels. Note that the arrangement order of the RGB pixels and the IR pixels and the ratio of the number of each pixel are arbitrary.

[0097] By arranging IR pixels between the RGB pixels, the first imaging unit 6 and the second imaging unit 7 can be formed on a single semiconductor chip. As a result, only one camera module 4, 5 needs to be provided, and the hardware cost of the electronic device 2 can be reduced.

[0098] As described above, in the third embodiment, by arranging pixels that receive light in the visible light wavelength band between pixels that receive light in the infrared light wavelength band, the reliability of the correction processing performed by the signal processing unit 33 can be improved. Also, by arranging pixels that receive light in the infrared light wavelength band between pixels that receive light in the visible light wavelength band, the first imaging unit 6 and the second imaging unit 7 can be formed on a single semiconductor chip.

[0099] (Fourth Embodiment) In the first embodiment, an example was shown in which the image data captured by the second imaging unit 7 is corrected using machine learning. However, even if machine learning is not necessarily used, the signal processing unit 33 can correct the image data captured by the second imaging unit 7.

[0100] FIG. 11 is a block diagram showing a first example of the internal configuration of the signal processing unit 33. The signal processing unit 33 in FIG. 11 includes a clamp unit 33a, a color output unit 33b, a defect correction unit 33c, a linear matrix unit 33d, a gamma correction unit 33e, a luminance chrominance signal generation unit 33f, a noise reduction unit 33g, and an edge enhancement unit 33h.

[0101] The clamp unit 33a performs a process of defining the black level. More specifically, the clamp unit 33a performs a process of subtracting the black level data from the digital pixel data. The color output unit 33b outputs pixel data for each color such as RGB, for example. The defect correction unit 33c performs a process of correcting the imaging data of specific pixels that could not be read correctly for some reason from the imaging data of surrounding pixels. The linear matrix unit 33d performs matrix operations on color information such as RGB to achieve more accurate color reproduction. The gamma correction unit 33e performs gamma correction so that a display with excellent visibility can be achieved according to the display characteristics of the display unit 3. For example, the gamma correction unit 33e performs a conversion from 10 bits to 8 bits while changing the gradient. The luminance chrominance signal generation unit 33f generates a luminance chrominance signal for display on the display unit 3 based on the output data of the gamma correction unit 33e. The noise reduction unit 33g performs a process of reducing the noise included in the luminance chrominance signal. The edge enhancement unit 33h performs a process of enhancing the edges of the subject image based on the luminance chrominance signal. The noise reduction process by the noise reduction unit 33g and the edge enhancement process by the edge enhancement unit 33h may be performed only when a predetermined condition is satisfied. The output unit 35 outputs the luminance chrominance signal after the noise reduction process.

[0102] The noise reduction unit 33g determines whether a pixel region has little luminance change based on the image data captured by the first imaging unit 6, and increases the noise removal ratio for the pixel region with little luminance change. More specifically, the noise reduction unit 33g makes the noise removal degree of the pixel region in the image data captured by the second imaging unit 7, where the luminance change is equal to or less than a predetermined reference value, higher than the noise removal degree of the pixel region in the image data where the luminance change is greater than the reference value.

[0103] Thereby, the data amount of the image data captured by the second imaging unit 7 can be reduced. On the other hand, for the pixel region with a large luminance change, the noise removal ratio is weakened. The reason for this is that the higher the noise removal ratio, the less distinct the edges become and the resolution also decreases. Therefore, in order to make the edges distinct and improve the resolution, it is desirable to weaken the noise removal ratio. However, if the noise removal ratio is weakened for all pixels, the data amount of the image data will become extremely large. Therefore, it is desirable to weaken the noise removal ratio only for limited pixel regions such as the edge portions.

[0104] FIG. 12 is a block diagram showing a second example of the internal configuration of the signal processing unit 33. The signal processing unit 33 in FIG. 11 is obtained by providing a flare extraction unit 33i and a flare correction unit 33j to the signal processing unit 33 in FIG. 11.

[0105] The flare extraction unit 33i extracts the degree of influence by flare or diffraction by comparing the pixel values of the image data captured by the first imaging unit 6 with the pixel values of the image data captured by the second imaging unit 7. For example, the degree of influence by flare or diffraction may be extracted based on the difference between the average pixel value of the image data captured by the first imaging unit 6 and the average pixel value of the image data captured by the second imaging unit 7.

[0106] The flare correction unit 33j performs a process of subtracting the degree of influence by flare or diffraction extracted by the flare extraction unit 33i from the image data captured by the second imaging unit 7. Thereby, the image data captured by the second imaging unit 7 is less likely to be affected by flare and diffraction.

[0107] As described above, in the fourth embodiment, by referring to the image data captured by the first imaging unit 6 in the internal processing of the signal processing unit 33, it is possible to correct the sensitivity and resolution of the image data captured by the second imaging unit 7, and it is also possible to suppress the influence of flare or diffraction. Further, by avoiding unnecessary correction, the processing load on the signal processing unit 33 can be reduced, and the data amount of the image data can be reduced.

[0108] (Fifth Embodiment) In the first to fourth embodiments, an example in which the first imaging unit 6 that receives light of the infrared light wavelength component is provided has been shown. In addition to this, a light emitting unit that emits light of the infrared light wavelength component may be provided.

[0109] FIG. 13 is a block diagram showing a schematic configuration of the electronic device 2b according to the fifth embodiment. The electronic device 2b in FIG. 13 includes a light emitting unit 25 and a light emission control unit 26 in addition to the configuration of the electronic device 2 in FIG. 4. The light emitting unit 25 emits light in the infrared light wavelength band. The light emitting unit 25 is provided on the display surface side of the electronic device 2b. For example, the light emitting unit 25 may be provided in the bezel 3b portion of the display surface of the electronic device 2b.

[0110] The light emission control unit 26 controls the light emission timing of the light emitting unit 25 so that the subject is irradiated with the light emitted by the light emitting unit 25 when the first imaging unit 6 captures image data. Since the light in the infrared light wavelength band cannot be recognized by the human eye, even if the light from the light emitting unit 25 irradiates a human, that human does not feel dazzled. However, it is necessary to irradiate with a light intensity and continuous irradiation time that do not hurt the human eye.

[0111] In this embodiment, it is assumed that the first imaging unit 6 and the second imaging unit 7 capture the same subject at the same timing. Since the first imaging unit 6 captures light in the infrared light wavelength band, when the first imaging unit 6 captures the subject while illuminating the subject with the light from the light emitting unit 25, the pixel value of the image data received by the first imaging unit 6 can be increased, and the sensitivity can be increased.

[0112] FIG. 14 is a flowchart showing the processing operations when the electronic device 2b according to the fifth embodiment performs imaging with the first and second camera modules 4 and 5. First, it is determined whether it is the imaging timing of the first imaging unit 6 and the second imaging unit 7 (step S41). Imaging by the first imaging unit 6 and the second imaging unit 7 is performed during a period when no image is displayed on the display unit 3, for example, within the vertical blanking period of the display unit 3.

[0113] Wait at step S41 until the imaging timing is reached. When the imaging timing is reached, start emitting light in the infrared light band (IR light) by the light emitting unit 25 (step S42). For example, when the electronic device 2b is a smartphone, in order for the user to focus the first and second camera modules 4 and 5 on the subject, if the traveling direction of the light emitted from the light emitting unit 25 is made to coincide with the optical axis directions of the first and second camera modules 4 and 5, the light from the light emitting unit 25 will illuminate the subject without the user having to be particularly aware of it.

[0114] During the time when the light emitting unit 25 is emitting light, perform imaging with the first imaging unit 6 (step S43). Thereby, even when shooting in a dark surrounding situation, the pixel values of the image data captured by the first imaging unit 6 can be increased. When imaging by the first imaging unit 6 and the second imaging unit 7 is completed, the light emitting unit 25 stops emitting light (step S44). Thereafter, similar to the first embodiment and the like, based on the image data captured by the first imaging unit 6, the image data captured by the second imaging unit 7 is corrected (step S45).

[0115] In this way, in the fifth embodiment, since the first imaging unit 6 receives light in the infrared light wavelength band while the light emitting unit 25 emits infrared light, the pixel values of the image data captured by the first imaging unit 6 can be increased, the sensitivity can be increased, and even in a dark surrounding situation, the image data captured by the second imaging unit 7 can be appropriately corrected based on the image data captured by the first imaging unit 6.

[0116] (Sixth Embodiment) The sixth embodiment is a modification of the fifth embodiment, and is configured such that the emission wavelength band of the light emitted by the light emitting unit 25 can be switched.

[0117] The electronic device 2b according to the sixth embodiment has the same block configuration as that in FIG. 13, but the function of the light emitting unit 25 is different. The light emitting unit 25 according to the sixth embodiment includes a plurality of light sources that emit light in different emission wavelength bands within the infrared wavelength band. The light emission control unit 26 sequentially switches and controls the light emission by the plurality of light sources while the first imaging unit 6 performs imaging. The first imaging unit 6 outputs a plurality of image data captured in different emission wavelength bands.

[0118] FIG. 15 is a flowchart showing the processing operations when the electronic device 2b according to the sixth embodiment performs imaging using the first and second camera modules 4 and 5. First, it is determined whether it is the imaging timing of the first imaging unit 6 and the second imaging unit 7 (step S51). When the imaging timing is reached, the light emitting unit 25 selects one of the plurality of emission wavelength bands according to an instruction from the light emission control unit 26 (step S52), and emits infrared light in the selected emission wavelength band (step S53). Then, imaging is performed by the first and second imaging units 6 and 7 (step S54), and thereafter, the light emission of the light emitting unit 25 is stopped (step S55).

[0119] Next, it is determined whether there is infrared light in an emission wavelength band that has not yet been selected among the light that the light emitting unit 25 can emit (step S56). If there is infrared light that has not yet been selected, the processing from step S52 onward is repeated. When it is determined in step S56 that the processing of steps S52 to S55 has been performed for all the light that the light emitting unit 25 can emit, the image data captured by the second imaging unit 7 is corrected based on the plurality of infrared wavelength band image data captured by the first imaging unit 6 (step S57).

[0120] When the first imaging unit 6 captures an image while being illuminated in a plurality of different infrared light wavelength bands, the information captured in the captured image may change little by little. By comprehensively considering these and correcting the image data captured by the second imaging unit 7, the image data captured by the second imaging unit 7 can be corrected more appropriately.

[0121] Thus, in the sixth embodiment, since the light emitting unit 25 has a plurality of light sources that emit light in different light emission wavelength bands within the infrared light wavelength band, by capturing an image with the first imaging unit 6 while these light sources sequentially emit light, a plurality of captured images captured by the first imaging unit 6 can be obtained. By correcting the image data captured by the second imaging unit 7 based on these plurality of captured images, more appropriate correction can be performed.

[0122] (Seventh Embodiment) In the first to sixth embodiments described above, an example in which the light that has passed through the display surfaces of the electronic devices 2, 2a, and 2b is imaged by the first and second camera modules 4 and 5 has been described. However, when the first and second camera modules 4 and 5 are arranged in the bezel 3b portion of the display surface, or when they are arranged on the back side of the electronic devices 2, 2a, and 2b, since external light is directly incident on the first and second camera modules 4 and 5, there is no loss of light amount due to the subject light passing through the display unit 3, nor is there an influence due to flare or diffraction. However, when imaging is performed with the second imaging unit 7 in a situation where the surroundings are dark, the sensitivity and resolution of the captured image may decrease. Also, when imaging is performed with the second imaging unit 7 in a situation where strong sunlight is received, the captured image is likely to be affected by flare and diffraction. In such a case, by correcting the image data captured by the second imaging unit 7 based on the image data captured by the first imaging unit 6 that receives light in the infrared light wavelength band, the sensitivity and resolution can be improved, or the influence due to flare and diffraction can be suppressed. Therefore, the electronic devices 2, 2a, and 2b according to the first to sixth embodiments are also applicable when the first and second camera modules 4 and 5 are arranged in the bezel 3b portion of the display surface, or when they are arranged on the back side of the electronic devices 2, 2a, and 2b.

[0123] (Eighth Embodiment) As specific candidates for the electronic device 2 having the configurations described in the above-described first to seventh embodiments, various ones can be considered. For example, FIG. 16 is a plan view of the case where the electronic device 2 of the first to seventh embodiments is applied to a capsule endoscope 50. The capsule endoscope 50 in FIG. 16 includes, for example, a camera (ultra-small camera) 52 for photographing an image in a body cavity, a memory 53 for recording the image data photographed by the camera 52, and a wireless transmitter 55 for transmitting the recorded image data to the outside via an antenna 54 after the capsule endoscope 50 is discharged outside the subject's body, within a housing 51 having hemispherical both end faces and a cylindrical central portion.

[0124] Further, a CPU (Central Processing Unit) 56 and a coil (magnetic force - current conversion coil) 57 are provided within the housing 51. The CPU 56 controls photographing by the camera 52 and data accumulation operation in the memory 53, and also controls data transmission from the memory 53 to a data receiving device (not shown) outside the housing 51 by the wireless transmitter 55. The coil 57 supplies power to the camera 52, the memory 53, the wireless transmitter 55, the antenna 54, and a light source 52b described later.

[0125] Furthermore, the housing 51 is provided with a magnetic (reed) switch 58 for detecting when the capsule endoscope 50 is set in the data receiving device. The CPU 56 supplies power from the coil 57 to the wireless transmitter 55 when this reed switch 58 detects the setting in the data receiving device and data transmission becomes possible.

[0126] The camera 52 has, for example, an image sensor 52a including an objective optical system for photographing an image in a body cavity, and a plurality of light sources 52b for illuminating the body cavity. Specifically, the camera 52 is constituted by, for example, a CMOS (Complementary Metal Oxide Semiconductor) sensor equipped with an LED (Light Emitting Diode) as the light source 52b, a CCD (Charge Coupled Device), or the like.

[0127] (Application examples of the imaging device 1 and the electronic device 2 according to the present disclosure) (First application example) The imaging device 1 and the electronic device 2 according to the present disclosure can be used for various applications. FIGS. 16A and 16B are diagrams showing the internal configuration of a vehicle 100 which is a first application example of the electronic device 2 provided with the imaging device 1 according to the present disclosure. FIG. 16A is a diagram showing the interior of the vehicle 100 from the rear to the front, and FIG. 16B is a diagram showing the interior of the vehicle 100 from the diagonally rear to the diagonally front.

[0128] The vehicle 100 in FIGS. 16A and 16B includes a center display 101, a console display 102, a head-up display 103, a digital rearview mirror 104, a steering wheel display 105, and a rear entertainment display 106.

[0129] The center display 101 is disposed at a position facing the driver's seat 108 and the passenger seat 109 on the dashboard 107. In FIG. 16, an example of the horizontally long center display 101 extending from the driver's seat 108 side to the passenger seat 109 side is shown, but the screen size and the arrangement position of the center display 101 are arbitrary. Information detected by various sensors 5 can be displayed on the center display 101. As a specific example, the center display 101 can display a captured image captured by an image sensor, a distance image to an obstacle in front of or on the side of the vehicle measured by the ToF sensor 5, the body temperature of a passenger detected by the infrared sensor 5, and the like. The center display 101 can be used, for example, to display at least one of safety-related information, operation-related information, life log, health-related information, authentication / identification-related information, and entertainment-related information.

[0130] Safety-related information includes information such as drowsiness detection, looking-around detection, mischief detection of children riding along, seat belt wearing status, and abandonment detection of passengers. For example, it is information detected by sensor 5 disposed overlapping on the back side of center display 101. Operation-related information detects gestures related to the operations of passengers using sensor 5. The detected gestures may include operations of various facilities inside vehicle 100. For example, it detects operations of air conditioning facilities, navigation devices, AV devices, lighting devices, etc. The life log includes life logs of all passengers. For example, the life log includes action records of each passenger during the ride. By acquiring and storing the life log, it is possible to confirm what state the passengers were in at the time of an accident. Health-related information detects the body temperature of passengers using temperature sensor 5 and estimates the health status of passengers based on the detected body temperature. Alternatively, an image sensor may be used to capture the faces of passengers and the health status of passengers may be estimated from the expressions of the captured faces. Further, an automatic voice conversation may be conducted with the passengers and the health status of the passengers may be estimated based on the response content of the passengers. Authentication / identification-related information includes a keyless entry function that performs face authentication using sensor 5, an automatic adjustment function for seat height and position by face identification, etc. Entertainment-related information includes a function of detecting operation information of an AV device by a passenger using sensor 5, a function of recognizing the face of a passenger by sensor 5 and providing content suitable for the passenger on the AV device, etc.

[0131] Console display 102 can be used, for example, to display life log information. Console display 102 is disposed near shift lever 111 of center console 110 between driver's seat 108 and passenger seat 109. Information detected by various sensors 5 can also be displayed on console display 102. Further, console display 102 may display an image of the periphery of the vehicle captured by an image sensor, or may display a distance image to an obstacle around the vehicle.

[0132] The head-up display 103 is virtually displayed behind the windshield 112 in front of the driver's seat 108. The head-up display 103 can be used, for example, to display at least one of safety-related information, operation-related information, life log, health-related information, authentication / identification-related information, and entertainment-related information. Since the head-up display 103 is often virtually arranged in front of the driver's seat 108, it is suitable for displaying information directly related to the operation of the vehicle 100, such as the speed of the vehicle 100 and the remaining amount of fuel (battery).

[0133] The digital rearview mirror 104 can not only display the rear of the vehicle 100 but also the state of the passengers in the rear seats. Therefore, by arranging the sensor 5 on the back side of the digital rearview mirror 104, it can be used, for example, for displaying life log information.

[0134] The steering wheel display 105 is arranged near the center of the steering wheel 113 of the vehicle 100. The steering wheel display 105 can be used, for example, to display at least one of safety-related information, operation-related information, life log, health-related information, authentication / identification-related information, and entertainment-related information. In particular, since the steering wheel display 105 is near the driver's hand, it is suitable for displaying life log information such as the driver's body temperature and information related to the operation of AV devices, air conditioning equipment, etc.

[0135] The rear entertainment display 106 is attached to the back side of the driver's seat 108 and the front passenger's seat 109 and is for the passengers in the rear seats to view. The rear entertainment display 106 can be used, for example, to display at least one of safety-related information, operation-related information, life logs, health-related information, authentication / identification-related information, and entertainment-related information. In particular, since the rear entertainment display 106 is in front of the eyes of the passengers in the rear seats, information related to the passengers in the rear seats is displayed. For example, information regarding the operation of an AV device or air conditioning equipment may be displayed, or the result of measuring the body temperature etc. of the passengers in the rear seats with the temperature sensor 5 may be displayed.

[0136] As described above, by arranging the sensor 5 so as to overlap the back side of the display unit 3, the distance to an object existing in the surroundings can be measured. Optical distance measurement methods are roughly classified into passive types and active types. The passive type measures the distance by receiving light from an object without projecting light from the sensor 5 to the object. Examples of the passive type include the lens focus method, the stereo method, and the monocular vision method. The active type measures the distance by projecting light onto an object and receiving the reflected light from the object with the sensor 5. Examples of the active type include the lidar method, the active stereo method, the illuminance difference stereo method, the moire topography method, and the interference method. The imaging device 1 according to the present disclosure is applicable to distance measurement by any of these methods. By using the sensor 5 arranged so as to overlap the back side of the display unit 3 according to the present disclosure, the above-described passive or active distance measurement can be performed.

[0137] (Second application example) The imaging device 1 according to the present disclosure is applicable not only to various displays used in vehicles but also to displays mounted on various electronic devices 2.

[0138] FIG. 17A is a front view of a digital camera 120 which is a second application example of the electronic device 2, and FIG. 17B is a rear view of the digital camera 120. The digital camera 120 in FIGS. 17A and 17B shows an example of a single-lens reflex camera with an interchangeable lens 121, but it is also applicable to a camera with a non-interchangeable lens 121.

[0139] When the photographer looks through the electronic viewfinder 124 while gripping the grip 123 of the camera body 122 to compose the picture and presses the shutter 125 in a state where focus adjustment has been performed, the photographed data is stored in the memory inside the camera. On the back side of the camera, as shown in FIG. 17B, a monitor screen 126 for displaying photographed data and the like, and a live image and the like, and an electronic viewfinder 124 are provided. Also, on the upper surface of the camera, a sub-screen for displaying setting information such as shutter speed and exposure value may be provided.

[0140] By disposing the sensor 5 so as to overlap the back side of the monitor screen 126, the electronic viewfinder 124, the sub-screen, etc. used in the camera, it can be used as the imaging device 1 according to the present disclosure.

[0141] (Third application example) The imaging device 1 and the electronic device 2 according to the present disclosure are also applicable to a head-mounted display (hereinafter referred to as an HMD). The HMD can be used for VR (Virtual Reality), AR (Augmented Reality), MR (Mixed Reality), or SR (Substitutional Reality), etc.

[0142] Figure 18A is an external view of the HMD 130 which is a third application example of the electronic device 2. The HMD 130 in Figure 18A has a wearing member 131 for wearing so as to cover a human eye. This wearing member 131 is fixed, for example, by hooking onto a human ear. A display device 132 is provided inside the HMD 130, and the wearer of the HMD 130 can visually recognize a stereoscopic image or the like on this display device 132. The HMD 130 is equipped with, for example, a wireless communication function and an acceleration sensor, etc., and can switch the stereoscopic image or the like displayed on the display device 132 according to the posture and gesture of the wearer, etc.

[0143] Also, a camera may be provided in the HMD 130 to capture an image of the surroundings of the wearer, and an image obtained by synthesizing the captured image of the camera and an image generated by a computer may be displayed on the display device 132. For example, a camera is arranged so as to overlap with the back side of the display device 132 that the wearer of the HMD 130 visually recognizes, and the area around the wearer's eyes is captured by this camera, and the captured image is displayed on another display provided on the outer surface of the HMD 130, so that people around the wearer can grasp the facial expression and eye movement of the wearer in real time.

[0144] Note that various types of the HMD 130 are conceivable. For example, as shown in Figure 18B, the imaging device 1 and the electronic device 2 according to the present disclosure are also applicable to the smart glasses 130a that project various information onto the glasses 134. The smart glasses 130a in Figure 18B have a main body part 135, an arm part 136, and a lens barrel part 137. The main body part 135 is connected to the arm part 136. The main body part 135 is detachable from the glasses 134. The main body part 135 incorporates a control board and a display part for controlling the operation of the smart glasses 130a. The main body part 135 and the lens barrel are connected to each other via the arm part 136. The lens barrel part 137 emits the image light emitted from the main body part 135 via the arm part 136 to the lens 138 side of the glasses 134. This image light enters the human eye through the lens 138. The wearer of the smart glasses 130a in Figure 18B can visually recognize not only the surrounding situation but also various information emitted from the lens barrel part 137, just like with ordinary glasses.

[0145] (Fourth Application Example) The imaging device 1 and the electronic device 2 according to the present disclosure are also applicable to a television device (hereinafter referred to as a TV). Recently, TVs tend to make the frame as small as possible from the viewpoints of miniaturization and designability. Therefore, when a camera for photographing viewers is provided in a TV, it is desirable to arrange it overlapping on the back side of the display panel 2 of the TV.

[0146] FIG. 19 is an external view of a TV 140 which is a fourth application example of the electronic device 2. The TV 140 in FIG. 19 has a minimized frame, and almost the entire front side is a display area. The TV 140 incorporates a sensor 5 such as a camera for photographing viewers. The sensor 5 in FIG. 19 is arranged on the back side of a part (for example, the portion of the broken line) inside the display panel 2. The sensor 5 may be an image sensor module, and various sensors such as a face authentication sensor, a distance measurement sensor, and a temperature sensor are applicable, and a plurality of types of sensors may be arranged on the back side of the display panel 2 of the TV 140.

[0147] As described above, according to the imaging device 1 and the electronic device 2 of the present disclosure, since the image sensor module 9 can be arranged overlapping on the back side of the display panel 2, there is no need to arrange a camera or the like on the frame, the TV 140 can be miniaturized, and there is no risk of the design being impaired by the frame.

[0148] (Fifth Application Example) The imaging device 1 and the electronic device 2 according to the present disclosure are also applicable to smartphones and mobile phones. FIG. 20 is an external view of a smartphone 150 which is a fifth application example of the electronic device 2. In the example of FIG. 20, the display surface 2z extends close to the outer size of the electronic device 2, and the width of the bezel 2y around the display surface 2z is set to several millimeters or less. Usually, a front camera is often mounted on the bezel 2y. However, in FIG. 20, as shown by the dashed line, an image sensor module 9 functioning as a front camera is arranged on the back side of, for example, substantially the center of the display surface 2z. By providing the front camera on the back side of the display surface 2z in this way, it becomes unnecessary to arrange the front camera on the bezel 2y, and the width of the bezel 2y can be narrowed.

[0149] Note that the present technology can adopt the following configurations. (1) A display unit, A first imaging unit arranged on the side opposite to the display surface of the display unit and capable of imaging light in the infrared wavelength band that has passed through the display unit, A second imaging unit arranged on the side opposite to the display surface of the display unit and capable of imaging light in the visible light wavelength band that has passed through the display unit, An electronic device comprising: a correction unit that corrects the image data captured by the second imaging unit based on the image data captured by the first imaging unit. (2) The electronic device according to (1), wherein the correction unit corrects the sensitivity of the image data captured by the second imaging unit based on the image data captured by the first imaging unit. (3) A learning unit that learns the correlation between the sensitivity of the image data captured by the first imaging unit and the sensitivity of the image data captured by the second imaging unit, The electronic device according to (2), wherein the correction unit corrects the sensitivity of the image data captured by the second imaging unit based on the image data captured by the first imaging unit with reference to the learning result of the learning unit. (4) The electronic device according to (1), wherein the correction unit corrects the resolution of the image data captured by the second imaging unit based on the image data captured by the first imaging unit. A learning unit that learns the correlation between the resolution of the image data captured by the first imaging unit and the resolution of the image data captured by the second imaging unit. The correction unit refers to the learning result of the learning unit and corrects the resolution of the image data captured by the second imaging unit based on the image data captured by the first imaging unit. The electronic device according to (4). (6) The correction unit corrects at least one of the flare component and the diffracted light component included in the image data captured by the second imaging unit based on the image data captured by the first imaging unit. The electronic device according to (1). (7) A learning unit that learns the correlation between at least one of the flare component and the diffracted light component included in the image data captured by the first imaging unit and at least one of the flare component and the diffracted light component included in the image data captured by the second imaging unit. The correction unit refers to the learning result of the learning unit and corrects at least one of the flare component and the diffracted light component included in the image data captured by the second imaging unit based on the image data captured by the first imaging unit. The electronic device according to (6). (8) A reference determination unit that determines whether at least one of the sensitivity, resolution, flare component, and diffracted light component of the image data captured by the second imaging unit satisfies a predetermined first reference. An imaging start instruction unit that starts imaging by the first imaging unit when it is determined by the reference determination unit that the first reference is not satisfied. Based on the result of comparing at least one of the sensitivity, resolution, flare component, and diffracted light component between the image data captured by the first imaging unit and the image data captured by the second imaging unit, a correction procedure determination unit that determines whether to perform correction by the correction unit and the type of image data serving as the correction reference when performing correction by the correction unit. The electronic device according to (7). (9) When the type of image data serving as a correction standard is determined by the correction procedure determination unit in the learning unit, the learning unit learns the correlation between at least one of the sensitivity, resolution, flare component, and diffraction light component of the determined image data and at least one of the sensitivity, resolution, flare component, and diffraction light component of the image data captured by the second imaging unit. The electronic device according to (8). (10) A sensor that detects at least one of the shape and color of an object, A reliability estimation unit that estimates the reliability of the learning by the learning unit, When the reliability estimated by the reliability estimation unit is equal to or lower than a predetermined second standard, an object identification determination unit that determines whether the object can be identified based on the detection data of the sensor, When it is determined by the object identification determination unit that the object can be identified, a color identification determination unit that determines whether the color of the object identified by the sensor can be specified, When it is determined by the color identification determination unit that the color of the object can be specified, the correction unit corrects the image data captured by the second imaging unit so as to approach the specified color. The electronic device according to any one of (7) to (9). (11) The correction unit makes the noise removal degree of a pixel region where the luminance change is equal to or lower than a predetermined reference value in the image data captured by the second imaging unit higher than the noise removal degree of a pixel region where the luminance change is greater than the reference value in the image data. The electronic device according to any one of (1) to (10). (12) A light emitting unit that emits light in an infrared light wavelength band, A light emission control unit that controls the light emission timing of the light emitting unit so that the subject is illuminated by the light emitted by the light emitting unit when the first imaging unit captures image data. The electronic device according to any one of (1) to (11). (13) The light emitting unit has a plurality of light sources that emit light in different light emission wavelength bands within the infrared light wavelength band, The light emission control unit sequentially switches and controls the light emission by the plurality of light sources while the first imaging unit is capturing an image, The first imaging unit outputs a plurality of image data captured in different light emission light wavelength bands, The correction unit corrects the image data captured by the second imaging unit based on the plurality of image data, the electronic device according to (12). (14) The light emitting unit is disposed on the display surface side of the display unit, the electronic device according to (12) or (13). (15) At least one of the first imaging unit and the second imaging unit includes a pixel that captures light in the infrared wavelength band and a pixel that captures light in the visible wavelength band, the electronic device according to any one of (1) to (14). (16) The first imaging unit has sensitivity to light of 550 nm or more, the electronic device according to (15). (17) The correction unit increases the degree of correction toward the short wavelength side for the image data captured by the second imaging unit, the electronic device according to any one of (1) to (16). (18) The first imaging unit has a photoelectric conversion unit that is arranged longer in the normal direction of the light incident surface than the second imaging unit, the electronic device according to any one of (1) to (17). (19) The area in the light incident surface direction per pixel of the first imaging unit is larger than the area in the light incident surface direction per pixel of the second imaging unit, and The area in the light incident surface direction for all the pixels of the first imaging unit is smaller than the area in the light incident surface direction for all the pixels of the second imaging unit, the electronic device according to any one of (1) to (18). (20) A first imaging unit disposed on the side opposite to the display surface of the display unit and capable of imaging light in the infrared wavelength band, A second imaging unit disposed on the side opposite to the display surface of the display unit and capable of imaging light in the visible wavelength band, An imaging device including a correction unit that corrects the image data captured by the second imaging unit based on the image data captured by the first imaging unit. (21) A first imaging unit capable of imaging light in the infrared wavelength band, A second imaging unit capable of imaging light in the visible wavelength band, An imaging device including a correction unit that corrects the image data captured by the second imaging unit based on the image data captured by the first imaging unit.

[0150] Aspects of the present disclosure are not limited to the individual embodiments described above, but also include various modifications that can be conceived by those skilled in the art, and the effects of the present disclosure are not limited to the above-described content. That is, various additions, changes, and partial deletions are possible without departing from the conceptual ideas and spirit of the present disclosure derived from the content defined in the claims and their equivalents.

Description of Reference Numerals

[0151] 1 Imaging device, 2, 2a, 2b Electronic device, 3 Display unit, 3a Display surface, 3b Bezel, 4 Camera module, 5 Camera module, 6 First imaging unit, 7 Second imaging unit, 8 First optical system, 9 Second optical system, 11 Semiconductor substrate, 12a First photoelectric conversion unit, 12b Second photoelectric conversion unit, 13 Element isolation layer, 14 Planarization layer, 15a On-chip lens, 15b On-chip lens, 16 Color filter layer, 17 Readout circuit, 21 Application processor, 22 Video signal generation unit, 23 A / D conversion unit, 24 Display control unit, 31 First A / D conversion unit, 32 Second A / D conversion unit, 33 Signal processing unit, 100 Vehicle, 101 Center display, 102 Console display, 103 Head-up display, 104 Digital mirror, 105 Steering wheel display, 106 Rear entertainment display, 107 Dashboard, 108 Driver's seat, 109 Passenger seat, 110 Center console, 111 Shift lever, 112 Windshield, 113 Steering wheel, 120 Digital camera, 121 Lens, 122 Camera body, 123 Grip, 124 Electronic viewfinder, 125 Shutter, 126 Monitor screen, 130 Smart glasses, 131 Mounting member, 132 Display device, 134 Glasses, 135 Main body, 136 Arm, 137 Lens barrel, 138 Lens, 150 Smartphone

Claims

1. A display unit, a first imaging unit disposed on the side opposite to the display surface of the display unit and capable of imaging light in the infrared wavelength band that has passed through the display unit, a second imaging unit disposed on the side opposite to the display surface of the display unit and capable of imaging light in the visible light wavelength band that has passed through the display unit, and a correction unit that corrects the image data captured by the second imaging unit based on the image data captured by the first imaging unit. An electronic device.

2. The correction unit corrects the sensitivity of the image data captured by the second imaging unit based on the image data captured by the first imaging unit. The electronic device according to Claim 1.

3. A learning unit that learns the correlation between the sensitivity of the image data captured by the first imaging unit and the sensitivity of the image data captured by the second imaging unit, wherein the correction unit corrects the sensitivity of the image data captured by the second imaging unit based on the image data captured by the first imaging unit with reference to the learning result of the learning unit. The electronic device according to Claim 2.

4. The correction unit corrects the resolution of the image data captured by the second imaging unit based on the image data captured by the first imaging unit. The electronic device according to Claim 1.

5. A learning unit that learns the correlation between the resolution of the image data captured by the first imaging unit and the resolution of the image data captured by the second imaging unit, wherein the correction unit corrects the resolution of the image data captured by the second imaging unit based on the image data captured by the first imaging unit with reference to the learning result of the learning unit. The electronic device according to Claim 4.

6. The correction unit corrects at least one of the flare component and the diffracted light component included in the image data captured by the second imaging unit based on the image data captured by the first imaging unit. The electronic device according to Claim 1.

7. A learning unit that learns the correlation between at least one of the flare component and the diffracted light component included in the image data captured by the first imaging unit and at least one of the flare component and the diffracted light component included in the image data captured by the second imaging unit, wherein the correction unit corrects at least one of the flare component and the diffracted light component included in the image data captured by the second imaging unit based on the image data captured by the first imaging unit with reference to the learning result of the learning unit. The electronic device according to Claim 6.

8. A reference determination unit that determines whether at least one of the sensitivity, resolution, flare component, and diffracted light component of the image data captured by the second imaging unit satisfies a predetermined first reference; An imaging start instruction unit that starts imaging by the first imaging unit when the reference determination unit determines that the first reference is not satisfied; Based on the result of comparing at least one of the sensitivity, resolution, flare component, and diffracted light component between the image data captured by the first imaging unit and the image data captured by the second imaging unit, a correction procedure determination unit that determines whether to perform correction by the correction unit and the type of image data serving as a reference for correction when performing correction by the correction unit; The electronic device according to claim 7.

9. When the type of image data serving as a reference for correction is determined by the correction procedure determination unit, the learning unit learns the correlation between at least one of the sensitivity, resolution, flare component, and diffracted light component of the determined image data and at least one of the sensitivity, resolution, flare component, and diffracted light component of the image data captured by the second imaging unit. The electronic device according to claim 8.

10. A sensor that detects at least one of the shape and color of an object; A reliability estimation unit that estimates the reliability of the learning by the learning unit; An object identification determination unit that determines whether the object can be identified based on the detection data of the sensor when the reliability estimated by the reliability estimation unit is equal to or lower than a predetermined second reference; Comprising a color specification determination unit that determines whether the color of the object identified by the sensor can be specified when the object identification determination unit determines that the object can be identified; When the color specification determination unit determines that the color of the object can be specified, the correction unit corrects the image data captured by the second imaging unit so as to approach the specified color. The electronic device according to any one of claims 7 to 9.

11. The correction unit increases the degree of noise removal of a pixel region where the luminance change is equal to or lower than a predetermined reference value in the image data captured by the second imaging unit compared to the degree of noise removal of a pixel region where the luminance change is greater than the reference value in the image data. The electronic device according to any one of claims 1 to 10.

12. A light emitting unit that emits light in an infrared light wavelength band; A light emission control unit that controls the light emission timing of the light emitting unit so that the subject is illuminated with the light emitted by the light emitting unit when the first imaging unit captures image data. The electronic device according to any one of claims 1 to 11.

13. The light emitting unit has a plurality of light sources that emit light in different light emission wavelength bands within the infrared light wavelength band, During imaging by the first imaging unit, the light emission control unit sequentially switches and controls light emission by the plurality of light sources, The first imaging unit outputs a plurality of image data captured in different light emission light wavelength bands, The correction unit corrects the image data captured by the second imaging unit based on the plurality of image data. The electronic device according to claim 12.

14. The light emitting unit is disposed on the display surface side of the display unit. The electronic device according to claim 12 or 13.

15. At least one of the first imaging unit and the second imaging unit includes pixels that image light in the infrared light wavelength band and pixels that image light in the visible light wavelength band. The electronic device according to any one of claims 1 to 14.

16. The first imaging unit has sensitivity to light of 550 nm or more. The electronic device according to claim 15.

17. The correction unit increases the degree of correction toward the short wavelength side for the image data captured by the second imaging unit. The electronic device according to any one of claims 1 to 16.

18. The first imaging unit has a photoelectric conversion unit that is arranged longer in the normal direction of the light incident surface than the second imaging unit. The electronic device according to any one of claims 1 to 17.

19. The area of the light incident surface direction per pixel of the first imaging unit is larger than the area of the light incident surface direction per pixel of the second imaging unit, and the area of the light incident surface direction for all pixels of the first imaging unit is smaller than the area of the light incident surface direction for all pixels of the second imaging unit. The electronic device according to any one of claims 1 to 18.

Images