Electronic device and imaging apparatus

Abstract

[Problem] A high-quality captured image is obtained even when the amount of incident light is small. [Solution] The electronic device is provided with a display unit, a first imaging unit provided on the side opposite the display surface of the display unit and capable of imaging light of an infrared wavelength band passing through the display unit, a second imaging unit provided on the side opposite the display surface of the display unit and capable of imaging light of a visible wavelength band passing through the display unit, and a correction unit that corrects image data obtained by imaging performed by the second imaging unit based on image data obtained by imaging performed by the first imaging unit.

Description

TECHNICAL FIELD

[0001] The disclosure relates to an electronic device and an imaging device. BACKGROUND

[0002] In recent electronic devices such as a smart phone, a mobile phone, and a personal computer (PC), various sensors such as a camera are installed in a frame (a bezel) of a display panel. On the other hand, there is a demand to make the external size of the electronic device as compact as possible without affecting the screen size, and the bezel width tends to be narrowed. In view of such a background, a technology of disposing a camera module right below a display panel and capturing an image of an object light passing through the display panel by the camera module is proposed.

[0003] LIST OF CITATIONS

[0004] PATENT LITERATURE

[0005] Patent Literature 1: U.S. Patent Publication No. 2018 / 0069060 SUMMARY

[0006] PROBLEMS TO BE SOLVED BY THE INVENTION

[0007] However, the display panel includes a plurality of layers, and some layers have a low visible light transmittance. Therefore, when the object light passing through the display panel is captured by the camera module, the captured image becomes dark or an overall blurred image. In addition, when the object light passes through the display panel, there is also a possibility that the image quality of the captured image is deteriorated due to the influence of flare or diffraction.

[0008] In addition, in the case where the camera module is disposed on the surface of a small electronic device such as a smart phone, since the lens is thin and the diameter of the lens cannot be increased, the captured image becomes dark and it is easy to obtain an unsharp image in the case where the surrounding environment is dark.

[0009] The disclosure provides an electronic device and an imaging device capable of obtaining a high-quality captured image even in the case where the amount of incident light is small.

[0010] SOLUTION TO PROBLEM

[0011] To solve the above problem, according to the disclosure, there is provided an electronic device including:

[0012] a display unit;

[0013] a first imaging unit disposed on a side opposite to a display surface of the display unit and capable of capturing an image of light of an infrared light wavelength band passing through the display unit;

[0014] a second imaging unit disposed on a side opposite to the display surface of the display unit and capable of capturing an image of light of a visible light wavelength band passing through the display unit; and

[0015] The correction unit corrects the image data imaged by the second imaging unit based on the image data imaged by the first imaging unit.

[0016] The correction unit can correct the sensitivity of the image data imaged by the second imaging unit based on the image data imaged by the first imaging unit.

[0017] A learning unit that learns a correlation between the sensitivity of the image data imaged by the first imaging unit and the sensitivity of the image data imaged by the second imaging unit can be provided, and the correction unit can correct the sensitivity of the image data imaged by the second imaging unit based on the image data imaged by the first imaging unit with reference to a learning result in the learning unit.

[0018] The correction unit can correct the resolution of the image data imaged by the second imaging unit based on the image data imaged by the first imaging unit.

[0019] A learning unit that learns a correlation between the resolution of the image data imaged by the first imaging unit and the resolution of the image data imaged by the second imaging unit can be provided, and the correction unit can correct the resolution of the image data imaged by the second imaging unit based on the image data imaged by the first imaging unit with reference to a learning result in the learning unit.

[0020] The correction unit can correct at least one of a flare component and a diffracted light component included in the image data imaged by the second imaging unit based on the image data imaged by the first imaging unit.

[0021] A learning unit that learns a correlation between at least one of a flare component and a diffracted light component included in the image data imaged by the first imaging unit and at least one of a flare component and a diffracted light component included in the image data imaged by the second imaging unit can be provided, and the correction unit can correct at least one of a flare component and a diffracted light component included in the image data imaged by the second imaging unit based on the image data imaged by the first imaging unit with reference to a learning result in the learning unit.

[0022] A learning unit that learns a correlation between at least one of a flare component and a diffracted light component included in the image data imaged by the first imaging unit and at least one of a flare component and a diffracted light component included in the image data imaged by the second imaging unit can be provided, and the correction unit can correct at least one of a flare component and a diffracted light component included in the image data imaged by the second imaging unit based on the image data imaged by the first imaging unit with reference to a learning result in the learning unit.

[0023] A reference determination unit that determines whether at least one of the sensitivity, the resolution, the flare component, and the diffracted light component of the image data imaged by the second imaging unit satisfies a predetermined first reference;

[0024] An imaging start instruction unit that starts imaging by the first imaging unit when the reference determination unit determines that the first reference is not satisfied; and

[0025] The correction process determination unit determines whether or not correction is performed by the correction unit and a type of image data serving as a correction reference when correction is performed by the correction unit, based on a result of comparing at least one of sensitivity, resolution, flare component, and diffracted light component between image data imaged by the first imaging unit and image data imaged by the second imaging unit.

[0026] When the correction process determination unit determines the type of image data serving as a correction reference, the learning unit can learn a correlation between at least one of the sensitivity, the resolution, the flare component, and the diffracted light component of the determined image data and at least one of the sensitivity, the resolution, the flare component, and the diffracted light component of the image data imaged by the second imaging unit.

[0027] a sensor that detects at least one of a shape and a color of the object;

[0028] a reliability estimation unit that estimates reliability of learning by the learning unit;

[0029] an object recognition determination unit that determines whether or not the object can be recognized based on the detection data of the sensor in a case where the reliability estimated by the reliability estimation unit is equal to or less than a predetermined second reference; and

[0030] a color designation determination unit that determines whether or not a color of the object recognized by the sensor can be designated in a case where the object recognition determination unit determines that the object can be recognized, and

[0031] In a case where the color designation determination unit determines that the color of the object can be designated, the correction unit can correct the image data imaged by the second imaging unit to approximate the designated color.

[0032] The correction unit can set a degree of noise removal of a pixel region in which a luminance variation is less than or equal to a predetermined reference value in the image data imaged by the second imaging unit to be higher than a degree of noise removal of a pixel region in which a luminance variation is greater than the reference value in the image data.

[0033] It can be provided that:

[0034] an emission unit that emits light of an infrared light wavelength band; and

[0035] an emission control unit that controls emission timing of the emission unit so that a subject is irradiated with light emitted by the emission unit when image data is imaged by the first imaging unit.

[0036] The emission unit can include a plurality of light sources that emit light of emission wavelength bands different from each other in the infrared light wavelength band;

[0037] The emission control unit can sequentially switch and control emission of the plurality of light sources while the first imaging unit performs imaging,

[0038] The first imaging unit can output a plurality of image data imaged in different emission light wavelength bands from each other; and

[0039] The correction unit can correct the image data imaged by the second imaging unit based on the plurality of image data.

[0040] The light emitting unit can be disposed at a display surface side of the display unit.

[0041] At least one of the first imaging unit and the second imaging unit can include a pixel that captures an image of an infrared light wavelength band and a pixel that captures an image of a visible light wavelength band.

[0042] The first imaging unit can have sensitivity to light of 550 nm or more.

[0043] The correction unit can increase a correction degree toward a shorter wavelength side with respect to the image data imaged by the second imaging unit.

[0044] The first imaging unit can include a photoelectric conversion unit disposed to be longer in a normal direction of a light incident surface than the second imaging unit.

[0045] An area of each pixel of the first imaging unit in a light incident surface direction can be greater than an area of each pixel of the second imaging unit in the light incident surface direction; and

[0046] An area of all pixels of the first imaging unit in the light incident surface direction can be less than an area of all pixels of the second imaging unit in the light incident surface direction.

[0047] According to another aspect of the disclosure, there is provided an imaging apparatus including:

[0048] A first imaging unit disposed at a side opposite to a display surface of a display unit and capable of capturing an image of an infrared light wavelength band;

[0049] A second imaging unit disposed at a side opposite to the display surface of the display unit and capable of capturing an image of a visible light wavelength band; and

[0050] A correction unit that corrects image data imaged by the second imaging unit based on image data imaged by the first imaging unit.

[0051] According to another aspect of the disclosure, there is provided an imaging apparatus including:

[0052] A first imaging unit capable of capturing an image of an infrared light wavelength band;

[0053] A second imaging unit capable of capturing an image of a visible light wavelength band; and

[0054] A correction unit corrects image data imaged by the second imaging unit based on image data imaged by the first imaging unit. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 is an external view of an electronic device equipped with an imaging device and a cross-sectional view taken along line A-A.

[0056] Figure 2 is a cross-sectional view taken along Figure 1 line B-B in FIG. 1.

[0057] Figure 3A is a cross-sectional view showing an example of a cross-sectional structure of the first imaging unit.

[0058] Figure 3B is a cross-sectional view showing an example of a cross-sectional structure of the second imaging unit.

[0059] Figure 4 is a block diagram showing an internal configuration of an electronic device according to the first embodiment.

[0060] Figure 5 is a flowchart showing a first example of a processing operation of a signal processing unit according to the first embodiment.

[0061] Figure 6 is a flowchart showing a second example of a processing operation of a signal processing unit.

[0062] Figure 7 is a block diagram showing an internal configuration of an electronic device according to the second embodiment.

[0063] Figure 8 is a flowchart showing a first example of a processing operation performed by a signal processing unit in an electronic device according to the second embodiment.

[0064] Figure 9 is a flowchart showing a second example of a processing operation performed by a signal processing unit in a case where a multi-spectral sensor is used.

[0065] Figure 10A is a diagram showing a first example of a pixel arrangement.

[0066] Figure 10B is a diagram showing a second example of a pixel arrangement.

[0067] Figure 11 is a block diagram showing a first example of an internal configuration of a signal processing unit.

[0068] Figure 12 is a block diagram showing a second example of an internal configuration of a signal processing unit.

[0069] Figure 13 is a block diagram showing a schematic configuration of an electronic apparatus 2b according to a fifth embodiment.

[0070] Figure 14 is a flowchart showing a processing operation when the electronic apparatus 2b according to the fifth embodiment performs imaging with the first camera module and the second camera module.

[0071] Figure 15 is a flowchart showing a processing operation when the electronic apparatus 2b according to the sixth embodiment performs imaging with the first camera module and the second camera module.

[0072] Figure 16A is a view showing an internal state of a vehicle from a rear side to a front side of the vehicle.

[0073] Figure 16B is a view showing an internal state of a vehicle from an oblique rear side to an oblique front side of the vehicle.

[0074] Figure 17A is a front view of a digital camera as a second application example of the electronic apparatus.

[0075] Figure 17B is a rear view of the digital camera.

[0076] Figure 18A is an external view of an HMD as a third application example of the electronic apparatus.

[0077] Figure 18B is an external view of smart glasses.

[0078] Figure 19 is an external view of a TV as a fourth application example of the electronic apparatus.

[0079] Figure 20 is an external view of a smart phone as a fifth application example of the electronic apparatus. DETAILED DESCRIPTION

[0080] Hereinafter, embodiments of an electronic apparatus and an imaging apparatus will be described with reference to the accompanying drawings. Hereinafter, main components of the electronic apparatus and the imaging apparatus will mainly be described, but the electronic apparatus and the imaging apparatus can have components and functions not shown or described. The following description does not exclude components and functions not shown or described.

[0081] (First Embodiment)

[0082] Figure 1 is an external view and a cross-sectional view taken along a line A-A of an electronic apparatus 2 on which an imaging apparatus 1 according to the first embodiment is mounted, and Figure 2 is a cross-sectional view taken along Figure 1a cross-sectional view taken along the line B-B. The electronic device 2 according to the present embodiment is any electronic device 2 having both a display function and a capturing function, such as a smartphone, a mobile phone, a tablet, or a PC. Figure 1 The electronic device 2 includes the camera modules (first and second camera modules) 4 and 5 arranged on opposite sides of the display surface 3a of the display unit 3. As described above, in the electronic device 2 according to the present embodiment, the camera modules 4 and 5 are arranged on the back surface side of the display surface 3a of the display unit 3. Thus, the camera modules 4 and 5 perform capturing through the display unit 3. Figure 1 In the electronic device 2 according to the present embodiment, the camera modules 4 and 5 are arranged on the back surface side of the display surface 3a of the display unit 3. Thus, the camera modules 4 and 5 perform capturing through the display unit 3.

[0083] The imaging device 1 according to the present embodiment includes a plurality of camera modules 4 and 5. In the present specification, an example in which two camera modules 4 and 5 are provided will be mainly described, but three or more camera modules can be provided. Hereinafter, an example in which the imaging device 1 includes two camera modules 4 and 5 will be mainly described. As described later, the camera module 4 includes a first imaging unit 6, and the camera module 5 includes a second imaging unit 7. The first imaging unit 6 can capture an image of light of an infrared light wavelength band that passes through the display unit 3. The infrared light wavelength band is, for example, a wavelength band in the range of 780 nm to 1000 nm. The second imaging unit 7 can capture an image of light of a visible light wavelength band that passes through the display unit 3. The visible light wavelength band is, for example, a wavelength band in the range of 380 nm to 750 nm.

[0084] In the electronic device 2 according to the present embodiment, the display surface 3a extends to the vicinity of the outer dimension of the electronic device 2, and the width of the bezel 3b around the display surface 3a is set to several mm or less. Generally, a front camera is usually mounted on the bezel 3b, but in the electronic device 2 according to the present embodiment, the camera modules 4 and 5 are arranged on the back surface side of the display surface 3a, and thus the front camera is not arranged in the bezel 3b. Thus, the width of the bezel 3b can be made narrow. Figure 1 In the electronic device 2 according to the present embodiment, the camera modules 4 and 5 arranged on the back surface side of the substantially central portion of the display surface 3a serve as front cameras, as indicated by the dotted circles. In this way, by providing the front cameras on the back surface side of the display surface 3a, it is not necessary to provide the front cameras within the bezel 3b, and the width of the bezel 3b can be made narrow.

[0085] Note that in the electronic device 2 according to the present embodiment, the camera modules 4 and 5 are arranged on the back surface side of the substantially central portion of the display surface 3a, but the camera modules 4 and 5 can also be arranged at any position on the back surface side of the display surface 3a, for example, the camera modules 4 and 5 can be arranged on the back surface side of the peripheral portion of the display surface 3a. In this way, the plurality of camera modules 4 and 5 in the present embodiment are arranged at any position on the back surface side overlapping the display surface 3a. Figure 1 As described above, in the electronic device 2 according to the present embodiment, the camera modules 4 and 5 are arranged on the back surface side of the display surface 3a of the display unit 3. Thus, the camera modules 4 and 5 perform capturing through the display unit 3.

[0086] Figure 2 and Figure 1 ​As shown, the display unit 3 is a laminate of a protective film 3c, a polyimide substrate 3d, a display layer 3e, a barrier layer 3f, a touch sensor layer 3g, an adhesive layer 3h, a circularly polarizing plate 3i, an optical clear adhesive (OCA) 3j, and a cover glass 3k, which are sequentially laminated. The display layer 3e can be, for example, an organic light emitting device (OLED) display layer, a liquid crystal display layer, a micro-LED, or a display layer based on another display principle. The display layer 3e can include a plurality of layers. For example, the display layer 3e can include a color filter layer, a backlight layer, and the like. The display unit 3 performs display using light in the visible light wavelength range, but the light displayed on the display unit 3 can include an infrared light component.

[0087] The barrier layer 3f is a layer that prevents oxygen and moisture from entering the display layer 3e. A touch sensor is incorporated in the touch sensor layer 3g. There are various types of touch sensors, such as capacitive and resistive film types, but any type can be employed. Furthermore, the touch sensor layer 3g and the display layer 3e can be integrated.

[0088] The adhesive layer 3h is provided to adhere the circularly polarizing plate 3i and the touch sensor layer 3g. A material having a high visible light transmittance is used for the adhesive layer 3h. The circularly polarizing plate 3i is provided to reduce glare and enhance the visibility of the display surface 3a even in bright environments. The optical clear adhesive 3j is provided to enhance the adhesiveness of the circularly polarizing plate 3i to the cover glass 3k. A material having a high visible light transmittance is used for the optical clear adhesive 3j. The cover glass 3k is provided to protect the display layer 3e and the like. Note that the layer configuration of the display unit 3 is not necessarily limited to the configuration shown in FIG. 1. Figure 2 and Figure 2 the configuration shown in FIG. 1.

[0089] As Figure 1 shown, the camera module 4 includes a first optical system 8 associated with the first imaging unit 6. Similarly, the camera modules 4 and 5 have a second optical system 9 associated with the second imaging unit 7. The first optical system 8 and the second optical system 9 are arranged on the light-incident surface side of the first imaging unit 6 and the second imaging unit 7, i.e., the side close to the display unit 3, and condense light passing through the display unit 3 to the first imaging unit 6 and the second imaging unit 7. The optical system is generally composed of a plurality of lenses, but the specific optical configuration of the optical system is not limited.

[0090] As described later, the first imaging unit 6 includes a first photoelectric conversion unit, and the second imaging unit 7 includes a second photoelectric conversion unit. The first photoelectric conversion unit and the second photoelectric conversion unit photoelectrically convert light incident via the display unit 3. The first photoelectric conversion unit and the second photoelectric conversion unit have different light wavelength bands capable of performing photoelectric conversion. The first photoelectric conversion unit mainly photoelectrically converts light of an infrared light wavelength band, and the second photoelectric conversion unit mainly photoelectrically converts light of a visible light wavelength band. The first photoelectric conversion unit and the second photoelectric conversion unit can be complementary metal-oxide semiconductor (CMOS) sensors or charge-coupled device (CCD) sensors. Furthermore, the photoelectric conversion unit can be a photodiode or an organic photoelectric conversion film.

[0091] The first photoelectric conversion unit and the second photoelectric conversion unit each have a photoelectric conversion element such as a CMOS sensor for each pixel. Each pixel can be arranged in any manner. Specifically, the arrangement system of each pixel can be a Bayer arrangement, an interline arrangement, a quad arrangement, a stripe arrangement, or another arrangement.

[0092] As shown in Figure 2 and Figure 1 In the electronic device 2 according to the present embodiment, the plurality of camera modules 4 and 5 capture images of an object light transmitted through the display unit 3. As shown in Figure 3A The display unit 3 includes a plurality of layers, and each layer is not a problem as long as the transmittance of light of a wavelength band having sufficient sensitivity of the first imaging unit 6 and the second imaging unit 7 is high. However, in reality, there is a possibility that the transmittance of certain layers is low. For example, in the polyimide substrate 3d, although the visible light transmittance is not high, the infrared light transmittance is higher than the visible light transmittance.

[0093] Therefore, in the present embodiment, a correction unit described later is provided, and the captured image data of the second imaging unit 7 that captures light of a visible light wavelength band is corrected based on the captured image data of the first imaging unit 6 that captures light of an infrared light wavelength band.

[0094] Figure 3B is a cross-sectional view showing an example of a cross-sectional structure of the first imaging unit 6, and Figure 3A is a cross-sectional view showing an example of a cross-sectional structure of the second imaging unit 7. Figure 3B The first imaging unit 6 shown in includes a first photoelectric conversion unit 12a formed in a semiconductor substrate 11, and for each pixel, the first photoelectric conversion unit 12a is divided by an element isolation layer 13. A planarization layer 14 is provided on the first photoelectric conversion unit 12a, and an on-chip lens 15a is provided on the planarization layer 14. Light is incident through the on-chip lens 15a. Therefore, the on-chip lens 15a becomes a light incident surface. In the present specification, the side on which the on-chip lens 15a is provided is referred to as the back surface side of the first imaging unit 6.

[0095] On the other hand, in Figure 3A The second imaging unit 7 illustrated in FIG. 1 includes a second photoelectric conversion unit 12b formed in a semiconductor substrate, and for each pixel, the second photoelectric conversion unit 12b is divided by an element isolation layer 13. A planarization layer 14 is provided on the second photoelectric conversion unit 12b, and a color filter layer 16 is provided on the planarization layer 14. The color filter layer 16 can have a color filter layer of RGB three colors, or can have a color filter layer of cyan, magenta, and yellow which are complementary colors thereof. An on-chip lens 15b is provided on the color filter layer 16.

[0096] As can be seen by comparing Figure 3B and Figure 3A A cross-sectional view of FIG. 1, the first photoelectric conversion unit 12a of the first imaging unit 6 is longer than the second photoelectric conversion unit 12b of the second imaging unit 7 in the normal direction of the light incident surface. This is because the photoelectric conversion efficiency of infrared light of the infrared light wavelength band is poorer than the photoelectric conversion efficiency of light of the visible light wavelength band, and the length of the light incident surface of the first photoelectric conversion unit 12a in the normal direction is made longer to improve the photoelectric conversion efficiency.

[0097] In addition, since infrared light is more difficult to refract than visible light, as illustrated in Figure 3B and Figure 3A The curvature of the on-chip lens 15a of the first imaging unit 6 can be made larger than the curvature of the on-chip lens 15b of the second imaging unit 7.

[0098] It is also possible to make the area of each pixel of the first imaging unit 6 in the light incident surface direction larger than the area of each pixel of the second imaging unit 7 in the light incident surface direction. The first imaging unit 6 is provided for correcting the sensitivity and the like of the captured image of the second imaging unit 7, and since it is possible to improve the sensitivity, a larger area is desired.

[0099] On the other hand, the area in the light incident surface direction (effective pixel area) of all pixels including the second imaging unit 7 can be larger than the area in the light incident surface direction (effective pixel area) of all pixels including the first imaging unit 6. The first imaging unit 6 is used for correcting the captured image of the second imaging unit 7, and for example, can be provided in a ratio of one pixel to a plurality of pixels of the second imaging unit 7. Note that the size and the number of pixels of the first imaging unit 6 and the second imaging unit 7 are not limited to the above-described size and number of pixels. Specific layout arrangement examples of the first imaging unit 6 and the second imaging unit 7 will be described later.

[0100] On the surface side of the first imaging unit 6 and the second imaging unit 7 (the side opposite to the on-chip lenses 15a and 15b), a read circuit 17 is formed on the semiconductor substrate 11, and the periphery of the read circuit 17 is covered with an interlayer insulating film 18. The read circuit 17 includes a transfer transistor, a reset transistor, an amplification transistor, a selection transistor, and the like. Note that the cross-sectional structure of the first imaging unit 6 and the second imaging unit 7 is not limited to those shown in Figure 3B and Figure 4 .

[0101] Figure 4 is a block diagram showing an internal configuration of the electronic apparatus 2 according to the first embodiment. As shown in Figure 5 , the electronic apparatus 2 includes the imaging apparatus 1, an application processor 21, a video signal generation unit 22, an A / D conversion unit 23, a display control unit 24, and a display unit 3.

[0102] The imaging apparatus 1 can include one or a plurality of semiconductor devices. More specifically, the imaging apparatus 1 includes the first camera module 4 and the second camera module 5, a first optical system 8 corresponding to the first imaging unit 6, a second optical system 9 corresponding to the second imaging unit 7, and an infrared (IR) cut filter 10, a first A / D conversion unit 31 corresponding to the first imaging unit 6, a second A / D conversion unit 32 corresponding to the second imaging unit 7, a signal processing unit 33, an imaging control unit 34, and an output unit 35. The first and second camera modules 4 and 5 can be mounted on a common substrate.

[0103] The first A / D conversion unit 31 converts an analog pixel signal photoelectrically converted by the first photoelectric conversion unit 12a into digital pixel data. The second A / D conversion unit 32 converts an analog pixel signal photoelectrically converted by the second photoelectric conversion unit 12b into digital pixel data.

[0104] As described later, the signal processing unit 33 generates image data corresponding to captured images of the first imaging unit 6 and the second imaging unit 7. The signal processing unit 33 functions as a correction unit that corrects the sensitivity of image data imaged by the second imaging unit 7 based on image data imaged by the first imaging unit 6. The imaging control unit 34 controls whether or not the first imaging unit 6 performs imaging in accordance with an instruction from the signal processing unit 33.

[0105] The application processor 21 is a semiconductor device separate from the first camera module 4 and the second camera module 5, and is mounted on the same or different substrate as the first camera module 4 and the second camera module 5. The application processor 21 includes a central processing unit (CPU) or the like therein, and executes programs such as an operating system and various application software. The application processor 21 can have a function of performing image processing, signal processing, or the like of a graphic processing unit (GPU), a baseband processor, or the like. The application processor 21 performs various processing on input image data and calculation results as needed, performs control to display an image on the display unit 3 of the electronic device 2, or performs transmission to an external cloud server via a predetermined network.

[0106] The video signal generation unit 22 generates a video signal to be displayed on the display unit 3. The A / D conversion unit 23 converts the video signal into digital pixel data. The display control unit 24 performs control to display the digital pixel data on the display unit 3.

[0107] Figure 5 is a flowchart showing a first example of a processing operation of the signal processing unit 33 according to the first embodiment. Figure 5 The flowchart of corrects the sensitivity of the image data imaged by the second imaging unit 7 based on the image data imaged by the first imaging unit 6.

[0108] First, it is determined whether or not the pixel value of the image data imaged by the second imaging unit 7 is equal to or less than a predetermined threshold value (step S1). Here, the average pixel value of the image data imaged by the second imaging unit 7 can be compared with the predetermined threshold value, or the pixel value of a partial pixel region of the image data imaged by the second imaging unit 7 can be compared with the predetermined threshold value.

[0109] In a case where it is determined in step S1 that the pixel value is not equal to or less than the predetermined threshold value, it can be estimated that the image data imaged by the second imaging unit 7 has sufficient sensitivity, and thus predetermined signal processing is performed based on the image data imaged by the second imaging unit 7 to generate and output RGB data (step S2).

[0110] On the other hand, in a case where it is determined in step S1 that the pixel value is equal to or less than the predetermined threshold value, it can be estimated that the sensitivity of the image data imaged by the second imaging unit 7 is insufficient, and thus the imaging start is instructed to the first imaging unit 6 via the imaging control unit (step S3). The first imaging unit 6 can include a plurality of imaging sensors capable of individually capturing images of light in a plurality of narrow wavelength bands (for example, 750 nm, 800 nm, 850 nm, or the like) in the infrared light wavelength band.

[0111] Then, the pixel value of the image data imaged by the first imaging unit 6 and the pixel value of the image data imaged by the second imaging unit 7 are compared (step S4). In step S4, for example, it is determined whether the pixel value of the image data imaged by the first imaging unit 6 is X times (X is a predetermined reference value larger than 1) or more of the pixel value of the image data imaged by the second imaging unit 7. The value of X can be set and changed by the user.

[0112] When determined as "No" in step S4, it can be estimated that the image data imaged by the first imaging unit 6 is not sensitive, and thus the processing of the above-described step S2 is performed without correction processing. When determined as "Yes" in step S4, it can be estimated that the sensitivity of the image data imaged by the first imaging unit 6 is sufficiently high, and thus it is determined whether there is a pixel having a pixel value equal to or greater than a predetermined threshold in the image data imaged by the second imaging unit 7 (step S5). The reason for setting the determination processing of step S5 is that if there is pixel data having a large pixel value in the captured image data of the second imaging unit 7 that captures light of the visible light wavelength band, the pixel data is considered to be valid data of high sensitivity.

[0113] When determined as "Yes" in step S5, the pixel data having a pixel value equal to or greater than the predetermined threshold in the image data imaged by the second imaging unit 7 and the image data imaged by the first imaging unit 6 are selected as valid data (step S6).

[0114] When determined as "No" in step S5, it can be estimated that the image data imaged by the second imaging unit 7 is not reliable, and thus the image data imaged by the first imaging unit 6 is selected as valid data (step S7).

[0115] Next, the image data imaged by the second imaging unit 7 is corrected based on the valid data selected in step S6 or S7 (step S8). Here, for example, a learning model that learns the correlation between the sensitivity of the image data imaged by the first imaging unit 6 and the sensitivity of the image data imaged by the second imaging unit 7 is generated in advance. For example, a convolutional neural network (CNN) is used to extract features of the image to appropriately perform sensitivity correction, and a learning model that can automatically correct the sensitivity of the image data imaged by the second imaging unit 7 is generated using the image data imaged by the first imaging unit 6. The series of processes for generating the learning model is called machine learning. By inputting the valid data selected in step S6 or S7 to the learning model that has been sufficiently learned to perform operation processing, the corrected image data is output from the learning model. The output image data is data in which the sensitivity is appropriately corrected for each color component of RGB.

[0116] As described above, in a case where the pixel value of the image data imaged by the second imaging unit 7 is large and the reliability is high, the signal processing unit 33 outputs the image data without correcting the sensitivity, and in a case where the pixel value of the image data imaged by the second imaging unit 7 is small and the reliability is high, the signal processing unit 33 corrects the sensitivity of the image data imaged by the second imaging unit 7 on the basis of the image data imaged by the first imaging unit 6. At the time of the correction, the correlation between the sensitivity of the image data imaged by the first imaging unit 6 and the sensitivity of the image data imaged by the second imaging unit 7 is learned in advance, and the correction is performed using the learning result.

[0117] In Figure 5 , an example of correcting the sensitivity of the image data imaged by the second imaging unit 7 has been described. However, in a case of correcting the resolution of the image data imaged by the second imaging unit 7, a similar process can also be applied. When a part of the light incident on the second camera module 5 is lost due to the display unit 3, a decrease in the resolution occurs. Therefore, by performing a similar process to that in Figure 6 , the image data imaged by the second imaging unit 7 is corrected on the basis of the image data imaged by the first imaging unit 6, so that the resolution of the image data imaged by the second imaging unit 7 can be improved. At this time, the correction process in step S8 requires a different process from the sensitivity correction. That is, in the case of the sensitivity correction, a learning model for appropriately correcting the sensitivity is generated, but in the case of the resolution correction, a learning model for appropriately correcting the resolution is generated, and the effective data selected in step S6 or S7 can be input to the learning model to obtain the image data of the resolution correction at a stage at which the learning model is sufficiently learned. More specifically, the learning model in the case of correcting the resolution learns the correlation between the resolution of the image data imaged by the first imaging unit 6 and the resolution of the image data imaged by the second imaging unit 7. The signal processing unit 33 serving as the correction unit refers to the learning result in the learning unit, and corrects the resolution of the image data imaged by the second imaging unit 7 on the basis of the image data imaged by the first imaging unit 6.

[0118] The signal processing unit 33 serving as the correction unit can increase the degree of correction toward the shorter wavelength side for the image data imaged by the second imaging unit 7. This is because the blue component of the subject light passing through the display unit 3 is more easily absorbed by the display unit 3.

[0119] Although the example of performing the sensitivity correction or the resolution correction on the image data imaged by the second imaging unit 7 has been described above, there is a possibility that the object light is reflected or diffracted when passing through the display unit 3, and the object light affected by the flare due to the reflection or the diffraction is incident on the first camera module 4 and the second camera module 5. Therefore, the signal processing unit 33 can perform the processing of correcting the effect of the flare or the diffracted light.

[0120] Figure 6 is a flowchart showing a second example of the processing operation of the signal processing unit 33. Figure 5 The flowchart of corrects the effect of the flare or the diffraction of the image data imaged by the second imaging unit 7 based on the image data imaged by the first imaging unit 6.

[0121] First, it is determined whether the pixel value of the image data imaged by the second imaging unit 7 is a predetermined threshold value or more (step Sll). In a case where the light incident on the second imaging unit 7 is affected by the flare or the diffraction, the pixel value generally increases. Therefore, in step Sll, it is determined whether the pixel value of the image data imaged by the second imaging unit 7 is a predetermined threshold value or more, and if the pixel value is less than the threshold value, it is determined that the image data is not affected by the flare or the diffraction, and predetermined signal processing is performed based on the image data imaged by the second imaging unit 7 to generate and output the RGB data (step S12).

[0122] On the other hand, in a case where it is determined in step Sll that the pixel value is equal to or more than the predetermined threshold value, it is estimated that the image data imaged by the second imaging unit 7 is affected by the flare or the diffraction, and therefore the imaging start is instructed to the first imaging unit 6 via the imaging control unit (step S13).

[0123] Then, the pixel value of the image data imaged by the first imaging unit 6 is compared with the pixel value of the image data imaged by the second imaging unit 7 (step S14). Here, as in step S4 in Figure 5 , for example, it is determined whether the pixel value of the image data imaged by the first imaging unit 6 is X times (X is a predetermined reference value larger than 1) or less than the pixel value of the image data imaged by the second imaging unit 7.

[0124] When "No" is determined in step S14, it can be estimated that the flare or the diffraction cannot be corrected even if the image data imaged by the first imaging unit 6 is used. Therefore, the process of step S12 is performed without correction processing. When "Yes" is determined in step S14, it can be estimated that the image data imaged by the first imaging unit 6 is not affected by the flare or the diffraction, and thus it is determined whether there is a pixel whose pixel value is equal to or less than a predetermined threshold in the image data imaged by the second imaging unit 7 (step S15).

[0125] When "Yes" is determined in step S15, the pixel data whose pixel value is equal to or less than the predetermined threshold in the image data imaged by the second imaging unit 7 and the image data imaged by the first imaging unit 6 are selected as valid data (step S16).

[0126] When "No" is determined in step S15, it can be estimated that the image data imaged by the second imaging unit 7 is unreliable, and thus the image data imaged by the first imaging unit 6 is selected as valid data (step S17).

[0127] Next, the image data imaged by the second imaging unit 7 is corrected on the basis of the valid data selected in step S16 or step S17 (step S18). Here, for example, a learning model in which correlation between at least one of a flare component and a diffraction light component included in the image data imaged by the first imaging unit 6 and at least one of a flare component and a diffraction light component included in the image data imaged by the second imaging unit 7 is learned in advance is generated, and the valid data selected in step S16 or S17 is input to the learning model in which sufficient learning has been performed, so that corrected image data is output from the learning model. The output image data is data in which the influence of the flare or the diffraction is appropriately corrected for each color component of RGB.

[0128] SUMMARY Figure 6 and Figure 5The signal processing unit 33 executes the processes of the flowchart of FIG. 9. The signal processing unit 33 executes the processes of the reference determination unit (steps S1, S11), the imaging start instruction unit (steps S2, S12), and the correction process determination unit (steps S3 to S8, S13 to S18). The reference determination unit determines whether at least one of the sensitivity, the resolution, the flare component, and the diffracted light component of the image data imaged by the second imaging unit 7 satisfies a predetermined first reference. When the reference determination unit determines that the first reference is not satisfied, the imaging start instruction unit starts imaging by the first imaging unit 6. The correction process determination unit determines whether correction is executed by the correction unit and the type of the image data serving as a reference for the correction when the correction is executed by the correction unit based on the result of comparing at least one of the sensitivity, the resolution, the flare component, and the diffracted light component between the image data imaged by the first imaging unit 6 and the image data imaged by the second imaging unit 7. When the type of the image data serving as a reference for the correction is determined by the correction process determination unit, the signal processing unit 33 (learning unit) learns the correlation between at least one of the sensitivity, the resolution, the flare component, and the diffracted light component of the determined image data and at least one of the sensitivity, the resolution, the flare component, and the diffracted light component of the image data imaged by the second imaging unit 7.

[0129] Note that the process of Figure 6 and the process of Figure 5 may be executed in combination. That is, the signal processing unit 33 can correct two or more of the influence of the sensitivity, the resolution, the flare component, and the influence of the diffracted light component of the image data imaged by the second imaging unit 7 based on the image data imaged by the first imaging unit 6.

[0130] As described above, in the first embodiment, the image data imaged by the second imaging unit 7 that captures light of the visible light wavelength band is corrected as necessary based on the image data imaged by the first imaging unit 6 that captures light of the infrared light wavelength band. Therefore, even if the object light is incident on the first camera module 4 and the second camera module 5 through the display unit 3, there is no possibility of a decrease in sensitivity or resolution, and the object light is not affected by flare or diffraction.

[0131] (Second Embodiment)

[0132] In the second embodiment, measures are taken in the case where sufficient learning is not executed in the machine learning executed in step S8 of Figure 6 or step S18 of Figure 7 .

[0133] Figure 4 is a block diagram showing the internal configuration of the electronic apparatus 2a according to the second embodiment. In addition to the configuration of Figure 7 , Figure 1The electronic device 2a further includes a depth sensor 36. The depth sensor 36 is a sensor that detects distance information of an object. The depth sensor 36 can detect distance information by, for example, an indirect time-of-flight (ToF) method. The depth sensor 36 is provided, for example, in a portion of the frame 3b on the display surface side of the electronic device 2 in Figure 8 The distance information detected by the depth sensor 36 is transmitted to the signal processing unit 33.

[0134] Figure 5 is a flowchart showing a first example of a processing operation performed by the signal processing unit 33 in the electronic device 2a according to the second embodiment. After the processing of Figure 6 or Figure 8 is performed, the signal processing unit 33 performs the processing of Figure 5

[0135] First, it is determined whether or not machine learning has been sufficiently performed (step S21). When input data is given to a learning model generated through machine learning, arithmetic processing is performed using the learning model, and output data is generated. If the input data is data within a range assumed in advance, appropriate output data reflecting the result of machine learning can be obtained by performing interpolation processing. However, in the case where the input data is data outside the range assumed in advance, extrapolation processing needs to be performed, and the accuracy of the output data decreases. Therefore, in step S21, it is determined whether or not machine learning has been sufficiently performed, for example, on the basis of whether or not extrapolation processing is necessary for the input data. Note that whether or not machine learning has been sufficiently performed can be determined by another method.

[0136] In the case where it is determined in step S21 that machine learning has been sufficiently performed, data subjected to arithmetic processing by the learning model is output without performing color correction (step S22).

[0137] In the case where it is determined in step S21 that machine learning has not been sufficiently performed, distance information is detected by the depth sensor 36 (step S23). Next, on the basis of the distance information detected by the depth sensor 36, the shape of an object captured in the corrected imaging data in the processing of Figure 6 or Figure 5 is grasped to identify the object (step S24). In the case where the signal processing unit 33 alone cannot identify the object, the imaging data corrected in the processing of Figure 6 or Figure 5 and the distance information detected by the depth sensor 36 can be transmitted to a cloud server or the like via a network, the shape of the object can be specified by performing shape analysis of the object using a large amount of data (big data) managed by the cloud server or the like, and information of the specified object can be received by the signal processing unit 33 via the network. Alternatively, in the case where Figure 6 or​Figure 8 The imaging data corrected in the process of the application processor 30 and the distance information detected by the depth sensor 36 can be transmitted to the application processor, and the application processor can specify the object.

[0138] Next, it is determined whether the color of the object identified in step S24 is known (step S25). When the color of the object is not known, color correction is abandoned, and the process of step S22 is executed. When the color of the object is known, color correction is performed to approach the known color (step S26). Here, a database for managing color information of the identified object can be provided, and the database can be accessed to acquire the color information of the identified object.

[0139] As described above, in the process of the application processor 30, Figure 7 in the process of the application processor 30, in a case where learning is not sufficiently performed by machine learning, the shape of the object is detected using the depth sensor 36 to identify the object, and color correction is performed based on known color information about the identified object.

[0140] In the process of the application processor 30, Figure 8 and Figure 9 In the process of the application processor 30, an example of detecting the shape of the object by the depth sensor 36 is shown, but information of the object can be detected using another sensor. For example, a multispectral sensor can be provided instead of or in addition to the depth sensor 36.

[0141] Figure 8 is a flowchart showing a second example of the processing operation performed by the signal processing unit 33 in a case where a multispectral sensor is used. The processes in steps S31 to S32 are similar to the processes in steps S21 to S22 in the process of the application processor 30. Figure 8 in the process of the application processor 30. When it is determined in step S31 that machine learning has not been sufficiently performed, color information is detected by the multispectral sensor (step S33). Since the multispectral sensor is capable of detecting light of a plurality of wavelength bands respectively, it is possible to detect color information of the object included in the image data imaged by the second imaging unit 7 in detail. Next, the object is identified from the color information detected in step S33 (step S34). At this time, as described above, the object can be identified by the cloud server or the application processor.

[0142] Next, it is determined whether the color of the object identified in step S34 is known (step S35). In a case where the color of the object is not known, color correction is performed based on the color information detected by the multispectral sensor (step S36). In a case where the color of the object is known, color correction is performed to approach the known color (step S37).

[0143] In summary Figure 9 and Figure 8The signal processing unit 33 executes the processes of the flowchart of FIG. 9. The signal processing unit 33 executes the processes of the reliability estimation unit (steps S21 and S31), the object recognition determination unit (steps S23 and S33), and the color designation determination unit (steps S24 and S34). The reliability estimation unit estimates the reliability of the learning of the learning unit. In a case where the reliability estimated by the reliability estimation unit is equal to or less than a predetermined second reference, the object recognition determination unit determines whether or not the object can be recognized on the basis of the detection data of the sensor. In a case where it is determined by the object recognition determination unit that the object can be recognized, the color designation determination unit determines whether or not the color of the object recognized by the sensor can be designated. In a case where it is determined by the color designation determination unit that the color of the object can be designated, the signal processing unit 33 serving as the correction unit corrects the image data imaged by the second imaging unit 7 to approximate the designated color.

[0144] Note that the processes of Figure 9 and the processes of Figure 10A may be executed in combination. If the object is designated on the basis of the result of detecting the shape information and the color information of the object by the depth sensor 36, the multispectral sensor, or the like, the object can be more accurately designated, and the color correction of the designated object can be accurately executed.

[0145] As described above, in the second embodiment, in a state where the learning by the machine learning is insufficient, it is not preferable to correct the image data imaged by the second imaging data. Therefore, in a case where the learning by the machine learning is insufficient, the color correction of the image data imaged by the second imaging unit 7 is performed using another sensor such as the depth sensor 36 or the multispectral sensor. Therefore, it is possible to avoid the possibility of performing the correction with low reliability in a state where the learning by the machine learning is insufficient.

[0146] (Third Embodiment)

[0147] In the third embodiment, the pixel arrangement of the first imaging unit 6 and the second imaging unit 7 is characterized. The first imaging unit 6 that receives and photoelectrically converts light of the infrared light wavelength band includes a plurality of pixels. Similarly, the second imaging unit 7 that receives and photoelectrically converts light of the visible light wavelength band also includes a plurality of pixels. A part of the plurality of pixels constituting the first imaging unit 6 can include a pixel capable of receiving light of the visible light wavelength band.

[0148] Figure 10A is a diagram showing a first example of the pixel arrangement. In the first example shown in Figure 10A , a pixel that receives and photoelectrically converts light of the infrared light wavelength band is denoted as IR, and a pixel that receives and photoelectrically converts light of a wavelength band corresponding to red is denoted as R. In the first example shown in Figure 10BIn the example, IR pixels and R pixels are arranged in an alternating manner. Note that the arrangement order of IR pixels and R pixels, as well as the ratio of their respective numbers, is arbitrary.

[0149] Red light has a wavelength component close to that of infrared light (e.g., 550 nm or greater). By arranging R pixels between IR pixels, the wavelength band of light that the first imaging unit 6 can receive can be further extended. When the signal processing unit 33 corrects the image data of the second imaging unit 7 based on the image data of the first imaging unit 6, the image data of the first imaging unit 6 includes a red component, thus facilitating the adjustment of the red component included in the image data of the second imaging unit 7, and enabling highly reliable correction processing.

[0150] Figure 10B This is a diagram illustrating a second example of pixel arrangement. Figure 10B In the second example shown, a portion of the pixels constituting the second imaging unit 7 includes pixels capable of receiving light in the infrared wavelength band. Figure 11 In this design, pixels that receive and photoelectrically convert visible light wavelengths are represented as R, G, or B, and pixels that receive and photoelectrically convert infrared light wavelengths are represented as IR. One IR pixel is assigned for every three RGB pixels. Note that the arrangement order of the RGB and IR pixels, as well as the ratio of their respective numbers, is arbitrary.

[0151] By arranging IR pixels between RGB pixels, a first imaging unit 6 and a second imaging unit 7 can be formed on a single semiconductor chip. Therefore, only one camera module 4 or 5 is required, and the hardware cost of the electronic device 2 can be reduced.

[0152] As described above, in the third embodiment, by arranging pixels that receive light in the visible light wavelength band among pixels that receive light in the infrared light wavelength band, the reliability of the correction processing performed by the signal processing unit 33 can be improved. Furthermore, by arranging pixels that receive light in the infrared light wavelength band among pixels that receive light in the visible light wavelength band, the first imaging unit 6 and the second imaging unit 7 can be formed on a single semiconductor chip.

[0153] (Fourth Embodiment)

[0154] In the first embodiment, an example of using machine learning to correct image data imaged by the second imaging unit 7 has been described. However, the image data imaged by the second imaging unit 7 can be corrected by the signal processing unit 33 without the need for machine learning.

[0155] Figure 11 This is a block diagram showing a first example of the internal configuration of the signal processing unit 33. Figure 12The signal processing unit 33 in the image processing unit 32 includes a clamp unit 33a, a color output unit 33b, a defect correction unit 33c, a linear matrix unit 33d, a gamma correction unit 33e, a luminance chrominance signal generation unit 33f, a noise reduction unit 33g, and an edge enhancement unit 33h.

[0156] The clamp unit 33a performs processing that defines a black level. More specifically, the clamp unit 33a performs processing that subtracts black level data from digital pixel data. The color output unit 33b outputs, for example, pixel data of each color of RGB. The defect correction unit 33c performs processing that corrects imaging data of a specific pixel that cannot be properly read for some reason, based on imaging data of surrounding pixels. The linear matrix unit 33d performs matrix operation on color information such as RGB to perform more correct color reproduction. The gamma correction unit 33e performs gamma correction to enable display with excellent visibility, based on display characteristics of the display unit 3. For example, the gamma correction unit 33e performs conversion from 10 bits to 8 bits while changing the gradient. The luminance chrominance signal generation unit 33f generates luminance chrominance signals to be displayed on the display unit 3, based on output data of the gamma correction unit 33e. The noise reduction unit 33g performs processing that reduces noise included in the luminance chrominance signals. The edge enhancement unit 33h performs processing that enhances edges of the subject image, based on the luminance chrominance signals. The noise reduction processing of the noise reduction unit 33g and the edge enhancement processing of the edge enhancement unit 33h can be performed only in a case where a predetermined condition is satisfied. The output unit 35 outputs the luminance chrominance signals subjected to the noise reduction processing.

[0157] The noise reduction unit 33g determines whether it is a pixel region in which the luminance variation is small, based on the image data imaged by the first imaging unit 6, and increases the noise removal ratio for the pixel region in which the luminance variation is small. Specifically, the noise reduction unit 33g sets the degree of noise removal of a pixel region in which the luminance variation is equal to or smaller than a predetermined reference value in the image data imaged by the second imaging unit 7 to be higher than the degree of noise removal of a pixel region in which the luminance variation is larger than the reference value in the image data.

[0158] Accordingly, it is possible to reduce the data amount of the image data imaged by the second imaging unit 7. On the other hand, for a pixel region in which the luminance variation is large, the noise removal ratio is attenuated. The reason for this is that as the noise removal ratio increases, the edges become unclear and the resolution also becomes lower, and in order to sharpen the edges and improve the resolution, it is desirable to attenuate the noise removal ratio. However, if the noise removal ratio is attenuated for all pixels, the data amount of the image data becomes enormous. Therefore, it is desirable to reduce the noise removal ratio only in a limited pixel region such as an edge portion.

[0159] Figure 11 is a block diagram showing a second example of the internal configuration of the signal processing unit 33. Figure 11The signal processing unit 33 obtains by providing a flare extraction unit 33i and a flare correction unit 33j in the signal processing unit 33. Figure 13 The signal processing unit 33 obtains by providing a flare extraction unit 33i and a flare correction unit 33j in the signal processing unit 33.

[0160] The flare extraction unit 33i compares the pixel value of the image data imaged by the first imaging unit 6 with the pixel value of the image data imaged by the second imaging unit 7, and extracts the degree of influence due to flare or diffraction. For example, the degree of influence due to flare or diffraction can be extracted from the difference between the average pixel value of the image data imaged by the first imaging unit 6 and the average pixel value of the image data imaged by the second imaging unit 7.

[0161] The flare correction unit 33j performs a process of subtracting the degree of influence due to flare or diffraction extracted by the flare extraction unit 33i from the image data imaged by the second imaging unit 7. Therefore, the image data imaged by the second imaging unit 7 is less likely to be affected by flare or diffraction.

[0162] As described above, in the fourth embodiment, by referring to the image data imaged by the first imaging unit 6 in the internal processing of the signal processing unit 33, the sensitivity and resolution of the image data imaged by the second imaging unit 7 can be corrected, and the influence due to flare or diffraction can also be suppressed. Furthermore, by avoiding unnecessary correction, the processing load of the signal processing unit 33 can be reduced, and the data amount of the image data can be reduced.

[0163] (Fifth Embodiment)

[0164] In the first embodiment to the fourth embodiment, an example in which the first imaging unit 6 that receives light of an infrared light wavelength component is provided has been described, but in addition thereto, a light emitting unit that emits light of an infrared light wavelength component can be provided.

[0165] Figure 4 is a block diagram showing a schematic configuration of the electronic device 2b according to the fifth embodiment. In addition to the configuration of the electronic device 2 in Figure 13 Figure 14 The electronic device 2b in The electronic device 2b in

[0166] The light emission control unit 26 controls the light emission timing of the light emission unit 25 so that the subject is irradiated with the light emitted by the light emission unit 25 when the first imaging unit 6 images the image data. Since the human eye cannot recognize the light of the infrared light wavelength band, there is no possibility that a person feels flare even if the person is irradiated with the light of the light emission unit 25. However, it is necessary to irradiate with the light intensity and the continuous irradiation time that do not impair the human eye.

[0167] Note that, in the present embodiment, it is assumed that the first imaging unit 6 and the second imaging unit 7 image the same subject at the same time. Since the first imaging unit 6 captures the light of the infrared light wavelength band, when the subject is imaged by the first imaging unit 6 in the state where the subject is irradiated with the light from the light emission unit 25, it is possible to improve the pixel value of the image data received by the first imaging unit 6, and it is possible to improve the sensitivity.

[0168] Figure 13 is a flowchart showing the processing operation when the electronic apparatus 2b according to the fifth embodiment performs imaging with the first camera module 4 and the second camera module 5. First, it is determined whether it is the imaging timing of the first imaging unit 6 and the second imaging unit 7 (step S41). The imaging of the first imaging unit 6 and the second imaging unit 7 is performed in a period in which the image is not displayed on the display unit 3, for example, in the vertical blanking period of the display unit 3.

[0169] The processing waits until the imaging timing at step S41, and at the imaging timing, the light emission unit 25 starts to emit the light of the infrared light band (IR light) (step S42). For example, in the case where the electronic apparatus 2b is a smartphone, in order to make the user focus the first camera module 4 and the second camera module 5 on the object, if the traveling direction of the light emitted from the light emission unit 25 coincides with the optical axis direction of the first camera module 4 and the second camera module 5, the light from the light emission unit 25 irradiates the subject even if the user does not particularly realize it.

[0170] The first imaging unit 6 performs imaging when the light emission unit 25 emits light (step S43). Therefore, even in the case where the capture is performed in a situation where the surroundings are dark, it is possible to improve the pixel value of the image data imaged by the first imaging unit 6. At the end of the imaging of the first imaging unit 6 and the second imaging unit 7, the light emission unit 25 stops emitting light (step S44). After that, as with the first embodiment and the like, the image data imaged by the second imaging unit 7 is corrected on the basis of the image data imaged by the first imaging unit 6 (step S45).

[0171] As described above, in the fifth embodiment, since the light of the infrared light wavelength band is received by the first imaging unit 6 in a state where the infrared light is emitted by the light emitting unit 25, even in a case where the surroundings are dark, it is possible to improve the pixel value of the image data imaged by the first imaging unit 6, it is possible to improve the sensitivity, and it is possible to appropriately correct the image data imaged by the second imaging unit 7 based on the image data imaged by the first imaging unit 6.

[0172] (Sixth Embodiment)

[0173] The sixth embodiment is a modification of the fifth embodiment, and it is possible to switch the light emitting wavelength band of the light emitted by the light emitting unit 25.

[0174] The electronic apparatus 2b according to the sixth embodiment has a similar block configuration to that in Figure 15 , but the function of the light emitting unit 25 is different. The light emitting unit 25 according to the sixth embodiment includes a plurality of light sources that emit light of different light emitting wavelength bands in the infrared light wavelength band. The light emitting control unit 26 sequentially switches and controls the light emission of the plurality of light sources while the first imaging unit 6 is imaging. The first imaging unit 6 outputs a plurality of image data imaged in the different light emitting wavelength bands.

[0175] Figure 16A is a flowchart showing the processing operation when the electronic apparatus 2b according to the sixth embodiment performs imaging with the first camera module 4 and the second camera module 5. First, it is determined whether it is the imaging timing of the first imaging unit 6 and the second imaging unit 7 (step S51). At the imaging timing, the light emitting unit 25 selects one of the plurality of light emitting wavelength bands according to the instruction from the light emitting control unit 26 (step S52), and emits the infrared light of the selected light emitting wavelength band (step S53). Then, imaging is performed by the first imaging unit 6 and the second imaging unit 7 (step S54), and then the light emission of the light emitting unit 25 is stopped (step S55).

[0176] Next, it is determined whether there is infrared light of a light emitting wavelength band that has not been selected among the light that the light emitting unit 25 can emit (step S56). In a case where there is infrared light that has not been selected, the processing in steps S52 and subsequent steps is repeated. In a case where it is determined in step S56 that the processing of steps S52 to S55 is performed on all of the light that the light emitting unit can emit, the image data imaged by the second imaging unit 7 is corrected based on the image data of the plurality of infrared light wavelength bands imaged by the first imaging unit 6 (step S57).

[0177] When the first imaging unit 6 images in a state in which the different plurality of infrared light wavelength bands are illuminated, there is a possibility that the information captured in the captured images hardly changes, and in consideration of these, by correcting the image data imaged by the second imaging unit 7, it is possible to more appropriately correct the image data imaged by the second imaging unit 7.

[0178] As described above, in the sixth embodiment, since the light emitting unit 25 includes a plurality of light sources that emit light of different light emitting wavelength bands in the infrared light wavelength band, in a state in which these light sources sequentially emit light, it is possible to acquire a plurality of captured images imaged by the first imaging unit 6 by imaging by the first imaging unit 6. By correcting the image data imaged by the second imaging unit 7 based on the plurality of captured images, it is possible to perform more appropriate correction.

[0179] (SEVENTH EMBODIMENT)

[0180] In the above-described first embodiment to the sixth embodiment, an example in which the first camera module 4 and the second camera module 5 capture images of light passing through the display surface of the electronic device 2, 2a, 2b has been described. However, in a case in which the first camera module 4 and the second camera module 5 are arranged in the bezel 3b portion of the display surface, or in a case in which the first camera module 4 and the second camera module 5 are arranged on the back surface side of the electronic device 2, 2a, 2b, light from the outside directly enters the first camera module 4 and the second camera module 5, and thus there is no loss of the amount of light due to subject light passing through the display unit 3, and there is no influence due to flare or diffraction. However, in a case in which imaging is performed by the second imaging unit 7 in a state in which the surroundings are dark, there is a case in which the sensitivity and the resolution of the captured image decrease. In addition, in a case in which imaging is performed by the second imaging unit 7 in a state in which strong sunlight is received, the captured image is easily affected by flare or diffraction. In this case, by correcting the image data imaged by the second imaging unit 7 based on the image data imaged by the first imaging unit 6 that receives light of the infrared light wavelength band, it is possible to improve the sensitivity and the resolution, and it is possible to suppress the influence due to flare or diffraction. Thus, the electronic device 2, 2a, 2b according to the first embodiment to the sixth embodiment is also applicable to a case in which the first camera module 4 and the second camera module 5 are arranged in the bezel 3b portion of the display surface, or a case in which the first camera module 4 and the second camera module 5 are arranged on the back surface side of the electronic device 2, 2a, 2b.

[0181] (EIGHTH EMBODIMENT)

[0182] As specific candidates of the electronic apparatus 2 having the configurations described in the first to seventh embodiments, various candidates can be envisaged. For example, FIG. 16 is a plan view when the electronic apparatus 2 in the first to seventh embodiments is applied to a capsule endoscope 50. The capsule endoscope 50 of FIG. 16, for example, includes a case 51 having hemispherical both end surfaces and a cylindrical central portion, a camera (ultra-small camera) 52 for capturing an image in a body cavity, a memory 53 for recording image data imaged by the camera 52, and a wireless transmitter 55 for transmitting the recorded image data to the outside via an antenna 54 after the capsule endoscope 50 is excreted to the outside of a subject.

[0183] Further, a central processing unit (CPU) 56 and a coil (magnetic force / current conversion coil) 57 are provided in the case 51. The CPU 56 controls the capturing of the camera 52 and the data accumulation operation in the memory 53, and controls the data transmission of the wireless transmitter 55 from the memory 53 to a data reception device (not shown) outside the case 51. The coil 57 supplies power to the camera 52, the memory 53, the wireless transmitter 55, the antenna 54, and a light source 52b described later.

[0184] In addition, a magnetic (reed) switch 58 is provided on the case 51 for detecting when the capsule endoscope 50 is disposed at the data reception device. When the reed switch 58 detects the disposition of the data reception device and the data transmission becomes feasible, the CPU 56 supplies power from the coil 57 to the wireless transmitter 55.

[0185] The camera 52, for example, has an imaging element 52a including an objective optical system for capturing an image in a body cavity, and a plurality of light sources 52b for illuminating the body cavity. Specifically, as the light sources 52b, the camera 52, for example, includes a complementary metal-oxide semiconductor (CMOS) sensor, a light-emitting diode (LED), a charge-coupled device (CCD), or the like.

[0186] (Application Examples of the Imaging Apparatus 1 and the Electronic Apparatus 2 According to the Present Disclosure)

[0187] (First Application Example)

[0188] The imaging apparatus 1 and the electronic apparatus 2 according to the present disclosure can be used for various purposes. Figure 16B and Figure 16A is a view showing an internal configuration of a vehicle 100 as a first application example of the electronic apparatus 2 including the imaging apparatus 1 according to the present disclosure. Figure 16B is a view showing an internal state of the vehicle 100 from the rear side to the front side of the vehicle 100, and Figure 16A is a view showing an internal state of the vehicle 100 from the oblique rear side to the oblique front side of the vehicle 100. is a view showing an internal state of the vehicle 100 from the oblique rear side to the oblique front side of the vehicle 100.

[0189] Figure 16B and Figure 16A The vehicle 100 of the present embodiment has a center display 101, a console display 102, a head-up display 103, a digital rearview mirror 104, a steering wheel display 105, and a rear entertainment display 106.

[0190] The center display 101 is disposed on an instrument panel 107 at a position facing a driver seat 108 and a passenger seat 109. Figure 16B and Figure 17A An example is shown in which the center display 101 has a horizontally long shape extending from the driver seat 108 side to the passenger seat 109 side, but the screen size and the arrangement position of the center display 101 are arbitrary. The center display 101 can display information detected by various sensors 5. As specific examples, the center display 101 can display a captured image captured by an image sensor, a distance image to an obstacle in front of or to the side of the vehicle measured by a ToF sensor 5, a body temperature of a passenger detected by an infrared sensor 5, and the like. The center display 101 can be used to display at least one of, for example, safety-related information, operation-related information, a life log, health-related information, authentication / recognition-related information, and entertainment-related information.

[0191] The safety-related information is information such as drowsiness detection, outward looking detection, mischief detection of a child riding together, whether or not a seat belt is worn, and detection of departure of a passenger, and is, for example, information detected by a sensor 5 disposed so as to overlap with the rear surface side of the center display 101. The operation-related information detects a gesture related to the operation of an occupant using the sensor 5. The detected gesture can include the operation of various devices in the vehicle 100. For example, the operation of an air conditioning device, a navigation device, an AV device, an illumination device, and the like is detected. The life log includes a life log of all the occupants. For example, the life log includes a record of the actions of each occupant in the vehicle. By acquiring and storing the life log, the state of the occupant at the time of an accident can be confirmed. The health-related information detects the body temperature of the occupant using a temperature sensor 5, and estimates the health state of the occupant based on the detected body temperature. Alternatively, the face of the occupant can be imaged using an image sensor, and the health condition of the occupant can be estimated from the imaged facial expression. Furthermore, a dialogue can be conducted with the occupant in an automatic voice, and the health state of the occupant can be estimated based on the content of the answer of the occupant. The authentication / recognition-related information includes a keyless entry function that performs face authentication using the sensor 5, a function that automatically adjusts the seat height and position in face recognition, and the like. The entertainment-related information includes a function that detects operation information of an AV device by the occupant using the sensor 5, a function that recognizes the face of the occupant by the sensor 5 and provides content suitable for the occupant by the AV device, and the like.

[0192] The console display 102 can be used to display, for example, life log information. The console display 102 is disposed near a shift lever 111 of a center console 110 between the driver's seat 108 and the passenger's seat 109. The console display 102 can also display information detected by various sensors 5. In addition, the console display 102 can display an image of the periphery of the vehicle imaged by an image sensor, or can display a distance image to an obstacle in the periphery of the vehicle.

[0193] The head-up display 103 is virtually displayed behind the windshield 112 in front of the driver's seat 108. The head-up display 103 can be used to display, for example, at least one of safety-related information, operation-related information, a life log, health-related information, authentication / recognition-related information, and entertainment-related information. Since the head-up display 103 is virtually disposed in front of the driver's seat 108 in many cases, it is suitable to display information directly related to the operation of the vehicle 100, such as the speed of the vehicle 100 and the remaining amount of fuel (battery).

[0194] The digital rearview mirror 104 is capable of displaying not only the rear of the vehicle 100 but also the state of the occupant of the rear seat, and thus can be used to display life log information, for example, by overlapping sensors 5 on the back side of the digital rearview mirror 104.

[0195] The steering wheel display 105 is disposed near the center of the steering wheel 113 of the vehicle 100. The steering wheel display 105 can be used to display, for example, at least one of safety-related information, operation-related information, a life log, health-related information, authentication / recognition-related information, and entertainment-related information. Specifically, since the steering wheel display 105 is close to the hand of the driver, it is suitable to display life log information such as the body temperature of the driver, or to display information related to the operation of an AV device, an air conditioning device, or the like.

[0196] The rear entertainment display 106 is attached to the rear side of the driver's seat 108 and the passenger's seat 109, and is used to be viewed by the occupant of the rear seat. The rear entertainment display 106 can be used to display, for example, at least one of safety-related information, operation-related information, a life log, health-related information, authentication / recognition-related information, and entertainment-related information. Specifically, since the rear entertainment display 106 is in front of the occupant of the rear seat, it displays information related to the occupant of the rear seat. For example, information on the operation of an AV device or an air conditioning device can be displayed, or the result of the measurement of the body temperature of the occupant of the rear seat, or the like, by a temperature sensor 5 can be displayed.

[0197] As described above, by providing the sensor 5 on the back side of the display unit 3 in an overlapping manner, the distance of an object present in the surrounding environment can be measured. Optical distance measurement methods are roughly classified into a passive type and an active type. In the passive type, distance measurement is performed by receiving light from an object without projecting light from the sensor 5 onto the object. The passive type includes a lens focusing method, a stereoscopic method, a monocular vision method, and the like. In the active type, light is projected onto an object, and reflected light from the object is received by the sensor 5 to measure the distance. The active type includes an optical radar method, an active stereo method, an illumination difference stereo method, a moire topography method, an interference method, and the like. The imaging device 1 according to the present disclosure can be applied to any of these types of distance measurement. By using the sensor 5 arranged on the back side of the display unit 3 according to the present disclosure in an overlapping manner, the passive or active distance measurement described above can be performed.

[0198] (Second Application Example)

[0199] The imaging device 1 according to the present disclosure can be applied not only to various displays used in vehicles but also to displays mounted on various electronic devices 2.

[0200] Figure 17B is a front view of a digital camera 120 as a second application example of the electronic device 2, and Figure 17A is a rear view of the digital camera 120. Figure 17B and Figure 17A The digital camera 120 in

[0201] In the cameras in Figure 17B and Figure 17B When a photographer observes the electronic viewfinder 124 while holding the grip 123 of the camera body 122 to determine the composition, and presses the shutter 125 while adjusting the focus, the captured data is stored in a memory in the camera. As shown in Figure 18A , a monitor screen 126 that displays captured data and the like, a real-time image and the like, and the electronic viewfinder 124 are provided on the back of the camera. Further, there is also a case where a sub screen for displaying setting information such as a shutter speed and an exposure value is provided on the upper surface of the camera.

[0202] By providing the sensor 5 to overlap with the back side of the monitor screen 126, the electronic viewfinder 124, the sub screen, and the like for the camera, the camera can function as the imaging device 1 according to the present disclosure.

[0203] (Third Application Example)

[0204] The imaging device 1 and the electronic device 2 according to the present disclosure can also be applied to a head-mounted display (hereinafter, referred to as an HMD). The HMD can be used for virtual reality (VR), augmented reality (AR), mixed reality (MR), surrogate reality (SR), or the like.

[0205] Figure 18A is an external view of an HMD 130 as a third application example of the electronic device 2. Figure 18B The HMD 130 of is provided with an attachment member 131 for attachment to cover a human eye. For example, the attachment member 131 is hooked and fixed to a human ear. A display device 132 is provided inside the HMD 130, and a wearer of the HMD 130 can visually recognize a stereoscopic image or the like with the display device 132. The HMD 130, for example, includes a wireless communication function, an acceleration sensor, or the like, and can switch a stereoscopic image or the like displayed on the display device 132 according to a posture, a gesture, or the like of the wearer.

[0206] Further, a camera can be provided in the HMD 130 to capture an image of the surroundings of the wearer, and an image obtained by combining the captured image of the camera and an image generated by a computer can be displayed on the display device 132. For example, the camera is provided to be superimposed on the back surface side of the display device 132 visually recognized by the wearer of the HMD 130, the periphery of the eye of the wearer is captured by the camera, and the captured image is displayed on another display provided on the outer surface of the HMD 130, so that a person around the wearer can grasp the expression of the face and the movement of the eyes of the wearer in real time.

[0207] Note that various types of HMD 130 are conceivable. For example, as shown in Figure 18B The imaging device 1 and the electronic device 2 according to the present disclosure can also be applied to a smart glasses 130a that displays various types of information on glasses 134. Figure 18B The smart glasses 130a in includes a main body 135, an arm 136, and a barrel portion 137. The main body 135 is connected to the arm 136. The main body 135 is detachable from the glasses 134. The main body 135 includes a control board for controlling the operation of the smart glasses 130a and a display unit. The main body 135 and the barrel are connected to each other via the arm 136. The barrel portion 137 emits image light emitted from the main body 135 through the arm 136 to a lens 138 of the glasses 134. This image light enters a human eye through the lens 138. Similar to ordinary glasses, Figure 19 The wearer of the smart glasses 130a in can not only visually recognize the surrounding situation, but also visually recognize various information emitted from the barrel portion 137.

[0208] (Fourth Application Example)

[0209] The imaging device 1 and the electronic device 2 according to the present disclosure are also applicable to a television device (hereinafter, TV). From the viewpoint of miniaturization and design characteristics, recent TVs tend to make the frame as small as possible. Therefore, in a case where a camera for capturing an image of a viewer is provided on the TV, it is desirable to arrange the camera to overlap with the back surface side of the display panel 2 of the TV.

[0210] Figure 19 is an external view of a TV 140 as a fourth application example of the electronic device 2. In Figure 19 In the TV 140, the frame is minimized, and almost the entire area of the front side is a display area. The TV 140 includes a sensor 5, such as a camera for capturing an image of a viewer. Figure 20 The sensor 5 in the TV 140 is provided on the back surface of a portion (for example, a dotted portion) in the display panel 2. The sensor 5 can be an image sensor module, or various sensors such as a sensor for face authentication, a sensor for distance measurement, and a temperature sensor can be applied, and multiple types of sensors can be arranged on the back surface side of the display panel 2 of the TV 140.

[0211] As described above, according to the imaging device 1 and the electronic device 2 of the present disclosure, because the image sensor module 9 can be provided to overlap with the back surface side of the display panel 2, it is not necessary to provide a camera or the like in the frame, the TV 140 can be reduced in size, and there is no possibility that the frame impairs the design.

[0212] (Fifth Application Example)

[0213] The imaging device 1 and the electronic device 2 according to the present disclosure are also applicable to a smartphone and a mobile phone. Figure 20 is an external view of a smartphone 150 as a fifth application example of the electronic device 2. In Figure 20 In the example, the display surface 2z extends close to the external size of the electronic device 2, and the width of the bezel 2y around the display surface 2z is set to several mm or less. Generally, a front camera is usually mounted on the bezel 2y, but in ​ In the smartphone 150, as indicated by a dotted line, for example, the image sensor module 9 serving as a front camera is provided on the back surface side of the substantially central portion of the display surface 2z. By providing the front camera on the back surface side of the display surface 2z in this way, it is not necessary to provide the front camera on the bezel 2y, and the width of the bezel 2y can be narrowed.

[0214] Note that the present technology can also adopt the following configurations.

[0215] (1) An electronic device, comprising:

[0216] a display unit;

[0217] a first imaging unit disposed on a side opposite to a display surface of the display unit and capable of capturing an image of light of an infrared light wavelength band that has passed through the display unit;

[0218] a second imaging unit disposed on a side opposite to a display surface of the display unit and capable of capturing an image of light of a visible light wavelength band that has passed through the display unit; and

[0219] a correction unit that corrects image data imaged by the second imaging unit based on image data imaged by the first imaging unit.

[0220] (2) The electronic apparatus according to (1), wherein the correction unit corrects a sensitivity of the image data imaged by the second imaging unit based on the image data imaged by the first imaging unit.

[0221] (3) The electronic apparatus according to (2), further comprising: a learning unit that learns a correlation between a sensitivity of the image data imaged by the first imaging unit and a sensitivity of the image data imaged by the second imaging unit, wherein

[0222] the correction unit corrects a sensitivity of the image data imaged by the second imaging unit based on the image data imaged by the first imaging unit with reference to a result of the learning by the learning unit.

[0223] (4) The electronic apparatus according to (1), wherein the correction unit corrects a resolution of the image data imaged by the second imaging unit based on the image data imaged by the first imaging unit.

[0224] (5) The electronic apparatus according to (4), further comprising: a learning unit that learns a correlation between a resolution of the image data imaged by the first imaging unit and a resolution of the image data imaged by the second imaging unit, wherein

[0225] the correction unit corrects a resolution of the image data imaged by the second imaging unit based on the image data imaged by the first imaging unit with reference to a result of the learning by the learning unit.

[0226] (6) The electronic apparatus according to (1), wherein the correction unit corrects at least one of a flare component and a diffracted light component included in the image data imaged by the second imaging unit based on the image data imaged by the first imaging unit.

[0227] (7) The electronic apparatus according to (6), further comprising:

[0228] a learning unit that learns a correlation between at least one of a flare component and a diffracted light component included in the image data imaged by the first imaging unit and at least one of a flare component and a diffracted light component included in the image data imaged by the second imaging unit, wherein

[0229] The correction unit corrects at least one of a flare component and a diffracted light component included in the image data imaged by the second imaging unit based on the image data imaged by the first imaging unit with reference to the learning result in the learning unit.

[0230] (8) The electronic apparatus according to (7), further comprising:

[0231] a reference determination unit that determines whether at least one of a sensitivity, a resolution, a flare component, and a diffracted light component of the image data imaged by the second imaging unit satisfies a predetermined first reference;

[0232] an imaging start instruction unit that starts imaging by the first imaging unit when the reference determination unit determines that the first reference is not satisfied; and

[0233] a correction process determination unit that determines, based on a result of comparing at least one of a sensitivity, a resolution, a flare component, and a diffracted light component between the image data imaged by the first imaging unit and the image data imaged by the second imaging unit, whether correction is performed by the correction unit and a type of image data serving as a correction reference when the correction is performed by the correction unit.

[0234] (9) The electronic apparatus according to (8), wherein, when the type of image data serving as the correction reference is determined by the correction process determination unit, the learning unit learns a correlation between at least one of a sensitivity, a resolution, a flare component, and a diffracted light component of the determined image data and at least one of a sensitivity, a resolution, a flare component, and a diffracted light component of the image data imaged by the second imaging unit.

[0235] (10) The electronic apparatus according to any one of (7) to (9), further comprising:

[0236] a sensor that detects at least one of a shape and a color of an object;

[0237] a reliability estimation unit that estimates reliability of learning by the learning unit;

[0238] an object recognition determination unit that determines, in a case where the reliability estimated by the reliability estimation unit is equal to or less than a predetermined second reference, whether the object can be recognized based on detection data of the sensor; and

[0239] a color designation determination unit that determines, in a case where it is determined by the object recognition determination unit that the object can be recognized, whether a color of the object recognized by the sensor can be designated, and

[0240] in a case where it is determined by the color designation determination unit that the color of the object can be designated, the correction unit corrects the image data imaged by the second imaging unit to approximate the designated color.

[0241] (11) The electronic apparatus according to any one of (1) to (10), wherein the correction unit sets a noise removal degree of a pixel region in which a luminance variation is equal to or less than a predetermined reference value in the image data imaged by the second imaging unit to be higher than a noise removal degree of a pixel region in which a luminance variation is greater than the reference value in the image data.

[0242] (12) The electronic apparatus according to any one of (1) to (11), further comprising:

[0243] a light emitting unit that emits light in an infrared light wavelength band; and

[0244] a light emission control unit that controls a light emission timing of the light emitting unit so that a subject is irradiated with light emitted by the light emitting unit when the image data is imaged by the first imaging unit.

[0245] (13) The electronic apparatus according to (12), wherein

[0246] the light emitting unit includes a plurality of light sources that emit light in light emission wavelength bands different from each other in the infrared light wavelength band,

[0247] the light emission control unit sequentially switches and controls light emission of the plurality of light sources while the first imaging unit performs imaging,

[0248] the first imaging unit outputs a plurality of image data imaged in the light emission wavelength bands different from each other; and

[0249] the correction unit corrects the image data imaged by the second imaging unit based on the plurality of image data.

[0250] (14) The electronic apparatus according to (12) or (13), wherein the light emitting unit is provided on a display surface side of the display unit.

[0251] (15) The electronic apparatus according to any one of (1) to (14), wherein at least one of the first imaging unit and the second imaging unit includes a pixel that captures an image of light in an infrared light wavelength band and a pixel that captures an image of light in a visible light wavelength band.

[0252] (16) The electronic apparatus according to (15), wherein the first imaging unit has sensitivity to light of 550 nm or more.

[0253] (17) The electronic apparatus according to any one of (1) to (16), wherein the correction unit increases a correction degree toward a shorter wavelength side with respect to the image data imaged by the second imaging unit.

[0254] (18) The electronic apparatus according to any one of (1) to (17), wherein the first imaging unit includes a photoelectric conversion unit arranged to be longer in a normal line direction of a light incident plane than the second imaging unit.

[0255] (19) The electronic apparatus according to any one of (1) to (18), in which

[0256] an area in a light-incident surface direction of each pixel of the first imaging unit is larger than an area in a light-incident surface direction of each pixel of the second imaging unit, and

[0257] an area in a light-incident surface direction of all pixels of the first imaging unit is smaller than an area in a light-incident surface direction of all pixels of the second imaging unit.

[0258] (20) An imaging apparatus comprising:

[0259] a first imaging unit provided on a side opposite a display surface of a display unit and capable of capturing an image of light of an infrared light wavelength band;

[0260] a second imaging unit provided on a side opposite the display surface of the display unit and capable of capturing an image of light of a visible light wavelength band; and

[0261] a correction unit that corrects image data imaged by the second imaging unit based on image data imaged by the first imaging unit.

[0262] (21) An imaging apparatus comprising:

[0263] a first imaging unit capable of capturing an image of light of an infrared light wavelength band;

[0264] a second imaging unit capable of capturing an image of light of a visible light wavelength band; and

[0265] a correction unit that corrects image data imaged by the second imaging unit based on image data imaged by the first imaging unit.

[0266] Aspects of the present disclosure are not limited to the above-described embodiments, but include various modifications that can be conceived by those skilled in the art, and the effects of the present disclosure are not limited to the above-described content. That is, various additions, modifications, and partial deletions can be made without departing from the conceptual idea and spirit of the present disclosure derived from the content defined in the claims and the equivalents thereof.

[0267] List of Reference Signs

[0268] 1 Imaging apparatus

[0269] 2, 2a, 2b Electronic apparatus

[0270] 3 Display unit

[0271] 3a Display surface

[0272] 3b Bezel

[0273] 4 camera module

[0274] 5 camera module

[0275] 6 first imaging unit

[0276] 7 second imaging unit

[0277] 8 first optical system

[0278] 9 second optical system

[0279] 11 semiconductor substrate

[0280] 12a first photoelectric conversion unit

[0281] 12b second photoelectric conversion unit

[0282] 13 element isolation layer

[0283] 14 planarization layer

[0284] 15a on-chip lens

[0285] 15b on-chip lens

[0286] 16 color filter layer

[0287] 17 read circuit

[0288] 21 application processor

[0289] 22 video signal generation unit

[0290] 23 A / D conversion unit

[0291] 24 display control unit

[0292] 31 first A / D conversion unit

[0293] 32 second A / D conversion unit

[0294] 33 signal processing unit

[0295] 100 vehicle

[0296] 101 central display

[0297] 102 console display

[0298] 103 head-up display

[0299] 104 digital rearview mirror

[0300] 105 steering wheel display

[0301] 106 rear-seat entertainment display

[0302] 107 instrument panel

[0303] 108 driver's seat

[0304] 109 passenger seat

[0305] 110 center console

[0306] 111 gear shift lever

[0307] 112 windshield

[0308] 113 steering wheel

[0309] 120 digital camera

[0310] 121 lens

[0311] 122 camera body

[0312] 123 grip

[0313] 124 electronic viewfinder

[0314] 125 shutter

[0315] 126 monitor screen

[0316] 130a smart glasses

[0317] 131 attachment member

[0318] 132 display device

[0319] 134 eyeglasses

[0320] 135 body

[0321] 136 arm

[0322] 137 lens barrel portion

[0323] 138 lens

[0324] 150 smart phone

Claims

1. An electronic device comprising: Display unit; The first imaging unit is disposed on the side opposite to the display surface of the display unit and is capable of capturing an image of light passing through the infrared wavelength band of the display unit; The second imaging unit is disposed on the side opposite to the display surface of the display unit and is capable of capturing an image of light in the visible light wavelength band passing through the display unit; The learning unit learns the correlation between at least one of the flare component and the diffraction component included in the image data imaged by the first imaging unit and at least one of the flare component and the diffraction component included in the image data imaged by the second imaging unit. as well as The correction unit, referring to the learning results learned by the learning unit, corrects at least one of the flare component and diffraction component included in the image data imaged by the second imaging unit based on the image data imaged by the first imaging unit.

2. The electronic device according to claim 1, further comprising: The reference determination unit determines whether at least one of the flare component and the diffraction component of the image data imaged by the second imaging unit satisfies a predetermined first reference. An imaging start indication unit initiates imaging by the first imaging unit when the reference determination unit determines that the first reference is not met. as well as The correction process determination unit determines, based on a comparison of at least one of the flare component and the diffraction component between the image data imaged by the first imaging unit and the image data imaged by the second imaging unit, whether correction should be performed by the correction unit and the type of image data used as the correction reference when correction is performed by the correction unit.

3. The electronic device according to claim 1, further comprising: A sensor that detects at least one of the shape and color of an object; A reliability estimation unit estimates the reliability of the learning unit's learning. The object identification and determination unit determines whether the object can be identified based on the detection data of the sensor if the reliability estimated by the reliability estimation unit is equal to or less than a predetermined second benchmark. as well as The color designation determination unit, when the object recognition determination unit determines that the object can be recognized, determines whether the color of the object recognized by the sensor can be designated, and When the color designation determination unit determines that the color of the object can be designated, the correction unit corrects the image data imaged by the second imaging unit to approximate the designated color.

4. The electronic device according to claim 1, wherein, The correction unit sets the noise removal level of pixel regions in the image data imaged by the second imaging unit to be less than or equal to a predetermined reference value, which is higher than the noise removal level of pixel regions in the image data whose brightness changes are greater than the reference value.

5. The electronic device according to claim 1, further comprising: The light-emitting unit emits light in the infrared wavelength band; as well as The light emission control unit controls the light emission timing of the light emission unit so that when image data is imaged by the first imaging unit, the subject is illuminated by light emitted by the light emission unit.

6. The electronic device according to claim 5, wherein, The light-emitting unit includes multiple light sources, which emit light in different infrared wavelength bands. The light emission control unit sequentially switches and controls the emission of the multiple light sources while the first imaging unit performs imaging. The first imaging unit outputs multiple image data that are imaged in different emission wavelength bands; and The correction unit corrects the image data imaged by the second imaging unit based on the plurality of image data.

7. The electronic device according to claim 5, wherein, The light-emitting unit is disposed on the display surface side of the display unit.

8. The electronic device according to claim 1, wherein, At least one of the first imaging unit and the second imaging unit includes pixels that capture images of light in the infrared wavelength band and pixels that capture images of light in the visible wavelength band.

9. The electronic device according to claim 8, wherein, The first imaging unit is sensitive to light of 550 nm or greater.

10. The electronic device according to claim 1, wherein, The correction unit increases the degree of correction relative to the image data imaged by the second imaging unit toward the shorter wavelength side.

11. The electronic device according to claim 1, wherein, Compared to the second imaging unit, the first imaging unit includes a photoelectric conversion unit that is arranged to be longer along the normal direction of the light incident surface.

12. The electronic device according to claim 1, wherein, The area of ​​each pixel in the first imaging unit along the light incident surface is larger than the area of ​​each pixel in the second imaging unit along the light incident surface. The area of ​​all pixels in the first imaging unit in the direction of light incident surface is smaller than the area of ​​all pixels in the second imaging unit in the direction of light incident surface.

13. An imaging device, comprising: The first imaging unit is disposed on the side opposite to the display surface of the display unit and is capable of capturing images of light in the infrared wavelength band; The second imaging unit is disposed on the side opposite to the display surface of the display unit and is capable of capturing images of light in the visible light wavelength band; The learning unit learns the correlation between at least one of the flare component and the diffraction component included in the image data imaged by the first imaging unit and at least one of the flare component and the diffraction component included in the image data imaged by the second imaging unit. as well as The correction unit, referring to the learning results learned by the learning unit, corrects at least one of the flare component and diffraction component included in the image data imaged by the second imaging unit based on the image data imaged by the first imaging unit.

14. An imaging device, comprising: The first imaging unit is capable of capturing images of light in the infrared wavelength band; The second imaging unit is capable of capturing images of light in the visible light wavelength band; The learning unit learns the correlation between at least one of the flare component and the diffraction component included in the image data imaged by the first imaging unit and at least one of the flare component and the diffraction component included in the image data imaged by the second imaging unit. as well as The correction unit, referring to the learning results learned by the learning unit, corrects at least one of the flare component and diffraction component included in the image data imaged by the second imaging unit based on the image data imaged by the first imaging unit.

Images