Electronic equipment

Abstract

[Problem] An electronic device is provided which, while decreasing the bezel width, can also suppress reductions in image quality of images taken by camera. [Solution] This electronic device is provided with a display unit and an imaging unit which is arranged on the side of the display unit opposite of the display surface, wherein the imaging unit has multiple photoelectric conversion units which photoelectrically convert light incident through the display unit, and multiple polarizing elements which are arranged on the light incident side of at least one of the multiple photoelectric conversion units.

Description

[Technical field]

[0001] The present disclosure relates to electronic devices. [Background technology]

[0002] Recent electronic devices such as smartphones, mobile phones, and PCs (Personal Computers) are equipped with cameras in the bezel of the display, making it easy to make video calls and take videos. Smartphones and mobile phones are often carried in a pocket or bag, so their external size needs to be as compact as possible. On the other hand, if the display screen size is small, the higher the display resolution, the smaller the displayed text size becomes, making it difficult to see. For this reason, studies are being conducted to reduce the width of the bezel around the display screen, thereby making it possible to make the display screen size as large as possible without increasing the external size of the electronic device.

[0003] However, since a camera or the like is often mounted on the bezel of an electronic device, the bezel width cannot be made smaller than the outer diameter size of the camera.

[0004] Furthermore, when the camera is placed on the bezel, for example when talking on a videophone, the eye line is often placed near the center of the display screen, which causes the eye line to deviate from the optical axis of the camera, resulting in an unnatural-looking captured image where the eye line is not aligned.

[0005] In order to avoid the above-mentioned problems, it has been proposed to place a camera module on the opposite side of the display unit from the display surface, and to use the camera to capture the subject light that passes through the display unit. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] U.S. Patent Publication 2018 / 0069060 Summary of the Invention [Problem to be solved by the invention]

[0007] However, a portion of the light passing through the display unit is reflected or diffracted before entering the camera, which causes a problem of degradation in the quality of the captured image due to the effects of flare and diffraction caused by reflection. One aspect of the present disclosure provides an electronic device that can reduce the width of a bezel while suppressing degradation in image quality of an image captured by a camera. [Means for solving the problem]

[0008] In order to solve the above problems, the present disclosure provides a display unit, an imaging unit disposed on the opposite side to a display surface of the display unit; The imaging unit includes: a plurality of photoelectric conversion units that perform photoelectric conversion on light incident via the display unit; and a plurality of polarizing elements arranged on a light incident side of at least one of the plurality of photoelectric conversion units.

[0009] The optical fiber may further include a correction unit that corrects captured images photoelectrically converted by the plurality of photoelectric conversion units based on polarization information obtained by polarizing the captured images by the plurality of polarizing elements and photoelectrically converting the captured images by the photoelectric conversion units.

[0010] The correction section may remove at least one of reflected light and diffracted light generated when passing through the display section and incident on the plurality of photoelectric conversion sections and captured.

[0011] The correction unit may correct the digital pixel data by performing a subtraction process on the digital pixel data that has been photoelectrically converted and digitized by the photoelectric conversion unit, by applying a correction amount based on polarization information data that has been digitized from the polarization information.

[0012] a polarizing member that is provided on the display unit and polarizes incident light in a specific direction; At least one of the plurality of polarizing elements transmits light polarized by the polarizing member; The correction unit may correct the captured images photoelectrically converted by the multiple photoelectric conversion units based on the polarization information that is polarized by the polarizing member, passes through the polarizing element, and photoelectrically converted by the photoelectric conversion unit.

[0013] a light source estimation unit that estimates a type of light source of light incident on the display unit; The correction unit may adjust a correction amount of the captured images photoelectrically converted by the plurality of photoelectric conversion units, based on the type of light source estimated by the light source estimation unit.

[0014] a brightness estimation unit that estimates a display brightness of the display unit, The correction section may adjust a correction amount of the captured image photoelectrically converted by the plurality of photoelectric conversion sections in accordance with a display luminance of the display section.

[0015] The image capturing apparatus may further include a signal processing unit that performs at least one of exposure processing, focus adjustment processing, and white balance adjustment processing on the captured image corrected by the correction unit.

[0016] a signal processing unit that performs at least one type of signal processing on the captured image corrected by the correction unit; The image capturing apparatus may further include a parameter adjustment unit that adjusts a parameter used when the captured image is subjected to signal processing in accordance with an amount of correction by the correction unit to correct the captured image.

[0017] the signal processing performed by the signal processing unit includes at least one of a noise reduction process for removing noise components included in the captured image and an edge enhancement process for enhancing edges of the captured image; The parameter adjustment section may adjust a parameter for performing at least one of the noise reduction processing and the edge enhancement processing, according to the correction amount.

[0018] The parameter adjustment section may perform at least one of a process of increasing a degree of noise removal in the noise reduction process and a process of increasing a degree of edge enhancement in the edge enhancement process for a captured image having a larger correction amount.

[0019] The signal processing unit may perform signal processing including exposure adjustment processing, and when performing the exposure adjustment processing, may perform exposure adjustment such that output values ​​of the plurality of photoelectric conversion units are not saturated.

[0020] The optical fiber may further include a defect correction unit that corrects an output value of the photoelectric conversion unit to which light polarized by the polarizing element is incident, using output values ​​of two or more of the surrounding photoelectric conversion units.

[0021] an imaging device including the photoelectric conversion unit and an optical system that focuses light on the photoelectric conversion unit; The imaging device may be disposed on the opposite side of a substantially central portion of a display surface of the display unit.

[0022] The photoelectric conversion unit may be provided with a defect correction unit that corrects the output value of a photoelectric conversion unit whose output value is saturated, based on polarization information polarized by one or more polarizing elements arranged around the photoelectric conversion unit and photoelectrically converted by the photoelectric conversion unit, among the multiple photoelectric conversion units.

[0023] The photoelectric conversion unit has a plurality of divided photoelectric conversion units, The light polarized by the plurality of polarizing elements may be incident on the plurality of split photoelectric conversion units in a portion of the photoelectric conversion unit.

[0024] The photoelectric conversion unit has a plurality of divided photoelectric conversion units, The light polarized by the polarizing element may be incident on a part of the split photoelectric conversion units in any one of the photoelectric conversion units.

[0025] The photoelectric conversion unit has a plurality of divided photoelectric conversion units, The light polarized by the polarizing element may be incident on each of the divided photoelectric conversion units, which are portions of the two or more photoelectric conversion units.

[0026] The plurality of photoelectric conversion units include a plurality of split photoelectric conversion units, Light polarized by the plurality of polarizing elements may be incident on the plurality of photoelectric conversion units.

[0027] the photoelectric conversion unit has a plurality of split photoelectric conversion units that are split into a plurality of parts in one direction and are capable of detecting phase difference information, The light polarized by the polarizing element may be incident on the plurality of split photoelectric conversion units in any one of the photoelectric conversion units. [Brief description of the drawings]

[0028] [Figure 1] 1 is a schematic cross-sectional view of an electronic device according to a first embodiment. [Diagram 2] 2(a) is a schematic external view of the electronic device in FIG. 1, and (b) is a cross-sectional view taken along line AA in (a). [Diagram 3] FIG. 2 is a plan view showing an example of a pixel array of a plurality of polarizing pixels and a plurality of non-polarizing pixels. [Figure 4A] FIG. 4 is a diagram showing a detailed cross-sectional structure of an imaging unit according to the present embodiment. [Figure 4B] FIG. 13 is a cross-sectional view showing a case where a plurality of polarization pixels are arranged on an on-chip lens. [Diagram 5] FIG. 4 is a perspective view showing an example of a detailed structure of each polarization pixel. [Figure 6A] 2 is a schematic diagram showing a state in which flare occurs when an object is photographed by the electronic device of FIG. 1; [Figure 6B] 6B is a diagram showing signal components contained in the captured image of FIG. 6A. [Figure 7A] 5A to 5C are diagrams conceptually explaining correction processing according to the embodiment. [Figure 7B] 5A to 5C are diagrams conceptually explaining correction processing according to the embodiment. [Figure 7C] 5A to 5C are diagrams conceptually explaining correction processing according to the embodiment. [Figure 8]FIG. 2 is a block diagram showing the internal configuration of the electronic device according to the present embodiment. [Figure 9] 5 is a flowchart showing the procedure of a photographing process performed by the electronic device according to the present embodiment. [Figure 10] 5 is a flowchart showing the procedure of a photographing process performed by the electronic device according to the present embodiment. [Figure 11] FIG. 13 is a plan view showing an example of a pixel array of a plurality of polarizing pixels and a plurality of non-polarizing pixels according to a third embodiment. [Figure 12] FIG. 11 is a plan view showing a first modified example of a division structure. [Figure 13] FIG. 11 is a plan view showing a second modified example of the division structure. [Figure 14] FIG. 11 is a plan view showing a third modified example of the division structure. [Figure 15] FIG. 13 is a plan view showing a fourth modified example of the division structure. [Figure 16] FIG. 13 is a schematic cross-sectional view of an electronic device according to a fourth embodiment. [Figure 17] FIG. 11 is a cross-sectional view of a first modified example in which the structure of the display unit 2 is given a distinctive feature. [Figure 18] 13 is a cross-sectional view of a second modified example in which the structure of the display unit 2 is distinctive. [Figure 19] 1 is a plan view showing a case where the electronic device according to the first to fourth embodiments is applied to a capsule endoscope. [Figure 20] 1 is a rear view of a digital single-lens reflex camera to which the electronic device according to the first to fourth embodiments is applied. [Figure 21A] FIG. 1 is a diagram showing an example in which the electronic device 1 is applied to an HMD. [Figure 21B] A diagram showing the current HMD. [Figure 22] FIG. 13 is a block diagram showing a schematic configuration of an electronic device according to a sixth embodiment. [Figure 23] FIG. 23 is a block diagram showing a schematic configuration of an electronic device according to a modification of the sixth embodiment. [Figure 24] 13 is a diagram showing a cross-sectional structure of an imaging section of a camera module mounted on an electronic device according to a seventh embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0029] Hereinafter, an embodiment of an electronic device will be described with reference to the drawings. The following description will focus on the main components of the electronic device, but the electronic device may include components and functions that are not shown or described. The following description does not exclude components and functions that are not shown or described.

[0030] (First embodiment) FIG. 1 is a schematic cross-sectional view of an electronic device 1 according to a first embodiment. The electronic device 1 in FIG. 1 is any electronic device having both a display function and an image capturing function, such as a smartphone, a mobile phone, a tablet, or a PC. The electronic device 1 in FIG. 1 includes a camera module (image capturing unit) disposed on the opposite side to the display surface of the display unit 2. In this manner, the electronic device 1 in FIG. 1 includes a camera module 3 disposed on the rear side of the display surface of the display unit 2. Therefore, the camera module 3 captures images through the display unit 2.

[0031] FIG. 2(a) is a schematic external view of the electronic device 1 in FIG. 1, and FIG. 2(b) is a cross-sectional view taken along line AA in FIG. 2(a). In the example of FIG. 2(a), the display screen 1a extends to nearly the outer size of the electronic device 1, and the width of the bezel 1b around the display screen 1a is set to a few mm or less. Usually, a front camera is often mounted on the bezel 1b, but in FIG. 2(a), as shown by the dashed line, a camera module 3 functioning as a front camera is disposed on the rear side of the display screen 1a at approximately the center. In this way, by providing the front camera on the rear side of the display screen 1a, it becomes unnecessary to dispose the front camera on the bezel 1b, and the width of the bezel 1b can be narrowed.

[0032] 2(a), the camera module 3 is disposed on the rear side of the display screen 1a at approximately the center thereof, but in this embodiment, the camera module 3 may be disposed on any rear side of the display screen 1a, for example, on the rear side near the periphery of the display screen 1a. In this manner, the camera module 3 in this embodiment is disposed at any position on the rear side that overlaps with the display screen 1a.

[0033] As shown in FIG. 1, the display unit 2 is a structure in which a display panel 4, a circular polarizer 5, a touch panel 6, and a cover glass 7 are laminated in this order. The display panel 4 may be, for example, an OLED (Organic Light Emitting Device) unit, a liquid crystal display unit, a MicroLED, or a display unit 2 based on other display principles. The display panel 4, such as an OLED unit, is composed of multiple layers. The display panel 4 is often provided with a member having low transmittance, such as a color filter layer. As will be described later, a through hole may be formed in the member having low transmittance in the display panel 4 in accordance with the arrangement location of the camera module 3. If the subject light passing through the through hole is made incident on the camera module 3, the image quality of the image captured by the camera module 3 can be improved.

[0034] The circular polarizer 5 is provided to reduce glare and improve the visibility of the display screen 1a even in a bright environment. A touch sensor is incorporated in the touch panel 6. There are various types of touch sensors, such as a capacitance type and a resistive film type, and any type may be used. The touch panel 6 and the display panel 4 may be integrated together. The cover glass 7 is provided to protect the display panel 4 and the like.

[0035] The camera module 3 has an imaging unit 8 and an optical system 9. The optical system 9 is disposed on the light incident surface side of the imaging unit 8, i.e., on the side closer to the display unit 2, and collects light that has passed through the display unit 2 onto the imaging unit 8. The optical system 9 is usually composed of multiple lenses.

[0036] The imaging unit 8 has a plurality of photoelectric conversion units 8a and a plurality of polarizing elements 8b. The photoelectric conversion unit 8a photoelectrically converts light incident through the display unit 2. The photoelectric conversion unit 8a may be a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor. The photoelectric conversion unit 8a may be a photodiode or an organic photoelectric conversion film. The plurality of photoelectric conversion units 8a may be arranged in any manner. The arrangement of the plurality of photoelectric conversion units 8a may be a Bayer arrangement, an interline arrangement, a checkerboard arrangement, a stripe arrangement, or any other arrangement.

[0037] The multiple polarizing elements 8b are disposed on the light incident side of at least one of the multiple photoelectric conversion units 8a. The polarizing element 8b polarizes the light that passes through the display unit 2 and is incident. The light polarized by the polarizing element 8b is incident on the corresponding photoelectric conversion unit 8a and photoelectrically converted. In this specification, the output value of the photoelectric conversion unit 8a that photoelectrically converts the light polarized by the polarizing element 8b is referred to as polarization information, and the output value of the photoelectric conversion unit 8a that photoelectrically converts the light that is incident without passing through the polarizing element 8b is referred to as pixel value or pixel information. In addition, in this specification, the polarizing element 8b and the photoelectric conversion unit 8a that photoelectrically converts the light polarized by the polarizing element 8b are collectively referred to as a polarizing pixel, and the photoelectric conversion unit 8a that photoelectrically converts the light that is incident without passing through the polarizing element 8b is referred to as a non-polarizing pixel.

[0038] Fig. 3 is a plan view showing an example of a pixel array of a plurality of polarizing pixels 81 and a plurality of non-polarizing pixels 82. In the example of Fig. 3, a plurality of non-polarizing pixels 82 for three colors, RGB, are arranged in a two-dimensional direction, and a plurality of polarizing pixels 81 are arranged spaced apart from each other between the plurality of non-polarizing pixels 82.

[0039] Fig. 3 is a plan view of multiple non-polarized pixels 82 and multiple polarized pixels 81 viewed from the light incident side. In the example of Fig. 3, four types of polarized pixels 81 with different polarization directions are arranged one by one within an arrangement area of ​​8 × 8 = 64 non-polarized pixels 82, but the ratio of the number of polarized pixels 81 to the number of non-polarized pixels 82, the types of polarized pixels 81, and the locations where the polarized pixels 81 are arranged are arbitrary.

[0040] Fig. 4A is a diagram showing a detailed cross-sectional structure of the imaging unit 8 according to this embodiment. As shown in Fig. 4A, a plurality of photoelectric conversion units 8a are arranged in a substrate 11. A plurality of wiring layers 12 are arranged on the first surface 11a side of the substrate 11. An interlayer insulating film 13 is arranged around the plurality of wiring layers 12. Contacts (not shown) are provided to connect these wiring layers 12 to each other and between the wiring layers 12 and the photoelectric conversion units 8a, but are omitted in Fig. 4A.

[0041] On the second surface 11b side of the substrate 11, a light-shielding layer 15 is disposed near the boundary between pixels via a planarizing layer 14, and an underlying insulating layer 16 is disposed around the light-shielding layer 15. A plurality of polarizing elements 8b are disposed at a distance from each other on the underlying insulating layer 16. Each polarizing element 8b in FIG. 4A is a wire-grid polarizing element with a line-and-space structure that is disposed on a part of the insulating layer 17.

[0042] FIG. 5 is a perspective view showing an example of a detailed structure of each polarizing element 8b. As shown in FIG. 5, each of the polarizing elements 8b has a plurality of line portions 8d of a convex shape extending in one direction and a space portion 8e between each line portion 8d. There are a plurality of types of polarizing elements 8b, each of which has a different extending direction of the line portion 8d. More specifically, there are three or more types of polarizing elements 8b, and for example, the angle between the arrangement direction of the photoelectric conversion portion 8a and the extension direction of the line portion 8d may be three types of 0 degrees, 60 degrees, and 120 degrees. Alternatively, the angle between the arrangement direction of the photoelectric conversion portion 8a and the extension direction of the line portion 8d may be four types of 0 degrees, 45 degrees, 90 degrees, and 135 degrees, or may be any other angle. Alternatively, the plurality of polarizing elements 8b may polarize light in only one direction. The material of the plurality of polarizing elements 8b may be a metal material such as aluminum or tungsten, or an organic photoelectric conversion film.

[0043] In this way, each polarizing element 8b has a structure in which a plurality of line portions 8d extending in one direction are arranged at intervals in a direction intersecting the one direction. There are a plurality of types of polarizing elements 8b in which the directions in which the line portions 8d extend are different. These polarizing elements 8b are arranged at intervals so as to overlap with a portion of the two-dimensional array of a plurality of photoelectric conversion units 8a.

[0044] The line portion 8d has a laminated structure in which a light reflecting layer 8f, an insulating layer 8g, and a light absorbing layer 8h are laminated. The light reflecting layer 8f is made of a metal material such as aluminum. The insulating layer 8g is made of SiO2. The light absorbing layer 8h is made of a metal material such as tungsten.

[0045] Returning to FIG. 4A, a planarization layer 20 is disposed on the insulating layer 17 on which the plurality of polarizing elements 8b are disposed, with protective layers 18 and 19 interposed therebetween. A color filter layer 21 is disposed on the planarization layer 20. The color filter layer 21 may have filter layers of three colors, RGB, or may have filter layers of their complementary colors, cyan, magenta, and yellow. Alternatively, the color filter layer 21 may have a filter layer that transmits colors other than visible light, such as infrared light, a filter layer having multispectral characteristics, or a filter layer of subtractive colors, such as white. By transmitting light other than visible light, such as infrared light, sensing information, such as depth information, can be detected. An on-chip lens 22 is disposed on the color filter layer 21.

[0046] In the cross-sectional structure of Fig. 4A, the on-chip lens 22 is disposed on the multiple polarizing elements 8b, but as shown in Fig. 4B, the multiple polarizing elements 8b may be disposed on the on-chip lens 22. The cross-sectional structure of Fig. 4B is the same as Fig. 4A except that the stacking order of the on-chip lens 22 and the multiple polarizing elements 8b is different.

[0047] In addition, some of the layer configurations in the cross-sectional structures of Figures 4A and B may be changed arbitrarily. For example, the insulating layer 16 in Figure 4A on which the multiple polarizing elements 8b are arranged and the color filter layer 21 may be integrated into one layer.

[0048] Next, a characteristic operation of the electronic device 1 according to this embodiment will be described. Fig. 6A is a diagram showing a typical state in which flare occurs when an object is photographed by the electronic device 1 of Fig. 1. Flare occurs when a part of light incident on the display unit 2 of the electronic device 1 is repeatedly reflected by any member in the display unit 2, and then incident on the imaging unit 8 and is captured in the captured image. When flare occurs in the captured image, a difference in brightness or a change in color occurs as shown in Fig. 6A, degrading the image quality.

[0049] Fig. 6B is a diagram showing signal components contained in the captured image of Fig. 6A. As shown in Fig. 6B, the captured image contains a subject signal and a flare component.

[0050] 7A, 7B, and 7C are diagrams conceptually explaining the correction process according to this embodiment. As shown in FIG. 7A, the imaging unit 8 according to this embodiment has a plurality of polarizing pixels 81 and a plurality of non-polarizing pixels 82. The pixel information photoelectrically converted by the plurality of non-polarizing pixels 82 shown in FIG. 7A includes an object signal and a flare component as shown in FIG. 6B. In contrast, the polarization information photoelectrically converted by the plurality of polarizing pixels 81 shown in FIG. 7A is information of the flare component. Therefore, by subtracting the polarization information photoelectrically converted by the plurality of polarizing pixels 81 from the pixel information photoelectrically converted by the plurality of non-polarizing pixels 82, the flare component is removed as shown in FIG. 7B to obtain an object signal. When an image based on this object signal is displayed on the display unit 2, an object image from which the flare that existed in FIG. 6A has been removed is displayed as shown in FIG. 7C.

[0051] 6A to 6B and 7A to 7C have described examples in which the subject image is affected by flare, but external light incident on the display unit 2 may be diffracted by wiring patterns or the like within the display unit 2, and the diffracted light may be incident on the imaging unit 8. In this way, at least one of flare and diffracted light may be captured in the captured image.

[0052] Fig. 8 is a block diagram showing the internal configuration of electronic device 1 according to this embodiment. Electronic device 1 in Fig. 8 includes an optical system 9, an imaging unit 8, an A / D converter 31, a clamp unit 32, a color output unit 33, a polarized output unit 34, a flare extraction unit 35, a flare correction signal generation unit 36, a defect correction unit 37, a linear matrix unit 38, a gamma correction unit 39, a luminance chroma signal generation unit 40, a focus adjustment unit 41, an exposure adjustment unit 42, a noise reduction unit 43, an edge enhancement unit 44, and an output unit 45.

[0053] The optical system 9 includes one or more lenses 9a and an IR (Infrared Ray) cut filter 9b. The IR cut filter 9b may be omitted. The imaging unit 8 includes a photoelectric conversion unit 8a including a plurality of non-polarized pixels 82 and a plurality of polarized pixels 81, as described above.

[0054] The output values ​​of the multiple polarizing pixels 81 and the output values ​​of the multiple non-polarizing pixels 82 are input to the A / D converter 31. The A / D converter 31 outputs polarization information data obtained by digitizing the output values ​​of the multiple polarizing pixels 81, and digital pixel data obtained by digitizing the output values ​​of the multiple non-polarizing pixels 82.

[0055] The clamping unit 32 performs a process for defining a black level, and subtracts black level data from both the digital pixel data and the polarization information data. The output data of the clamping unit 32 is branched, and RGB digital pixel data is output from the color output unit 33, and polarization information data is output from the polarization output unit 34. The flare extraction unit 35 extracts at least one of a flare component and a diffracted light component from the polarization information data. In this specification, at least one of the flare component and the diffracted light component extracted by the flare extraction unit 35 may be referred to as a correction amount.

[0056] The flare correction signal generation unit 36 ​​corrects the digital pixel data output from the color output unit 33 by subtracting the correction amount extracted by the flare extraction unit 35 from the digital pixel data. The output data of the flare correction signal generation unit 36 ​​is digital pixel data from which at least one of the flare component and the diffracted light component has been removed. In this way, the flare correction signal generation unit 36 ​​functions as a correction unit that corrects the captured image photoelectrically converted by the plurality of non-polarized pixels 82 based on the polarization information.

[0057] The signal level of the digital pixel data at the pixel position of polarization pixel 81 is lowered by the amount that the data has passed through polarization element 8b. For this reason, defect correction unit 37 regards polarization pixel 81 as defective and performs a predetermined defect correction process. In this case, the defect correction process may be an interpolation process using digital pixel data from surrounding pixel positions.

[0058] The linear matrix section 38 performs a matrix operation on color information such as RGB, to reproduce colors more accurately. The linear matrix section 38 is also called a color matrix section.

[0059] Gamma correction unit 39 performs gamma correction so as to enable display with excellent visibility in accordance with the display characteristics of display unit 2. For example, gamma correction unit 39 performs conversion from 10 bits to 8 bits while changing the gradient.

[0060] The luminance chrominance signal generating section 40 generates a luminance chrominance signal to be displayed on the display section 2 based on the output data of the gamma correction section 39 .

[0061] The focus adjustment unit 41 performs autofocus processing based on the luminance chroma signal after the defect correction processing. The exposure adjustment unit 42 performs exposure adjustment based on the luminance chroma signal after the defect correction processing. When performing exposure adjustment, the exposure adjustment may be performed by setting an upper limit clip so that the pixel values ​​of each non-polarized pixel 82 do not saturate. Furthermore, if the pixel values ​​of each non-polarized pixel 82 are saturated even after exposure adjustment, the pixel values ​​of the saturated non-polarized pixels 82 may be estimated based on the pixel values ​​of the polarized pixels 81 surrounding the non-polarized pixel 82.

[0062] The noise reduction unit 43 performs a process of reducing noise contained in the luminance chroma signal. The edge enhancement unit 44 performs a process of enhancing the edge of the subject image based on the luminance chroma signal. The noise reduction process by the noise reduction unit 43 and the edge enhancement process by the edge enhancement unit 44 may be performed only when a predetermined condition is satisfied. The predetermined condition is, for example, when the correction amount of the flare component or the diffracted light component extracted by the flare extraction unit 35 exceeds a predetermined threshold value. The more the flare component or the diffracted light component contained in the captured image, the more noise there will be in the image when the flare component or the diffracted light component is removed, and the more the edges will be blurred. For this reason, the frequency of the noise reduction process or the edge enhancement process can be reduced by performing the noise reduction process or the edge enhancement process only when the correction amount exceeds a threshold value.

[0063] At least some of the signal processing of the defect correction unit 37, the linear matrix unit 38, the gamma correction unit 39, the luminance chroma signal generation unit 40, the focus adjustment unit 41, the exposure adjustment unit 42, the noise reduction unit 43, and the edge enhancement unit 44 in FIG. 8 may be executed by a logic circuit in an image sensor having an image capture unit 8, or may be executed by a signal processing circuit in an electronic device 1 equipped with an image sensor. Alternatively, at least some of the signal processing of FIG. 8 may be executed by a server on a cloud that transmits and receives information to and from the electronic device 1 via a network. As shown in the block diagram of FIG. 8, the electronic device 1 according to this embodiment performs various signal processing on digital pixel data from which at least one of the flare component and the diffracted light component has been removed in the flare correction signal generation unit 36. In particular, some of the signal processing such as the exposure processing, the focus adjustment processing, and the white balance adjustment processing cannot obtain good signal processing results even if the signal processing is performed in a state in which the flare component and the diffracted light component are included.

[0064] 9 is a flowchart showing the procedure of the photographing process performed by the electronic device 1 according to this embodiment. First, the camera module 3 is started (step S1). This causes a power supply voltage to be supplied to the imaging section 8, and the imaging section 8 starts capturing an image of the incident light. More specifically, the multiple non-polarizing pixels 82 photoelectrically convert the incident light, and the multiple polarizing pixels 81 acquire polarization information of the incident light (step S2). The A / D converter 31 outputs polarization information data obtained by digitizing the output values ​​of the multiple polarizing pixels 81, and digital pixel data obtained by digitizing the output values ​​of the multiple non-polarizing pixels 82 (step S3).

[0065] Next, the flare extraction unit 35 determines whether or not flare or diffraction occurs based on the polarization information data (step S4). Here, for example, if the polarization information data exceeds a predetermined threshold, it is determined that flare or diffraction occurs. If it is determined that flare or diffraction occurs, the flare extraction unit 35 extracts a correction amount for the flare component or the diffracted light component based on the polarization information data (step S5). The flare correction signal generation unit 36 ​​subtracts the correction amount from the digital pixel data to generate digital pixel data from which the flare component or the diffracted light component has been removed (step S6).

[0066] Next, various signal processing is performed on the digital pixel data corrected in step S6 or on the digital pixel data determined in step S4 to be free of flare or diffraction (step S7). More specifically, in step S7, as shown in Fig. 8, processes such as defect correction processing, linear matrix processing, gamma correction processing, luminance chroma signal generation processing, exposure processing, focus adjustment processing, white balance adjustment processing, noise reduction processing, edge enhancement processing, etc. are performed. Note that the types and execution order of signal processing are arbitrary, and signal processing of some blocks shown in Fig. 8 may be omitted, or signal processing other than that of blocks shown in Fig. 8 may be performed.

[0067] The digital pixel data that has been subjected to the signal processing in step S7 is output from the output unit 45 and may be stored in a memory (not shown) or may be displayed on the display unit 2 as a live image (step S8).

[0068] In this way, in the first embodiment, the camera module 3 is disposed on the opposite side to the display surface of the display unit 2, and the polarization information of the light passing through the display unit 2 is acquired by the multiple polarization pixels 81. A part of the light passing through the display unit 2 is repeatedly reflected within the display unit 2 and then incident on the multiple non-polarization pixels 82 in the camera module 3. According to this embodiment, by acquiring the above-mentioned polarization information, it is possible to generate a captured image in a state in which the flare components and diffracted light components contained in the light that is repeatedly reflected within the display unit 2 and then incident on the multiple non-polarization pixels 82 are simply and reliably removed. More specifically, in this embodiment, a correction amount based on the polarization information acquired by the multiple polarization pixels 81 is subtracted from the digital pixel data captured and A / D-converted by the multiple non-polarization pixels 82 that photoelectrically convert the light that is incident without passing through the polarizing element 8b, thereby generating digital pixel data from which the flare components and diffracted light components have been removed.

[0069] Second Embodiment 8 performs subtraction processing of a correction amount based on polarization information acquired by the plurality of polarization pixels 81 on the digital pixel data captured by the plurality of non-polarization pixels 82, but this subtraction processing may deteriorate the S / N ratio of the digital pixel data after subtraction. This is because the optical shot noise superimposed on the flare component cannot be removed by the above-mentioned subtraction processing.

[0070] Therefore, the degree of noise reduction processing may be adjusted according to the magnitude of the correction amount due to the flare component or the diffracted light component. More specifically, the greater the correction amount, the higher the degree of noise reduction processing may be. In addition, if the edge becomes unclear by increasing the degree of noise reduction processing, the degree of edge enhancement processing may be increased.

[0071] FIG. 10 is a flowchart showing the procedure of the photographing process performed by the electronic device 1 according to this embodiment. The processes from steps S11 to S16 are the same as steps S1 to S6 in FIG. 9, and therefore will not be described. When the process of subtracting the correction amount from the digital pixel data is completed in step S16, it is then determined whether or not the correction amount extracted in step S14 exceeds a predetermined threshold value (step S17). If it is determined that the correction amount exceeds the threshold value, at least one of the noise removal degree of the noise reduction process and the edge enhancement degree of the edge enhancement process is increased (step S18). That is, when the correction amount of the flare component or the diffracted light component is large, the noise removal degree is increased or the edge is enhanced. As a result, even if the S / N ratio decreases by subtracting the correction amount from the digital pixel data, the decrease in the S / N ratio can be suppressed by appropriately performing the noise removal process and the edge enhancement process.

[0072] On the other hand, if it is determined that the correction amount does not exceed the threshold value, normal noise reduction processing and edge enhancement processing are performed (step S19).

[0073] If it is determined in step S14 that no flare or diffraction has occurred, or if the process of step S18 has been completed, or if the process of step S19 has been completed, other signal processing is performed (step S20). After all signal processing is completed, digital pixel data is output (step S21).

[0074] FIG. 10 shows an example in which the degree of noise reduction processing and edge enhancement processing is increased when the correction amount exceeds a threshold value. However, for any signal processing other than these processing, the specific content of the signal processing may be switched depending on whether the correction amount exceeds a threshold value or not. In this case, a parameter adjustment unit may be provided that adjusts parameters when performing any signal processing depending on the correction amount. Since the amount of parameter adjustment may differ depending on the content of the signal processing, the parameter adjustment unit may set optimal parameters depending on the correction amount for each unit performing each signal processing. In the above example, the noise reduction unit 43 and the edge enhancement unit 44 serve as parameter adjustment units. The noise reduction unit 43 adjusts parameters that determine the degree of noise removal, and the edge enhancement unit 44 adjusts parameters that determine the degree of edge enhancement.

[0075] In addition, in steps S17 to S19 in FIG. 10, the degree of noise reduction processing and edge enhancement processing is switched between two levels depending on whether the correction amount exceeds a threshold value or not. However, the degree of noise reduction processing and edge enhancement processing may be switched finely in three or more levels depending on the correction amount of the flare component or diffracted light component, the state of the subject, etc.

[0076] In this way, in the second embodiment, when there is a risk that the S / N ratio of the digital pixel data after subtraction of the correction amount based on the flare component or the diffracted light component from the digital pixel data may decrease, a process is performed to further increase the degree of noise removal and edge emphasis. This eliminates the risk of the S / N ratio of the captured image decreasing even when the flare component or the diffracted light component is removed.

[0077] (Third embodiment) In the third embodiment, each pixel has a divided structure.

[0078] Fig. 11 is a plan view showing an example of a pixel array of a plurality of polarizing pixels 81 and a plurality of non-polarizing pixels 82 according to the third embodiment. As shown in Fig. 11, each polarizing pixel 81 and each non-polarizing pixel 82 is divided into four. More specifically, each photoelectric conversion unit 8a has a plurality of split photoelectric conversion units, and light polarized by a plurality of polarizing elements 8b is incident on a plurality of split photoelectric conversion units in some of the photoelectric conversion units 8a. Note that, although Fig. 11 shows an example in which each polarizing pixel 81 and each non-polarizing pixel 82 are divided into four, they may be divided into any number of parts equal to or greater than two.

[0079] The multiple polarizing pixels 81 are provided corresponding to the positions and sizes of the multiple non-polarizing pixels 82, and in the example of FIG. 11, one polarizing pixel 81 is provided for every five non-polarizing pixels 82 in both the horizontal and vertical directions. Each polarizing pixel 81 is divided into four, just like each non-polarizing pixel 82. The polarization patterns of the four divided pixels in the same polarizing pixel 81 are the same. In the example of FIG. 11, four types of polarizing pixels 81 each having a different polarization pattern are provided, but the number of types of polarizing pixels 81 may be any number as long as there are three or more types. In this specification, each pixel obtained by dividing each polarizing pixel 81 or each non-polarizing pixel 82 into four is called a divided pixel.

[0080] As shown enlarged on the right side of FIG. 11, the four divided pixels constituting each non-polarized pixel 82 may each be provided with an on-chip lens 22 separately.

[0081] In the example of Figure 11, the size of each polarizing pixel 81 is the same as the size of each non-polarizing pixel 82, and each polarizing pixel 81 is arranged to match the location of each non-polarizing pixel 82, but the size and location of each polarizing pixel 81 may be different from the size and location of each non-polarizing pixel 82.

[0082] 11, light polarized by the corresponding polarizing element 8b is input to a pixel region in the photoelectric conversion unit 8a assigned as a polarization pixel 81, so the output value of the photoelectric conversion unit 8a in this pixel region becomes small. For this reason, the defect correction unit 37 shown in FIG. 8 needs to perform defect correction that corrects the output value of the polarization pixel 81 using the output values ​​of the non-polarizing pixels 82 surrounding each polarization element 8b.

[0083] FIG. 12 is a plan view showing a first modified example of the division structure. In FIG. 12, three of the four division pixels constituting one pixel are used for photoelectric conversion of the corresponding non-polarized pixel 82, and the remaining one is used as a polarized pixel 81. More specifically, the photoelectric conversion unit 8a has a plurality of divided photoelectric conversion units, and light polarized by a plurality of polarizing elements 8b is incident on some of the divided photoelectric conversion units in any of the photoelectric conversion units 8a. As a result, polarized pixels 81 and non-polarized pixels 82 are partially arranged inside each pixel. In this case, as shown in an enlarged view on the right side of FIG. 12, on-chip lenses are provided corresponding to each of the four divided pixels constituting one pixel.

[0084] In the example of Fig. 11, one polarizing pixel 81 is provided for each of a plurality of non-polarizing pixels 82, but in the example of Fig. 12, a non-polarizing pixel 82 and a polarizing pixel 81 are provided for each pixel. Therefore, in the example of Fig. 12, polarization information in photoelectric conversion unit 8a can be obtained more accurately than in the example of Fig. 11.

[0085] In addition, in the example of FIG. 12, four types of polarization pixels 81 having four different polarization patterns are evenly arranged, but the number of types of polarization patterns is not limited to four as long as it is three or more.

[0086] 12, for each pixel, of the two vertically and two horizontally divided pixels, the pixel output value is determined based on the output value of the divided photoelectric conversion unit for three divided pixels used as non-polarized pixels 82, and the correction amount is determined based on the output value of the divided photoelectric conversion unit for one divided pixel used as a polarized pixel 81. In this way, in the example of FIG. 12, since a portion of each pixel includes a non-polarized pixel 82, there is no need to perform defect correction, and signal processing can be simplified.

[0087] In this way, by allocating polarized pixels 81 and non-polarized pixels 82 to a plurality of divided pixels obtained by dividing each pixel into a plurality of parts, color information and polarization information can be output by the non-polarized pixel 82c in each pixel, and since there is no loss of resolution, defect correction processing is unnecessary. In addition, the polarized pixels 81 allocated to a portion of each pixel can calculate a correction amount based on the flare component and diffracted light component, and by subtracting this correction amount from the digital pixel data, the flare component and diffracted light component can be removed from the captured image.

[0088] 12, each pixel is divided into four divided pixels, and thus the polarization pixel 81 is assigned to one of these four divided pixels, making the size of the polarization pixel 81 extremely small. The polarization pixel 81 is, for example, composed of a line-and-space portion as shown in FIG. 5, but it is not easy to precisely align the shape of the line portion 8d and the spacing between two adjacent line portions 8d using microfabrication technology. Therefore, the size of the polarization pixel 81 may be made larger than the size of the non-polarization pixel 82.

[0089] FIG. 13 is a plan view showing a second modified example of the division structure. In FIG. 13, one polarizing pixel 81 is arranged so as to span four adjacently arranged pixels. More specifically, the photoelectric conversion unit 8a has a plurality of divided photoelectric conversion units, and light polarized by the polarizing element 8b is incident on the divided conversion units of the two or more photoelectric conversion units 8a, one by one. The size of the polarizing pixel 81 is twice the size of the pixel in length and width, and the area of ​​the polarizing pixel 81 is four times the size of the pixel. In the example of FIG. 13, three of the four divided pixels constituting one pixel are assigned to the non-polarizing pixel 82 and used for normal photoelectric conversion, and the remaining one is assigned to the polarizing pixel 81 and used for acquiring polarization information. Therefore, as in FIG. 12, since each pixel has a divided pixel used as the non-polarizing pixel 82, there is no need to perform defect correction.

[0090] 13, of the four split pixels obtained by dividing one pixel, three split pixels used as non-polarized pixels 82 are provided with separate on-chip lenses 22. The remaining split pixel, together with the other three adjacent split pixels, is provided with one large on-chip lens 22 and used as a polarized pixel 81. In this way, by using the regions of the four adjacent split pixels to create a large polarizing element 8b and providing an on-chip lens 22 corresponding to the size of the large polarizing element 8b, it is possible to easily manufacture the polarizing element 8b and to obtain highly reliable polarization information.

[0091] Fig. 14 is a plan view showing a third modified example of the divided structure. In Fig. 14, polarizing pixels 81 are arranged in the entire pixel region. More specifically, the multiple photoelectric conversion units 8a have multiple divided photoelectric conversion units, and light polarized by multiple polarizing elements 8b is incident on the multiple photoelectric conversion units 8a. That is, in the example of Fig. 14, only polarizing pixels 81 exist, and no non-polarizing pixels 82 exist. In the example of Fig. 14, each pixel is divided into four divided pixels, and all of the divided pixels are used as polarizing pixels 81.

[0092] In the case of Fig. 14, since a polarization pixel 81 is assigned to every divided pixel of every pixel, the output value of the photoelectric conversion unit 8a is uniformly small. For this reason, a gain adjustment process may be performed to increase the overall output value of the photoelectric conversion unit 8a. Although performing a gain adjustment process may reduce the S / N ratio, highly reliable digital pixel data can be generated by performing processes such as increasing the degree of noise reduction processing and edge emphasis processing.

[0093] 14, by providing a polarizing element 8b corresponding to the entire area of ​​the photoelectric conversion unit 8a, it is possible to obtain polarization information that takes into account the influence of flare and diffracted light incident on any location of the photoelectric conversion unit 8a, thereby improving the accuracy of the correction amount extracted by the flare extraction unit 35.

[0094] FIG. 15 is a plan view showing a fourth modified example of the divided structure. In the example of FIG. 15, the polarizing pixel 81 is arranged between the non-polarizing pixels 82, which is common to FIG. 3 and the like, but the non-polarizing pixel 82 is divided into two divided pixels, one on the left and the other on the right, which is different from the other divided structures described above. More specifically, the photoelectric conversion unit 8a has a plurality of divided photoelectric conversion units that are divided into a plurality in one direction and can detect phase difference information, and light polarized by the polarizing element 8b is incident on the plurality of divided photoelectric conversion units in any one of the photoelectric conversion units 8a. Each divided pixel is, for example, a vertically long rectangle. The two divided pixels can be used to detect phase difference information. Based on this phase difference information, for example, focus adjustment can be performed.

[0095] Each polarizing pixel 81, provided at a ratio of one for each of a number of non-polarizing pixels 82 (for example, five non-polarizing pixels 82) in FIG. 15, has the size of two divided pixels. As described above, since the polarizing element 8b has a line-and-space portion, microfabrication is not easy. However, since the polarizing element 8b in FIG. 15 has twice the size of the divided pixels, it is relatively easy to manufacture and manufacturing defects are unlikely to occur. However, in the pixel region where the polarizing element 8b is arranged, the output value of the photoelectric conversion unit 8a decreases, so it is necessary to perform defect correction processing using the pixel values ​​of the photoelectric conversion units 8a corresponding to the surrounding non-polarizing pixels 82.

[0096] 11 to 15 described above show typical examples of division structures, and division structures other than those shown in the drawings are also possible. This embodiment is applicable to all division structures, and the photoelectric conversion unit 8a may be divided into any shape and size. Therefore, the division shape, division size, and number of divisions of the non-polarization pixels 82 are arbitrary. In addition, the polarization pixels 81 may or may not be divided.

[0097] In this way, in the third embodiment, at least the non-polarized pixels 82 have a split structure, so that some of the split pixels within each pixel are assigned to polarized pixels 81 to obtain polarization information, and the remaining split pixels are assigned to non-polarized pixels 82 to obtain color information, eliminating the need to perform defect correction for each pixel.

[0098] Also, by making each pixel into a divided structure, it may be possible to detect not only polarization information but also phase difference information. Furthermore, if the non-polarized pixels 82 have a divided structure and the polarized pixels 81 do not have a divided structure, the size of the polarized pixels 81 can be made larger than the non-polarized pixels 82, which makes it easier to manufacture the polarizing element 8b.

[0099] (Fourth embodiment) In the fourth embodiment, in an electronic device 1 in which a camera module 3 is disposed on the side opposite to the display surface of a display unit 2, the structure of the display unit 2 is given a distinctive feature.

[0100] As described above, the circular polarizer 5 is provided inside the display unit 2 to improve visibility. However, since the circular polarizer 5 blocks at least a part of light, when the subject light that has passed through the display unit 2 is captured by the camera module 3, problems such as a dark captured image occur. Similarly, the transmittance of the touch sensor and the display panel 4 such as the OLED unit inside the display unit 2 is not very high. Therefore, in this embodiment, a through hole is provided in an area of ​​the display unit 2 that overlaps with the camera module 3 in the front-back direction.

[0101] FIG. 16 is a schematic cross-sectional view of an electronic device 1 according to a fourth embodiment. The display unit 2 in the electronic device 1 in FIG. 16 has a plurality of through holes 2a in a portion overlapping with the camera module 3 in the front-back direction. These through holes 2a are provided in a plurality of layers with low transmittance, such as the touch sensor, the circular polarizer 5, and the display panel 4 in the display unit 2. The diameter size of the through holes 2a varies for each layer. For example, the circular polarizer 5 has a through hole 2a with the same diameter size as the imaging unit 8 of the camera module 3. On the other hand, the touch sensor and the display panel 4 may have a plurality of small through holes 2a within the diameter size of the imaging unit 8 so as to have as little adverse effect as possible on the display and touch sensitivity of the display unit 2.

[0102] Providing these through holes 2a may slightly affect the visibility and touch sensitivity of the display screen 1a of the display unit 2, but this poses almost no practical problem since the diameter size of the camera module 3 is only about a few mm. Providing these through holes 2a improves the transmittance of subject light passing through the display unit 2, and can improve the image quality of the image captured by the camera module 3.

[0103] The example of the display unit 2 having a characteristic structure is not limited to that shown in FIG. 16, and various other examples are possible. FIG. 17 is a cross-sectional view of a first modified example of the display unit 2 having a characteristic structure. In the display unit 2 of FIG. 17, a polarizing member 2b that polarizes light in a specific direction is disposed on the rear surface of the display panel 4. In the example of FIG. 17, the polarizing member 2b is disposed on the rear surface side of the display panel 4 in an area that does not vertically overlap with the camera module 3.

[0104] When light incident on the display unit 2 is incident on the polarizing member 2b, it is polarized in a specific direction when it is reflected. This reflected light is further reflected, for example, by another layer of the display unit 2, and is incident on the imaging unit 8. Since this reflected light is polarized in a specific direction when it is incident on the imaging unit 8, if at least a part of the multiple polarizing elements 8b is given the same polarization characteristics as the polarizing member 2b, it is possible to reliably obtain polarization information of the light polarized by the polarizing member 2b.

[0105] In this way, by providing a polarizing member 2b polarized in a specific direction within the display unit 2, the light incident on the imaging unit 8 can be polarized in a specific direction, and flare components and diffraction components can be more reliably extracted by the multiple polarizing pixels 81.

[0106] FIG. 18 is a cross-sectional view of a second modified example having a characteristic structure for the display unit 2. The display unit 2 in FIG. 18 has a polarizing member 2b polarized in a specific direction in an area of ​​the circular polarizing plate 5 that overlaps with the camera module 3 in the front-back direction. The camera module 3 is originally intended to capture light incident in a direction approximately parallel to the optical axis. For this reason, for example, a polarizing member 2b polarized in a direction approximately parallel to the optical axis is provided in an area of ​​the circular polarizing plate 5 that overlaps with the camera module 3 in the front-back direction. As a result, the light passing through this area is polarized in a direction approximately parallel to the optical axis and then incident on the imaging unit 8. If the multiple polarizing elements 8b in the imaging unit 8 are provided with the same polarization characteristics as the polarizing member 2b, the light incident on the imaging unit 8 will be reliably received by the multiple polarizing pixels 81.

[0107] In this way, in the fourth embodiment, by processing a part of the display unit 2, it is possible to improve the transmittance when light passes through the display unit 2, and improve the quality of the captured image captured by the camera module 3. As a more specific example, by providing an opening in a region overlapping with the camera module 3 in the front-back direction, it is possible to improve the transmittance when light passes through the display unit 2, and improve the quality of the captured image captured by the camera module 3. Furthermore, by providing a polarizing member 2b polarized in a specific direction in at least a part of the display unit 2, it is possible to polarize light that is repeatedly reflected by the display unit 2 and enters the imaging unit 8 in a specific direction, and this light can be reliably received by the polarization pixel 81.

[0108] Fifth embodiment There are various possible specific candidates for the electronic device 1 having the configurations described in the first to fourth embodiments. For example, Fig. 19 is a plan view of the electronic device 1 of the first to fourth embodiments applied to a capsule endoscope 50. The capsule endoscope 50 in Fig. 19 includes, in a housing 51 having, for example, hemispherical end faces and a cylindrical center, a camera (ultra-miniature camera) 52 for capturing images inside a body cavity, a memory 53 for recording image data captured by the camera 52, and a wireless transmitter 55 for transmitting the recorded image data to the outside via an antenna 54 after the capsule endoscope 50 is discharged from the subject's body.

[0109] Also provided within housing 51 are a CPU (Central Processing Unit) 56 and a coil (magnetic force-current conversion coil) 57. CPU 56 controls the shooting by camera 52 and the data storage operation in memory 53, and also controls the data transmission from memory 53 to a data receiving device (not shown) outside housing 51 by wireless transmitter 55. Coil 57 supplies power to camera 52, memory 53, wireless transmitter 55, antenna 54, and light source 52b described below.

[0110] Furthermore, the housing 51 is provided with a magnetic (reed) switch 58 for detecting when the capsule endoscope 50 is set in the data receiving device. When the reed switch 58 detects that the capsule endoscope 50 is set in the data receiving device and data transmission becomes possible, the CPU 56 supplies power from the coil 57 to the wireless transmitter 55.

[0111] The camera 52 has, for example, an imaging element 52a including an objective optical system 9 for capturing an image inside the body cavity, and a plurality of light sources 52b for illuminating the inside of the body cavity. Specifically, the camera 52 is configured with, for example, a CMOS (Complementary Metal Oxide Semiconductor) sensor equipped with an LED (Light Emitting Diode) or a CCD (Charge Coupled Device) as the light source 52b.

[0112] The display unit 2 in the electronic device 1 of the first to fourth embodiments is a concept including a light emitting body such as the light source 52b in Fig. 19. The capsule endoscope 50 in Fig. 19 has, for example, two light sources 52b, but these light sources 52b can be configured with the display panel 4 having a plurality of light source units or an LED module having a plurality of LEDs. In this case, by arranging the imaging unit 8 of the camera 52 below the display panel 4 or the LED module, restrictions on the layout arrangement of the camera 52 are reduced, and a smaller capsule endoscope 50 can be realized.

[0113] 20 is a rear view of the electronic device 1 according to the first to fourth embodiments applied to a digital single-lens reflex camera 60. The digital single-lens reflex camera 60 and compact cameras are provided with a display unit 2 that displays a preview screen on the rear surface opposite the lens. A camera module 3 may be disposed on the opposite side of the display surface of the display unit 2 so that a facial image of the photographer can be displayed on the display screen 1a of the display unit 2. In the electronic device 1 according to the first to fourth embodiments, the camera module 3 can be disposed in an area overlapping with the display unit 2, so that it is not necessary to provide the camera module 3 in the frame portion of the display unit 2, and the size of the display unit 2 can be made as large as possible.

[0114] Fig. 21A is a plan view showing an example in which the electronic device 1 according to the first to fourth embodiments is applied to a head mounted display (hereinafter, HMD) 61. The HMD 61 in Fig. 21A is used for VR (Virtual Reality), AR (Augmented Reality), MR (Mixed Reality), SR (Substituional Reality), or the like. Current HMDs, as shown in Fig. 21B, are equipped with a camera 62 on the outer surface, and while a person wearing the HMD can view images of the surroundings, there is a problem in that people around cannot see the eyes or facial expressions of the person wearing the HMD.

[0115] 21A, the display surface of the display unit 2 is provided on the outer surface of the HMD 61, and the camera module 3 is provided on the opposite side to the display surface of the display unit 2. This allows the facial expression of the wearer captured by the camera module 3 to be displayed on the display surface of the display unit 2, allowing people around the wearer to grasp the facial expression and eye movements of the wearer in real time.

[0116] 21A, since the camera module 3 is provided on the back side of the display unit 2, there are no restrictions on the installation location of the camera module 3, which increases the freedom of design of the HMD 61. In addition, since the camera can be placed in an optimal position, it is possible to prevent problems such as the wearer's line of sight not being aligned with the display surface.

[0117] In this way, in the fifth embodiment, the electronic device 1 according to the first to fourth embodiments can be used for various purposes, thereby increasing the utility value.

[0118] Sixth embodiment The likelihood of flare or diffraction may vary depending on the type of light source of the light incident on the electronic device 1 and the display luminance of the display unit 2. For this reason, the type of light source may be estimated when calculating the correction amount for the flare component or the diffracted light component.

[0119] FIG. 22 is a block diagram showing a schematic configuration of the electronic device 1 according to the sixth embodiment. The electronic device 1 of FIG. 22 includes a light source estimation unit 46 in addition to the configuration of FIG. 8. The light source estimation unit 46 estimates the light source of the light incident on the display unit 2. The light source estimation unit 46 may estimate the light source based on the sensitivity difference of each color of RGB photoelectrically converted by the photoelectric conversion unit 8a, or may estimate the light source using an illuminance sensor, a multispectral sensor, or the like. In addition, a GPS sensor mounted on the electronic device 1 may determine whether the electronic device 1 is located indoors or outdoors, and estimate whether the light source is sunlight or illumination light. In addition, a gyro sensor mounted on the electronic device 1 may determine the tilt angle of the electronic device 1, and estimate the light source based on the tilt angle.

[0120] The flare correction signal generation unit 36 ​​adjusts the amount of correction for the captured image photoelectrically converted by the non-polarized pixels 82, based on the light source estimated by the light source estimation unit 46. More specifically, a formula for calculating the amount of correction may be changed for each type of light source. Furthermore, the amount of correction may be adjusted individually for each type of light source and for each RGB color.

[0121] Fig. 23 is a block diagram showing a schematic configuration of an electronic device 1 according to a modified example of the sixth embodiment. The electronic device 1 in Fig. 23 includes a luminance estimation unit 47 in addition to the configuration in Fig. 8. The luminance estimation unit 47 estimates the display luminance of the display unit 2. The luminance estimation unit 47 estimates the display luminance of the display unit 2, for example, based on a signal from a display control unit (not shown) that controls the display of the display unit 2.

[0122] The flare correction signal generation unit 36 ​​adjusts the amount of correction for the captured image photoelectrically converted by the multiple non-polarized pixels 82, based on the display luminance of the display unit 2 estimated by the luminance estimation unit 47. More specifically, the amount of correction may be adjusted for each color individually, based on the luminance of each color of RGB displayed on the display unit 2.

[0123] 22 and the luminance estimation unit 47 of Fig. 23. In this case, the flare correction signal generation unit 36 ​​adjusts the amount of correction based on the type of light source and the display luminance of the display unit 2.

[0124] In this way, in the sixth embodiment, the amount of correction for correcting the effects of flare components and diffracted light components is adjusted taking into consideration the type of light source incident on the electronic device 1 and the display brightness of the display unit 2, so that the flare components and diffracted light components can be more appropriately removed from the captured image.

[0125] Seventh embodiment In the electronic device 1 according to the seventh embodiment, the optical system 9 of the camera module 3 is different from those in the first to sixth embodiments.

[0126] Fig. 24 is a diagram showing a cross-sectional structure of an imaging unit 8 of a camera module 3 mounted on an electronic device 1 according to a seventh embodiment. The imaging unit 8 in Fig. 24 has a microlens array 64, rather than a single lens or a lens group in which single lenses are arranged in the optical axis direction.

[0127] More specifically, the imaging unit 8 in Figure 24 has a photoelectric conversion unit 8a arranged along the bottom surface of the housing 63, a microlens array 64 arranged above the photoelectric conversion unit 8a, a plurality of light shields 66 arranged between adjacent microlenses 65, and a light guide plate 67 arranged above the microlens array 64.

[0128] In this way, by providing the microlens array 64 as the optical system 9 of the imaging unit 8, it is possible to prevent the influence of adjacent pixels and reduce color bleeding.

[0129] Eighth embodiment In the above-mentioned first to seventh embodiments, examples of removing flare and diffracted light have been described, but the electronic device 1 according to the present disclosure can be applied not only to removing flare and diffracted light, but also, for example, to removing shine on a subject. In the electronic device 1 according to the present embodiment, the camera module 3 can be disposed on the opposite side of the display surface of the display unit 2, so that when a wide-angle or fisheye lens is attached to the camera module 3, the entire display surface of the display unit 2 can be captured by the camera module 3. This not only enables detection of wet fingers, which often results in erroneous recognition with the existing touch panel 6, but also enables touch operation and fingerprint authentication by removing shine caused by wet fingers and detecting unevenness of fingerprints.

[0130] More specifically, a plurality of polarization pixels 81 having a plurality of polarization elements 8b may acquire polarization information caused by the shine component contained in the subject light. In this case, the flare extractor 35 in Fig. 8 extracts a correction amount based on the shine component based on the polarization information. Then, the flare correction signal generator 36 generates an image from which the shine component has been removed from the subject image.

[0131] In addition, in each of the above-described embodiments, it is assumed that the camera module 3 is arranged on the opposite side to the display surface of the display unit 2. However, if a reflector is present in the bottom layer of the display panel 4, such as an OLED unit, a part of the reflector may be cut out and the camera module 3 may be arranged in the cut-out portion, or the camera module 3 may be arranged so that it fits into a part of the display panel 4.

[0132] The present technology can be configured as follows. (1) A display unit and an imaging unit disposed on the opposite side of a display surface of the display unit, The imaging unit includes: a plurality of photoelectric conversion units that perform photoelectric conversion on light incident via the display unit; and a plurality of polarizing elements arranged on a light incident side of at least one of the plurality of photoelectric conversion units. (2) The electronic device according to (1), further comprising a correction unit that corrects a captured image that has been photoelectrically converted based on polarization information that has been polarized by the multiple polarizing elements and photoelectrically converted by the photoelectric conversion unit. (3) The electronic device described in (2), wherein the correction unit removes components of at least one of reflected light and diffracted light generated when passing through the display unit and imaged by being incident on the multiple photoelectric conversion units. (4) The electronic device described in (2) or (3), wherein the correction unit corrects the digital pixel data by performing a subtraction process of a correction amount based on polarization information data obtained by digitizing the polarization information on the digital pixel data that has been photoelectrically converted and digitized by the photoelectric conversion unit. (5) A polarizing member is provided on the display unit and polarizes incident light in a specific direction, and at least one of the plurality of polarizing elements transmits light polarized by the polarizing member, The electronic device described in any one of (2) to (4), wherein the correction unit corrects the captured image photoelectrically converted by the multiple photoelectric conversion units based on the polarization information polarized by the polarizing member, passed through the polarizing element, and photoelectrically converted by the photoelectric conversion unit. (6) A light source estimation unit that estimates a type of light source of light incident on the display unit, The electronic device according to any one of (2) to (4), wherein the correction unit adjusts a correction amount of the captured image photoelectrically converted by the multiple photoelectric conversion units based on the type of light source estimated by the light source estimation unit. (7) A brightness estimation unit that estimates a display brightness of the display unit, The electronic device according to any one of (2) to (4), wherein the correction unit adjusts a correction amount of the captured image photoelectrically converted by the plurality of photoelectric conversion units in accordance with a display luminance of the display unit. (8) The electronic device according to any one of (2) to (7), further comprising a signal processing unit that performs at least one of exposure processing, focus adjustment processing, and white balance adjustment processing on the captured image corrected by the correction unit. (9) a signal processing unit that performs at least one type of signal processing on the captured image corrected by the correction unit; The electronic device described in any one of (2) to (7), further comprising: a parameter adjustment unit that adjusts parameters when signal processing the captured image according to an amount of correction by the correction unit to correct the captured image. (10) The signal processing performed by the signal processing unit includes at least one of a noise reduction process for removing noise components included in the captured image and an edge enhancement process for enhancing edges of the captured image, The electronic device according to (9), wherein the parameter adjustment unit adjusts parameters for performing at least one of the noise reduction processing and the edge enhancement processing in accordance with the amount of correction. (11) The electronic device described in (10), wherein the parameter adjustment unit performs at least one of a process of increasing the degree of noise removal in the noise reduction processing and a process of increasing the degree of edge enhancement in the edge enhancement processing for an image having a larger amount of correction. (12) The electronic device described in any one of (9) to (11), wherein the signal processing unit performs signal processing including exposure adjustment processing, and when performing the exposure adjustment processing, performs exposure adjustment so that the output values ​​of the multiple photoelectric conversion units are not saturated. (13) An electronic device described in any one of (1) to (12), comprising a defect correction unit that corrects the output value of the photoelectric conversion unit to which light polarized by the polarizing element is incident, using output values ​​of two or more of the surrounding photoelectric conversion units. (14) An imaging device including the photoelectric conversion unit and an optical system that focuses light on the photoelectric conversion unit, The electronic device according to any one of (1) to (13), wherein the imaging device is disposed on an opposite side of a display surface of the display unit near a substantially central portion thereof. (15) The electronic device described in any one of (1) to (14), further comprising a defect correction unit that corrects the output value of a photoelectric conversion unit whose output value is saturated among the plurality of photoelectric conversion units, based on polarization information polarized by one or more polarizing elements arranged around the photoelectric conversion unit and photoelectrically converted by the photoelectric conversion unit. (16) The photoelectric conversion unit has a plurality of divided photoelectric conversion units, The electronic device according to any one of (1) to (15), wherein the light polarized by the plurality of polarizing elements is incident on the plurality of split photoelectric conversion units in a portion of the photoelectric conversion unit. (17) The photoelectric conversion unit has a plurality of divided photoelectric conversion units, The electronic device according to any one of (1) to (15), wherein the light polarized by the polarizing element is incident on a part of the split photoelectric conversion units in any of the photoelectric conversion units. (18) The photoelectric conversion unit has a plurality of divided photoelectric conversion units, The electronic device according to any one of (1) to (15), wherein the light polarized by the polarizing element is incident on the split photoelectric conversion unit, which is a portion of each of the two or more photoelectric conversion units. (19) The plurality of photoelectric conversion units each include a plurality of split photoelectric conversion units, The electronic device according to any one of (1) to (15), wherein light polarized by the plurality of polarizing elements is incident on the plurality of photoelectric conversion units. (20) The photoelectric conversion unit includes a plurality of split photoelectric conversion units that are split into a plurality of parts in one direction and can detect phase difference information, The electronic device according to any one of (1) to (15), wherein the light polarized by the polarizing element is incident on the plurality of split photoelectric conversion units in any of the photoelectric conversion units.

[0133] The aspects of the present disclosure are not limited to the above-mentioned individual embodiments, but include various modifications that may be conceived by a person skilled in the art, and the effects of the present disclosure are not limited to the above-mentioned contents. In other words, various additions, modifications, and partial deletions are possible within the scope of the conceptual idea and intent of the present disclosure derived from the contents defined in the claims and their equivalents. [Explanation of symbols]

[0134] 1 electronic device, 1a display screen, 1b bezel, 2 display section, 3 camera module, 4 display panel, 5 circular polarizer, 6 touch panel, 7 cover glass, 8 imaging section, 8a photoelectric conversion section, 8b polarizing element, 8d line section, 8f light reflecting layer, 8g insulating layer, 8h light absorbing layer, 9 optical system, 11 substrate, 11a first surface, 11b second surface, 12 wiring layer, 13 interlayer insulating film, 14 planarization layer, 15 light shielding layer, 16 base insulating layer, 17 insulating layer, 31 A / D converter, 32 clamp section, 33 color output section, 34 polarization output section, 35 flare extraction section, 36 flare correction signal generation section, 37 defect correction section, 38 linear matrix section, 39 gamma correction section, 40 luminance chroma signal generation section, 41 focus adjustment section, 42 Exposure adjustment section, 43 noise reduction section, 44 edge enhancement section, 45 output section, 81 polarization conversion element, 82 non-polarized pixel

Claims

1. A display unit; an imaging unit disposed on the opposite side to a display surface of the display unit; The imaging unit includes: a plurality of photoelectric conversion units that perform photoelectric conversion on light incident via the display unit; a plurality of polarizing elements arranged on a light incident side of at least one of the plurality of photoelectric conversion units; a correction unit that corrects the captured images photoelectrically converted by the plurality of photoelectric conversion units based on polarization information polarized by the plurality of polarizing elements and photoelectrically converted by the photoelectric conversion units; a light source estimation unit that estimates a type of light source of light incident on the display unit, The correction unit adjusts a correction amount of the captured image photoelectrically converted by the plurality of photoelectric conversion units based on the type of light source estimated by the light source estimation unit. electronic equipment.

2. A display unit; an imaging unit disposed on the opposite side to a display surface of the display unit; The imaging unit includes: a plurality of photoelectric conversion units that perform photoelectric conversion on light incident via the display unit; a plurality of polarizing elements arranged on a light incident side of at least one of the plurality of photoelectric conversion units; a correction unit that corrects the captured images photoelectrically converted by the plurality of photoelectric conversion units based on polarization information polarized by the plurality of polarizing elements and photoelectrically converted by the photoelectric conversion units; a brightness estimation unit that estimates a display brightness of the display unit, the correction unit adjusts a correction amount of the captured image photoelectrically converted by the plurality of photoelectric conversion units in accordance with a display luminance of the display unit. electronic equipment.

3. a signal processing unit that performs at least one type of signal processing on the captured image corrected by the correction unit; a parameter adjustment unit that adjusts a parameter when the captured image is subjected to signal processing according to a correction amount by the correction unit to correct the captured image, 3. The electronic device according to claim 1 or 2.

4. the signal processing performed by the signal processing unit includes at least one of a noise reduction process for removing noise components included in the captured image and an edge enhancement process for enhancing edges of the captured image; the parameter adjustment unit adjusts a parameter for performing at least one of the noise reduction processing and the edge enhancement processing in accordance with the correction amount.

4. The electronic device according to claim 3.

5. A display unit; an imaging unit disposed on the opposite side to a display surface of the display unit; The imaging unit includes: a plurality of photoelectric conversion units that perform photoelectric conversion on light incident via the display unit; a plurality of polarizing elements arranged on a light incident side of at least one of the plurality of photoelectric conversion units; a correction unit that corrects the captured images photoelectrically converted by the plurality of photoelectric conversion units based on polarization information polarized by the plurality of polarizing elements and photoelectrically converted by the photoelectric conversion units; a signal processing unit that performs at least one type of signal processing on the captured image corrected by the correction unit; a parameter adjustment unit that adjusts a parameter for signal processing of the captured image according to a correction amount by the correction unit; the signal processing performed by the signal processing unit includes at least one of a noise reduction process for removing noise components included in the captured image and an edge enhancement process for enhancing edges of the captured image; the parameter adjustment unit adjusts a parameter for performing at least one of the noise reduction processing and the edge enhancement processing in accordance with the correction amount. electronic equipment.

6. a polarizing member that is provided on the display unit and polarizes incident light in a specific direction; At least one of the plurality of polarizing elements transmits light polarized by the polarizing member; The correction unit corrects the captured images photoelectrically converted by the plurality of photoelectric conversion units based on the polarization information polarized by the polarizing member, passed through the polarizing element, and photoelectrically converted by the photoelectric conversion unit.

6. The electronic device according to claim 5.

7. the parameter adjustment unit performs at least one of a process of increasing a degree of noise removal in the noise reduction process and a process of increasing a degree of edge enhancement in the edge enhancement process for a captured image having a larger correction amount.

7. The electronic device according to claim 4.

8. the signal processing unit performs signal processing including an exposure adjustment process, and when performing the exposure adjustment process, performs exposure adjustment so that output values ​​of the plurality of photoelectric conversion units are not saturated. The electronic device according to claim 3 .

9. the correction unit removes at least one of reflected light and diffracted light generated when passing through the display unit and incident on the plurality of photoelectric conversion units to be imaged.

9. The electronic device according to claim 1.

10. the correction unit corrects the digital pixel data by performing a subtraction process of a correction amount based on polarization information data obtained by digitizing the polarization information, on the digital pixel data that has been photoelectrically converted and digitized by the photoelectric conversion unit.

10. The electronic device according to claim 1.

11. a signal processing unit that performs at least one of an exposure process, a focus adjustment process, and a white balance adjustment process on the captured image corrected by the correction unit; The electronic device according to claim 1 .

12. a defect correction unit that corrects an output value of the photoelectric conversion unit to which light polarized by the polarizing element is incident, using output values ​​of two or more of the surrounding photoelectric conversion units, 12. The electronic device according to claim 1.

13. an imaging device including the photoelectric conversion unit and an optical system that focuses light on the photoelectric conversion unit; The imaging device is disposed on the opposite side of the display surface of the display unit, near a substantially central portion of the display surface.

13. The electronic device according to claim 1.

14. a defect correction unit that corrects an output value of a photoelectric conversion unit having a saturated output value among the plurality of photoelectric conversion units, based on polarization information obtained by polarizing the photoelectric conversion unit by one or more of the polarizing elements disposed around the photoelectric conversion unit and photoelectrically converting the photoelectric conversion unit; 14. The electronic device according to claim 1.

15. The photoelectric conversion unit has a plurality of divided photoelectric conversion units, The light polarized by the plurality of polarizing elements is incident on the plurality of split photoelectric conversion units in a part of the photoelectric conversion unit.

15. The electronic device according to claim 1.

16. The photoelectric conversion unit has a plurality of divided photoelectric conversion units, The light polarized by the polarizing element is incident on a part of the divided photoelectric conversion units in any one of the photoelectric conversion units.

15. The electronic device according to claim 1.

17. The photoelectric conversion unit has a plurality of divided photoelectric conversion units, The light polarized by the polarizing element is incident on each of the divided photoelectric conversion units, which are a part of the two or more photoelectric conversion units.

15. The electronic device according to claim 1.

18. The plurality of photoelectric conversion units include a plurality of split photoelectric conversion units, The light polarized by the plurality of polarizing elements is incident on the plurality of photoelectric conversion units.

15. The electronic device according to claim 1.

19. the photoelectric conversion unit has a plurality of split photoelectric conversion units that are split into a plurality of parts in one direction and are capable of detecting phase difference information, The light polarized by the polarizing element is incident on the plurality of split photoelectric conversion units in any one of the photoelectric conversion units.

15. The electronic device according to claim 1.

Images