electronic devices

Abstract

The present application provides an electronic device capable of reducing the width of its frame while simultaneously suppressing poor image quality in images captured by a camera. The electronic device comprises: a display unit; and a camera unit disposed on a side opposite to a display surface of the display unit, the camera unit comprising: a plurality of photoelectric conversion units for performing photoelectric conversion on light incident through the display unit; and a plurality of polarizing elements disposed on the light-incident side of at least one of the plurality of photoelectric conversion units.

Description

Technical Field

[0001] The present disclosure relates to an electronic device. Background Art

[0002] In recent years, electronic devices such as smart phones, mobile phones, and PCs (personal computers) have been equipped with cameras on the outer edge (frame) of the display, making it easy to make video calls, shoot animations, etc. Smart phones and mobile phones are often carried in pockets or bags, so they need to be as compact as possible. On the other hand, when the size of the display screen is small, the higher the display resolution, the smaller the size of the displayed text becomes, and it becomes difficult to visually recognize. Therefore, research is being conducted to make the size of the display screen as large as possible without increasing the size of the electronic device by reducing the width of the frame surrounding the display screen.

[0003] However, the frame of an electronic device often includes a camera or the like, and therefore the frame width cannot be made smaller than the outer diameter of the camera.

[0004] In addition, when the camera is configured in the frame, for example, when having a conversation via video call, the line of sight is often located near the center of the display screen, so the line of sight is offset from the optical axis of the camera, resulting in an awkward captured image that is not aligned with the line of sight.

[0005] In order to avoid the above problem, the following solution has been proposed: a camera module is arranged on the side opposite to the display surface of the display unit, and the camera is used to capture the subject light passing through the display unit.

[0006] Prior art literature

[0007] Patent Literature

[0008] Patent Document 1: U.S. Patent Publication No. 2018 / 0069060 Summary of the Invention

[0009] Technical problem to be solved by the invention

[0010] However, part of the light that has passed through the display unit is reflected or diffracted before entering the camera. This causes a problem in that the quality of the captured image is degraded due to the influence of flare and diffraction caused by the reflection.

[0011] According to one embodiment of the present invention, an electronic device is provided that can reduce the width of a frame and suppress degradation in the quality of images captured by a camera.

[0012] Means for solving technical problems

[0013] In order to solve the above technical problems, the present disclosure provides an electronic device comprising: a display unit; and a shooting unit, which is arranged on the side opposite to the display surface of the display unit, and the shooting unit has: a plurality of photoelectric conversion units, which perform photoelectric conversion on light incident through the display unit; and a plurality of polarizing elements, which are arranged on the light incident side of at least one photoelectric conversion unit among the plurality of photoelectric conversion units.

[0014] The electronic device may further include a correction unit that corrects a captured image photoelectrically converted by the plurality of photoelectric conversion units based on polarization information of light polarized by the plurality of polarizing elements and photoelectrically converted by the photoelectric conversion units.

[0015] The correction unit may remove a component of at least one of reflected light and diffracted light generated when passing through the display unit and incident on the plurality of photoelectric conversion units and captured.

[0016] The correction section may correct the digital pixel data by subtracting a correction amount based on polarization information data obtained by digitizing the polarization information from the digital pixel data photoelectrically converted and digitized by the photoelectric conversion section.

[0017] The electronic device may include a polarizing component that is arranged on the display unit and polarizes the incident light in a specific direction. At least one polarizing element among the multiple polarizing elements allows the light polarized by the polarizing component to pass through. The correction unit may correct the captured image after photoelectric conversion by the multiple photoelectric conversion units based on the polarization information after the light is polarized by the polarizing component, passes through the polarizing element, and is photoelectrically converted by the photoelectric conversion unit.

[0018] The electronic device may include a light source estimating unit that estimates the type of light source of light incident on the display unit, and the correction unit may adjust the correction amount of the captured image photoelectrically converted by the multiple photoelectric conversion units based on the type of light source estimated by the light source estimating unit.

[0019] The electronic device may include a brightness estimation unit that estimates display brightness of the display unit, and the correction unit may adjust a correction amount of the captured image photoelectrically converted by the plurality of photoelectric conversion units according to the display brightness of the display unit.

[0020] The electronic device may include a signal processing unit configured to perform at least one of exposure processing, focus adjustment processing, and white balance adjustment processing on the captured image corrected by the correction unit.

[0021] The electronic device may include: a signal processing unit that performs at least one signal processing on the captured image corrected by the correction unit; and a parameter adjustment unit that adjusts parameters when performing signal processing on the captured image according to the correction amount of the captured image corrected by the correction unit.

[0022] The signal processing performed by the signal processing unit may include at least one of noise reduction processing for removing noise components contained in the captured image and edge emphasis processing for emphasizing edges of the captured image. The parameter adjustment unit may adjust parameters for performing at least one of the noise reduction processing and the edge emphasis processing based on the correction amount.

[0023] The parameter adjustment unit may perform at least one of a higher degree of noise removal in the noise reduction process and a higher degree of edge emphasis in the edge emphasis process on the captured image having a larger correction amount.

[0024] The signal processing section may perform exposure adjustment so that output values ​​of the plurality of photoelectric conversion sections are not saturated when performing signal processing including exposure adjustment processing and performing the exposure adjustment processing.

[0025] The electronic device may include a defect correction unit that corrects an output value of the photoelectric conversion unit on which light polarized by the polarization element is incident, using output values ​​of two or more surrounding photoelectric conversion units.

[0026] The electronic device may include a camera having the photoelectric conversion unit and an optical system for focusing light on the photoelectric conversion unit, and the camera may be arranged on a side opposite to a substantially central portion of a display surface of the display unit.

[0027] The electronic device may include a defect correction unit, which corrects the output value of the photoelectric conversion unit whose output value is saturated among the multiple photoelectric conversion units based on polarization information after polarization is performed by one or more polarization elements arranged around the photoelectric conversion unit and photoelectric conversion is performed by the photoelectric conversion unit.

[0028] The photoelectric conversion section may include a plurality of divided photoelectric conversion sections, and the light polarized by the plurality of polarization elements may be incident on the plurality of divided photoelectric conversion sections in a part of the photoelectric conversion section.

[0029] The photoelectric conversion section may include a plurality of divided photoelectric conversion sections, and the light polarized by the polarization element may be incident on a portion of the divided photoelectric conversion sections among any of the photoelectric conversion sections.

[0030] The photoelectric conversion section may include a plurality of divided photoelectric conversion sections, and the light polarized by the polarization element may be incident on a portion of each of the divided photoelectric conversion sections in two or more of the photoelectric conversion sections.

[0031] The plurality of photoelectric conversion sections may include a plurality of divided photoelectric conversion sections, and the light polarized by the plurality of polarization elements may be incident on the plurality of photoelectric conversion sections.

[0032] The photoelectric conversion section may include a plurality of divided photoelectric conversion sections that are divided in one direction and capable of detecting phase difference information, and the light polarized by the polarization element may be incident on any of the plurality of divided photoelectric conversion sections. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a schematic cross-sectional view of the electronic device according to the first embodiment.

[0034] Figure 2 (a) is Figure 1 A schematic diagram of the appearance of an electronic device, Figure 2 (b) is a cross-sectional view of (a) along line AA.

[0035] Figure 3 It is a plan view showing an example of a pixel arrangement of a plurality of polarized pixels and a plurality of non-polarized pixels.

[0036] Figure 4A 1 is a diagram showing a detailed interface structure of the imaging unit according to this embodiment.

[0037] Figure 4B This is a cross-sectional view showing a case where a plurality of polarization pixels are arranged on the upper surface of an on-chip lens.

[0038] Figure 5 It is a perspective view showing an example of the detailed structure of each polarization pixel.

[0039] Figure 6A is schematically shown by Figure 1 A diagram showing the situation where light spots are generated when an electronic device captures a subject.

[0040] Figure 6B It shows Figure 6A A diagram showing signal components contained in a captured image.

[0041] Figure 7A This is a diagram conceptually illustrating the correction process according to this embodiment.

[0042] Figure 7B This is a diagram conceptually illustrating the correction process according to this embodiment.

[0043] Figure 7C This is a diagram conceptually illustrating the correction process according to this embodiment.

[0044] Figure 8 is a block diagram showing the internal structure of the electronic device according to the present embodiment.

[0045] Figure 9 This is a flowchart showing the processing procedure of the imaging process performed by the electronic device according to this embodiment.

[0046] Figure 10 This is a flowchart showing the processing procedure of the imaging process performed by the electronic device according to this embodiment.

[0047] Figure 11 This is a plan view showing an example of a pixel arrangement of a plurality of polarization pixels and a plurality of non-polarization pixels according to the third embodiment.

[0048] Figure 12 It is a plan view showing a first modification example of the segmentation structure.

[0049] Figure 13 It is a plan view showing a second modification of the segmented structure.

[0050] Figure 14 It is a plan view showing a third modification example of the segmented structure.

[0051] Figure 15 1 is a plan view showing a fourth modification of the segmented structure.

[0052] Figure 16 is a schematic cross-sectional view of an electronic device according to a fourth embodiment.

[0053] Figure 17 It is a cross-sectional view of a first modified example having a characteristic structure of the display unit 2 .

[0054] Figure 18 It is a cross-sectional view of a second modified example having a characteristic structure of the display unit 2 .

[0055] Figure 19 These are plan views of the electronic devices according to the first to fourth embodiments when applied to a capsule endoscope.

[0056] Figure 20 This is a rear view of the electronic device according to the first to fourth embodiments applied to a digital single-lens reflex camera.

[0057] Figure 21A 1 is a diagram showing a case where the electronic device 1 is applied to an HMD.

[0058] Figure 21B is a diagram showing a current HMD.

[0059] Figure 22 It is a block diagram showing a schematic configuration of an electronic device according to a sixth embodiment.

[0060] Figure 23 This is a block diagram showing a schematic configuration of an electronic device according to a modified example of the sixth embodiment.

[0061] Figure 24 1 is a diagram showing a cross-sectional structure of an imaging unit of a camera module mounted on an electronic device according to a seventh embodiment.

[0062] Description of reference numerals:

[0063] 1: Electronic device, 1a: Display screen, 1b: Frame, 2: Display unit, 3: Camera module, 4: Display panel, 5: Circular polarizer, 6: Touch panel, 7: Cover glass, 8: Imaging unit, 8a: Photoelectric conversion unit, 8b: Polarizing element, 8d: Line unit, 8f: Light reflection layer, 8g: Insulation layer, 8h: Light absorption layer, 9: Optical system, 11: Substrate, 11a: First surface, 11b: Second surface, 12: Wiring layer, 13: Interlayer insulation film, 14: Planarization layer, 15: Light shielding Layer, 16: Base insulating layer, 17: Insulating layer, 31: A / D converter, 32: Clamping unit, 33: Color output unit, 34: Polarization output unit, 35: Spot extraction unit, 36: Spot correction signal generation unit, 37: Defect correction unit, 38: Linear matrix unit, 39: Gamma correction unit, 40: Brightness and chromaticity signal generation unit, 41: Focus adjustment unit, 42: Exposure adjustment unit, 43: Noise reduction unit, 44: Edge emphasis unit, 45: Output unit, 81: Changing element, 82: Non-polarized pixel. DETAILED DESCRIPTION

[0064] Below, embodiments of electronic devices are described with reference to the accompanying drawings. The following description focuses on the main components of the electronic device. However, the electronic device may contain components and functions that are not shown or described. The following description does not exclude components and functions that are not shown or described.

[0065] (First embodiment)

[0066] Figure 1 It is a schematic cross-sectional view of the electronic device 1 according to the first embodiment. Figure 1 The electronic device 1 is any electronic device having both a display function and a photographing function, such as a smartphone, a mobile phone, a tablet computer, or a PC. Figure 1 The electronic device 1 includes a camera module (photographing unit) disposed on the side opposite to the display surface of the display unit 2. Figure 1 The electronic device 1 is provided with a camera module 3 on the inner side of the display surface of the display unit 2. Therefore, the camera module 3 takes a picture through the display unit 2.

[0067] Figure 2 (a) is Figure 1 A schematic external view of an electronic device 1 is shown. Figure 2 (b) is Figure 2 (a) is a cross-sectional view along the A-A line. Figure 2 In the example (a), the display screen 1a is extended to a size close to the outer dimensions of the electronic device 1, so that the width of the frame 1b around the display screen 1a is less than a few mm. Usually, the frame 1b is often equipped with a front camera, but in Figure 2 In (a), as shown by the dotted line, a camera module 3 functioning as a front camera is disposed on the inner surface side of the substantially central portion of the display screen 1a. By placing the front camera on the inner surface side of the display screen 1a, it is no longer necessary to dispose a front camera on the frame 1b, thereby reducing the width of the frame 1b.

[0068] It should be pointed out that in Figure 2 In (a), the camera module 3 is arranged on the inner surface side of the substantially central portion of the display screen 1a. However, in this embodiment, the camera module 3 may be arranged on the inner surface side near the peripheral portion of the display screen 1a, for example, as long as it is on the inner surface side of the display screen 1a. In this way, the camera module 3 in this embodiment is arranged at any position on the inner surface side overlapping the display screen 1a.

[0069] like Figure 1 As shown, the display unit 2 is a structure in which a display panel 4, a circular polarizer 5, a touch panel 6 and a cover glass 7 are stacked in sequence. The display panel 4 can be, for example, an OLED (Organic Light Emitting Device) unit, a liquid crystal display unit, a MicroLED, or a display unit 2 based on other display principles. The display panel 4 of the OLED unit, etc. is composed of multiple layers. The display panel 4 is often provided with components with low transmittance such as a color filter layer. As described later, a through hole can also be formed in the component with low transmittance in the display panel 4 to match the configuration position of the camera module 3. If the subject light passing through the through hole is incident on the camera module 3, the image quality of the image captured by the camera module 3 can be improved.

[0070] A circular polarizer 5 is provided to reduce glare and improve visibility of the display screen 1a even in bright environments. A touch sensor is incorporated into the touch panel 6. Touch sensors come in various types, such as capacitive and resistive, and any type may be used. Alternatively, the touch panel 6 and display panel 4 may be integrated. A cover glass 7 is provided to protect the display panel 4, etc.

[0071] The camera module 3 includes an imaging unit 8 and an optical system 9. The optical system 9 is disposed on the light incident side of the imaging unit 8, i.e., on the side close to the display unit 2, and focuses light passing through the display unit 2 onto the imaging unit 8. The optical system 9 is generally composed of a plurality of lenses.

[0072] The shooting unit 8 has a plurality of photoelectric conversion units 8a and a plurality of polarizing elements 8b. The photoelectric conversion unit 8a performs photoelectric conversion on the light incident through the display unit 2. The photoelectric conversion unit 8a can be either a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor. In addition, the photoelectric conversion unit 8a can be either a photodiode or an organic photoelectric conversion film. The plurality of photoelectric conversion units 8a can be arranged in any manner. The arrangement of the plurality of photoelectric conversion units 8a can be a Bayer arrangement, an inter-row arrangement, a checkered arrangement, a stripe arrangement, or other arrangements.

[0073] A plurality of polarizing elements 8b are arranged on the light incident side of at least one photoelectric conversion unit 8a among the plurality of photoelectric conversion units 8a. The polarizing element 8b polarizes the light incident through the display unit 2. The light polarized by the polarizing element 8b is incident on the corresponding photoelectric conversion unit 8a for photoelectric conversion. In this specification, the output value of the photoelectric conversion unit 8a that performs photoelectric conversion on the light polarized by the polarizing element 8b is referred to as polarization information, and the output value of the photoelectric conversion unit 8a that performs photoelectric conversion on the light not incident by the polarizing element 8b is referred to as pixel value or pixel information. In addition, in this specification, the polarizing element 8b and the photoelectric conversion unit 8a that performs photoelectric conversion on the light polarized by the polarizing element 8b are collectively referred to as polarized pixels, and the photoelectric conversion unit 8a that performs photoelectric conversion on the light not incident by the polarizing element 8b is referred to as non-polarized pixels.

[0074] Figure 3 81 and 82 are plan views showing an example of a pixel arrangement of a plurality of polarized pixels 81 and a plurality of non-polarized pixels 82. Figure 3 In the example of FIG, a plurality of non-polarized pixels 82 for three colors of RGB are arranged in a two-dimensional direction, and a plurality of polarized pixels 81 are arranged between the plurality of non-polarized pixels 82 at intervals.

[0075] Figure 3 This is a plan view of a plurality of non-polarized pixels 82 and a plurality of polarized pixels 81 as viewed from the light incident side. Figure 3 In the example, four types of polarized pixels 81 with different polarization directions are arranged one by one in the arrangement area of ​​8×8=64 non-polarized pixels 82, but the ratio of the number of polarized pixels 81 to the number of non-polarized pixels 82, the type of polarized pixels 81, and the arrangement position of the polarized pixels 81 are arbitrary.

[0076] Figure 4A 1 is a diagram showing a detailed cross-sectional structure of the imaging unit 8 according to this embodiment. Figure 4A As shown, a plurality of photoelectric conversion units 8a are arranged in a substrate 11. A plurality of wiring layers 12 are arranged on the first surface 11a side of the substrate 11. An interlayer insulating film 13 is arranged around the plurality of wiring layers 12. Contacts (not shown) are provided to connect these wiring layers 12 to each other and to the wiring layers 12 and the photoelectric conversion units 8a, but are not shown. Figure 4A Omitted in.

[0077] On the second surface 11b side of the substrate 11, a light shielding layer 15 is arranged near the pixel boundary via the planarization layer 14, and a base insulating layer 16 is arranged around the light shielding layer 15. On the base insulating layer 16, a plurality of polarizers 8b are arranged at intervals. Figure 4A Each polarizing element 8 b is a wire grid polarizing element having a line and space structure and is disposed locally on the insulating layer 17 .

[0078] Figure 5 1 is a perspective view showing an example of the detailed structure of each polarizing element 8b. Figure 5 The figure shows a plurality of line portions 8d having a convex shape extending in one direction and spacers 8e between the line portions 8d. The polarizing element 8b has a plurality of types in which the extension directions of the line portions 8d are different. More specifically, the polarizing element 8b has three or more types. For example, the angles formed by the arrangement direction of the photoelectric conversion portion 8a and the extension direction of the line portion 8d may be 0 degrees, 60 degrees, and 120 degrees. Alternatively, the angles formed by the arrangement direction of the photoelectric conversion portion 8a and the extension direction of the line portion 8d may be 0 degrees, 45 degrees, 90 degrees, and 135 degrees, or angles other than these. Alternatively, the plurality of polarizing elements 8b may only polarize in a single direction. The material of the plurality of polarizing elements 8b may be a metal material such as aluminum or tungsten, or an organic photoelectric conversion film.

[0079] Thus, each polarizing element 8b has a structure in which a plurality of linear portions 8d extending in one direction are spaced apart and arranged in a direction intersecting the one direction. There are multiple types of polarizing elements 8b in which the linear portions 8d extend in different directions. These polarizing elements 8b are spaced apart and arranged so as to overlap a portion of the two-dimensional arrangement of the plurality of photoelectric conversion sections 8a.

[0080] The line portion 8d has a laminated structure comprising a light-reflecting layer 8f, an insulating layer 8g, and a light-absorbing layer 8h. The light-reflecting layer 8f is formed of a metal material such as aluminum. The insulating layer 8g is formed of SiO2, for example. The light-absorbing layer 8h is formed of a metal material such as tungsten.

[0081] return Figure 4AOn the insulating layer 17 on which a plurality of polarizing elements 8b are arranged, a planarizing layer 20 is arranged via protective layers 18 and 19. A color filter layer 21 is arranged on the planarizing layer 20. The color filter layer 21 may also include filter layers for the three colors of RGB, or may include filter layers for cyan, magenta, and yellow as their complementary colors. Alternatively, it may include a filter layer that allows colors other than visible light, such as infrared light, to pass through, a filter layer with multi-spectral characteristics, or a filter layer for subtractive colors, such as white. By allowing light other than visible light, such as infrared light, to pass through, it is possible to detect sensing information such as depth of field information. An on-chip lens 22 is arranged on the color filter layer 21.

[0082] exist Figure 4A In the cross-sectional structure of , the on-chip lens 22 is arranged on the plurality of polarizing elements 8b, but as Figure 4B As shown, a plurality of polarizing elements 8 b may be arranged on the on-chip lens 22 . Figure 4B The cross-sectional structure of the chip is different from that of the chip except that the stacking order of the on-chip lens 22 and the plurality of polarizing elements 8b is different. Figure 4A same.

[0083] In addition, you can also change Figure 4A and Figure 4B For example, the local layer structure of the cross-sectional structure can also be Figure 4A The insulating layer 16 and the color filter layer 21 in which the plurality of polarizing elements 8b are arranged are combined into one layer.

[0084] Next, characteristic operations of the electronic device 1 according to this embodiment will be described. Figure 6A is schematically shown in the Figure 1 The figure shows the appearance of flare (glare) when the electronic device 1 is photographing a subject. Flare is caused by a part of the light incident on the display unit 2 of the electronic device 1 being repeatedly reflected by any component in the display unit 2, and then incident on the imaging unit 8 and written into the captured image. If flare is generated in the captured image, Figure 6A As shown, differences in brightness (luminance) and changes in color tone will occur, resulting in poor image quality.

[0085] Figure 6B It shows Figure 6A A diagram showing the signal components contained in the captured image. Figure 6B As shown, the photographic image includes a subject signal and a light spot component.

[0086] Figure 7A 、 Figure 7B and Figure 7C FIG is a diagram for roughly explaining the correction process based on this embodiment. Figure 7A As shown in FIG. 1 , the imaging unit 8 according to this embodiment includes a plurality of polarized pixels 81 and a plurality of non-polarized pixels 82. Figure 7AThe pixel information of the photoelectric conversion of the plurality of non-polarized pixels 82 shown is as follows: Figure 6B The image shown includes the subject signal and the light spot component. Figure 7A The polarization information photoelectrically converted by the plurality of polarized pixels 81 shown is the information of the light spot component. Therefore, by deducting the polarization information photoelectrically converted by the plurality of polarized pixels 81 from the pixel information photoelectrically converted by the plurality of non-polarized pixels 82, as shown in FIG. Figure 7B As shown in FIG, the light spot component is removed to obtain the subject signal. If the image based on the subject signal is displayed on the display unit 2, the image is as follows: Figure 7C As shown, it is shown that the Figure 6A The subject image with light spots existing in the image.

[0087] pass Figure 6A-6B and Figures 7A to 7C While the subject image has been described as being affected by flare, there are also cases where external light incident on the display unit 2 is diffracted by wiring patterns within the display unit 2, and the diffracted light is incident on the imaging unit 8. In this case, at least one of the flare or the diffracted light may be reflected in the captured image.

[0088] Figure 8 This is a block diagram showing the internal structure of the electronic device 1 according to the present embodiment. Figure 8 The electronic device 1 includes: an optical system 9, a shooting unit 8, an A / D converter 31, a clamping unit 32, a color output unit 33, a polarization output unit 34, a spot extraction unit 35, a spot correction signal generation unit 36, a defect correction unit 37, a linear matrix unit 38, a gamma correction unit 39, a brightness and color signal generation unit 40, a focus adjustment unit 41, an exposure adjustment unit 42, a noise reduction unit 43, an edge emphasis unit 44 and an output unit 45.

[0089] The optical system 9 includes one or more lenses 9a and an IR (Infrared Ray) cut filter 9b. The IR cut filter 9b may be omitted. As described above, the imaging unit 8 includes a photoelectric conversion unit 8a and a plurality of polarized pixels 81. The photoelectric conversion unit 8a includes a plurality of non-polarized pixels 82.

[0090] The output values ​​of the polarized pixels 81 and the non-polarized pixels 82 are input to the A / D converter 31. The A / D converter 31 outputs polarization information data obtained by digitizing the output values ​​of the polarized pixels 81 and digital pixel data obtained by digitizing the output values ​​of the non-polarized pixels 82.

[0091] The clamping unit 32 performs processing to define a black level, subtracting the black level data from the digital pixel data and the polarization information data. The output data from the clamping unit 32 is branched, with the color output unit 33 outputting the RGB digital pixel data and the polarization output unit 34 outputting the polarization information data. The speckle extraction unit 35 extracts at least one of the speckle component and the diffracted light component from the polarization information data. In this specification, the at least one of the speckle component and the diffracted light component extracted by the speckle extraction unit 35 is sometimes referred to as the correction amount.

[0092] The speckle correction signal generator 36 corrects the digital pixel data output from the color output unit 33 by subtracting the correction amount extracted by the speckle extraction unit 35. The output data from the speckle correction signal generator 36 is digital pixel data from which at least one of the speckle component and the diffracted light component has been removed. In this way, the speckle correction signal generator 36 functions as a correction unit that corrects the captured image photoelectrically converted by the plurality of non-polarized pixels 82 based on polarization information.

[0093] The digital pixel data at the polarized pixel 81 passes through the polarizing element 8b, causing the signal level to decrease accordingly. Therefore, the defect correction unit 37 treats the polarized pixel 81 as a defect and performs a predetermined defect correction process. In this case, the defect correction process may be interpolation using the digital pixel data at surrounding pixel locations.

[0094] The linear matrix unit 38 performs matrix operations on color information such as RGB, thereby achieving more accurate color reproduction. The linear matrix unit 38 is also called a color matrix unit.

[0095] The gamma correction unit 39 performs gamma correction to obtain a display with excellent visibility in accordance with the display characteristics of the display unit 2. For example, the gamma correction unit 39 performs conversion from 10 bits to 8 bits while changing the gradient.

[0096] The luminance and chrominance signal generating section 40 generates luminance and chrominance signals for display on the display section 2 based on the output data of the gamma correction section 39 .

[0097] The focus adjustment unit 41 performs autofocus based on the luminance and chromaticity signals after defect correction. The exposure adjustment unit 42 performs exposure adjustment based on the luminance and chromaticity signals after defect correction. When performing exposure adjustment, an upper limit may be set to ensure that the pixel values ​​of each non-polarized pixel 82 are not saturated. Furthermore, even if exposure adjustment is performed, if the pixel values ​​of each non-polarized pixel 82 are saturated, the pixel values ​​of the saturated non-polarized pixels 82 may be estimated based on the pixel values ​​of the polarized pixels 81 surrounding the non-polarized pixels 82.

[0098] The noise reduction unit 43 performs processing to reduce the noise contained in the luminance and chrominance signals. The edge emphasis unit 44 performs processing to emphasize the edges of the subject image based on the luminance and chrominance signals. The noise reduction processing by the noise reduction unit 43 and the edge emphasis processing by the edge emphasis unit 44 can also be performed only when predetermined conditions are met. The predetermined conditions refer, for example, to the case where the correction amount of the spot component and the diffracted light component extracted by the spot extraction unit 35 exceeds a predetermined threshold. The more spot components and diffracted light components contained in the captured image, the more noise there is or the blurred the edges are in the image when the spot components and the diffracted light components are removed. Therefore, the noise reduction processing and the edge emphasis processing are performed only when the correction amount exceeds the threshold, thereby reducing the frequency of performing the noise reduction processing and the edge emphasis processing.

[0099] Figure 8 The signal processing of at least a part of the defect correction unit 37, the linear matrix unit 38, the gamma correction unit 39, the luminance and chrominance signal generation unit 40, the focus adjustment unit 41, the exposure adjustment unit 42, the noise reduction unit 43, and the edge emphasis unit 44 can be performed by a logic circuit in the imaging sensor having the imaging unit 8, or by a signal processing circuit in the electronic device 1 equipped with the imaging sensor. Alternatively, it can be performed by a server on the cloud that transmits and receives information via the electronic device 1 and the network. Figure 8 At least part of the signal processing. Figure 8 As shown in the block diagram, electronic device 1 according to this embodiment performs various signal processing on digital pixel data from which at least one of the speckle component and the diffracted light component has been removed, using speckle correction signal generating section 36. This is because, in particular, for some signal processing processes such as exposure processing, focus adjustment processing, and white balance adjustment processing, even if signal processing is performed while speckle and diffracted light components are present, good signal processing results cannot be obtained.

[0100] Figure 9 1 is a flowchart illustrating the processing sequence of the photographic processing performed by the electronic device 1 according to the present embodiment. First, the camera module 3 is activated (step S1). As a result, a power supply voltage is supplied to the imaging unit 8, and the imaging unit 8 begins to capture incident light. More specifically, the plurality of non-polarized pixels 82 perform photoelectric conversion on the incident light, and the plurality of polarized pixels 81 obtain polarization information of the incident light (step S2). The A / D converter 31 outputs polarization information data obtained by digitizing the output values ​​of the plurality of polarized pixels 81 and digital pixel data obtained by digitizing the output values ​​of the plurality of non-polarized pixels 82 (step S3).

[0101] Next, the flare extraction unit 35 determines whether flare or diffraction has occurred based on the polarization information data (step S4). For example, if the polarization information data exceeds a predetermined threshold, it is determined that flare or diffraction has occurred. If flare or diffraction has occurred, the flare extraction unit 35 extracts correction amounts for the flare component and the diffracted light component based on the polarization information data (step S5). The flare correction signal generation unit 36 ​​subtracts the correction amount from the digital pixel data to generate digital pixel data with the flare and diffracted light components removed (step S6).

[0102] Next, various signal processing is performed on the digital pixel data corrected in step S6 or the digital pixel data determined to be free of light spots and diffraction in step S4 (step S7). More specifically, in step S7, Figure 8 As shown, defect correction processing, linear matrix processing, gamma correction processing, brightness and color signal generation processing, exposure processing, focus adjustment processing, white balance adjustment processing, noise reduction processing, edge emphasis processing, etc. It should be noted that the type of signal processing and the execution order are arbitrary and can also be omitted. Figure 8 The signal processing shown as part of the box can also be performed except Figure 8 Signal processing outside the box shown.

[0103] The digital pixel data subjected to the signal processing in step S7 is output from the output unit 45 and may be stored in a memory (not shown) or displayed as a moving image on the display unit 2 (step S8 ).

[0104] Thus, in the first embodiment, the camera module 3 is arranged on the side opposite the display surface of the display unit 2, and the polarization information of the light passing through the display unit 2 is obtained by the plurality of polarizing pixels 81. A portion of the light passing through the display unit 2 is repeatedly reflected within the display unit 2 and is incident on the plurality of non-polarizing pixels 82 within the camera module 3. According to this embodiment, by obtaining this polarization information, it is possible to simply and reliably remove the speckle component and diffracted light component included in the light that is repeatedly reflected within the display unit 2 and thus incident on the plurality of non-polarizing pixels 82. More specifically, in this embodiment, by processing the digital pixel data captured by the plurality of non-polarizing pixels 82 that perform photoelectric conversion on the incident light without passing through the polarizing element 8b and then undergoing A / D conversion, and then subtracting the correction amount based on the polarization information obtained by the plurality of polarizing pixels 81, it is possible to generate digital pixel data from which the speckle component and diffracted light component have been removed.

[0105] (Second embodiment)

[0106] exist Figure 8The speckle correction signal generator 36 subtracts the correction amount based on the polarization information acquired by the polarized pixels 81 from the digital pixel data captured by the non-polarized pixels 82. However, this subtraction process may deteriorate the S / N ratio of the digital pixel data after subtraction. This is because the photon shot noise superimposed on the speckle component cannot be removed by the subtraction process.

[0107] Therefore, the degree of noise reduction in the noise reduction process can be adjusted based on the magnitude of the correction applied to the speckle and diffracted light components. More specifically, the greater the correction, the greater the degree of noise reduction. Furthermore, by increasing the degree of noise reduction, the degree of edge emphasis in the edge enhancement process can be increased when edges become blurred.

[0108] Figure 10 This is a flowchart showing the processing procedure of the photographing process performed by the electronic device 1 according to this embodiment. The processing of steps S11 to S16 is related to Figure 9 Since steps S1 to S6 are the same, their description is omitted. In step S16, when the process of subtracting the correction amount from the digital pixel data is completed, it is next determined whether the correction amount extracted in step S14 exceeds a predetermined threshold value (step S17). If it is determined that the correction amount exceeds the threshold value, at least one of the noise removal degree of the noise reduction process and the edge emphasis degree of the edge emphasis process is made higher (step S18). That is, when the correction amount of the spot component and the diffracted light component is large, a process is performed to make the noise removal degree higher or to further emphasize the edge. Thus, even if the S / N ratio decreases due to subtracting the correction amount from the digital pixel data, the decrease in the S / N ratio can be suppressed by appropriately performing noise removal processing and edge emphasis processing.

[0109] On the other hand, if it is determined that the correction amount does not exceed the threshold value, normal noise reduction processing and edge emphasis processing are performed (step S19).

[0110] If it is determined in step S14 that no flare or diffraction has occurred, or if the processing in step S18 is completed, or if the processing in step S19 is completed, other signal processing is performed (step S20). If all signal processing is completed, digital pixel data is output (step S21).

[0111] exist Figure 10, an example is shown in which the degree of noise reduction processing and edge emphasis processing is increased when the correction amount exceeds a threshold value. However, for any signal processing other than these processes, the specific content of the signal processing can also be switched when the correction amount exceeds or does not exceed the threshold value. In this case, a parameter adjustment unit can also be provided to adjust the parameters when performing any signal processing according to the correction amount. Since the adjustment amount of the parameters varies depending on the content of the signal processing, the parameter adjustment unit can also set the optimal parameters for each part of each signal processing according to the correction amount. In addition, in the above example, the noise reduction unit 43 and the edge emphasis unit 44 also serve as the parameter adjustment unit. The noise reduction unit 43 adjusts the parameters that determine the degree of noise removal, and the edge emphasis unit 44 adjusts the parameters that determine the degree of edge emphasis.

[0112] In addition, Figure 10 In steps S17 to S19, the degree of noise reduction processing and edge emphasis processing is switched to two levels depending on whether the correction amount exceeds the threshold, but the degree of noise reduction processing and edge emphasis processing can also be finely switched to three or more stages depending on the correction amount of the light spot component, the diffraction light component, the condition of the subject, etc.

[0113] Thus, in the second embodiment, by subtracting the correction amount based on the speckle component and the diffracted light component from the digital pixel data, processing is performed to further enhance the degree of noise reduction and edge emphasis when the S / N ratio of the digital pixel data after the subtraction is likely to decrease. As a result, even when the speckle component and the diffracted light component are removed, the S / N ratio of the captured image does not decrease.

[0114] (Third embodiment)

[0115] In the third embodiment, each pixel has a divided structure.

[0116] Figure 11 1 is a plan view showing an example of a pixel arrangement of a plurality of polarized pixels 81 and a plurality of non-polarized pixels 82 according to the third embodiment. Figure 11 As shown, each polarized pixel 81 and each non-polarized pixel 82 is divided into four. More specifically, each photoelectric conversion unit 8a has a plurality of divided photoelectric conversion units, and the light polarized by the plurality of polarizing elements 8b is incident on the plurality of divided photoelectric conversion units in a part of the photoelectric conversion unit 8a. Figure 11 , an example is shown in which each polarized pixel 81 and each non-polarized pixel 82 is divided into four, but the number of divisions may be any number of divisions, two or more.

[0117] The positions and sizes of the plurality of polarized pixels 81 and the plurality of non-polarized pixels 82 are correspondingly arranged. Figure 11In the example, polarized pixels 81 are arranged in the horizontal direction and the vertical direction at a ratio of 1 polarized pixel 81 to 5 non-polarized pixels 82. Each polarized pixel 81 is divided into four in the same manner as each non-polarized pixel 82. The polarization patterns of the four divided pixels in the same polarized pixel 81 are the same. Figure 11 In the example of FIG, four polarization pixels 81 having different polarization patterns are provided. However, the number of polarization pixels 81 is arbitrary as long as there are three or more types. In this specification, each pixel that divides each polarization pixel 81 or each non-polarization pixel 82 into four is referred to as a split pixel.

[0118] like Figure 11 As shown in the enlarged diagram on the right side of FIG. 8 , the four divided pixels constituting each non-polarized pixel 82 are each provided with an individual on-chip lens 22 .

[0119] exist Figure 11 In the example, the size of each polarized pixel 81 is made the same as the size of each non-polarized pixel 82, and each polarized pixel 81 is configured to match the configuration position of each non-polarized pixel 82, but the size and configuration position of each polarized pixel 81 can also be different from the size and configuration position of each non-polarized pixel 82.

[0120] exist Figure 11 In the case of , since the pixel area in the photoelectric conversion unit 8a assigned as the polarized pixel 81 is input with the light polarized by the corresponding polarizing element 8b, the output value of the photoelectric conversion unit 8a in the pixel area becomes small. Figure 8 The defect correction unit 37 shown performs defect correction to correct the output value of the polarization pixel 81 using the output value of the non-polarization pixel 82 around each polarization element 8 b.

[0121] Figure 12 : is a plan view showing a first modification of the split structure. Figure 12 In the example, three of the four split pixels constituting one pixel are utilized for photoelectric conversion of the corresponding non-polarized pixel 82, and the remaining split pixel is utilized as the polarized pixel 81. In more detail, the photoelectric conversion unit 8a has a plurality of split photoelectric conversion units, and the light polarized by the plurality of polarizing elements 8b is incident on a portion of the split photoelectric conversion units in any photoelectric conversion unit 8a. As a result, polarized pixels 81 and non-polarized pixels 82 are locally configured inside each pixel. In this case, as Figure 12 As shown in the enlarged diagram on the right side of FIG, on-chip lenses are provided corresponding to each of the four divided pixels constituting one pixel.

[0122] exist Figure 11 In the example, the polarized pixels 81 are set at a ratio of one polarized pixel 81 to multiple non-polarized pixels 82, but in Figure 12 In the example of FIG, a non-polarized pixel 82 and a polarized pixel 81 are provided in each pixel. Figure 11 Compared to the example in Figure 12 In the example, the polarization information in the photoelectric conversion section 8a can be obtained more accurately.

[0123] In addition, Figure 12 In the example, four types of polarization pixels 81 having four types of polarization patterns are evenly arranged, but the types of polarization patterns may be three or more and are not limited to four.

[0124] exist Figure 12 In the case of , for each pixel, the pixel output value is determined based on the output values ​​of the split photoelectric conversion units for the three split pixels used as the non-polarized pixel 82 in each of the two split pixels in the vertical and horizontal directions, and the correction amount is determined based on the output value of the split photoelectric conversion unit for the one split pixel used as the polarized pixel 81. Figure 12 In the example of , since a portion of each pixel includes the non-polarized pixel 82, defect correction is not required, and signal processing can be simplified.

[0125] By allocating polarized pixels 81 and non-polarized pixels 82 to the multiple divided pixels formed by dividing each pixel, color information and polarization information can be output from the non-polarized pixels 82c within each pixel, eliminating the need for defect correction. Furthermore, correction amounts based on flare and diffracted light components can be calculated using the polarized pixels 81 allocated to a portion of each pixel. By subtracting these correction amounts from the digital pixel data, these flare and diffracted light components can be removed from the captured image.

[0126] exist Figure 12 In the example of , since the polarization pixel 81 is allocated to one of the four divided pixels after each pixel is divided, the size of the polarization pixel 81 is very small. Figure 5 As shown, the polarized pixel 81 is composed of lines and spacers, but it is not easy to precisely align the shape of the line 8d and the spacing between two adjacent line 8d using fine processing technology. Therefore, the size of the polarized pixel 81 can be made larger than the size of the non-polarized pixel 82.

[0127] Figure 13 : is a plan view showing a second modification of the split structure. Figure 13In the embodiment, a polarized pixel 81 is arranged so as to span four adjacent pixels. In more detail, the photoelectric conversion unit 8a has a plurality of split photoelectric conversion units, and the light polarized by the polarizing element 8b is incident on the split conversion units of two or more photoelectric conversion units 8a. The size of the polarized pixel 81 is twice the size of the split pixel in terms of length and width, and the area of ​​the polarized pixel 81 is four times the area of ​​the split pixel. Figure 13 In the example of FIG, three of the four segmented pixels constituting one pixel are assigned as non-polarized pixels 82 and used for normal photoelectric conversion, and the remaining segmented pixel is assigned as polarized pixel 81 and used for obtaining polarization information. Figure 12 Likewise, since each pixel has a divided pixel utilized as a non-polarized pixel 82 , defect correction is not required.

[0128] like Figure 13 As shown in the enlarged diagram on the right side of FIG, a separate on-chip lens 22 is provided for each of the three segments of the four segments formed by dividing a single pixel and used as non-polarized pixels 82. The remaining segment is provided with a single large on-chip lens 22 along with the other three adjacent segments and used as a polarized pixel 81. By utilizing the area of ​​the four adjacent segments to create a large polarizing element 8b and providing an on-chip lens 22 of a corresponding size, the polarizing element 8b can be manufactured more easily and highly reliable polarization information can be obtained.

[0129] Figure 14 : is a plan view showing a third modified example of the split structure. Figure 14 In FIG, polarization pixels 81 are arranged in the full pixel area. More specifically, the plurality of photoelectric conversion units 8a have a plurality of split photoelectric conversion units, and the light polarized by the plurality of polarization elements 8b is incident on the plurality of photoelectric conversion units 8a. Figure 14 In the example of , there are only polarized pixels 81, and no non-polarized pixels 82. Figure 14 In the example of , each pixel is divided into four divided pixels, but any divided pixel can also be used as the polarization pixel 81.

[0130] exist Figure 14 In this case, since the polarization pixel 81 is assigned to each segmented pixel, the output value of the photoelectric converter 8a is uniformly reduced. Therefore, gain adjustment processing can be performed to increase the output value of the photoelectric converter 8a as a whole. While gain adjustment processing may reduce the S / N ratio, it is possible to generate more reliable digital pixel data by performing processing such as increasing the degree of noise reduction and edge enhancement.

[0131] In addition, if Figure 14As shown, by providing the polarizing element 8b across the entire region of the photoelectric converter 8a, it is possible to obtain polarization information that takes into account the effects of flare and diffracted light incident on any portion of the photoelectric converter 8a. This improves the accuracy of the correction amount extracted by the flare extraction unit 35.

[0132] Figure 15 : is a plan view showing a fourth modified example of the split structure. Figure 15 For example, the polarized pixels 81 are arranged between the non-polarized pixels 82. Figure 3 The structure is similar to the above-mentioned structure, but differs from the other segmented structures in that the non-polarized pixel 82 is divided into two left and right segmented pixels. More specifically, the photoelectric conversion section 8a includes multiple segmented photoelectric conversion sections that are divided in one direction and capable of detecting phase difference information. Light polarized by the polarizing element 8b enters any of the multiple segmented photoelectric conversion sections in the photoelectric conversion section 8a. Each segmented pixel is, for example, a vertically elongated rectangular shape. Two segmented pixels can be used to detect phase difference information. Based on this phase difference information, focus adjustment can be performed, for example.

[0133] Figure 15 Each polarized pixel 81 arranged at a ratio of one polarized pixel 81 to a plurality of non-polarized pixels 82 (for example, five non-polarized pixels 82) has a size of two divided pixels. As described above, the polarizing element 8b has lines and spacers, so fine processing is not easy, but Figure 15 Polarizing element 8b is twice the size of the divided pixel, making it easier to manufacture and less prone to manufacturing defects. However, the output value of photoelectric converter 8a decreases in the pixel area where polarizing element 8b is located. Therefore, it is necessary to use the pixel values ​​of photoelectric converter 8a corresponding to the surrounding non-polarized pixels 82 to perform defect correction.

[0134] The above Figures 11 to 15 While a segmented structure is shown as an example, other segmented structures are also possible. This embodiment is applicable to all segmented structures, and the photoelectric conversion unit 8a can be segmented into any shape and size. Therefore, the segmented shape, segmented size, and number of segments of the non-polarized pixels 82 are arbitrary. Furthermore, the polarized pixels 81 may or may not be segmented.

[0135] Thus, in the third embodiment, since at least the non-polarized pixels 82 are segmented structures, if a portion of the segmented pixels within each pixel are allocated as polarized pixels 81 to obtain polarization information and the remaining segmented pixels are allocated as non-polarized pixels 82 to obtain color information, there is no need to perform defect correction on each pixel.

[0136] Furthermore, by providing a segmented structure for each pixel, it is possible to detect not only polarization information but also phase difference information. Furthermore, if the non-polarized pixels 82 are segmented and the polarized pixels 81 are not segmented, the polarized pixels 81 can be made larger than the non-polarized pixels 82, making it easier to manufacture the polarizing element 8b.

[0137] (Fourth embodiment)

[0138] The fourth embodiment is an embodiment in which the structure of the display unit 2 is characterized in an electronic device 1 in which a camera module 3 is arranged on the side opposite to the display surface of the display unit 2 .

[0139] As described above, a circular polarizer 5 is provided inside the display unit 2 to improve visual recognition. However, the circular polarizer 5 blocks at least a portion of the light, so when the camera module 3 captures the subject light that has passed through the display unit 2, undesirable effects such as darkening of the captured image may occur. Similarly, the transmittance of the display panel 4, such as the touch sensor and OLED portion, inside the display unit 2 is not that high. Therefore, this embodiment is characterized in that a through hole is provided in the area of ​​the display unit 2 that overlaps with the camera module 3 in the front-to-back direction.

[0140] Figure 16 It is a schematic cross-sectional view of an electronic device 1 according to a fourth embodiment. Figure 16 The portion of the display unit 2 in the electronic device 1 that overlaps with the camera module 3 in the front-to-back direction has a plurality of through holes 2a. These through holes 2a are provided in a plurality of layers with low transmittance, such as the touch sensor, circular polarizer 5, and display panel 4 in the display unit 2. The radial dimensions of the through holes 2a are different in each layer. For example, the circular polarizer 5 has a through hole 2a with the same radial dimensions as the radial dimensions of the shooting unit 8 of the camera module 3. On the other hand, the touch sensor and the display panel 4 may also be provided with a plurality of smaller through holes 2a within the radial dimensions of the shooting unit 8 so as not to adversely affect the display and touch sensitivity of the display unit 2.

[0141] The provision of these through holes 2a may slightly affect the visibility and touch sensitivity of the display screen 1a of the display unit 2, but this is practically unrelated to the minimal radial dimension of the camera module 3, which is only a few millimeters. The provision of these through holes 2a increases the transmittance of subject light passing through the display unit 2, thereby improving the quality of images captured by the camera module 3.

[0142] Examples of characterizing the structure of the display unit 2 include not only Figure 16 , and various examples are also considered. Figure 17 It is a cross-sectional view of a first modified example that features the structure of the display unit 2 . Figure 17The display unit 2 is provided with a polarizing component 2b that polarizes light in a specific direction on the back of the display panel 4. Figure 17 In the example of FIG. 4 , the polarizing member 2 b is arranged on the inner surface side of a region of the display panel 4 that does not overlap with the camera module 3 in the vertical direction.

[0143] When light incident on the display unit 2 strikes the polarizing element 2b, it is polarized in a specific direction upon reflection. This reflected light is further reflected by, for example, another layer of the display unit 2 and then strikes the imaging unit 8. Since this reflected light is polarized in a specific direction upon striking the imaging unit 8, if at least some of the multiple polarizing elements 8b have the same polarization characteristics as the polarizing element 2b, it is possible to reliably obtain polarization information of the light polarized by the polarizing element 2b.

[0144] Thus, by providing the polarizing member 2 b that polarizes light in a specific direction in the display unit 2 , the light incident on the imaging unit 8 can be polarized in a specific direction, and the speckle component and the diffraction component can be extracted more reliably using the plurality of polarization pixels 81 .

[0145] Figure 18 It is a cross-sectional view of a second modified example that features the structure of the display unit 2 . Figure 18 The display unit 2 is provided with a polarizing component 2b that polarizes light in a specific direction in the area of ​​the circular polarizing plate 5 that overlaps with the camera module 3 in the front-back direction. Originally, the camera module 3 is intended to capture light that is incident approximately parallel to the optical axis. Therefore, for example, a polarizing component 2b that polarizes light in a direction approximately parallel to the optical axis is provided in the area of ​​the circular polarizing plate 5 that overlaps with the camera module 3 in the front-back direction. As a result, the light passing through this area is polarized in a direction approximately parallel to the optical axis, and is incident on the shooting unit 8. If the multiple polarizing elements 8b in the shooting unit 8 have the same polarization characteristics as the polarizing component 2b, the multiple polarized pixels 81 can reliably receive the light incident on the shooting unit 8.

[0146] Thus, in the fourth embodiment, by locally processing the display unit 2, the transmittance of light passing through the display unit 2 can be increased, thereby improving the quality of images captured by the camera module 3. As a more specific example, by providing an opening in the area that overlaps with the camera module 3 in the front-to-back direction, the transmittance of light passing through the display unit 2 can be increased, thereby improving the quality of images captured by the camera module 3. Furthermore, by providing a polarizing member 2b that polarizes light in a specific direction on at least a portion of the display unit 2, light that is repeatedly reflected by the display unit 2 and incident on the imaging unit 8 can be polarized in a specific direction, allowing the polarized pixels 81 to reliably receive this light.

[0147] (Fifth embodiment)

[0148] As a specific supplement to the electronic device 1 having the configuration described in the first to fourth embodiments, various electronic devices can be considered. For example, Figure 19 This is a plan view of the case where the electronic device 1 according to the first to fourth embodiments is applied to a capsule endoscope 50 . Figure 19 The capsule endoscope 50 includes, within a shell 51, for example, whose two end faces are hemispherical and whose central portion is cylindrical: a camera (ultra-small camera) 52 for capturing images inside the body cavity; a memory 53 for recording image data captured by the camera 52; and a wireless transmitter 55 for transmitting the recorded image data to the outside via an antenna 54 after the capsule endoscope 50 is expelled from the body of the subject.

[0149] Furthermore, a CPU (Central Processing Unit) 56 and a coil (magnetic or current conversion coil) 57 are provided within the housing 51. The CPU 56 controls the camera 52's image capture and data storage in the memory 53, and controls the wireless transmitter 55 to transmit data from the memory 53 to a data receiving device (not shown) outside the housing 51. The coil 57 supplies power to the camera 52, the memory 53, the wireless transmitter 55, the antenna 54, and the light source 52b (described later).

[0150] Furthermore, the housing 51 is provided with a magnetic (guidance) switch 58 for detecting the placement of the capsule endoscope 50 on the data receiving device. The guidance switch 58 detects the placement of the capsule endoscope 50 on the data receiving device, and the CPU 56 supplies power from the coil 57 to the wireless transmitter 55 when data transmission is possible.

[0151] The camera 52 includes, for example, an imaging element 52a including an objective optical system 9 for capturing images within the body cavity, and multiple light sources 52b for illuminating the body cavity. Specifically, the camera 52 is constructed using, for example, a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) equipped with an LED (Light Emitting Diode) as the light source 52b.

[0152] The display unit 2 in the electronic device 1 of the first to fourth embodiments includes Figure 19 The concept of light source 52b and other luminous bodies. Figure 19The capsule endoscope 50 has, for example, two light sources 52b. However, these light sources 52b may be formed by a display panel 4 having multiple light source units or an LED module having multiple LEDs. In this case, by arranging the imaging unit 8 of the camera 52 below the display panel 4 and the LED module, the restrictions on the layout of the camera 52 are reduced, and a more compact capsule endoscope 50 can be realized.

[0153] in addition, Figure 20 This is a rear view of the electronic device 1 according to the first to fourth embodiments, applied to a digital SLR camera 60. The digital SLR camera 60 or compact camera includes a display unit 2 on the back side opposite the lens, which displays a preview screen. A camera module 3 may also be disposed on the side opposite the display surface of the display unit 2, enabling the user's facial image to be displayed on the display screen 1a of the display unit 2. According to the electronic device 1 according to the first to fourth embodiments, the camera module 3 can be disposed in the area overlapping the display unit 2. This eliminates the need to locate the camera module 3 at the outer edge of the display unit 2, allowing the display unit 2 to be maximized in size.

[0154] Figure 21A 1 is a plan view showing an example in which the electronic device 1 according to the first to fourth embodiments is applied to a head-mounted display (hereinafter simply referred to as HMD) 61 . Figure 21A The HMD 61 is used for VR (Virtual Reality), AR (Augmented Reality), MR (Mixed Reality) or SR (Substituional Reality). Figure 21B As shown, the camera 62 is mounted on the outer surface. However, there is a problem that the wearer of the HMD can visually recognize the surrounding images, but the people around him cannot know the eyes and facial expressions of the wearer of the HMD.

[0155] To this end, in Figure 21A In the embodiment, the display surface of the display unit 2 is provided on the outer surface of the HMD 61, and the camera module 3 is provided on the opposite side of the display surface of the display unit 2. In this way, the facial expression of the wearer captured by the camera module 3 can be displayed on the display unit 2, and people around the wearer can understand the wearer's facial expression and eye movements in real time.

[0156] exist Figure 21A In the case of a wearable device, the camera module 3 is provided on the inner surface of the display unit 2. This eliminates restrictions on the location of the camera module 3, thereby increasing the design freedom of the HMD 61. Furthermore, the camera can be placed at an optimal position, thereby preventing problems such as the wearer's line of sight being misaligned with the display surface.

[0157] As described above, in the fifth embodiment, the electronic device 1 according to the first to fourth embodiments can be used for various purposes, thereby improving its utility value.

[0158] (Sixth embodiment)

[0159] The difficulty of generating flare and diffraction may vary depending on the type of light source incident on electronic device 1 and the display brightness of display unit 2. Therefore, the type of light source can be estimated when calculating the correction amount of flare components and diffracted light components.

[0160] Figure 22 This is a block diagram showing a schematic configuration of an electronic device 1 according to the sixth embodiment. Figure 22 Electronic devices 1 except Figure 8 In addition to the configuration of the display unit 2, the display unit 2 further includes a light source estimation unit 46. The light source estimation unit 46 estimates the light source of the light incident on the display unit 2. The light source estimation unit 46 can estimate the light source based on the sensitivity differences of the RGB colors photoelectrically converted by the photoelectric conversion unit 8a. Alternatively, the light source estimation unit 46 can use an illuminance sensor, a multi-spectral sensor, or the like. Furthermore, a GPS sensor mounted on the electronic device 1 can determine whether the electronic device 1 is indoors or outdoors and estimate that the light source is sunlight or lighting. Furthermore, a gyro sensor mounted on the electronic device 1 can determine the tilt angle of the electronic device 1 and estimate the light source based on the tilt angle.

[0161] The speckle correction signal generator 36 adjusts the correction amount for the captured image, which has undergone photoelectric conversion by the plurality of non-polarized pixels 82, based on the light source estimated by the light source estimation unit 46. More specifically, the formula for calculating the correction amount can be changed depending on the type of light source. Furthermore, the correction amount can be adjusted individually for each of the RGB colors depending on the type of light source.

[0162] Figure 23 This is a block diagram showing a schematic configuration of an electronic device 1 according to a modified example of the sixth embodiment. Figure 23 Electronic devices 1 except Figure 8 In addition to the configuration of the display unit 2, the display unit 2 further includes a brightness estimation unit 47. The brightness estimation unit 47 estimates the display brightness of the display unit 2. The brightness estimation unit 47 estimates the display brightness of the display unit 2 based on a signal from a display control unit (not shown) that controls the display of the display unit 2, for example.

[0163] The flare correction signal generator 36 adjusts the correction amount for the captured image photoelectrically converted by the plurality of non-polarized pixels 82 based on the display brightness of the display unit 2 estimated by the brightness estimation unit 47. More specifically, the correction amount may be adjusted individually for each color based on the brightness of each of the RGB colors displayed on the display unit 2.

[0164] In addition, you may also consider having Figure 22 The light source estimation unit 46 and Figure 23 In this case, the spot correction signal generating unit 36 ​​adjusts the correction amount based on the light source type and the display brightness of the display unit 2.

[0165] In this way, in the sixth embodiment, the correction amount for correcting the influence of the spot component and the diffracted light component is adjusted taking into account the type of light source incident on the electronic device 1 and the display brightness of the display unit 2, so that the spot component and the diffracted light component can be more appropriately removed from the captured image.

[0166] (Seventh embodiment)

[0167] The optical system 9 of the camera module 3 of the electronic device 1 according to the seventh embodiment is different from those in the first to sixth embodiments.

[0168] Figure 24 1 is a diagram showing a cross-sectional structure of an imaging unit 8 of a camera module 3 mounted on an electronic device 1 according to a seventh embodiment. Figure 24 The imaging unit 8 includes a microlens array 64 rather than a single lens or a lens group in which single lenses are arranged in the optical axis direction.

[0169] In more detail, Figure 24 The shooting unit 8 includes: a photoelectric conversion unit 8a, which is arranged along the bottom surface of the shell 63; a microlens array 64, which is arranged above the photoelectric conversion unit 8a; a plurality of light-shielding bodies 66, which are arranged between adjacent microlenses 65; and a light guide plate 67, which is arranged above the microlens array 64.

[0170] In this manner, by providing the microlens array 64 as the optical system 9 of the imaging unit 8 , the influence of adjacent pixels can be prevented, and the occurrence of color bleeding can be reduced.

[0171] (Eighth Embodiment)

[0172] While the first through seventh embodiments described above illustrate examples of removing speckle and diffracted light, the electronic device 1 according to the present disclosure can be used not only to remove speckle and diffracted light but can also be used, for example, to remove reflections from a subject. The electronic device 1 according to this embodiment can be configured with a camera module 3 on the side opposite the display surface of the display unit 2. Therefore, when a wide-angle or fisheye lens is attached to the camera module 3, the entire display surface of the display unit 2 can be captured via the camera module 3. This allows for the detection of wet fingers, which are often misidentified by existing touch panels 6, as well as for touch operations and fingerprint authentication by removing reflections from wet fingers and detecting the unevenness of fingerprints.

[0173] More specifically, the plurality of polarization pixels 81 having the plurality of polarization elements 8b can also obtain polarization information caused by the reflected light component contained in the subject light. In this case, Figure 8 The flare extraction unit 35 extracts a correction amount based on the reflected light component based on the polarization information. Then, the flare correction signal generation unit 36 ​​generates an image from which the reflected light component is removed from the subject image.

[0174] In addition, in each of the above-mentioned embodiments, the premise is that the camera module 3 is arranged on the opposite side of the display surface of the display unit 2. However, in the case where there is a reflective plate at the bottom layer of the display panel 4 such as the OLED unit, a portion of the reflective plate can be hollowed out and the camera module 3 can be arranged in the hollowed portion. The camera module 3 can also be arranged in a manner surrounded by a portion of the display panel 4.

[0175] In addition, the present technology can take the following configurations.

[0176] (1) An electronic device includes a display unit and a camera unit, wherein the camera unit is arranged on a side opposite to a display surface of the display unit, and the camera unit has: a plurality of photoelectric conversion units that perform photoelectric conversion on light incident through the display unit; and a plurality of polarizing elements arranged on a light incident side of at least one of the plurality of photoelectric conversion units.

[0177] (2) Regarding the electronic device described in (1), the electronic device includes a correction unit that corrects a captured image after photoelectric conversion by the multiple photoelectric conversion units based on polarization information after polarization by the multiple polarizing elements and photoelectric conversion by the photoelectric conversion units.

[0178] (3) Regarding the electronic device described in (2), the correction unit removes a component of at least one of reflected light and diffracted light generated when passing through the display unit, which is incident on the plurality of photoelectric conversion units and captured.

[0179] (4) Regarding the electronic device described in (2) or (3), the correction unit corrects the digital pixel data by subtracting a correction amount based on the polarization information data obtained by digitizing the polarization information from the digital pixel data photoelectrically converted and digitized by the photoelectric conversion unit.

[0180] (5) Regarding the electronic device described in any one of (2) to (4), the electronic device includes a polarization component arranged on the display unit and polarizing the incident light in a specific direction, at least one polarization element among the multiple polarization elements allows the light polarized by the polarization component to pass through, and the correction unit corrects the captured image after photoelectric conversion by the multiple photoelectric conversion units based on the polarization information after the light is polarized by the polarization component, passes through the polarization element, and is photoelectrically converted by the photoelectric conversion unit.

[0181] (6) Regarding the electronic device described in any one of (2) to (4), the electronic device includes a light source estimating unit, which estimates the type of light source of the light incident on the display unit, and the correction unit adjusts the correction amount of the captured image after photoelectric conversion by the multiple photoelectric conversion units based on the type of light source estimated by the light source estimating unit.

[0182] (7) Regarding the electronic device described in any one of (2) to (4), the electronic device includes a brightness estimation unit, which estimates the display brightness of the display unit, and the correction unit adjusts the correction amount of the captured image after photoelectric conversion by the multiple photoelectric conversion units according to the display brightness of the display unit.

[0183] (8) Regarding the electronic device described in any one of (2) to (7), the electronic device includes a signal processing unit that performs at least one of exposure processing, focus adjustment processing, and white balance adjustment processing on the captured image corrected by the correction unit.

[0184] (9) Regarding the electronic device described in any one of (2) to (7), the electronic device comprises: a signal processing unit that performs at least one signal processing on the captured image corrected by the correction unit; and a parameter adjustment unit that adjusts the parameters when the signal processing is performed on the captured image according to the correction amount of the captured image corrected by the correction unit.

[0185] (10) Regarding the electronic device described in (9), the signal processing performed by the signal processing unit includes at least one of noise reduction processing for removing noise components contained in the captured image and edge emphasis processing for emphasizing edges of the captured image. The parameter adjustment unit adjusts parameters for performing at least one of the noise reduction processing and the edge emphasis processing based on the correction amount.

[0186] (11) Regarding the electronic device described in (10), the parameter adjustment unit performs at least one of a higher degree of noise removal in the noise reduction processing and a higher degree of edge emphasis in the edge emphasis processing on the captured image with a larger correction amount.

[0187] (12) Regarding the electronic device described in any one of (9) to (11), when the signal processing unit performs signal processing including exposure adjustment processing and performs the exposure adjustment processing, the signal processing unit performs exposure adjustment so that the output values ​​of the multiple photoelectric conversion units are not saturated.

[0188] (13) Regarding the electronic device described in any one of (1) to (12), the electronic device includes a defect correction unit, which uses the output values ​​of two or more of the surrounding photoelectric conversion units to correct the output value of the photoelectric conversion unit into which the light polarized by the polarizing element is incident.

[0189] (14) Regarding the electronic device described in any one of (1) to (13), the electronic device includes a camera, the camera has the photoelectric conversion unit and an optical system that converges light on the photoelectric conversion unit, and the camera is arranged on the side opposite to the approximately central part of the display surface of the display unit.

[0190] (15) Regarding the electronic device described in any one of (1) to (14), the electronic device may include a defect correction unit, which corrects the output value of the photoelectric conversion unit with a saturated output value among the multiple photoelectric conversion units based on polarization information after polarization is performed by one or more polarization elements arranged around the photoelectric conversion unit and photoelectric conversion is performed by the photoelectric conversion unit.

[0191] (16) Regarding the electronic device described in any one of (1) to (15), the photoelectric conversion unit has a plurality of split photoelectric conversion units, and the light polarized by the plurality of polarizing elements is incident on the plurality of split photoelectric conversion units in a part of the photoelectric conversion units.

[0192] (17) Regarding the electronic device described in any one of (1) to (15), the photoelectric conversion unit has a plurality of split photoelectric conversion units, and the light polarized by the polarizing element can be incident on any of the split photoelectric conversion units.

[0193] (18) Regarding the electronic device described in any one of (1) to (15), the photoelectric conversion unit has a plurality of split photoelectric conversion units, and the light polarized by the polarizing element is incident on each of a portion of the split photoelectric conversion units in two or more of the photoelectric conversion units.

[0194] (19) Regarding the electronic device described in any one of (1) to (15), the plurality of photoelectric conversion sections include a plurality of divided photoelectric conversion sections, and the light polarized by the plurality of polarizing elements is incident on the plurality of photoelectric conversion sections.

[0195] (20) Regarding the electronic device described in any one of (1) to (15), the photoelectric conversion unit has a plurality of split photoelectric conversion units that are divided into a plurality of units in one direction and can detect phase difference information, and the light polarized by the polarizing element is incident on any of the plurality of split photoelectric conversion units.

[0196] The present disclosure is not limited to the above-described embodiments, but also encompasses various modifications that can be conceived by those skilled in the art. The effects of the present disclosure are not limited to the above-described contents. In other words, various additions, modifications, and partial deletions may be made without departing from the conceptual ideas and purpose of the present disclosure as defined by the claims and their equivalents.

Claims

1. An electronic device, characterized in that: have: Display unit; an imaging unit disposed on a side opposite to the display surface of the display unit and on an inner surface side of the display screen of the display unit, the imaging unit comprising: a plurality of photoelectric conversion units for performing photoelectric conversion on light incident through the display unit; and a plurality of polarizing elements disposed on a light incident side of at least one of the plurality of photoelectric conversion sections; a correction section that corrects the captured image photoelectrically converted by the plurality of photoelectric conversion sections based on polarization information of light polarized by the plurality of polarizing elements and photoelectrically converted by the photoelectric conversion sections; and a light source estimating unit for estimating a type of light source of light incident on the display unit; The correction unit adjusts a correction amount of a captured image photoelectrically converted by the plurality of photoelectric conversion units based on the type of the light source estimated by the light source estimation unit.

2. The electronic device according to claim 1, wherein The correction unit removes a component of at least one of reflected light and diffracted light generated when passing through the display unit and incident on the plurality of photoelectric conversion units and captured.

3. The electronic device according to claim 1, wherein The correction section corrects the digital pixel data by performing a process of subtracting a correction amount based on polarization information data obtained by digitizing the polarization information from the digital pixel data photoelectrically converted and digitized by the photoelectric conversion section.

4. The electronic device according to claim 1, wherein: The electronic device includes a polarizing member provided on the display unit and configured to polarize incident light in a specific direction. At least one of the plurality of polarizing elements allows light polarized by the polarizing component to pass therethrough. The correction section corrects the captured image photoelectrically converted by the plurality of photoelectric conversion sections based on polarization information of light polarized by the polarizing member, passing through the polarizing element, and photoelectrically converted by the photoelectric conversion section.

5. The electronic device according to claim 1, wherein The electronic device includes a brightness estimation unit configured to estimate display brightness of the display unit. The correction unit adjusts a correction amount of a captured image photoelectrically converted by the plurality of photoelectric conversion units, according to display brightness of the display unit.

6. The electronic device according to claim 1, wherein: The electronic device includes a signal processing unit configured to perform at least one of exposure processing, focus adjustment processing, and white balance adjustment processing on the captured image corrected by the correction unit.

7. The electronic device according to claim 1, wherein: have: a signal processing unit that performs at least one signal processing on the captured image corrected by the correction unit; and The parameter adjustment unit adjusts parameters for signal processing of the captured image based on the correction amount of the captured image corrected by the correction unit.

8. The electronic device according to claim 7, wherein: The signal processing performed by the signal processing unit includes at least one of a noise reduction process for removing noise components included in the captured image and an edge emphasis process for emphasizing edges of the captured image. The parameter adjustment unit adjusts a parameter when performing at least one of the noise reduction process and the edge emphasis process according to the correction amount.

9. The electronic device according to claim 8, wherein: The parameter adjustment unit performs at least one of a higher degree of noise removal in the noise reduction process and a higher degree of edge emphasis in the edge emphasis process on the captured image having a larger correction amount.

10. The electronic device according to claim 7, wherein: The signal processing section performs exposure adjustment so that output values ​​of the plurality of photoelectric conversion sections are not saturated when performing signal processing including exposure adjustment processing and performing the exposure adjustment processing.

11. The electronic device according to claim 1, wherein The electronic device includes a defect correction unit that corrects an output value of the photoelectric conversion unit on which light polarized by the polarization element is incident, using output values ​​of two or more surrounding photoelectric conversion units.

12. The electronic device according to claim 1, wherein The electronic device includes a camera having the photoelectric conversion unit and an optical system for focusing light on the photoelectric conversion unit. The imaging device is arranged on a side opposite to the vicinity of a center portion of a display surface of the display unit.

13. The electronic device according to claim 1, wherein The electronic device includes a defect correction unit, which corrects the output value of the photoelectric conversion unit whose output value is saturated among the multiple photoelectric conversion units based on polarization information after polarization is performed by one or more polarization elements arranged around the photoelectric conversion unit and photoelectric conversion is performed by the photoelectric conversion unit.

14. The electronic device according to claim 1, wherein The photoelectric conversion unit includes a plurality of divided photoelectric conversion units. The light polarized by the plurality of polarization elements is incident on the plurality of divided photoelectric conversion sections in a part of the photoelectric conversion sections.

15. The electronic device according to claim 1, wherein The photoelectric conversion unit includes a plurality of divided photoelectric conversion units. The light polarized by the polarizing element is incident on the divided photoelectric conversion section, which is a part of any of the photoelectric conversion sections.

16. The electronic device according to claim 1, wherein The photoelectric conversion unit includes a plurality of divided photoelectric conversion units. The light polarized by the polarizing element is incident on the divided photoelectric conversion section, which is a part of each of the two or more photoelectric conversion sections.

17. The electronic device according to claim 1, wherein: The plurality of photoelectric conversion sections include a plurality of divided photoelectric conversion sections, The light polarized by the plurality of polarization elements is incident on the plurality of photoelectric conversion sections.

18. The electronic device according to claim 1, wherein The photoelectric conversion section includes a plurality of divided photoelectric conversion sections that are divided into a plurality of sections in one direction and are capable of detecting phase difference information. The light polarized by the polarization element is incident on the plurality of divided photoelectric conversion sections in any of the photoelectric conversion sections.

Images