Electronic device

Abstract

An electronic device is provided that is capable of taking pictures regardless of abnormalities on the display surface. The electronic device includes: a display unit; a camera unit disposed on the side opposite to the display surface of the display unit; an abnormality detection unit that detects abnormalities on the display surface; and a display control unit that highlights the location of the abnormality detected by the abnormality detection unit on the display unit.

Description

Technical Field

[0001] This disclosure relates to electronic devices. Background Technology

[0002] In recent electronic devices such as smartphones, mobile phones, and PCs (Personal Computers), various sensors, including cameras, are mounted on the bezel (outer frame) of the display. On the other hand, there is a growing demand to make the form factor of electronic devices as compact as possible without affecting screen size, leading to a trend towards narrower bezel widths. Against this backdrop, a technique has been proposed that places a camera module directly below the display and uses the camera to capture light from a subject passing through the display.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: US Patent Publication 2018 / 0069060 Summary of the Invention

[0006] The technical problem that the invention aims to solve

[0007] Smartphones and other devices have built-in touch sensors in their displays, which often accumulate fingerprints, hand oil, and other impurities. Additionally, defects may sometimes exist on the display surface, the protective film, or air bubbles may form between the display surface and the film.

[0008] Thus, if there is an abnormality in the display surface, when a camera positioned directly below the display is used to take a picture through the display, it may cause foreign objects to be reflected in the picture or the focus to be on the foreign object, making it impossible to take the picture as expected.

[0009] Therefore, this disclosure provides an electronic device capable of taking pictures without being affected by anomalies in the display surface.

[0010] Solutions for solving technical problems

[0011] To solve the above-mentioned technical problems, according to this disclosure, an electronic device is provided, comprising: a display unit; a camera unit disposed on the side opposite to the display surface of the display unit; an anomaly detection unit for detecting anomalies on the display surface; and a display control unit for highlighting the location of the anomaly detected by the anomaly detection unit on the display unit.

[0012] Alternatively, the display control unit may cause the display unit to display information reminding the user to remove the abnormality.

[0013] Alternatively, the information may include information corresponding to the type of the anomaly.

[0014] Alternatively, the display control unit may cause the display unit to display a flag indicating the location where the abnormality occurred.

[0015] According to this disclosure, an electronic device is provided, comprising: a display unit capable of emitting light at multiple different emission wavelengths; a camera unit disposed on a side opposite to the display surface of the display unit; and an anomaly detection unit that detects anomalies on the display surface based on multiple images captured by the camera unit while at least a portion of the display surface is emitting light at each of the multiple emission wavelengths.

[0016] Alternatively, the anomaly detection unit can detect anomalies on the display surface based on multiple images captured by the camera unit in a state where the area on the display surface that overlaps with the viewing angle of the camera unit emits light at each of the multiple emission wavelengths.

[0017] According to this disclosure, an electronic device is provided, comprising: a display unit; a camera unit disposed on the side opposite to the display surface of the display unit; an anomaly detection unit for detecting anomalies on the display surface; and a correction processing unit for correcting images captured by the camera unit based on the anomalies.

[0018] Alternatively, the correction processing unit may correct the image captured by the camera unit based on at least one of the types, colors, sizes, locations, and quantities of the anomalies.

[0019] Alternatively, the electronic device may include an anomaly detection unit that can determine the type of the anomaly.

[0020] The correction processing unit performs correction processing on the image captured by the camera unit, corresponding to the type of anomaly determined by the anomaly discrimination unit.

[0021] Alternatively, the correction process may include at least one of edge enhancement processing, deformation correction processing, and defect correction processing.

[0022] Alternatively, the electronic device may include a model generation unit that generates a model for correcting the image captured by the camera unit based on the information of the anomaly. This model is obtained by learning from at least one of the types, colors, sizes, locations, and quantities of the anomaly and the images before and after correction by the correction processing unit.

[0023] The correction processing unit provides the learned model with images captured by the camera unit and information about the anomalies to correct the images captured by the camera unit.

[0024] Alternatively, the electronic device may include: a correction determination unit that determines whether the correction performed by the correction processing unit is effective; an image communication unit that, if the correction performed by the correction processing unit is determined to be ineffective, sends the image captured by the camera unit and the abnormal information to an information processing device, and receives the image corrected by the information processing device; and an output unit that outputs the image corrected by the information processing device.

[0025] Alternatively, the camera unit may have multiple cameras that capture images from different perspectives, and the correction processing unit may remove the anomaly from the images captured by the camera unit based on the multiple images captured by the multiple cameras.

[0026] Alternatively, the camera unit may have a plurality of photoelectric conversion units that perform photoelectric conversion on light incident via the display unit, at least one of the plurality of photoelectric conversion units being capable of detecting phase difference information, and the anomaly detection unit detecting the anomaly based on the phase difference information.

[0027] Alternatively, the camera unit may have: a plurality of photoelectric conversion units that perform photoelectric conversion on light incident via the display unit; and a plurality of polarization elements disposed on the light incident side of at least one of the plurality of photoelectric conversion units, wherein the anomaly detection unit detects the anomaly based on polarization information polarized by the plurality of polarization elements and photoelectrically converted by the photoelectric conversion units.

[0028] Alternatively, the plurality of polarization elements may include multiple polarization elements that detect different polarization states respectively.

[0029] Alternatively, the camera unit may include: a plurality of photoelectric conversion units that perform photoelectric conversion on light incident via the display unit; and a microlens array that images the subject onto the plurality of photoelectric conversion units.

[0030] Alternatively, the electronic device may include a distance detection unit that detects the distance to the subject being photographed by the camera unit, and an anomaly detection unit that detects the anomaly based on the distance detected by the distance detection unit.

[0031] Alternatively, the imaging unit may have multiple cameras that capture images from different perspectives, and the distance detection unit may detect the distance based on the images captured by the multiple cameras.

[0032] Alternatively, the electronic device may include a fingerprint detection unit that detects the fingerprint of a finger that is in contact with the display surface based on an image captured by the camera unit. Attached Figure Description

[0033] Figure 1A This is a schematic cross-sectional view of the electronic device according to the first embodiment.

[0034] Figure 1B Is Figure 1A A schematic cross-sectional view of the electronic device with the opening has been added.

[0035] Figure 2 (a) is Figure 1A A schematic diagram of the appearance of an electronic device. Figure 2 (b) is Figure 2 (a) Cross-sectional view along line AA.

[0036] Figure 3 This is a cross-sectional view showing an example of the cross-sectional structure of the camera unit.

[0037] Figure 4A This is a block diagram showing the internal structure of the electronic device according to the first embodiment.

[0038] Figure 4B It is a block diagram showing the internal structure of the signal processing unit.

[0039] Figure 5 This is a flowchart illustrating the processing operations of the electronic device 1 according to the first embodiment.

[0040] Figure 6A This is a diagram that highlights specific examples of locations where an anomaly has occurred.

[0041] Figure 6B This is a diagram that highlights specific examples of locations where an anomaly has occurred.

[0042] Figure 6C This is a diagram that highlights specific examples of locations where an anomaly has occurred.

[0043] Figure 6D This is a diagram that highlights specific examples of locations where an anomaly has occurred.

[0044] Figure 7A This diagram schematically illustrates an example of exposure by a camera unit when a portion of the display surface emits light at different wavelengths.

[0045] Figure 7B This diagram schematically illustrates an example of exposure by a camera unit when a portion of the display surface emits light at different wavelengths.

[0046] Figure 7C This diagram schematically illustrates an example of exposure by a camera unit when a portion of the display surface emits light at different wavelengths.

[0047] Figure 8 This is a flowchart illustrating the processing operations of the electronic device according to the second embodiment.

[0048] Figure 9 This is a block diagram illustrating the internal structure of the electronic device according to the third embodiment.

[0049] Figure 10 This is a flowchart showing the processing sequence of the correction processing unit.

[0050] Figure 11 This is a block diagram illustrating an example of how electronic devices and information processing devices are connected.

[0051] Figure 12 This is a top view of the electronic device according to the fourth embodiment.

[0052] Figure 13 yes Figure 12 A cross-sectional view along line AA.

[0053] Figure 14 This is a top view showing an example of the pixel arrangement of the camera unit in the fifth embodiment.

[0054] Figure 15 This is a top view showing an example of the pixel arrangement of the camera unit in the sixth embodiment.

[0055] Figure 16 This is a cross-sectional view of the camera unit in the sixth embodiment.

[0056] Figure 17 This is a three-dimensional diagram showing an example of the detailed structure of each polarization element.

[0057] Figure 18 This is a block diagram showing the internal structure of the signal processing unit within an electronic device that has the function of suppressing the effects of flare light and diffraction light.

[0058] Figure 19 This is a block diagram showing the general configuration of the electronic device according to the seventh embodiment.

[0059] Figure 20 This is a diagram showing the cross-sectional structure of the camera unit of the camera module mounted on the electronic device according to the ninth embodiment.

[0060] Figure 21 This is a top view of the case where the electronic devices of the first to ninth embodiments are applied to a capsule endoscope.

[0061] Figure 22 This is a rear view of the case where the electronic devices of the first to ninth embodiments are applied to a digital SLR camera.

[0062] Figure 23A This is a top view showing an example of applying the electronic devices of the first to ninth embodiments to an HMD.

[0063] Figure 23B This is a diagram representing the current HMD.

[0064] Explanation of reference numerals in the attached figures

[0065] 1: Electronic device; 1a: Display surface; 1b: Outer frame; 2: Display unit; 2b: Protective film; 2c: Polyimide substrate; 2d: Display layer; 2e: Barrier layer; 2f: Touch sensor layer; 2g: Adhesive layer; 2h: Circular polarizer; 2i: Optical adhesive sheet; 2p: Abnormality; 2q: Mark; 2r: Film; 3: Camera module; 4: Imaging unit; 4a: Photoelectric conversion unit; 5: Optical system; 11: Semiconductor substrate; 12: Component separation layer; 13: Planarization layer; 14: Color filter layer; 15: On-chip lens; 16: Readout circuit ; 17: Interlayer insulating film; 21: Camera device; 22: Application processor; 23: Video signal generation unit; 24: A / D conversion unit; 25: Display control unit; 31: A / D conversion unit; 32: Signal processing unit; 32a: Clamping unit; 32b: Color output unit; 32c: Defect correction unit; 32d: Linear matrix unit; 32e: Gamma correction unit; 32f: Luminance and chrominance signal generation unit; 32g: Noise reduction unit; 32h: Edge enhancement unit; 33: Camera control unit; 34: Exposure adjustment unit; 35: Anomaly detection unit; 36: Output unit. Detailed Implementation

[0066] Hereinafter, embodiments of the electronic device will be described with reference to the accompanying drawings. The description focuses on the main components of the electronic device; however, there may be components and functions not shown or described in the electronic device. These unshown components and functions will not be omitted from the following description.

[0067] (First Implementation)

[0068] Figure 1A and Figure 1B This is a schematic cross-sectional view of the electronic device 1 according to the first embodiment. Figure 1A The electronic device 1 is any electronic device 1 that combines display and shooting functions, such as smartphones, mobile phones, tablets, and PCs. Figure 1A The electronic device 1 includes a camera module (image capture unit) 3 disposed on the side opposite to the display surface 1a of the display unit 2. Thus, Figure 1A The electronic device 1 has a camera module mounted on the back side of the display surface 1a of the display unit 2. Therefore, the camera module takes pictures through the display unit 2.

[0069] Figure 2 (a) is Figure 1A A schematic view of the appearance of electronic device 1. Figure 2 (b) is Figure 2 A cross-sectional view along line AA in (a). Figure 2 In example (a), the display surface 1a is expanded to be close to the overall dimensions of the electronic device 1, and the width of the outer frame 1b surrounding the display surface 1a is set to less than a few millimeters. Typically, the front-facing camera is mounted on the outer frame 1b, but... Figure 2 In (a), as shown by the dashed line, a camera module 3, which functions as a front-facing camera, is disposed on the rear side of the approximately central portion of the display surface 1a. In this way, by placing the front-facing camera on the rear side of the display surface 1a, the width of the outer frame 1b can be narrowed without having to place the front-facing camera on the outer frame 1b.

[0070] In addition, Figure 2 In (a), the camera module 3 is disposed on the back side of approximately the center of the display surface 1a. However, in this embodiment, any back side of the display surface 1a is acceptable; for example, the camera module 3 can be disposed on the back side near the periphery of the display surface 1a. Thus, in this embodiment, the camera module 3 is disposed at any position on the back side overlapping with the display surface 1a.

[0071] like Figure 1A As shown, the display unit 2 is a laminate formed by sequentially stacking a protective film 2b, a polyimide substrate 2c, a display layer 2d, a barrier layer 2e, a touch sensor layer 2f, an adhesive layer 2g, a circular polarizing plate 2h, an optically clear adhesive (OCA) 2i, and a protective glass 2j. The display layer 2d can be, for example, an OLED (Organic Light Emitting Device) display layer, a liquid crystal display layer, a MicroLED display layer, or a display layer based on other display principles. The display layer 2d may sometimes consist of multiple layers. For example, the display layer 2d may sometimes include a color filter layer, a backlight layer, etc. The display unit 2 displays light using light in the visible light wavelength range, but the light displayed by the display unit 2 may also include infrared light components.

[0072] The barrier layer 2e prevents oxygen and moisture from penetrating the display layer 2d. A touch sensor is assembled on the touch sensor layer 2f. The touch sensor can be of various types, such as capacitive or resistive, but any type can be used. Alternatively, the touch sensor layer 2f can be integrated with the display layer 2d.

[0073] The adhesive layer 2g is provided to bond the circular polarizer 2h to the touch sensor layer 2f. A material with high visible light transmittance is used for the adhesive layer 2g. The circular polarizer 2h is provided to reduce glare or improve the visibility of the display surface 1a in bright environments. The optical adhesive sheet 2i is provided to improve the adhesion between the circular polarizer 2h and the protective glass 2j. A material with high visible light transmittance can be used for the optical adhesive sheet 2i. The protective glass 2j is provided to protect the display layer 2d, etc. Furthermore, the layer structure of the display section 2 is not necessarily limited to... Figure 1A , Figure 2 The layers shown are structurally similar.

[0074] The camera module 3 is positioned on the side opposite to the display surface 1a of the display unit 2, i.e., on the back side of the display unit 2. The camera module 3 includes an image-capturing unit 4 and an optical system 5. The optical system 5 is positioned on the light-incident surface side of the image-capturing unit 4, i.e., the side closest to the display unit 2, to focus light passing through the display unit 2 onto the image-capturing unit 4. The optical system 5 is typically composed of multiple lenses. As described later, sometimes multiple camera modules 3 may be positioned on the side opposite to the display surface 1a of the display unit 2. In this case, the focal length of the optical system 5 of each camera module 3 can be different from each other, thereby enabling shooting with different perspectives such as telephoto and wide-angle.

[0075] The camera unit 4 includes a photoelectric conversion unit 4a. The photoelectric conversion unit 4a performs photoelectric conversion on light incident via the display unit 2. The photoelectric conversion unit 4a can be a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor. Alternatively, the photoelectric conversion unit 4a can be a photodiode or an organic photoelectric conversion film.

[0076] The photoelectric conversion unit 4a has photoelectric conversion elements such as a CMOS sensor in each pixel. The pixels can be arranged in any way. Specifically, the arrangement of the pixels can be Bayer arrangement, interlaced arrangement, patterned arrangement, striped arrangement, or other arrangements.

[0077] like Figure 1A and Figure 2 As shown in (b), in this embodiment, the electronic device 1 is configured such that the display unit 2 and the camera module 3 overlap in opposite directions. Therefore, the camera module 3 captures images of the subject light transmitted through the display unit 2. Figure 1AAs shown, the display unit 2 is formed of multiple layers. If the transmittance of light in a wavelength band with sufficient sensitivity is high in each layer, then the imaging unit 4 will not have a problem. However, in reality, the transmittance of some layers may be low. For example, the visible light transmittance of the polyimide substrate 2c is not that high. Therefore, as... Figure 1B As shown, one or more openings 2k can be formed in layers with relatively low transmittance among the multiple layers constituting the display section 2, and the camera module 3 can capture images of the light passing through the openings 2k. Figure 1B In this example, multiple openings 2k are provided that penetrate the layers other than the protective glass 2j among the multiple layers constituting the display unit 2. These openings 2k are located at positions where they overlap with the camera module 3 when viewed from above. The diameter and number of each opening 2k are set to appropriate values ​​considering the display quality of the display unit 2 and the image quality of the captured images of the camera unit 4.

[0078] Furthermore, when using a substrate with higher transparency instead of a polyimide substrate 2c, it is not necessarily required to provide an opening 2k.

[0079] Figure 3 This is a cross-sectional view showing an example of the cross-sectional structure of the camera unit 4. Figure 3 The imaging unit 4 includes a photoelectric conversion unit 4a formed within a semiconductor substrate 11, with each photoelectric conversion unit 4a separated by an element separation layer 12 for each pixel. A planarization layer 13 is disposed on the photoelectric conversion unit 4a, and a color filter layer 14 is disposed on the planarization layer 13. The color filter layer 14 may have filters for the three colors RGB, or filters for cyan, magenta, and yellow as their complementary colors. Alternatively, it may have filters that allow colors other than visible light, such as infrared light, to pass through, filters with multispectral characteristics, or subtractive filters such as white. By allowing light other than visible light, such as infrared light, to pass through, sensing information such as depth information can be detected. An on-chip lens 15 is disposed on the color filter layer 14. Light passes through the on-chip lens 15 and is incident. In this specification, the side on which the on-chip lens 15 is disposed is referred to as the back side of the imaging unit 4.

[0080] On the surface side of the imaging unit 4, a readout circuit 16 is formed on the semiconductor substrate 11, and the readout circuit 16 is surrounded by an interlayer insulating film 17. The readout circuit 16 includes a transfer transistor, a reset transistor, an amplification transistor, and a selection transistor. Furthermore, the cross-sectional structure of the imaging unit 4 is not limited to... Figure 3 The cross-sectional structure shown.

[0081] Figure 4A This is a block diagram showing the internal structure of the electronic device 1 according to the first embodiment. For example... Figure 4AAs shown, the electronic device 1 includes a camera device 21, an application processor 22, a video signal generation unit 23, an A / D conversion unit 24, a display control unit 25, and a display unit 2.

[0082] The camera device 21 can be composed of one or more semiconductor devices. In addition to the camera unit 4, optical system 5 and IR (Infrared Ray) cut-off filter 6 that constitute the camera module 3, it also has an A / D conversion unit 31, a signal processing unit 32, a camera control unit 33, an exposure adjustment unit 34, an anomaly detection unit 35 and an output unit 36.

[0083] The A / D converter 31 converts the analog pixel signals captured by the camera unit 4 into digital pixel data.

[0084] For example Figure 4B As shown in detail, the signal processing unit 32 includes: a clamping unit 32a, a color output unit 32b, a defect correction unit 32c, a linear matrix unit 32d, a gamma correction unit 32e, a luminance and chrominance signal generation unit 32f, a noise reduction unit 32g, and an edge enhancement unit 32h.

[0085] The clamping unit 32a performs a specified black level processing. More specifically, the clamping unit 32a performs a process of subtracting black level data from digital pixel data. The color output unit 32b outputs, for example, pixel data for each color of RGB. The defect correction unit 32c performs a process of correcting the image data of a specific pixel that cannot be accurately read for some reason based on the image data of the surrounding pixels. The linear matrix unit 32d performs more accurate color reproduction by performing row and column operations on color information such as RGB. The gamma correction unit 32e performs gamma correction based on the display characteristics of the display unit 2 to achieve a display with excellent visibility. For example, the gamma correction unit 32e performs a 10-bit to 8-bit conversion while changing the gradient. The luminance and chrominance signal generation unit 32f generates a luminance and chrominance signal for display on the display unit 2 based on the output data of the gamma correction unit 32e. The noise reduction unit 32g performs a process of reducing noise included in the luminance and chrominance signal. The edge enhancement unit 32h performs a process of enhancing the edges of the subject image based on the luminance and chrominance signal. The noise reduction processing performed by the noise reduction unit 32g and the edge enhancement processing performed by the edge enhancement unit 32h can be performed only under specified conditions. The output unit 36 ​​outputs the luminance and chrominance signals after noise reduction processing.

[0086] The camera control unit 33 sets the frame rate for recording by the camera unit 4 based on the luminance and chrominance signals. The exposure adjustment unit 34 adjusts the exposure time according to the frame rate set by the camera control unit 33. The photoelectric conversion unit 4a records images pixel by pixel according to the exposure time adjusted by the exposure adjustment unit 34.

[0087] The anomaly detection unit 35 detects anomalies on the display surface of the display unit 2. Anomalies include various deposits on the display surface, defects on the display surface, deposits or defects in protective materials such as the protective film on the display surface, and air bubbles between the display surface and the protective material. Deposits include dirt, hand oil, solidified substances, and liquefied substances. Defects include cracks, fissures, and fissures. Thus, the anomaly detection unit 35 detects a variety of anomalies. In this specification, anomalies are sometimes collectively referred to as foreign matter.

[0088] The anomaly detection unit 35 sends information about detected anomalies to the application processor 22. Here, the anomaly information includes, for example, information about the type and location of the anomaly. Additionally, the anomaly detection unit 35 sends camera image data, including the anomaly information, to the output unit 36. The output unit 36 ​​outputs the anomaly information along with the camera image data. The output unit 36 ​​may also sometimes send the camera image data containing the anomaly to the video signal generation unit 23.

[0089] The application processor 22 is a semiconductor device independent of the camera module 3, and is mounted on the same or a different substrate as the camera module 3. The application processor 22 internally contains a CPU (Central Processing Unit) and executes programs such as the operating system and various application software. The application processor 22 may also incorporate a GPU (Graphics Processing Unit), baseband processor, etc., to perform image processing, signal processing, and other functions. The application processor 22 performs various processing operations on the input image data and calculation results as needed, or controls the display of images on the display unit 2 of the electronic device 1, or sends them to an external cloud server via a specified network.

[0090] When the application processor 22 receives an anomaly information sent from the anomaly detection unit 35, it sends the information to the display control unit 25. The display control unit 25 then highlights the location on the display unit 2 where the anomaly detected by the anomaly detection unit 35 occurred. Specific examples of this highlighting will be described later.

[0091] The video signal generation unit 23 generates a video signal for display on the display unit 2. The A / D conversion unit 24 converts the video signal into digital pixel data. The display control unit 25 controls the display of the digital pixel data on the display unit 2. At this time, as described above, based on the information of the anomaly detected by the anomaly detection unit 35, the location where the anomaly occurred is highlighted.

[0092] Figure 5 This is a flowchart illustrating the processing operations of the electronic device 1 according to the first embodiment, showing, for example, the processing operations of the application processor 22, the camera control unit 33, and the display control unit 25. This flowchart is continuously repeated while the electronic device 1 is powered on.

[0093] First, it is determined whether the user of the electronic device 1 has activated the camera module 3 (step S1). If the user has not activated the camera module 3, the image generated by the video signal generation unit 23 based on the instruction from the application processor 22 is displayed on the display unit 2 (step S2).

[0094] When the user activates the camera module 3, exposure based on the camera unit 4 is performed during the display-off period of the display unit 2 (step S3). The display of the display unit 2 is updated at a frequency of 30 to 120 frames per second. There is a display-off period between two consecutive frames during which no display occurs. In addition, within a frame, while scanning each horizontal line of each horizontal line, the display is performed, and there is also a display-off period between each horizontal line during which no display occurs. In step S3, exposure based on the camera unit 4 is performed during these display-off periods.

[0095] The reason for performing exposure during the display-off period is that when the camera module 3 takes a picture while an image is displayed on the display unit 2, it may adversely affect the captured image. However, as will be described later, for the purpose of anomaly detection, it is sometimes possible to perform imaging while the display unit 2 is intentionally emitting light in a predetermined light-emitting color. Therefore, the display-off period in step S3 more accurately refers to the period during which an image already displayed on the display unit 2 before the camera module 3 is activated is not displayed.

[0096] Next, the signal processing unit 32 performs various signal processing on the image obtained in the exposure processing based on the camera unit 4 (step S4).

[0097] Next, the anomaly detection unit 35 detects whether there is an anomaly on the display surface based on the signal-processed camera image data (step S5). Various methods can be considered for anomaly detection, and specific examples will be described later. If no anomaly is detected, the camera image data output from the signal processing unit 32 is sent to the output unit 36 ​​(step S6).

[0098] When an anomaly is detected in step S5, the anomaly detection unit 35 sends anomaly information, including the location where the anomaly occurred, to the application processor 22, and sends the anomaly information and camera image data to the output unit 36 ​​(step S7).

[0099] The application processor 22 sends the abnormality information from the abnormality detection unit 35 to the display control unit 25. When the display control unit 25 displays an image based on the video signal generated by the video signal generation unit 23 on the display surface, it highlights the location where the abnormality occurred on the display surface (step S8).

[0100] Figure 6A , Figure 6B , Figure 6C as well as Figure 6D This is a diagram that highlights a specific example of the location where an anomaly 2p occurred. Figure 6A This example shows the display of the ring-shaped marker 2q surrounding the location where the anomaly 2p occurred. Figure 6A The logo 2q is preferably displayed in a color different from the background image to make it stand out. Alternatively, the line width of the logo 2q can be made thicker.

[0101] Figure 6B This example shows the display of the marker 2q for the arrow indicating the location where an error 2p occurred. Alternatively, a message such as "Please wipe" can be added to the base of the arrow.

[0102] Figure 6C This illustrates an example of enlarging and displaying anomaly 2p included in the camera image data on the display surface in order to notify the user of the type of anomaly 2p. Figure 6C This illustrates how anomaly 2p is an example of a fingerprint, allowing users to understand the type of anomaly by zooming in on the image.

[0103] Figure 6D This indicates that anomaly 2p is an example of damage to the film 2r attached to the display surface. In this case, a message such as "Please replace the film" can also be used.

[0104] Thus, in the first embodiment, when the camera unit 4, positioned on the side opposite to the display surface of the display unit 2, is recording an image, if an abnormality 2p is detected on the display surface, the location of the abnormality 2p is highlighted on the display unit 2. This allows the user to be notified in advance that the abnormality 2p may interfere with the recording operation of the camera unit 4, thus promoting the elimination of the abnormality 2p. Based on the highlighted display on the display unit 2, the user can perform recording based on the camera unit 4 by independently wiping the display surface or performing other actions to eliminate the abnormality 2p, thereby improving the image quality of the recorded image.

[0105] (Second Implementation)

[0106] The second implementation detects anomaly 2p based on multiple camera images captured while emitting light at multiple emission wavelengths.

[0107] Figure 7A , Figure 7B as well as Figure 7C This diagram schematically illustrates an example of exposure based on the imaging unit 4 performed when a display area 20, which is part of the viewing angle range of the display surface 1a of the camera module 3, emits light at different wavelengths. For example, Figures 7A to 7C Examples of emitting light and exposing the light at red, green, and blue wavelengths are shown respectively.

[0108] As described above, the shooting performed by camera module 3 is carried out during display off periods, such as between frames and between periods of the horizon line, but Figures 7A to 7C The light emission is used during a portion of the display process for foreign object detection.

[0109] The electronic device 1 of the second embodiment includes a... Figure 4A and Figure 4B Same internal structure, but with a portion of the same Figure 5 Different processing actions.

[0110] Figure 8 This is a flowchart illustrating the processing operations of the electronic device 1 according to the second embodiment. Hereinafter, it will be discussed in conjunction with... Figure 5 The different processing actions are explained in detail. This is done when the user has not activated camera module 3, and... Figure 5 The steps S1 and S2 are the same (steps S11 and S12).

[0111] When the user activates camera module 3, such as Figure 7A As shown, a portion of the display area 20 within the viewing angle range of the camera module 3 emits light at the first emission wavelength (step S13). Depending on the focal length of the lens provided by the camera module 3, the viewing angle range varies; therefore, the size of the display area 20 emitting light at the first emission wavelength also needs to be adjusted accordingly to the focal length of the lens. Alternatively, the entire area of ​​the display surface 1a can also emit light at the first emission wavelength.

[0112] During the period when the light is emitted in step S13, exposure is performed based on the camera unit 4 (step S14), and various signal processing is performed by the signal processing unit 32 to generate camera image data (step S15).

[0113] Next, as Figure 7B The same actions as steps S13 to S15 (steps S16 to S18) are performed while the display area 20 is emitting light at the second emission wavelength. Next, as shown... Figure 7C The same actions as steps S13 to S15 (steps S19 to S21) are performed when the display area 20 emits light at the third emission wavelength.

[0114] The anomaly detection unit 35 detects the presence or absence of anomaly 2p based on the three camera image data obtained in steps S15, S18, and S21 (steps S22 to S24).

[0115] Thus, in the second embodiment, imaging based on the camera unit 4 is performed while at least a portion of the display area 20 of the display unit 2 emits light at multiple emission wavelengths, and anomaly 2p is detected by the anomaly detection unit 35 based on the acquired multiple image data. Depending on the type of anomaly 2p, only light of a specific emission wavelength is reflected or transmitted. Therefore, by performing imaging while the display unit 2 emits light at multiple emission wavelengths, it is easy to determine the presence or absence of anomaly 2p and the type of anomaly 2p.

[0116] (Third Implementation)

[0117] In the third embodiment, when the anomaly detection unit 35 detects an anomaly 2p, the camera image data is corrected.

[0118] Figure 9 This is a block diagram showing the internal structure of the electronic device 1 according to the third embodiment. Figure 9 1 pair of electronic devices Figure 4A The electronic device 1 is supplemented with a correction processing unit 37. The correction processing unit 37 corrects the image displayed on the display unit 2 to make the abnormality 2p less noticeable. More specifically, the correction processing unit 37 performs correction processing on the camera image data output from the signal processing unit 32 based on the information of the abnormality 2p detected by the abnormality detection unit 35, and outputs the corrected image data. The image data output from the correction processing unit 37 is sent to the output unit 36. The correction processing performed by the correction processing unit 37 may also include at least one of, for example, edge enhancement processing, deformation correction processing, and defect correction processing.

[0119] The anomaly detection unit 35 in the electronic device 1 of the third embodiment can also detect anomalies 2p in the display surface 1a by emitting light at multiple emission wavelengths, similar to the second embodiment. The anomaly detection unit 35 of this embodiment can detect the type of anomaly 2p, and its absorption and scattering characteristics change for specific emission wavelengths depending on the type of anomaly 2p. The anomaly detection unit 35 can accurately estimate the type of anomaly 2p based on the correspondence between emission wavelength and absorption and scattering characteristics. If the anomaly detection unit 35 can detect the type of anomaly 2p, the correction processing unit 37 can perform appropriate correction processing according to the type of anomaly 2p.

[0120] Figure 10 This is a flowchart showing the processing sequence of the correction processing unit 37. Figure 10 The correction process is in Figure 5 Step S5 or Figure 8 In step S22, it is determined that if there is an abnormality in display surface 1a, it will be replaced. Figure 5 Steps S7-S8 or Figure 8The steps S24 to S25 are performed.

[0121] First, the type of anomaly 2p is determined based on the information of anomaly 2p detected by anomaly detection unit 35 (step S31).

[0122] Next, it is determined whether effective calibration processing can be performed using the calibration processing unit 37 within the electronic device 1 (step S32). For example, if the electronic device 1 is a portable device such as a smartphone, its hardware performance is inferior compared to a PC or server, making it difficult to perform high-level calibration processing. Therefore, if calibration processing to the user's satisfaction level cannot be performed within a few seconds, it is determined that effective calibration processing cannot be performed. The determination process in step S32 is performed, for example, by the calibration processing unit 37.

[0123] When it is determined in step S32 that effective correction processing can be performed, the correction processing unit 37 performs correction processing corresponding to the type of abnormality 2p (step S33). For example, when the abnormality detection unit 35 detects that abnormality 2p is hand oil adhering to the display surface 1a, there is a tendency for the edges of the image captured through the hand oil to be blurred, so edge enhancement processing is performed. In this case, edge enhancement processing can be performed inside the correction processing unit 37, or the correction processing unit 37 can instruct the edge enhancement unit 32h in the signal processing unit 32 to enhance the edge enhancement processing. In addition, when the abnormality detection unit 35 detects that abnormality 2p is a water droplet adhering to the display surface 1a, the subject light is refracted by the water droplet, so distorted image data is obtained. Therefore, the correction processing unit 37 can perform distortion correction processing. In addition, when the abnormality detection unit 35 detects that abnormality 2p is a defect on the display surface 1a, the correction processing unit 37 can perform defect removal correction processing. The corrected image data is sent to the output unit 36 ​​(step S34).

[0124] On the other hand, when it is determined in step S32 that effective correction processing cannot be performed, the camera image data output from the signal processing unit 32 and the information of the anomaly 2p detected by the anomaly detection unit 35 are sent to a specific information processing device for advanced correction processing (step S35). The information processing device can be, for example, a server device connected to a network or a high-performance PC. Afterwards, the correction processing performs correction processing on the camera image data via the information processing device, and the correction processing unit 37 receives the corrected image data (step S36). The received corrected image data is sent to the output unit 36.

[0125] Figure 11 This is a block diagram illustrating an example of the connection method between electronic device 1 and information processing device 41. Figure 11In this example, electronic device 1 and information processing device 41 exchange information via network 42. Information processing device 41 can be located in a location that is not specific to electronic device 1, i.e., a cloud environment, or it can be located in a location relatively close to electronic device 1, such as a base station, like MEC (Mobile Edge Computing).

[0126] Information processing device 41 is connected to database device 43, which stores information about the correspondence between the types of anomalies 2p and the correction processing, as well as information about the model for correction processing based on the types of anomalies 2p. Information processing device 41 learns the model, taking the types of anomalies 2p and the camera image data as input data, and outputs camera image data that has undergone appropriate correction processing. During the learning process, the model is updated by changing the weight information of the neural network in various ways. By storing the types of various anomalies 2p and the results of correction processing in database device 43 and repeatedly learning the model, information processing device 41 can generate a highly reliable model. By learning this model in advance, highly reliable correction processing can be performed on the types of anomalies 2p and the camera image data sent from electronic device 1 in a short time, and the corrected image data can be returned to electronic device 1.

[0127] In addition, Figure 10 The flowchart illustrates an example of using an external information processing device 41 to perform correction processing when it is determined that effective correction processing cannot be performed inside the electronic device 1. However, if the external information processing device 41 is not available, or if communication with the external information processing device 41 is not possible for some reason, the situation that effective correction processing cannot be performed can also be displayed on the display unit 2.

[0128] Thus, in the third embodiment, when the anomaly detection unit 35 detects anomaly 2p, it performs correction processing on the camera image data to make the anomaly 2p less noticeable. Therefore, even if the user does not perform the operation of eliminating the anomaly 2p on the display surface 1a, high-quality camera image data can be obtained.

[0129] (Fourth Implementation)

[0130] In the fourth embodiment, a plurality of camera modules 3 are arranged on the side opposite to the display surface 1a of the display unit 2.

[0131] Figure 12 This is a top view of the electronic device 1 according to the fourth embodiment. Figure 12The electronic device 1 has three camera modules 3 arranged on the side opposite to the display surface 1a of the display unit 2. Furthermore, the number of camera modules 3 is not limited to three. The focal lengths of each camera module 3 can be the same or different.

[0132] Figure 13 yes Figure 12 A cross-sectional view along line AA. The shooting range of each camera module 3 is indicated by dashed lines. In this embodiment, at least two camera modules 3 are designed to capture images of any location on the display surface 1a of the display unit 2. Thus, multiple camera modules 3 can be used to capture images of any anomaly 2p generated on the display surface 1a.

[0133] By capturing images of the abnormal areas on the display surface 1a from different directions, the type of abnormality 2p can be easily determined. The abnormality detection unit 35 detects the type of abnormality 2p by referring to multiple image data captured by multiple camera modules 3. At this time, similar to the second embodiment, images can also be captured while the display surface 1a emits light at multiple different wavelengths.

[0134] The correction processing unit 37 performs correction processing to remove anomalies 2p based on multiple camera image data captured by the multiple camera modules 3. Compared to correction processing based on camera image data captured from one direction, correction processing based on multiple camera image data captured from multiple directions allows for more effective correction. Furthermore, if it is determined that the correction processing unit 37 cannot perform effective correction processing, the multiple camera image data captured by the multiple camera modules 3 can be sent to an external information processing device 41, where the information processing device 41 performs the correction processing.

[0135] Thus, in the fourth embodiment, by setting multiple camera modules 3, at least two camera modules 3 can be used to capture images of any location of the abnormality 2p on the display surface 1a. Therefore, it is possible to detect the presence or absence of the abnormality 2p, the type of the abnormality 2p, and perform correction processing to eliminate the abnormality 2p with high precision.

[0136] (Fifth Implementation)

[0137] The fifth embodiment relates to a specific detection method for anomaly 2p, which can be applied to the anomaly detection unit 35 of the first to fourth embodiments.

[0138] Figure 14 This is a top view showing an example of the pixel arrangement of the camera unit 4 in the fifth embodiment. Figure 14In this example, RGB pixels are arranged in a two-dimensional direction. Each pixel has a photoelectric conversion unit 4a that converts incident light into an electrical signal corresponding to the amount of light. A portion of the pixels are used for phase difference detection, and the pixel region is divided into two. Each segmented pixel 10a, 10b used for phase difference detection performs photoelectric conversion independently. The difference between the electrical signals converted by the two segmented pixels 10a, 10b is phase difference information, and the amount of defocus can be detected using this phase difference information. This amount of defocus can be used for focus adjustment. Figure 14 The example shows pixels for detecting phase differences arranged at the four corners, but the extent to which the pixels for detecting phase differences are arranged and where they are located is arbitrary.

[0139] The presence or absence of anomaly 2p can be detected using the amount of defocus obtained by the two segmented pixels 10a and 10b constituting the pixel used for phase difference detection. That is, when there is a foreign object on the display surface 1a, the pixel used for phase difference detection identifies the foreign object as part of the subject and detects the amount of defocus. The foreign object can be detected using the detected amount of defocus. For example, in situations such as... Figure 12 In the case of multiple camera modules 3 as shown, at least two of each camera module 3 detect the amount of defocus caused by foreign objects, so that the detected amount of defocus can be determined to be caused by foreign objects on the display surface 1a, rather than by the originally expected subject.

[0140] Thus, in the fifth embodiment, by setting a pixel for detecting phase difference in a portion of the pixel arrangement used for imaging, it is possible to detect anomalies 2p on the display surface 1a with high precision.

[0141] (Sixth Implementation Method)

[0142] The sixth embodiment detects anomalies 2p on the display surface 1a using a method different from that of the fifth embodiment. The anomaly 2p detection processing of the sixth embodiment can be applied to the anomaly detection units 35 of the first to fourth embodiments.

[0143] Figure 15 This is a top view showing an example of the pixel arrangement of the camera unit 4 in the sixth embodiment. Figure 15 In the example, with Figure 14 Similarly, RGB three-color pixels are arranged in a two-dimensional direction. Each pixel has a photoelectric conversion unit 4a. In some pixels, a polarization element 8b is arranged on the light incident surface side of the photoelectric conversion unit 4a. Hereinafter, pixels with polarization element 8b are referred to as polarized pixels 10c, and pixels without polarization element 8b are referred to as unpolarized pixels 10d.

[0144] The polarizing element 8b polarizes the light incident through the display unit 2. The light polarized by the polarizing element 8b is incident on the corresponding photoelectric conversion unit 4a and is photoelectrically converted.

[0145] Figure 15 This example shows an 8×8=64 pixels, where 60 are unpolarized pixels 10d and 4 are polarized pixels 10c. However, the ratio of unpolarized pixels 10d to polarized pixels 10c and the location of polarized pixels 10c are arbitrary.

[0146] Figure 16 This is a cross-sectional view of the camera unit 4 according to the sixth embodiment. Figure 16 In China, for the sake of Figure 3 Shared parts are labeled with the same reference numerals, and the following description focuses on the differences. Figure 16 In the imaging unit 4, a light-shielding layer 18 and a substrate insulating layer 19 are disposed on the planarization layer 13, and a plurality of polarization elements 8b are disposed at intervals on the substrate insulating layer 19. The polarization elements 8b are, for example, wire grid polarization elements 8b with a line and gap structure disposed on a part of the substrate insulating layer 19.

[0147] Figure 17 This is a perspective view showing an example of the detailed structure of each polarization element 8b. For example... Figure 17 As shown, the multiple polarizing elements 8b each have multiple convex lines 8d extending in one direction and gaps 8e between the lines 8d. The polarizing elements 8b can have multiple types with different extension directions of the lines 8d. More specifically, the polarizing elements 8b can have three or more types; for example, the angle between the arrangement direction of the photoelectric conversion section 4a and the extension direction of the lines 8d can be 0 degrees, 60 degrees, or 120 degrees. Alternatively, the angle between the arrangement direction of the photoelectric conversion section 4a and the extension direction of the lines 8d can be 0 degrees, 45 degrees, 90 degrees, or 135 degrees, or other angles. Alternatively, the multiple polarizing elements 8b can also polarize in only one direction. The material of the multiple polarizing elements 8b can be a metal such as aluminum or tungsten, or an organic photoelectric conversion film.

[0148] like Figure 15 As shown, when the camera unit 4 is equipped with four types of polarization elements 8b, each polarization element 8b allows light in an independent polarization state to pass through, and the corresponding photoelectric conversion unit 4a performs photoelectric conversion. When there is an abnormality 2p on the display surface 1a, the type of light passing through the four polarization elements 8b changes depending on the type of the abnormality 2p. Therefore, by analyzing the electrical signals photoelectrically converted by the four polarization pixels 10c, the type of abnormality 2p can be determined.

[0149] In addition, the polarization pixel 10c can also be used to extract and remove components of flare and diffracted light incident on the display surface 1a.

[0150] Figure 18 This is a block diagram showing the internal structure of the signal processing unit 32A within an electronic device 1, which has the function of suppressing the effects of flare light and diffraction light. The structure of the electronic device 1 other than the signal processing unit 32A is similar to... Figure 4A same.

[0151] exist Figure 18 In China, for the sake of Figure 4B The common parts are labeled with the same reference numerals, and the following description focuses on the differences. Except... Figure 4B In addition to the internal structure of the signal processing unit 32, Figure 18 The signal processing unit 32A also includes a polarization output unit 32j, a flare extraction unit 32k, and a flare correction signal generation unit 32m.

[0152] The polarization output unit 32j outputs polarization information data. The flare extraction unit 32k extracts at least one of the flare component and the diffraction component from the polarization information data. At least one of the flare component and the diffraction component extracted by the flare extraction unit 32k is a correction amount. The flare correction signal generation unit 32m performs subtraction processing on the digital pixel data output from the color output unit 32b using the correction amount extracted by the flare extraction unit 32k, thereby correcting the digital pixel data. The output data of the flare correction signal generation unit 32m is digital pixel data in which at least one of the flare component and the diffraction component has been removed. In this way, the flare correction signal generation unit 32m functions as a correction unit that corrects the photographic image converted by multiple unpolarized pixels 10d based on polarization information.

[0153] The signal level of the digital pixel data at the pixel position of polarization pixel 10c is reduced by the amount that passes through polarization element 8b. Therefore, the defect correction unit 32c treats polarization pixel 10c as a defect and performs a prescribed defect correction process.

[0154] like Figures 15-18 As shown, by acquiring polarization information using multiple polarization elements 8b, it is possible to generate a photographic image in a simple and reliable manner by removing the flare and diffraction components included in the light incident on multiple unpolarized pixels 10d after repeated reflections within the display unit 2.

[0155] Thus, in the sixth embodiment, a polarized pixel 10c using a polarizing element 8b is provided in a portion of the pixel arrangement used for imaging, thereby enabling the polarized pixel 10c to detect anomalies 2p on the display surface 1a. Furthermore, the polarized pixel 10c can be used to extract and remove flares and diffracted light incident on the display surface 1a.

[0156] (Seventh Implementation)

[0157] The seventh embodiment uses a sensor that optically detects the distance to the subject to detect anomalies 2p on the display surface 1a.

[0158] Figure 19 This is a block diagram showing the general configuration of the electronic device 1 according to the seventh embodiment. Besides... Figure 4A In addition to the components of electronic device 1, Figure 19 The electronic device 1 also includes a distance measuring unit 27. The distance measuring unit 27 has a light-projecting unit 27a and a light-receiving unit 27b. The light-projecting unit 27a intermittently emits pulsed light of a predetermined wavelength, and the light-receiving unit 27b receives the light emitted by the light-projecting unit 27a after reflection from an object. The distance measuring unit 27 measures the distance to the object based on the time difference between the emission of light from the light-projecting unit 27a and the reception of light by the light-receiving unit 27b. The light-projecting unit 27a can scan the direction of light projection within a predetermined angular range. Therefore, light from the light-projecting unit 27a can be emitted over the entire area of ​​the display surface 1a.

[0159] If there is an abnormality 2p on the display surface 1a of the display unit 2, and if the abnormality 2p is a foreign object, light from the projection unit 27a will be reflected. Therefore, the distance from the distance measuring unit 27 to the display surface 1a is known in advance, so the abnormality detection unit 35 can accurately detect whether the object is a foreign object on the display surface 1a by using the distance measured by the distance measuring unit 27.

[0160] Thus, in the seventh embodiment, a distance measuring unit 27 is provided to measure the distance to the object in an optical manner, so that it is possible to easily and accurately detect whether there is an abnormality 2p on the display surface 1a.

[0161] Alternatively, multiple cameras can be set up to take pictures from different angles, and the distance to the object can be measured based on the images taken by these multiple cameras, instead of measuring the distance to the object by the time from when the light is projected from the light-projecting part 27a to when the light is received by the light-receiving part 27b.

[0162] (Eighth Implementation Method)

[0163] In the first to seventh embodiments described above, an example is shown where the camera unit 4 within the electronic device 1 is used for photographing a subject. However, the camera unit 4 can be used for biometric authentication purposes such as fingerprint detection, iris detection, retinal detection, vein detection, and biometric shape detection. The appropriate focal length varies depending on the biometric object being detected. Therefore, by setting a lens with a focal length corresponding to the object being detected and optimizing the prism, light source, etc., the electronic device 1 of the first to seventh embodiments can be applied to the detection of various biometric information.

[0164] (Ninth Implementation)

[0165] In the electronic device 1 of the ninth embodiment, the optical system 5 of the camera module 3 is different from that of the first to eighth embodiments.

[0166] Figure 20 This is a diagram showing the cross-sectional structure of the camera unit 4 of the camera module 3 mounted on the electronic device 1 according to the ninth embodiment. Figure 20 The camera unit 4 is not a single lens or a lens group in which a single lens is arranged in the optical axis direction, but has a microlens array 64.

[0167] In more detail, Figure 20 The camera unit 4 includes: a photoelectric conversion unit 4a disposed along the bottom surface of the housing 63; a microlens array 64 disposed above the photoelectric conversion unit 4a; a plurality of light shields 66 disposed between adjacent microlenses 65; and a light guide plate 67 disposed above the microlens array 64. Figure 20 The camera unit 4 can be applied to any one of the first to eighth embodiments described above.

[0168] (Tenth Implementation)

[0169] Various electronic devices can be considered as specific candidates for the electronic device 1 having the configuration described in the first to ninth embodiments above. For example, Figure 21 This is a top view of the case where the electronic device 1 of the first to ninth embodiments is applied to the capsule endoscope 50. Figure 21 The capsule endoscope 50, for example, includes within a housing 51 that is hemispherical at both ends and cylindrical in the center: a camera (miniature camera) 52 for capturing images inside the body cavity; a memory 53 for recording image data captured by the camera 52; and a wireless transmitter 55 for transmitting the recorded image data to the outside via an antenna 54 after the capsule endoscope 50 is ejected from the body of the subject.

[0170] Additionally, a CPU (Central Processing Unit) 56 and a coil (magnetic force / current conversion coil) 57 are disposed within the housing 51. The CPU 56 controls the shooting operations performed by the camera 52 and the data storage operations to the memory 53, and controls the transmission of data from the memory 53 to a data receiving device (not shown) outside the housing 51 via the wireless transmitter 55. The coil 57 provides power to the camera 52, memory 53, wireless transmitter 55, antenna 54, and light source 52b (described later).

[0171] Furthermore, a magnetic (wire) switch 58 is provided in the housing 51 for detecting when the capsule endoscope 50 is placed on the data receiving device. The CPU 56 detects the placement of the wire switch 58 on the data receiving device and supplies power from the coil 57 to the wireless transmitter 55 when data can be transmitted.

[0172] The camera 52 includes, for example, an imaging element 52a and multiple light sources 52b for illuminating the body cavity. The imaging element 52a includes an objective lens optical system for capturing images of the body cavity. Specifically, the light sources 52b of the camera 52 are, for example, composed of a CMOS (Complementary Metal Oxide Semiconductor) sensor equipped with an LED (Light Emitting Diode), a CCD (Charge Coupled Device), or the like.

[0173] The display unit 2 of the electronic device 1 in the first to ninth embodiments includes, for example: Figure 21 The concept of a light source like 52b. Figure 21 The capsule endoscope 50 may have two light sources 52b, but these light sources 52b can be configured using a display panel with multiple light source units and an LED module with multiple LEDs. In this case, by arranging the camera unit 4 of the camera 52 below the display panel and the LED module, the constraints related to the layout and configuration of the camera 52 are reduced, and a smaller capsule endoscope 50 can be realized.

[0174] in addition, Figure 22This is a rear view of an electronic device 1 according to the first to ninth embodiments applied to a digital SLR camera 60. The digital SLR camera 60, a compact camera, has a display unit 2 on the rear side opposite to the lens, for displaying a preview image. It is possible to configure the camera module 3 to be positioned on the side opposite to the display surface of the display unit 2, thereby displaying an image of the subject's face on the display surface 1a of the display unit 2. In the electronic device 1 of the first to ninth embodiments, the camera module 3 can be positioned in an area overlapping with the display unit 2; therefore, the camera module 3 does not need to be placed in the bezel portion of the display unit 2, allowing the size of the display unit 2 to be maximized.

[0175] Figure 23A This is a top view showing an example of applying the electronic device 1 of the first to ninth embodiments to a head-mounted display (hereinafter HMD) 61. Figure 23A The HMD61 is used for VR (Virtual Reality), AR (Augmented Reality), MR (Mixed Reality), or SR (Substitutive Reality), etc. For example... Figure 23B As shown, current HMDs have cameras 62 mounted on their outer surface. HMD wearers can visually recognize the surrounding images, but on the other hand, there is a problem that people around them cannot see the HMD wearer's eyes or facial expressions.

[0176] Therefore, in Figure 23A In this design, a display surface of the display unit 2 is provided on the outer surface of the HMD61, and a camera module 3 is provided on the side opposite to the display surface of the display unit 2. This allows the display surface of the display unit 2 to display the facial expressions of the wearer captured by the camera module 3, enabling people around the wearer to monitor the wearer's facial expressions and eye movements in real time.

[0177] exist Figure 23A In this case, the camera module 3 is provided on the back side of the display unit 2, so the placement of the camera module 3 is not restricted, which can increase the freedom of the HMD61's appearance design. In addition, the camera can be positioned in the optimal position, thus preventing defects such as inconsistent viewing angles for the wearer on the display surface.

[0178] Thus, in the tenth embodiment, the electronic device 1 of the first to ninth embodiments can be used for various purposes, thereby increasing its utilization value.

[0179] Furthermore, this technology can be configured as described below.

[0180] (1) An electronic device comprising: a display unit; a camera unit disposed on a side opposite to the display surface of the display unit; an anomaly detection unit for detecting anomalies on the display surface; and a display control unit for highlighting the location of the anomaly detected by the anomaly detection unit on the display unit.

[0181] (2) In the electronic device described in (1), the display control unit causes the display unit to display information reminding the removal of the abnormality.

[0182] (3) In the electronic device described in (2), the information includes information corresponding to the type of the anomaly.

[0183] (4) Among the electronic devices described in any of (1) to (3),

[0184] The display control unit causes the display unit to display a flag indicating the location where the abnormality occurred.

[0185] (5) An electronic device comprising: a display unit capable of emitting light at multiple different light emission wavelengths; a camera unit disposed on a side opposite to the display surface of the display unit; and an anomaly detection unit that detects an anomaly on the display surface based on multiple images captured by the camera unit in a state in which at least a portion of the display surface is emitting light at each of the multiple light emission wavelengths.

[0186] (6) In the electronic device described in (5), the anomaly detection unit detects anomalies on the display surface based on multiple images captured by the camera unit in a state in which the area on the display surface that overlaps with the viewing angle of the camera unit emits light at each of the multiple emission wavelengths.

[0187] (7) An electronic device comprising: a display unit; a camera unit disposed on a side opposite to the display surface of the display unit; an anomaly detection unit for detecting an anomaly on the display surface; and a correction processing unit for correcting an image captured by the camera unit based on the anomaly.

[0188] (8) In the electronic device described in (7), the correction processing unit corrects the image captured by the camera unit based on at least one of the types, colors, sizes, locations, and quantities of the anomalies.

[0189] (9) In the electronic device described in (7) or (8), the electronic device includes an anomaly discrimination unit that determines the type of the anomaly, and the correction processing unit performs correction processing on the image captured by the camera unit in accordance with the type of the anomaly determined by the anomaly discrimination unit.

[0190] (10) In the electronic device described in (9), the correction process includes at least one of edge enhancement processing, deformation correction processing and defect correction processing.

[0191] (11) In any one of (7) to (10) the electronic device, the electronic device includes a model generation unit that generates a model for correcting an image captured by the camera unit based on information about the anomaly. The model is obtained by learning from at least one of the types, colors, sizes, locations, and quantities of the anomaly and images before and after correction by the correction processing unit. The correction processing unit provides the learned model with images captured by the camera unit and information about the anomaly to correct the images captured by the camera unit.

[0192] (12) In any one of (7) to (11) the electronic device, the electronic device comprises: a correction determination unit that determines whether the correction performed by the correction processing unit is effective; an image communication unit that, when it is determined that the correction performed by the correction processing unit is ineffective, sends the image captured by the camera unit and the abnormal information to an information processing device, and receives the image corrected by the information processing device; and an output unit that outputs the image corrected by the information processing device.

[0193] (13) In any of the electronic devices described in (7) to (12), the camera unit has multiple cameras that each take pictures from different angles, and the correction processing unit removes the anomaly from the images taken by the camera unit based on the multiple images taken by the multiple cameras.

[0194] (14) In any of the electronic devices described in (1) to (13), the camera unit has a plurality of photoelectric conversion units that photoelectrically convert light incident via the display unit, at least one of the plurality of photoelectric conversion units is capable of detecting phase difference information, and the anomaly detection unit detects the anomaly based on the phase difference information.

[0195] (15) In any of the electronic devices described in (1) to (13), the camera unit has: a plurality of photoelectric conversion units that perform photoelectric conversion on light incident via the display unit; and a plurality of polarization elements disposed on the light incident side of at least one of the plurality of photoelectric conversion units, wherein the anomaly detection unit detects the anomaly based on polarization information polarized by the plurality of polarization elements and photoelectrically converted by the photoelectric conversion units.

[0196] (16) In the electronic device described in (15), the plurality of polarization elements include a variety of polarization elements that detect different polarization states respectively.

[0197] (17) In any of the electronic devices described in (1) to (16), the camera unit has: a plurality of photoelectric conversion units that perform photoelectric conversion on light incident via the display unit; and a microlens array that images the subject onto the plurality of photoelectric conversion units.

[0198] (18) In any one of (1) to (13) the electronic device includes a distance detection unit that detects the distance up to the subject captured by the camera unit, and the anomaly detection unit detects the anomaly based on the distance detected by the distance detection unit.

[0199] (19) In the electronic device described in (18), the camera unit has multiple cameras that each take pictures from different angles, and the distance detection unit detects the distance based on the pictures taken by the multiple cameras.

[0200] (20) In any of the electronic devices described in (1) to (19), the electronic device includes a fingerprint detection unit that detects the fingerprint of a finger in contact with the display surface based on an image captured by the camera unit.

[0201] The methods disclosed herein are not limited to the various embodiments described above, but also include various modifications that can be conceived by those skilled in the art, and the effects of this disclosure are not limited to the above content. That is, various additions, changes, and partial deletions can be made without departing from the conceptual idea and spirit of this disclosure derived from the content specified in the claims and their equivalents.

Claims

1. An electronic device, comprising: Possessing: a display section; a camera section configured on a side opposite to a display surface of the display section; an abnormality detection section that detects an abnormality on the display surface; and a display control section that emphasizes display on the display section of a position at which the abnormality detected by the abnormality detection section has occurred, the camera section has: a plurality of photoelectric conversion sections that photoelectrically convert light that has entered via the display section; and a plurality of polarization elements configured on a light incident side of at least one of the plurality of photoelectric conversion sections, the abnormality detection section detects the abnormality based on polarization information of light that has been polarized by the plurality of polarization elements and photoelectrically converted by the photoelectric conversion section, the plurality of polarization elements each have a plurality of linear sections of a convex shape that extend in one direction and a gap section between the linear sections, the plurality of polarization elements include a plurality of polarization elements that respectively detect different polarization states, the camera section further has: a microlens array that images object light on the plurality of photoelectric conversion sections; and a plurality of light shielding bodies configured between adjacent microlenses, a correction processing section that performs correction processing for removing the abnormality based on a plurality of camera image data captured by a plurality of the camera sections; a correction determination section that determines whether the correction processing section can perform effective correction processing; an image communication section that, in a case where it is determined that the correction processing section cannot perform effective correction processing, transmits an image captured by the camera section and information of the abnormality to an information processing device and receives an image corrected by the information processing device; and an output section that outputs the image corrected by the information processing device.

2. The electronic device according to claim 1, wherein the display control section causes the display section to display information that prompts removal of the abnormality.

3. The electronic device according to claim 2, wherein the information includes information corresponding to a kind of the abnormality.

4. The electronic device according to claim 1, wherein the display control section causes the display section to display a mark that indicates a position at which the abnormality has occurred.

5. The electronic device according to claim 1, wherein the electronic device possesses a fingerprint detection section that detects a fingerprint of a finger in contact with the display surface based on an image captured by the camera section. Possessing: a display section that can emit light at a plurality of emission wavelengths different from each other; 6. An electronic device, comprising: a camera section configured on a side opposite to a display surface of the display section; and an abnormality detection section that detects an abnormality on the display surface based on a plurality of images captured by the camera section in a state in which at least a part of the display surface is caused to emit light at each of the plurality of emission wavelengths, the camera section has: a plurality of photoelectric conversion sections that photoelectrically convert light that has entered via the display section; and a plurality of polarization elements configured on a light incident side of at least one of the plurality of photoelectric conversion sections, the abnormality detection section detects the abnormality based on polarization information of light that has been polarized by the plurality of polarization elements and photoelectrically converted by the photoelectric conversion section, ​ ​ ​ The plurality of polarizing elements each have a plurality of linear portions in a convex shape extending in one direction, and gap portions between the linear portions, The plurality of polarizing elements include a plurality of polarizing elements that respectively detect different polarization states, The imaging section further has: a microlens array that images an object light on the plurality of photoelectric conversion sections; and a plurality of light-shielding bodies disposed between adjacent microlenses, a correction processing section that performs correction processing for removing the abnormality based on a plurality of image data captured by the plurality of imaging sections; a correction determination section that determines whether the correction processing section can perform effective correction processing; an image communication section that, in a case where it is determined that the correction processing section cannot perform effective correction processing, transmits an image captured by the imaging section and information of the abnormality to an information processing device, and receives an image corrected by the information processing device; and an output section that outputs the image corrected by the information processing device.

7. The electronic device according to claim 6, wherein the abnormality detection section detects an abnormality on the display surface based on a plurality of images captured by the imaging section in a state where a region on the display surface overlapping a visual angle of the imaging section is caused to emit light in each of the plurality of light emission wavelengths.

8. The electronic device according to claim 1 or 6, wherein the correction processing section corrects an image captured by the imaging section based on at least one of information of a kind, a color, a size, a position, and a number of the abnormality.

9. The electronic device according to claim 8, wherein the electronic device includes an abnormality discrimination section that discriminates the kind of the abnormality, the correction processing section performs correction processing on an image captured by the imaging section corresponding to the kind of the abnormality discriminated by the abnormality discrimination section.

10. The electronic device according to claim 9, wherein the correction processing includes at least one of edge emphasis processing, distortion correction processing, and blemish correction processing.

11. The electronic device according to claim 1 or 6, wherein the electronic device includes a model generation section that generates a model for correcting an image captured by the imaging section based on information of the abnormality, the model being obtained based on at least one of information of a kind, a color, a size, a position, and a number of the abnormality and images before and after correction by the correction processing section is performed, the correction processing section provides the model after learning is completed with an image captured by the imaging section and information of the abnormality to correct the image captured by the imaging section.

12. The electronic device according to claim 1 or 6, wherein the imaging section includes a plurality of cameras that respectively perform imaging of different visual angles, the correction processing section removes the abnormality from an image captured by the imaging section based on a plurality of images captured by the plurality of cameras.

Images