Electronic device

Abstract

[Problem] To provide an electronic device wherein degradation in the picture quality of captured images and in the displaying quality of a display unit are prevented from occurring. [Solution] An electronic device according to the present invention comprises: a display unit; an image capture unit located on the opposite side to the display surface of the display unit; and a control unit that synchronizes the display timing of the display unit with the image capture timing of the image capture unit such that the image capture unit performs image captures at timings when the display unit performs no displaying.

Description

[Technical field]

[0001] The present disclosure relates to electronic devices. [Background technology]

[0002] Recent electronic devices such as smartphones, mobile phones, and PCs (Personal Computers) are equipped with cameras in the bezel of the display, making it easy to make video calls and take videos. Smartphones and mobile phones are often carried in a pocket or bag, so their external size needs to be as compact as possible. On the other hand, if the display screen size is small, the higher the display resolution, the smaller the displayed text size becomes, making it difficult to see. For this reason, studies are being conducted to reduce the width of the bezel around the display screen, thereby making it possible to make the display screen size as large as possible without increasing the external size of the electronic device.

[0003] However, since a camera or the like is often mounted on the bezel of an electronic device, the bezel width cannot be made smaller than the outer diameter size of the camera.

[0004] Furthermore, when the camera is placed on the bezel, for example when talking on a videophone, the eye line is often placed near the center of the display screen, which causes the eye line to deviate from the optical axis of the camera, resulting in an unnatural-looking captured image where the eye line is not aligned.

[0005] In order to avoid the above-mentioned problems, it has been proposed to place a camera module on the opposite side of the display unit from the display surface, and to use the camera to capture the subject light that passes through the display unit. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] U.S. Patent Publication 2018 / 0069060 Summary of the Invention [Problem to be solved by the invention]

[0007] However, if the camera module is placed on the opposite side to the display surface of the display unit, light that has passed through the display unit is incident on the camera module, which may affect the display brightness of the display unit and degrade the image quality of the captured image.

[0008] Another possible solution would be to reduce the pixel density of part of the display area of ​​the display unit that overlaps with the location of the camera module, but this would result in a partial degradation of the display quality of the display unit, resulting in an unnatural display.

[0009] One aspect of the present disclosure provides an electronic device that does not degrade the image quality of a captured image and the display quality of a display unit. [Means for solving the problem]

[0010] In order to solve the above problems, in one aspect of the present disclosure, a display unit and an imaging unit disposed on the opposite side to a display surface of the display unit; There is provided an electronic device including: a control unit that synchronizes a display timing of the display unit with an image capturing timing of the image capturing unit so that the image capturing unit captures an image when the display unit does not display an image.

[0011] The control unit may set a period during which the display unit does not perform display to be longer when the imaging unit is capturing an image, compared to a period during which the imaging unit is not capturing an image.

[0012] The control unit may, when the imaging unit is capturing an image, lower a frequency at which at least a part of pixels of the display unit is displayed compared to when the imaging unit is not capturing an image.

[0013] The control unit may set a frame rate of the display unit to be lower when the imaging unit is capturing an image, compared to when the imaging unit is not capturing an image.

[0014] The control unit may lower a display frequency of at least a part of horizontal pixel lines of the display unit when the imaging unit is capturing an image, compared to when the imaging unit is not capturing an image.

[0015] The control unit may, when the imaging unit is performing imaging, lower the horizontal pixel line display frequency of the portion including the display area that overlaps with the imaging unit when the display unit is viewed in a planar view from the normal direction of the display surface, compared to when the imaging unit is not performing imaging.

[0016] The control unit may set an exposure time of the imaging unit based on a period during which the display unit does not perform display.

[0017] The control unit may lower a display pixel density in at least a part of a display area of ​​the display unit when the imaging unit is capturing an image, compared to when the imaging unit is not capturing an image.

[0018] When the imaging unit is capturing an image, the control unit may lower the display pixel density within a portion of the display area, including a display area that overlaps with the imaging unit, when the display unit is viewed in a planar view from the normal direction of the display surface, compared to when the imaging unit is not capturing an image.

[0019] The control unit may increase a luminance value of pixels in at least a part of a display area of ​​the display unit when the imaging unit is capturing an image, compared to when the imaging unit is not capturing an image.

[0020] The control unit may, when the imaging unit is capturing an image, increase the brightness value of pixels in the portion of the display area including the display area that overlaps with the imaging unit when the display unit is viewed in a planar view from the normal direction of the display surface, compared to when the imaging unit is not capturing an image.

[0021] The control unit may, when the imaging unit is capturing an image, lower the frequency at which pixels in the portion of the display area are displayed and increase the brightness value of the pixels compared to when the imaging unit is not capturing an image.

[0022] the display unit has a first display surface and a second display surface which are stacked on each other and whose display is controlled separately; the first display surface displays a display area of ​​the display unit excluding the part of the display area; The second display surface may display the part of the display area so as not to overlap with the display area of ​​the first display surface when stacked on the first display surface.

[0023] A brightness detection unit is provided to detect the brightness of the surroundings, The control unit may control a frequency at which at least a part of pixels of the display unit is displayed when the imaging unit captures an image, in accordance with the surrounding brightness detected by the brightness detection unit.

[0024] The control unit may control a frequency at which at least a portion of the pixels of the display unit are displayed when the imaging unit captures an image, and a display pixel density, in accordance with the ambient brightness detected by the brightness detection unit.

[0025] A timing detection unit detects the timing of a blink of a person who directs his or her gaze to the display unit, The control unit may temporarily switch the display unit to a non-emitting state and perform imaging with the imaging unit in accordance with the timing detected by the timing detection unit.

[0026] The imaging section may perform photoelectric conversion of at least one of visible light and infrared light.

[0027] the display unit has an opening at a position overlapping with the imaging unit when the display unit is viewed in a plan view from a normal direction of the display surface, The opening may be disposed so as to pass through a layer, of a plurality of layers constituting the display unit, that has a transmittance for light in a predetermined wavelength band that is less than a predetermined value.

[0028] The display section may be made up of a plurality of layers having a transmittance of a predetermined value or more for light in a wavelength band capable of being photoelectrically converted by the imaging section.

[0029] The control unit is A first control unit that controls the display of the display unit; The image capturing apparatus may further include a second control unit that cooperates with the first control unit to control imaging of the imaging unit. [Brief description of the drawings]

[0030] [Figure 1A] 1 is a schematic cross-sectional view of an electronic device according to a first embodiment. [Figure 1B] FIG. 1B is a cross-sectional view of FIG. 1A with an opening. [Diagram 2] FIG. 2(a) is a schematic external view of the electronic device of FIG. 1A, and FIG. 2(b) is a cross-sectional view taken along line AA of FIG. 2(a). [Diagram 3] FIG. 2 is a cross-sectional view showing an example of a cross-sectional structure of an imaging unit 4. [Figure 4] FIG. 2 is a block diagram showing the internal configuration of the electronic device according to the present embodiment. [Diagram 5] 5 is a flowchart showing a processing operation of the electronic device according to the first embodiment. [Figure 6] FIG. 6 is an operation timing chart corresponding to the flowchart in FIG. 5 . [Figure 7] 10 is a flowchart showing a processing operation of an electronic device according to a second embodiment. [Figure 8] FIG. 8 is an operation timing chart corresponding to the flowchart in FIG. 7. [Figure 9] 10 is a flowchart showing a processing operation of an electronic device according to a third embodiment. [Figure 10] FIG. 10 is an operation timing chart corresponding to the flowchart in FIG. [Figure 11] 11 is a diagram showing an example of a screen display on the display unit 2 displayed by the processing of the flowchart in FIG. 10. [Figure 12] 10 is a flowchart showing a processing operation of an electronic device according to a fourth embodiment. [Figure 13] FIG. 13 is an operation timing chart corresponding to the flowchart of FIG. 12. [Figure 14A] FIG. 13 is a diagram illustrating a specific example of a second display area in the fourth embodiment. [Figure 14B] FIG. 13 is a diagram illustrating a specific example of a second display area in the fourth embodiment. [Figure 14C] FIG. 13 is a diagram illustrating a specific example of a second display area in the fourth embodiment. [Figure 15] 13 is a flowchart showing a processing operation of an electronic device according to a fifth embodiment. [Figure 16] FIG. 16 is an operation timing chart corresponding to the flowchart of FIG. [Figure 17A] FIG. 13 is a diagram illustrating a specific example of a second display area in the fifth embodiment. [Figure 17B] FIG. 13 is a diagram illustrating a specific example of a second display area in the fifth embodiment. [Figure 18] FIG. 13 is a block diagram showing the internal configuration of an electronic device according to a sixth embodiment. [Figure 19] 13 is a flowchart showing a processing operation of an electronic device according to a sixth embodiment. [Figure 20] FIG. 20 is an operation timing chart corresponding to the flowchart of FIG. 19 . [Figure 21] FIG. 13 is a block diagram showing the internal configuration of an electronic device according to a seventh embodiment. [Figure 22] 13 is a flowchart showing a processing operation of an electronic device according to a seventh embodiment. [Diagram 23] FIG. 23 is a diagram showing a cross-sectional structure of an imaging section of a camera module according to a ninth embodiment. [Figure 24] FIG. 23 is a diagram showing a cross-sectional structure of an imaging unit according to a tenth embodiment. [Diagram 25] FIG. 4 is a perspective view showing an example of a detailed structure of each polarizing element 8b. [Figure 26] FIG. 1 is a block diagram showing the internal configuration of an electronic device having a function of suppressing the effects of flare light and diffracted light. [Figure 27] 1 is a plan view showing a case where the electronic device according to the first to fourth embodiments is applied to a capsule endoscope. [Figure 28] 1 is a rear view of a digital single-lens reflex camera to which the electronic device according to the first to fourth embodiments is applied. [Figure 29A]1 is a plan view showing an example in which the electronic device according to the first to fourth embodiments is applied to an HMD. [Figure 29B] A diagram showing the current HMD. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0031] Hereinafter, an embodiment of an electronic device will be described with reference to the drawings. The following description will focus on the main components of the electronic device, but the electronic device may include components and functions that are not shown or described. The following description does not exclude components and functions that are not shown or described.

[0032] (First embodiment) 1A and 1B are schematic cross-sectional views of an electronic device 1 according to a first embodiment. The electronic device 1 in FIG. 1A is any electronic device 1 having both a display function and an image capturing function, such as a smartphone, a mobile phone, a tablet, or a PC. The electronic device 1 in FIG. 1A includes a camera module (image capturing unit) disposed on the opposite side to the display surface 1a of the display unit 2. In this manner, the electronic device 1 in FIG. 1A includes a camera module 3 on the rear side of the display surface 1a of the display unit 2. Therefore, the camera module 3 captures images through the display unit 2.

[0033] FIG. 2(a) is a schematic external view of the electronic device 1 in FIG. 1A, and FIG. 2(b) is a cross-sectional view taken along line AA in FIG. 2(a). In the example of FIG. 2(a), the display surface 1a extends to nearly the outer size of the electronic device 1, and the width of the bezel 1b around the display surface 1a is set to a few mm or less. Usually, a front camera is often mounted on the bezel 1b, but in FIG. 2(a), as shown by the dashed line, a camera module 3 functioning as a front camera is disposed on the rear side of the display surface 1a at approximately the center. In this way, by providing the front camera on the rear side of the display surface 1a, it becomes unnecessary to dispose the front camera on the bezel 1b, and the width of the bezel 1b can be narrowed.

[0034] 2(a), the camera module 3 is disposed on the rear side of the display surface 1a at approximately the center thereof, but in this embodiment, the camera module 3 may be disposed on any rear side of the display surface 1a, for example, on the rear side near the periphery of the display surface 1a. In this manner, the camera module 3 in this embodiment is disposed at any position on the rear side that overlaps with the display surface 1a.

[0035] As shown in FIG. 1A, the display unit 2 is a laminate in which a protective film 2b, a polyimide substrate 2c, a display layer 2d, a barrier layer 2e, a touch sensor layer 2f, an adhesive layer 2g, a circular polarizer 2h, an optical adhesive sheet (OCA: Optical Clear Adhesive) 2i, and a cover glass 2j are laminated in this order. The display layer 2d may be, for example, an OLED (Organic Light Emitting Device) display layer, a liquid crystal display layer, a MicroLED, or a display layer based on other display principles. The display layer 2d may be composed of multiple layers. For example, the display layer 2d may include a color filter layer, a backlight layer, and the like. The display unit 2 performs display using light in the visible light wavelength range, but the light displayed on the display unit 2 may contain an infrared light component.

[0036] The barrier layer 2e is a layer that prevents oxygen and moisture from penetrating into the display layer 2d. A touch sensor is incorporated in the touch sensor layer 2f. There are various types of touch sensors, such as a capacitance type and a resistive film type, and any type may be adopted. In addition, the touch sensor layer 2f and the display layer 2d may be integrated.

[0037] The adhesive layer 2g is provided to bond the circular polarizer 2h and the touch sensor layer 2f. A material having a high visible light transmittance is used for the adhesive layer 2g. The circular polarizer 2h is provided to reduce glare and to improve the visibility of the display surface 1a even in a bright environment. The optical adhesive sheet 2i is provided to improve the adhesion between the circular polarizer 2h and the cover glass 2j. A material having a high visible light transmittance is used for the optical adhesive sheet 2i. The cover glass 2j is provided to protect the display layer 2d and the like. The layer structure of the display unit 2 is not necessarily limited to those shown in FIG. 1A or FIG. 2.

[0038] The camera module 3 is disposed on the opposite side of the display surface 1a of the display unit 2, i.e., on the back side of the display unit 2. The camera module 3 has an imaging unit 4 and an optical system 5. The optical system 5 is disposed on the light incident surface side of the imaging unit 4, i.e., on the side closer to the display unit 2, and focuses light that has passed through the display unit 2 onto the imaging unit 4. The optical system 5 is usually composed of multiple lenses. As will be described later, multiple camera modules 3 may be disposed on the opposite side of the display surface 1a of the display unit 2. In this case, the focal lengths of the optical systems 5 of the camera modules 3 may be different from each other, which allows shooting with different angles of view, such as telephoto and wide angle.

[0039] The imaging unit 4 has a photoelectric conversion unit 4a. The photoelectric conversion unit 4a photoelectrically converts light incident through the display unit 2. The photoelectric conversion unit 4a may be a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor. The photoelectric conversion unit 4a may also be a photodiode or an organic photoelectric conversion film.

[0040] The photoelectric conversion unit 4a has a photoelectric conversion element such as a CMOS sensor for each pixel. The pixels can be arranged in any manner. Specifically, the arrangement of the pixels may be a Bayer arrangement, an interline arrangement, a checkerboard arrangement, a stripe arrangement, or any other arrangement.

[0041] As shown in FIG. 1A and FIG. 2(b), the electronic device 1 of this embodiment is arranged so that the display unit 2 and the camera module 3 overlap in the front-back direction. Therefore, the camera module 3 captures the subject light transmitted through the display unit 2. As shown in FIG. 1A, the display unit 2 is formed of multiple layers, and there is no problem if each layer has a high transmittance of light in a wavelength band to which the imaging unit 4 has sufficient sensitivity, but in reality, some layers may have a low transmittance. For example, the polyimide substrate 2c does not have a very high visible light transmittance. Therefore, as shown in FIG. 1B, one or more openings 2k may be formed in a layer that does not have a very high transmittance among the multiple layers constituting the display unit 2, and the light that passes through the openings 2k may be captured by the camera module 3. In the example of FIG. 1B, among the multiple layers constituting the display unit 2, multiple openings 2k are provided that penetrate the layers other than the cover glass 2j. These openings 2k are provided at positions that overlap with the camera module 3 when the display unit 2 is viewed in a plan view. The diameter and number of each opening 2k are set to appropriate values, taking into consideration the display on the display unit 2 and the image quality of the image captured by the imaging unit 4.

[0042] When a highly transparent substrate is used instead of the polyimide substrate 2c, it is not always necessary to provide the opening 2k.

[0043] FIG. 3 is a cross-sectional view showing an example of the cross-sectional structure of the imaging unit 4. The imaging unit 4 in FIG. 3 includes a photoelectric conversion unit 4a formed in a semiconductor substrate 11, and the photoelectric conversion unit 4a is divided by an element isolation layer 12 for each pixel. A planarization layer 13 is disposed on the photoelectric conversion unit 4a, and a color filter layer 14 is disposed thereon. The color filter layer 14 may have filter layers of three colors, RGB, or may have filter layers of their complementary colors, cyan, magenta, and yellow. Alternatively, it may have a filter layer that transmits colors other than visible light, such as infrared light, a filter layer having multispectral characteristics, or a filter layer of subtractive colors such as white. By transmitting light other than visible light, such as infrared light, sensing information such as depth information can be detected. An on-chip lens 15 is disposed on the color filter layer 14. Light is incident through the on-chip lens 15. In this specification, the side on which the on-chip lens 15 is disposed is called the back side of the imaging unit 4.

[0044] On the front surface side of the imaging section 4, a readout circuit 16 is formed on a semiconductor substrate 11, and the periphery of the readout circuit 16 is covered with an interlayer insulating film 17. The readout circuit 16 has a transfer transistor, a reset transistor, an amplification transistor, a selection transistor, etc. Note that the cross-sectional structure of the imaging section 4 is not limited to that shown in FIG.

[0045] 4 is a block diagram showing the internal configuration of the electronic device 1 according to the present embodiment. As shown in FIG. 4, the electronic device 1 has an imaging device 21, an application processor (first control unit) 22, a resolution conversion unit 23, an A / D conversion unit 24, a display control unit 25, and a display unit 2.

[0046] The imaging device 21 can be composed of one or more semiconductor devices, and in addition to the imaging section 4 (photoelectric conversion section 4a), optical system 5, and IR (Infrared Ray) cut filter 6 that constitute the camera module 3, has an A / D conversion section 31, a signal processing section 32, an imaging control section (second control section) 33, and an exposure adjustment section 34.

[0047] The A / D conversion unit 31 converts analog pixel signals captured by the imaging unit 4 into digital pixel data. The signal processing unit 32 has a clamp unit 32a, a color output unit 32b, a defect correction unit 32c, a linear matrix unit 32d, a gamma correction unit 32e, a luminance chroma signal generation unit 32f, a noise reduction unit 32g, an edge enhancement unit 32h, and an output unit 32i.

[0048] The clamping unit 32a performs a process of defining the black level. More specifically, the clamping unit 32a performs a process of subtracting black level data from digital pixel data. The color output unit 32b outputs pixel data for each color, for example, RGB. The defect correction unit 32c performs a process of correcting imaging data of a specific pixel that cannot be read correctly for some reason, using imaging data of surrounding pixels. The linear matrix unit 32d performs a matrix operation on color information such as RGB to perform more accurate color reproduction. The gamma correction unit 32e performs gamma correction so as to enable display with excellent visibility in accordance with the display characteristics of the display unit 2. For example, the gamma correction unit 32e performs conversion from 10 bits to 8 bits while changing the gradient. The luminance chroma signal generation unit 32f generates a luminance chroma signal to be displayed on the display unit 2 based on the output data of the gamma correction unit 32e. The noise reduction unit 32g performs a process of reducing noise contained in the luminance chroma signal. The edge enhancement unit 32h performs a process of enhancing the edge of the subject image based on the luminance chroma signal. The noise reduction process by the noise reduction unit 32g and the edge emphasis process by the edge emphasis unit 32h may be performed only when a predetermined condition is satisfied. The output unit 32i outputs the luminance chrominance signal after the noise reduction process.

[0049] The imaging control unit 33 sets a frame rate at which imaging is performed by the imaging unit 4 based on the luminance chroma signal. The exposure adjustment unit 34 adjusts the exposure time in accordance with the frame rate set by the imaging control unit 33. The photoelectric conversion unit 4a performs imaging for each pixel in accordance with the exposure time adjusted by the exposure adjustment unit 34.

[0050] The application processor 22 is a semiconductor device separate from the camera module 3, and is mounted on the same or a different substrate as the camera module 3. The application processor 22 has a CPU (Central Processing Unit) and the like therein, and executes programs such as an operating system and various application software. The application processor 22 may be equipped with a function for performing image processing, signal processing, and the like, such as a GPU (Graphics Processing Unit) or a baseband processor. The application processor 22 executes various processes on input image data and calculation results as necessary, controls the display of an image on the display unit 2 of the electronic device 1, and transmits the image data and calculation results to an external cloud server via a predetermined network.

[0051] The application processor 22 transmits and receives various control signals to and from the imaging control unit 33. The application processor 22 also receives a luminance chroma signal generated by the luminance chroma signal generating unit 32f and supplies it to the display control unit 25. More specifically, the application processor 22 receives information on the frame rate at which the imaging unit 4 captures images via the imaging control unit 33, and sets the frame rate at which the display unit 2 displays images.

[0052] The application processor 22 and the imaging control unit 33 may be integrated into one chip. In this specification, the application processor 22 and the imaging control unit 33 may be collectively referred to as a control unit. The control unit in this embodiment synchronizes the display timing of the display unit 2 with the imaging timing of the imaging unit 4 so that the imaging unit 4 captures an image when the display unit 2 does not display an image.

[0053] The control unit extends the period during which the display unit 2 does not display when the imaging unit 4 captures an image, compared to when the imaging unit 4 does not capture an image. The control unit may lower the frequency at which at least some pixels of the display unit 2 are displayed when the imaging unit 4 captures an image, compared to when the imaging unit 4 does not capture an image. The control unit may lower the frame rate of the display unit 2 when the imaging unit 4 captures an image, compared to when the imaging unit 4 does not capture an image. The control unit may lower the display frequency of at least some horizontal pixel lines of the display unit 2 when the imaging unit 4 captures an image, compared to when the imaging unit 4 does not capture an image. The control unit may lower the display frequency of some horizontal pixel lines including a display area overlapping with the imaging unit 4 when the display unit 2 is viewed in plan from the normal direction of the display surface 1a, compared to when the imaging unit 4 does not capture an image. The control unit may set the exposure time of the imaging unit 4 based on the period during which the display unit 2 does not display an image. Moreover, the control unit may lower the display pixel density in at least a part of the display area of ​​the display unit 2 when the imaging unit 4 captures an image, compared to when the imaging unit 4 does not capture an image. Moreover, the control unit may lower the display pixel density in a part of the display area including the display area overlapping with the imaging unit 4 when the display unit 2 is viewed in plan from the normal direction of the display surface 1a, compared to when the imaging unit 4 does not capture an image, when the imaging unit 4 captures an image. Moreover, the control unit may increase the luminance value of the pixels in at least a part of the display area of ​​the display unit 2 when the imaging unit 4 captures an image, compared to when the imaging unit 4 does not capture an image. Moreover, the control unit may increase the luminance value of the part of the display area including the display area overlapping with the imaging unit 4 when the display unit 2 is viewed in plan from the normal direction of the display surface 1a, compared to when the imaging unit 4 does not capture an image, when the imaging unit 4 captures an image. Moreover, the control unit may lower the frequency at which at least a part of the pixels of the display unit 2 is displayed and increase the luminance value, compared to when the imaging unit 4 does not capture an image, when the imaging unit 4 captures an image.

[0054] The resolution conversion unit 23 converts the resolution of a video signal to be displayed on the display unit 2. The A / D conversion unit 31 converts the video signal, the resolution of which has been converted by the resolution conversion unit 23, into digital video data. The display control unit 25 controls the display of the digital video data on the display unit 2 at a frame rate according to an instruction from the application processor 22.

[0055] Fig. 5 is a flowchart showing the processing operation of the electronic device 1 according to the first embodiment, and shows, for example, the processing operation executed by the application processor 22 and the imaging control unit 33. The flowchart in Fig. 5 is continuously repeated while the power of the electronic device 1 is turned on.

[0056] First, it is determined whether or not the user of the electronic device 1 has activated the camera module 3 (step S1). If the camera module 3 has not been activated, a normal frame rate is selected (step S2). The normal frame rate is a predetermined frame rate, such as 60 Hz. The frame rate is the frequency required to display the entire display area of ​​the display unit 2 once. Next, display on the display unit 2 starts at the normal frame rate (step S3).

[0057] On the other hand, if it is determined in step S1 that the camera module 3 has been activated, a low frame rate is selected (step S4). The low frame rate is a frame rate that is lower than the normal frame rate, such as 30 Hz. In the example of FIG. 5, the low frame rate is set to a fixed frequency, but the frequency of the low frame rate may be made changeable. Next, display on the display unit 2 starts at the low frame rate (step S5).

[0058] When displaying on the display unit 2 at a constant frame rate, the length of the period during which the display unit 2 does not display (hereinafter referred to as the display off period) changes depending on the frequency of the frame rate. The faster the frame rate, the longer the display off period. The display off period is provided between the end of displaying one horizontal pixel line and the start of displaying the next horizontal pixel line, or between the end of displaying one frame and the start of displaying the next frame.

[0059] After the display is started in step S5, when the display-off period begins, exposure is performed by the imaging unit 4 during the display-off period (step S6). The exposure time by the imaging unit 4 is determined according to the length of the display-off period. The lower the frame rate, the longer the display-off period, so when displaying at a low frame rate, the exposure time can be made longer.

[0060] Fig. 6 is an operation timing diagram corresponding to the flowchart of Fig. 5. Waveform w1 in Fig. 6 is a signal waveform indicating the display period of the normal frame rate, and waveform w2 is a signal waveform indicating the display period of the low frame rate. The high periods of waveforms w1 and w2 indicate the display periods, and the low periods are the display off periods. Waveform w3 is a signal waveform indicating the exposure period, and the high periods are the exposure periods. As shown by waveforms w1 to w3, at the low frame rate, the display off period is longer than at the normal frame rate, so a sufficient exposure time can be ensured.

[0061] In the first embodiment, as shown in Fig. 5 and Fig. 6, the imaging unit 4 captures images during the display-off period, so there is no risk of the image quality of the captured image being degraded due to the influence of the display luminance of the display unit 2. In addition, since the display is displayed at a low frame rate when capturing images, the display-off period can be lengthened, so that a sufficiently long exposure time can be secured during capturing images, and the problem of the captured image becoming dark due to insufficient exposure time does not occur. In other words, the imaging sensitivity can be improved.

[0062] Second embodiment If the frame rate of the display unit 2 is lowered when the camera module 3 is started up as in the first embodiment, the human eye senses brightness at an averaged luminance, and the display is perceived as darker than normal. Therefore, in the second embodiment, the luminance value of the pixel data is increased when the camera module 3 is started up.

[0063] The electronic device 1 according to the second embodiment has the same internal configuration as that shown in FIG. 4. FIG. 7 is a flowchart showing the processing operation of the electronic device 1 according to the second embodiment. The following mainly describes the differences from the flowchart in FIG. 5. In FIG. 7, when the camera module 3 is not started (NO in step S11), a normal luminance value is set (step S13) before or after the process of selecting the normal frame rate (step S12). The normal luminance value means that the pixel data is displayed at its original luminance value. Next, pixel data at the set display luminance is supplied to the display unit 2 at the normal frame rate to start displaying (step S14).

[0064] On the other hand, when the camera module 3 is started (YES in step S11), a luminance value higher than normal is set (step S16) before or after the process of selecting a low frame rate (step S15). A luminance value higher than normal means that the luminance value of the pixel data is adjusted so that the human eye perceives the display luminance as being equivalent to the display luminance when the display unit 2 is displaying at the normal frame rate when the camera module 3 is not started. More specifically, in step S16, the luminance value of the pixel data is made larger than the luminance value set in step S13.

[0065] Next, display is started with the luminance value set in step S16 (step S17). By increasing the luminance value of the pixel data, even if the frame rate is lowered, the human eye perceives the display luminance as being equivalent to that when a normal frame is selected. Then, within the display-off period, imaging is performed by the imaging unit 4 with an exposure time according to the length of the display-off period (step S18).

[0066] Fig. 8 is an operation timing diagram corresponding to the flowchart of Fig. 7. Waveforms w4 to w6 are similar to waveforms w1 to w3. Waveform w7 is a waveform that typically indicates the luminance value of pixel data displayed on the display unit 2 when the camera module 3 is not activated. Waveform w8 is a waveform that typically indicates the luminance value of pixel data displayed on the display unit 2 when the camera module 3 is activated.

[0067] As can be seen from waveforms w7 and w8, when a low frame rate is selected, the display luminance perceived by the human eye can be made equivalent to the display luminance when a normal frame rate is selected by increasing the luminance value of the pixel data.

[0068] In this way, in the second embodiment, when the camera module 3 is started, not only is the display unit 2 set to a low frame rate, but the luminance value of the pixel data is also set to a value higher than when the normal frame rate is selected, so that the display of the display unit 2 does not appear dark when the camera module 3 is started, and a natural display can be performed. Also, as in the first embodiment, imaging is performed by the imaging unit 4 during the display off period, so imaging can be performed with sufficient exposure time without being affected by the display of the display unit 2.

[0069] (Third embodiment) In the third embodiment, not only the frame rate is lowered but also the display pixel density is controlled when the camera module 3 is started up. The electronic device 1 according to the third embodiment has an internal configuration similar to that shown in FIG.

[0070] 9 is a flowchart showing the processing operation of the electronic device 1 according to the third embodiment. The following will mainly explain the differences from FIG. 7. The processing in steps S21 to S24 is the same as steps S11 to S14 in FIG.

[0071] When the camera module 3 is started, a frame rate and a display pixel density are set (step S25). Here, a low frame rate lower than the normal frame rate is set. Also, the display pixel density is set to be lower than the display pixel density when the camera module 3 is not started. As a more specific example, when the camera module 3 is started, every other horizontal pixel line or every third horizontal pixel line is selected and driven to reduce the display pixel density. This makes it possible to lengthen the display off period between the display of one horizontal pixel line and the display of the next horizontal pixel line. Alternatively, a plurality of pixels on the horizontal pixel line may be thinned out and driven. In this case, the interval for driving each pixel on the horizontal pixel line is widened, so that the display off period can be set within that interval to perform imaging by the imaging unit 4.

[0072] Next, a luminance value of the pixel data is set based on the set frame rate and display pixel density (step S26). The higher the display pixel density, the higher the display luminance perceived by the human eye. Also, the lower the frame rate, the lower the display luminance perceived by the human eye. For this reason, in step S26, taking into consideration both the set frame rate and display pixel density, the luminance value of the pixel data is set to be equivalent to the display luminance of the display unit 2 when the camera module 3 is not activated.

[0073] Next, display is started with the set luminance value (step S27) Next, within the display-off period, imaging is performed by the imaging unit 4 with an exposure time according to the length of the display-off period (step S28).

[0074] Fig. 10 is an operation timing diagram corresponding to the flowchart of Fig. 9. Waveform w9 is the waveform of the vertical synchronizing signal VSYNC at the normal frame rate, waveform w10 is the waveform of the horizontal synchronizing signal HSYNC at the normal frame rate, and waveform w11 is a waveform that typically indicates the luminance value of pixel data displayed on the display unit 2 at the normal frame rate. The high period of the horizontal synchronizing signal HSYNC of waveform w10 indicates the horizontal pixel line display period. Waveform w12 is the waveform of the vertical synchronizing signal VSYNC at the low frame rate, waveform w13 is the waveform of the horizontal synchronizing signal HSYNC at the low frame rate, waveform w14 is the waveform of the exposure time during which the imaging unit 4 captures an image, and waveform w15 is a waveform that typically indicates the luminance value of pixel data displayed on the display unit 2 at the low frame rate.

[0075] As shown in FIG. 10, when the camera module 3 is started, a low frame rate lower than the normal frame rate is selected and the display pixel density is also reduced, so that the display off period can be made longer and the exposure time when the imaging unit 4 captures an image can be made longer.

[0076] Fig. 11 is a diagram showing an example of a screen display on the display unit 2 that is displayed by the processing of the flowchart in Fig. 10. In this embodiment, when the camera module 3 is started, the frame rate and display pixel density are set for each frame, so that the entire display area of ​​the display unit 2 is displayed at a display pixel density that is different from when the camera module 3 is not started, as shown in Fig. 11. However, when the camera module 3 is started, the luminance value of the pixel data is made larger than when the camera module 3 is not started, so that the display luminance perceived by the human eye can be made equivalent to when the camera module 3 is not started.

[0077] In this way, in the third embodiment, when the camera module 3 is started, the frame rate and the display pixel density are set on a frame-by-frame basis, so that imaging can be performed with a lowered frame rate and display pixel density for the entire display area of ​​the display unit 2. By lowering the frame rate and the display pixel density, the exposure time when capturing an image with the imaging unit 4 can be lengthened, and the imaging sensitivity can be improved.

[0078] (Fourth embodiment) The fourth embodiment controls the display pixel density for some horizontal pixel lines.

[0079] The electronic device 1 according to the fourth embodiment has the same internal configuration as that shown in Fig. 4. However, the display unit 2 in the electronic device 1 has a first display area 2r1 and a second display area 2r2. The first display area 2r1 is an area displayed at a normal frame rate and a normal display pixel density. The second display area 2r2 is an area displayed at a low frame rate and a lower-than-normal display pixel density.

[0080] The second display region 2r2 is a display region that includes a display region that overlaps with the camera module 3 when the display unit 2 is viewed in a plan view from the normal direction of the display surface 1a. In this embodiment, the second display region 2r2 has a strip-shaped area that is an integer multiple of the horizontal pixel lines of the display unit 2. At least a portion of the second display region 2r2 overlaps with the location of the imaging unit 4 in the front-back direction, and the display-off period of the second display region 2r2 can be set to be longer by lowering the display pixel density in the second display region 2r2.

[0081] FIG. 12 is a flowchart showing the processing operation of the electronic device 1 according to the fourth embodiment, and shows the processing operation executed by, for example, the application processor 22 and the imaging control unit 33. The following mainly describes the differences from the flowchart in FIG. 9. Steps S31 to S34 in FIG. 12 perform the same processing as steps S21 to S24 in FIG. 9. When the camera module 3 is started, the display pixel density of some horizontal pixel lines in one frame is set (step S35). The some horizontal pixel lines are horizontal pixel lines included in the second display area 2r2 described above. Although the number of horizontal pixel lines is not limited, considering the size of the camera module 3, the second display area 2r2 includes a plurality of horizontal pixel lines. The display pixel density set in step S35 is set to, for example, a value smaller than the display pixel density when the camera module 3 is not started. More specifically, the display pixel density is reduced by thinning out and driving a plurality of horizontal pixel lines in the second display area 2r2. By reducing the display pixel density, the display off period of the second display area 2r2 can be extended, and the exposure time can be set to a longer period.

[0082] Next, a luminance value of the pixel data is set based on the display pixel density in the second display region 2r2 (step S36). Since lowering the display pixel density makes the display darker, the luminance value of the pixel data is increased so that the display luminance perceived by the human eye is the same as the display luminance when the camera module 3 is not activated. Next, when starting the display and displaying some horizontal pixel lines, i.e., when displaying the second display region 2r2, the display is performed with the set display pixel density and luminance value (step S37).

[0083] Next, during a display-off period of the second display region 2r2, the imaging section 4 captures an image with an exposure time determined by the display pixel density in step S35 (step S38).

[0084] Figure 13 is an operation timing diagram corresponding to the flowchart of Figure 12. Waveform w16 is the waveform of the vertical synchronizing signal VSYNC, waveform w17 is the waveform of the horizontal synchronizing signal HSYNC, waveform w18 is a waveform indicating an exposure period, and waveform w19 is a waveform that diagrammatically indicates the luminance value of pixel data displayed on the display unit 2. The high period of waveform w17 is the display period of each horizontal pixel line.

[0085] In the example of Fig. 13, the display pixel density is adjusted by making the display periods of some horizontal pixel lines in one frame longer than the display periods of the other horizontal pixel lines. That is, the display pixel density is lowered by driving every other horizontal pixel line or every third horizontal pixel line in the second display region 2r2. Also, the luminance value of the pixel data is increased during the display periods of the horizontal pixel lines in the second display region 2r2.

[0086] 14A, 14B, and 14C are diagrams showing schematic examples of the second display area 2r2. FIG. 14A shows an example in which the camera module 3 is present on the upper end side of the display unit 2, and the second display area 2r2 is provided in a horizontal strip shape including the placement location of the camera module 3. FIG. 14B shows an example in which the camera module 3 is present closer to the center of the display unit 2 than in FIG. 14A, and the second display area 2r2 is provided in a horizontal strip shape including the placement location of the camera module 3. FIG. 14C shows an example in which two camera modules 3 are placed, and two second display areas 2r2 are provided in a horizontal strip shape including the placement locations of these camera modules 3.

[0087] In this way, in the fourth embodiment, the display pixel density can be adjusted in units of horizontal pixel lines of the display unit 2. Therefore, when starting up the camera module 3, the display pixel density of the horizontally extending band-shaped second display area 2r2 can be adjusted to match the location of the camera module 3, thereby lengthening the exposure time used by the imaging unit 4 to capture an image during the display-off period of the second display area 2r2.

[0088] Fifth embodiment The electronic device 1 according to the fifth embodiment adjusts the display pixel density of the display area in accordance with the outer size of the camera module 3 arranged on the rear side of the display unit 2.

[0089] The electronic device 1 according to the fifth embodiment has the same internal configuration as that shown in FIG. 4. Similarly to the fourth embodiment, the display unit 2 has a first display region 2r1 and a second display region 2r2. While the second display region 2r2 in the fourth embodiment is an area corresponding to a plurality of horizontal pixel lines, the second display region 2r2 in this embodiment is a pixel block region including a plurality of pixels arranged adjacently in the horizontal and vertical directions. The second display region 2r2 in this embodiment includes some of the pixels in the plurality of horizontal pixel lines.

[0090] Fig. 15 is a flowchart showing the processing operation of the electronic device 1 according to the fifth embodiment, for example, the processing operation executed by the application processor 22. The following mainly describes the differences from the flowchart in Fig. 12. Steps S41 to S44 in Fig. 15 perform the same processing as steps S31 to S34 in Fig. 12. When starting up the camera module 3, the display pixel density of a part of the display area, i.e., the second display area 2r2, is set (step S45). Here, the display pixel density may be determined in advance, or may be set arbitrarily depending on the surrounding brightness, etc., as described later.

[0091] Next, based on the display pixel density set in step S45, the luminance value of the pixel data is set so that the display luminance perceived by the human eye is approximately the same as when the camera module 3 is not activated (step S46).

[0092] Next, when display is started and the second display region 2r2 is displayed, it is displayed with the set display pixel density and brightness value (step S47). In this embodiment, even when the camera module 3 is activated, when the first display region 2r1 is displayed, it is displayed with the same display pixel density and brightness value as when the camera module 3 is not activated. On the other hand, when the second display region 2r2 is displayed, it is displayed with the display pixel density and brightness value set in steps S45 and S46.

[0093] Next, during the display-off period in which the second display region 2r2 is being displayed, the imaging section 4 captures an image (step S48). The exposure time in this case corresponds to the length of the display-off period.

[0094] Fig. 16 is an operation timing diagram corresponding to the flowchart of Fig. 15. Waveform w20 is a waveform indicating the horizontal synchronization signal HSYNC, waveform w21 is a waveform indicating the display period of a horizontal pixel line when the camera module 3 is not activated, waveform w22 is a waveform indicating the display period of a horizontal pixel line when the camera module 3 is activated, waveform w23 is a waveform indicating the exposure period when the camera module 3 is activated, and waveform w24 is a waveform indicating the luminance value of pixel data.

[0095] 16, in this embodiment, when displaying the second display region 2r2 provided at a position overlapping with the arrangement location of the camera module 3, the display pixel density is reduced and the display off period is extended. This allows imaging by the imaging unit 4 to be performed during the display off period.

[0096] 17A and 17B are diagrams showing a specific example of the second display area 2r2 when the camera module 3 is activated. FIG. 17A shows an example in which the camera module 3 is disposed on the upper end side of the display unit 2, and FIG. 17B shows an example in which the camera module 3 is disposed closer to the center of the display unit 2. In the cases of FIGS. 17A and 17B, the second display area 2r2 is provided according to the outer shape of the camera module 3, and the display pixel density is reduced when the second display area 2r2 is displayed. However, by adjusting the display brightness, the difference in display brightness perceived by the human eye between the second display area 2r2 and the first display area 2r1 surrounding it can be suppressed.

[0097] In this way, in the fifth embodiment, the second display area 2r2 is set at a location overlapping with the location of the camera module 3, and the display pixel density and the luminance value of the pixel data of the second display area 2r2 are adjusted, so that imaging by the imaging unit 4 can be performed without being affected by the display of the display unit 2, thereby improving the image quality of the captured image. Furthermore, since the area of ​​the second display area 2r2 can be made smaller than that of the fourth embodiment, the second display area 2r2 becomes less noticeable and does not give the user a sense of incongruity in the display.

[0098] Sixth embodiment The sixth embodiment controls the frame rate according to the surrounding brightness.

[0099] Fig. 18 is a block diagram showing an internal configuration of an electronic device 1 according to a sixth embodiment. The electronic device 1 of Fig. 18 includes a brightness detection unit 26 in addition to the internal configuration of the electronic device 1 of Fig. 4. The brightness detection unit 26 detects, for example, an average brightness of light incident on the photoelectric conversion unit 4a, or a brightness based on an integrated value of the incident light, based on digital pixel data output from the A / D conversion unit 31. The brightness detection unit 26 can detect the brightness around the electronic device 1. Alternatively, the brightness detection unit 26 may be configured with a sensor that detects brightness separately from the imaging unit 4.

[0100] The imaging control unit 33 and the application processor 22 set the frame rate of the display unit 2 and the exposure time of the imaging unit 4 based on the brightness information detected by the brightness detection unit 26. For example, the darker the surroundings are, the lower the frame rate of the display unit 2 is set. This makes it possible to lengthen the display off period and therefore the exposure time, thereby improving the imaging sensitivity of the imaging unit 4.

[0101] FIG. 19 is a flowchart showing the processing operation of the electronic device 1 according to the sixth embodiment. The following mainly describes the differences from the flowchart in FIG. 9. Steps S51 to S54 in FIG. 19 are similar to steps S21 to S24 in FIG. 9. When starting up the camera module 3, the brightness detection unit 26 detects the brightness of the surroundings (step S55). Next, based on the detected brightness of the surroundings, a frame rate and an exposure time are set (step S56). Next, display is started at the set frame rate (step S57). Then, during the display-off period, imaging is performed by the imaging unit 4 with the set exposure time (step S58).

[0102] Fig. 20 is an operation timing chart corresponding to the flowchart of Fig. 19. The waveform w25 is a waveform indicating the vertical synchronizing signal VSYNC, the waveform w26 is a waveform indicating the horizontal synchronizing signal HSYNC, the waveform w27 is a waveform indicating the exposure time, and the waveform w28 is a luminance value of the pixel data.

[0103] 20 with FIG 10, in this embodiment, the display luminance is not adjusted. This is because improving the imaging sensitivity of the imaging unit 4 is given priority over the visibility of the display unit 2.

[0104] The flowchart in Figure 19 assumes that the frame rate of the entire display area of ​​the display unit 2 is changed in accordance with the ambient brightness, as in Figure 11, but the second display area 2r2 including some horizontal pixel lines as in Figures 14A to 14C, or the second display area 2r2 including some pixel blocks as in Figures 17A to 17B, may be displayed at a display pixel density in accordance with the ambient brightness.

[0105] In this way, in the sixth embodiment, the frame rate or the display pixel density is controlled according to the surrounding brightness, so that, for example, the darker the surroundings are, the lower the frame rate or the display pixel density can be, and the longer the exposure time can be, thereby improving the imaging sensitivity of the imaging unit 4.

[0106] Seventh embodiment In the seventh embodiment, blinking of a person looking at the display unit 2 of the electronic device 1 is detected, and while the eyes are closed, the display on the display unit 2 is turned off and an image is captured by the imaging unit 4.

[0107] Fig. 21 is a block diagram showing the internal configuration of an electronic device 1 according to the seventh embodiment. The electronic device 1 in Fig. 21 has an imaging device 21, a timing detection unit 27, a resolution conversion unit 23, an A / D conversion unit 31, a display control unit 25, and a display unit 2. In Fig. 21, the application processor 22, the imaging control unit 33, and the exposure adjustment unit 34 present in Fig. 4 are omitted, but the application processor 22, the imaging control unit 33, and the exposure adjustment unit 34 may be provided.

[0108] The timing detection unit 27 detects the timing of a blink of a person captured in a captured image based on the digital pixel data output from the A / D conversion unit 31. Then, the timing detection unit 27 notifies the display control unit 25 of information indicating the timing when the person closes his / her eyes and the timing when he / she opens his / her eyes. The display control unit 25 performs control to turn off the display of the display unit 2 while the person's eyes are closed and to turn on the display of the display unit 2 while the person's eyes are open.

[0109] 22 is a flowchart showing the processing operation of the electronic device 1 according to the seventh embodiment. If the camera module 3 is not activated (NO in step S61), display is started at the normal frame rate (step S62). On the other hand, if the camera module 3 is activated (YES in step S61), display is started at the normal frame rate (step S63). Then, based on information from the timing detection unit 27, it is determined whether the human eyes are closed (step S64). The processing of steps S63 and S64 is repeated until the human eyes are closed.

[0110] If it is determined that the human eye is closed, the display of the display unit 2 is turned off (step S65), and exposure by the imaging unit 4 is started (step S66). Then, based on information from the timing detection unit 27, it is determined whether the human eye is open (step S67). If it is determined that the human eye is open, exposure by the imaging unit 4 is ended (step S68).

[0111] In this way, in the seventh embodiment, a blink of a person is detected, and while the person's eyes are closed, the display of the display unit 2 is turned off, and an exposure time is set according to the period during which the person's eyes are closed, and then imaging is performed by the imaging unit 4. This allows imaging by the imaging unit 4 to be performed with the display of the display unit 2 turned off, improving the quality of the captured image. Furthermore, the display of the display unit 2 is turned off during the period during which the person's eyes are closed due to blinking, and since the person is not made aware that the display has been turned off, there is no risk of the display unit 2 giving a sense of discomfort.

[0112] Eighth embodiment In the fourth and fifth embodiments described above, an example was shown in which the first display area 2r1 and the second display area 2r2 are provided within the display surface 1a of the display unit 2, but the display surface 1a of the first display area 2r1 (hereinafter, the first display surface) and the display surface 1a of the second display area 2r2 (hereinafter, the second display surface) may be provided in separate layers. The first display surface layer and the second display surface layer are stacked in the front-to-back direction, and when the first display surface layer and the second display surface layer are stacked, the display areas are arranged so that they do not overlap. Therefore, when the line of sight is directed toward the display unit 2, the first display area 2r1 and the second display area 2r2 are viewed as being united, and the observer does not feel uncomfortable.

[0113] In this way, by providing the first display area 2r1 and the second display area 2r2 on separate display surfaces 1a, display control of the first display area 2r1 and display control of the second display area 2r2 can be performed separately, making it easier to process the display control unit 25.

[0114] Ninth embodiment In the electronic device 1 according to the ninth embodiment, the optical system 5 of the camera module 3 is different from those in the first to eighth embodiments.

[0115] Fig. 23 is a diagram showing a cross-sectional structure of an imaging unit 4 of a camera module 3 mounted on an electronic device 1 according to a ninth embodiment. The imaging unit 4 in Fig. 23 has a microlens array 64, rather than a single lens or a lens group in which single lenses are arranged in the optical axis direction.

[0116] 23 has a photoelectric conversion unit 4a arranged along the bottom surface of a housing 63, a microlens array 64 arranged above the photoelectric conversion unit 4a, a plurality of light blocking bodies 66 arranged between adjacent microlenses 65, and a light guide plate 67 arranged above the microlens array 64. The imaging unit 4 in FIG. 23 is applicable to any of the first to eighth embodiments described above.

[0117] In this way, by providing the microlens array 64 as the optical system 5 of the imaging unit 4, it is possible to prevent the influence of adjacent pixels and reduce color bleeding.

[0118] (Tenth embodiment) Since a part of the light passing through the display unit 2 is reflected or diffracted and enters the camera, the quality of the captured image may be degraded due to the influence of flare or diffraction caused by the reflection. Therefore, a polarizing element 8b may be provided in the imaging unit 4 of the first to ninth embodiments described above to obtain polarization information including components of flare light and diffracted light, and digital pixel data from which the components of flare light and diffracted light have been removed may be generated using this polarization information.

[0119] The cross-sectional structure of the imaging unit 4 in this case is shown in a cross-sectional view such as that shown in Fig. 24. In Fig. 24, parts common to Fig. 3 are given the same reference numerals, and the following mainly describes the differences. In the imaging unit 4 in Fig. 13, a light-shielding layer 18 and an underlying insulating layer 19 are disposed on the planarizing layer 13, and a plurality of polarizing elements 8b are disposed at a distance from each other on the underlying insulating layer 19. The polarizing elements 8b are, for example, wire-grid polarizing elements 8b with a line-and-space structure disposed on a part of the underlying insulating layer 19.

[0120] FIG. 25 is a perspective view showing an example of a detailed structure of each polarizing element 8b. As shown in FIG. 25, each of the polarizing elements 8b has a plurality of line portions 8d of a convex shape extending in one direction and a space portion 8e between each line portion 8d. There are a plurality of types of polarizing elements 8b, each of which has a different extending direction of the line portion 8d. More specifically, there are three or more types of polarizing elements 8b, and for example, the angle between the arrangement direction of the photoelectric conversion unit 4a and the extension direction of the line portion 8d may be three types of 0 degrees, 60 degrees, and 120 degrees. Alternatively, the angle between the arrangement direction of the photoelectric conversion unit 4a and the extension direction of the line portion 8d may be four types of 0 degrees, 45 degrees, 90 degrees, and 135 degrees, or may be any other angle. Alternatively, the plurality of polarizing elements 8b may polarize light in only one direction. The material of the plurality of polarizing elements 8b may be a metal material such as aluminum or tungsten, or an organic photoelectric conversion film.

[0121] The multiple polarizing elements 8b are disposed on the light incident side of at least one of the multiple photoelectric conversion units 4a. The polarizing element 8b polarizes the light that passes through the display unit 2 and is incident. The light polarized by the polarizing element 8b is incident on the corresponding photoelectric conversion unit 4a and photoelectrically converted. Hereinafter, the output value of the photoelectric conversion unit 4a that photoelectrically converts the light polarized by the polarizing element 8b is referred to as polarization information, and the output value of the photoelectric conversion unit 4a that photoelectrically converts the light that is incident without passing through the polarizing element 8b is referred to as pixel value or pixel information. Hereinafter, the polarizing element 8b and the photoelectric conversion unit 4a that photoelectrically converts the light polarized by the polarizing element 8b are collectively referred to as a polarizing pixel, and the photoelectric conversion unit 4a that photoelectrically converts the light that is incident without passing through the polarizing element 8b is referred to as a non-polarizing pixel.

[0122] 24, on the insulating layer 19 on which the plurality of polarizing elements 8b are arranged, a planarization layer 20 is arranged. On the planarization layer 20, a color filter layer 14 is arranged.

[0123] Fig. 26 is a block diagram showing the internal configuration of an electronic device having a function of suppressing the effects of flare light and diffracted light. In Fig. 26, the same reference numerals are used for the parts common to Fig. 4, and the following description will focus on the differences. The electronic device in Fig. 26 has a polarized light output section 32j, a flare extraction section 32k, and a flare correction signal generation section 32m in addition to the internal configuration in Fig. 4.

[0124] The polarization output unit 32j outputs the polarization information data. The flare extraction unit 32k extracts at least one of a flare component and a diffracted light component from the polarization information data. At least one of the flare component and the diffracted light component extracted by the flare extraction unit 32k is a correction amount. The flare correction signal generation unit 32m corrects the digital pixel data output from the color output unit 32b by subtracting the correction amount extracted by the flare extraction unit 32k from the digital pixel data. The output data of the flare correction signal generation unit 32m is digital pixel data from which at least one of the flare component and the diffracted light component has been removed. In this way, the flare correction signal generation unit 32m functions as a correction unit that corrects the captured image photoelectrically converted by a plurality of non-polarized pixels based on the polarization information.

[0125] The signal level of the digital pixel data at the pixel position of the polarization pixel is lowered by the amount that the data has passed through the polarizing element 8b. Therefore, the defect correction unit 32c regards the polarization pixel as defective and performs a predetermined defect correction process.

[0126] As shown in Figures 24 to 26, by acquiring polarization information using multiple polarizing elements 8b, it is possible to generate an captured image in a state in which flare components and diffracted light components contained in light that is repeatedly reflected within the display unit 2 and then incident on multiple non-polarized pixels are simply and reliably removed.

[0127] (Eleventh embodiment) There are various possible specific candidates for the electronic device 1 having the configurations described in the first to fourth embodiments. For example, Fig. 27 is a plan view of the electronic device 1 of the first to fourth embodiments applied to a capsule endoscope 50. The capsule endoscope 50 in Fig. 27 includes, in a housing 51 having, for example, hemispherical end faces and a cylindrical center, a camera (ultra-miniature camera) 52 for capturing images inside a body cavity, a memory 53 for recording image data captured by the camera 52, and a wireless transmitter 55 for transmitting the recorded image data to the outside via an antenna 54 after the capsule endoscope 50 is discharged from the subject's body.

[0128] Also provided within housing 51 are a CPU (Central Processing Unit) 56 and a coil (magnetic force-current conversion coil) 57. CPU 56 controls the shooting by camera 52 and the data storage operation in memory 53, and also controls the data transmission from memory 53 to a data receiving device (not shown) outside housing 51 by wireless transmitter 55. Coil 57 supplies power to camera 52, memory 53, wireless transmitter 55, antenna 54, and light source 52b described below.

[0129] Furthermore, the housing 51 is provided with a magnetic (reed) switch 58 for detecting when the capsule endoscope 50 is set in the data receiving device. When the reed switch 58 detects that the capsule endoscope 50 is set in the data receiving device and data transmission becomes possible, the CPU 56 supplies power from the coil 57 to the wireless transmitter 55.

[0130] The camera 52 has, for example, an imaging element 52a including an objective optical system for capturing an image inside the body cavity, and a plurality of light sources 52b for illuminating the inside of the body cavity. Specifically, the camera 52 is configured with, for example, a CMOS (Complementary Metal Oxide Semiconductor) sensor equipped with an LED (Light Emitting Diode) or a CCD (Charge Coupled Device) as the light source 52b.

[0131] The display unit 2 in the electronic device 1 of the first to fourth embodiments is a concept including a light emitting body such as the light source 52b in Fig. 27. The capsule endoscope 50 in Fig. 27 has, for example, two light sources 52b, but these light sources 52b can be configured with a display panel having a plurality of light source units or an LED module having a plurality of LEDs. In this case, by arranging the imaging unit 4 of the camera 52 below the display panel or the LED module, restrictions on the layout arrangement of the camera 52 are reduced, and a smaller capsule endoscope 50 can be realized.

[0132] 28 is a rear view of the electronic device 1 according to the first to fourth embodiments applied to a digital single-lens reflex camera 60. The digital single-lens reflex camera 60 and compact cameras are provided with a display unit 2 that displays a preview screen on the rear surface opposite the lens. A camera module 3 may be disposed on the opposite side of the display surface of the display unit 2 so that an image of the photographer's face can be displayed on the display surface 1a of the display unit 2. In the electronic device 1 according to the first to fourth embodiments, the camera module 3 can be disposed in an area overlapping with the display unit 2, so that it is not necessary to provide the camera module 3 in the frame portion of the display unit 2, and the size of the display unit 2 can be made as large as possible.

[0133] Fig. 29A is a plan view showing an example in which the electronic device 1 according to the first to fourth embodiments is applied to a head mounted display (hereinafter, HMD) 61. The HMD 61 in Fig. 29A is used for VR (Virtual Reality), AR (Augmented Reality), MR (Mixed Reality), SR (Substituional Reality), or the like. Current HMDs, as shown in Fig. 29B, are equipped with a camera 62 on the outer surface, and while a person wearing the HMD can view images of the surroundings, there is a problem in that people around cannot see the eyes or facial expressions of the person wearing the HMD.

[0134] 29A, the display surface of the display unit 2 is provided on the outer surface of the HMD 61, and the camera module 3 is provided on the opposite side to the display surface of the display unit 2. This allows the facial expression of the wearer captured by the camera module 3 to be displayed on the display surface of the display unit 2, allowing people around the wearer to grasp the facial expression and eye movements of the wearer in real time.

[0135] 29A, since the camera module 3 is provided on the back side of the display unit 2, there are no restrictions on where to install the camera module 3, which increases the freedom of design of the HMD 61. In addition, since the camera can be placed in an optimal position, it is possible to prevent problems such as the wearer's line of sight not being aligned with the display surface.

[0136] In this way, in the eleventh embodiment, the electronic device 1 according to the first to fourth embodiments can be used for various purposes, thereby increasing the utility value.

[0137] The present technology can be configured as follows. (1) a display unit; an imaging unit disposed on the opposite side to a display surface of the display unit; An electronic device comprising: a control unit that synchronizes a display timing of the display unit with an image capturing timing of the image capturing unit so that the image capturing unit captures an image when the display unit is not displaying an image. (2) The electronic device described in (1), wherein the control unit, when the imaging unit is capturing an image, makes the period during which the display unit does not display longer than when the imaging unit is not capturing an image. (3) The electronic device described in (1) or (2), wherein the control unit, when the imaging unit is capturing an image, lowers the frequency at which at least some of the pixels of the display unit are displayed compared to when the imaging unit is not capturing an image. (4) The electronic device according to (3), wherein the control unit, when the imaging unit is capturing an image, sets a frame rate of the display unit lower than when the imaging unit is not capturing an image. (5) The electronic device described in (3), wherein the control unit, when the imaging unit is capturing an image, lowers the display frequency of at least some of the horizontal pixel lines of the display unit compared to when the imaging unit is not capturing an image. (6) The electronic device described in (5), wherein, when the imaging unit is capturing an image, the control unit lowers the horizontal pixel line display frequency of the portion including the display area that overlaps with the imaging unit when the display unit is viewed in a planar view from the normal direction of the display surface, compared to when the imaging unit is not capturing an image. (7) The electronic device according to any one of (1) to (6), wherein the control unit sets an exposure time of the imaging unit based on a period during which the display unit does not display. (8) The electronic device described in any one of (1) to (7), wherein the control unit, when the imaging unit is capturing an image, reduces the display pixel density in at least a portion of the display area of ​​the display unit compared to when the imaging unit is not capturing an image. (9) The electronic device described in (8), wherein, when the imaging unit is capturing an image, the control unit reduces the display pixel density within a portion of the display area, including a display area that overlaps with the imaging unit, when the display unit is viewed in a planar view from the normal direction of the display surface, compared to when the imaging unit is not capturing an image. (10) The electronic device described in (8) or (9), wherein the control unit increases the brightness value of pixels in at least a portion of the display area of ​​the display unit when the imaging unit is capturing an image, compared to when the imaging unit is not capturing an image. (11) The electronic device described in (10), wherein when the imaging unit is capturing an image, the control unit increases the brightness value of pixels in the portion of the display area including the display area that overlaps with the imaging unit when the display unit is viewed in a planar view from the normal direction of the display surface, compared to when the imaging unit is not capturing an image. (12) The electronic device described in (11), wherein when the imaging unit is capturing an image, the control unit lowers the frequency at which pixels in the portion of the display area are displayed and increases the brightness value of the pixels compared to when the imaging unit is not capturing an image. (13) The display unit has a first display surface and a second display surface that are stacked on each other and whose display is controlled separately, the first display surface displays a display area of ​​the display unit excluding the part of the display area; The electronic device described in any one of (8) to (12), wherein the second display surface displays the portion of the display area so as not to overlap with the display area of ​​the first display surface when stacked on the first display surface. (14) A brightness detection unit is provided which detects the brightness of the surroundings. The electronic device described in any one of (1) to (13), wherein the control unit controls a frequency at which at least a portion of the pixels of the display unit are displayed when the imaging unit captures an image, in accordance with the ambient brightness detected by the brightness detection unit. (15) The electronic device described in (14), wherein the control unit controls the frequency at which at least some of the pixels of the display unit are displayed and the display pixel density when the imaging unit captures an image, in accordance with the ambient brightness detected by the brightness detection unit. (16) A timing detection unit is provided to detect the timing of a blink of a person who directs his / her gaze to the display unit, The electronic device described in any one of (1) to (15), wherein the control unit temporarily switches the display unit to a non-emitting state and captures an image of the imaging unit in accordance with the timing detected by the timing detection unit. (17) The electronic device according to any one of (1) to (16), wherein the imaging unit photoelectrically converts at least one of visible light and infrared light. (18) The display unit has an opening at a position overlapping with the imaging unit when the display unit is viewed in a plan view from a normal direction of the display surface, The electronic device described in any one of (1) to (17), wherein the opening is arranged to pass through a layer, among the multiple layers constituting the display unit, whose transmittance for light in a predetermined wavelength band is less than a predetermined value. (19) The electronic device described in any one of (1) to (18), wherein the display unit is composed of a plurality of layers having a transmittance of a predetermined value or more for light in a wavelength band capable of photoelectric conversion by the imaging unit. (20) The control unit: A first control unit that controls the display of the display unit; The electronic device according to any one of (1) to (19), further comprising: a second control unit that cooperates with the first control unit to control imaging of the imaging unit.

[0138] The aspects of the present disclosure are not limited to the above-mentioned individual embodiments, but include various modifications that may be conceived by a person skilled in the art, and the effects of the present disclosure are not limited to the above-mentioned contents. In other words, various additions, modifications, and partial deletions are possible within the scope of the conceptual idea and intent of the present disclosure derived from the contents defined in the claims and their equivalents. [Explanation of symbols]

[0139] REFERENCE SIGNS LIST 1 electronic device, 2 display section, 3 camera module, 4 imaging section, 4a photoelectric conversion section, 5 optical system, 11 semiconductor substrate, 12 element isolation layer, 13 planarization layer, 14 color filter layer, 15 on-chip lens, 16 readout circuit, 17 interlayer insulating film, 21 imaging device, 22 application processor, 23 resolution conversion section, 24 A / D conversion section, 25 display control section, 31 A / D conversion section, 32 signal processing section, 33 imaging control section, 34 exposure adjustment section

Claims

1. A display unit that repeats a first period in which a display is performed and a second period in which a display is not performed; an imaging unit disposed on the opposite side to a display surface of the display unit; a control unit that synchronizes a display timing of the display unit with an imaging timing of the imaging unit so that the imaging unit captures images during the second period, and makes the second period when the imaging unit captures images longer than the second period when the imaging unit does not capture images.

2. A display unit that repeats a first period in which a display is performed and a second period in which a display is not performed; an imaging unit disposed on the opposite side to a display surface of the display unit; a control unit that synchronizes a display timing of the display unit with an imaging timing of the imaging unit so that the imaging unit performs imaging during the second period, and sets a display pixel density in at least a part of a display area of ​​the display unit during the first period when the imaging unit performs imaging during the second period lower than a display pixel density in the display area during the first period when the imaging unit does not perform imaging during the second period.

3. The electronic device according to claim 2 , wherein the control unit sets the second period when the imaging unit captures an image to be longer than the second period when the imaging unit does not capture an image.

4. 4. The electronic device according to claim 1, wherein the control unit sets a display frequency of horizontal pixel lines during the first period when the imaging unit is imaging lower than the display frequency during the first period when the imaging unit is not imaging.

5. The electronic device according to claim 4 , wherein the control unit sets a frame rate of the display unit when the imaging unit is capturing an image to be lower than a frame rate when the imaging unit is not capturing an image.

6. 6. The electronic device according to claim 4 or 5, wherein the control unit, during the first period when the imaging unit is capturing images, lowers a horizontal pixel line display frequency of the portion including a display area that overlaps with the imaging unit when the display unit is viewed in a planar view from a normal direction to the display surface, compared to the first period when the imaging unit is not capturing images.

7. The electronic device according to claim 1 , wherein the control unit sets an exposure time of the imaging unit based on the second period when the imaging unit captures an image.

8. 2. The electronic device of claim 1, wherein the control unit reduces a display pixel density in at least a portion of a display area of ​​the display unit during the first period when the imaging unit is imaging during the second period to a value lower than a display pixel density in the display area during the first period when the imaging unit is not imaging during the second period.

9. 9. The electronic device of claim 2 or 8, wherein the control unit reduces a display pixel density in a part of a display area including a display area overlapping with the imaging unit when the display unit is viewed in a planar view from a normal direction to the display surface, in the first period when the imaging unit is capturing an image, compared to the first period when the imaging unit is not capturing an image.

10. 10. The electronic device according to claim 2, 8, or 9, wherein the control unit increases a luminance value of pixels in at least a portion of a display area of ​​the display unit during the first period when the imaging unit is capturing an image, compared to the first period when the imaging unit is not capturing an image.

11. 11. The electronic device of claim 10, wherein the control unit increases a luminance value of a pixel in the part of the display area including a display area overlapping with the imaging unit when the display unit is viewed in a planar view from a normal direction to the display surface, during the first period when the imaging unit is capturing an image, compared to the first period when the imaging unit is not capturing an image.

12. 12. The electronic device of claim 11, wherein the control unit, during the first period when the imaging unit is capturing an image, reduces a frequency at which pixels in the partial display area are displayed and increases a luminance value of the pixels compared to the first period when the imaging unit is not capturing an image.

13. the display unit includes a first display surface and a second display surface that are stacked on top of each other and whose display is controlled separately; the first display surface displays a display area of ​​the display unit excluding the part of the display area; 13. The electronic device according to claim 2, wherein the second display surface displays the part of the display area so as not to overlap with the display area of ​​the first display surface when the second display surface is laminated on the first display surface.

14. A brightness detection unit is provided to detect the brightness of the surroundings, 14. The electronic device according to claim 1, wherein the control unit controls a frequency at which at least a portion of pixels of the display unit are displayed during the first period when the imaging unit captures an image, in accordance with the ambient brightness detected by the brightness detection unit.

15. 15. The electronic device according to claim 14, wherein the control unit controls a frequency at which at least a portion of pixels of the display unit are displayed during the first period when the imaging unit is capturing an image, and a display pixel density, in accordance with the ambient brightness detected by the brightness detection unit.

16. A timing detection unit detects the timing of a blink of a person who directs his or her gaze to the display unit, The electronic device according to claim 1 , wherein the control unit temporarily switches the display unit to a non-emitting state and controls the imaging unit to capture an image in accordance with the timing detected by the timing detection unit.

17. The electronic device according to claim 1 , wherein the imaging section performs photoelectric conversion on at least one of visible light and infrared light.

18. the display unit has an opening at a position overlapping with the imaging unit when the display unit is viewed in a plan view from a normal direction of the display surface, The electronic device according to claim 1 , wherein the opening is disposed so as to penetrate a layer, of a plurality of layers constituting the display unit, the layer having a transmittance for light in a predetermined wavelength band less than a predetermined value.

19. The electronic device according to claim 1 , wherein the display section is made up of a plurality of layers, each of which has a transmittance of a predetermined value or more for light in a wavelength band capable of being photoelectrically converted by the imaging section.

20. The control unit is A first control unit that controls display of the display unit; The electronic device according to claim 1 , further comprising: a second control unit that cooperates with the first control unit to control imaging of the imaging unit.

21. A display unit; an imaging unit disposed on the opposite side to a display surface of the display unit; a control unit that synchronizes a display timing of the display unit with an imaging timing of the imaging unit so that the imaging unit captures an image when the display unit does not display an image, and that, when the imaging unit captures an image, lowers a display pixel density in at least a part of a display area of ​​the display unit and increases a luminance value of pixels in at least a part of the display area of ​​the display unit compared to when the imaging unit does not capture an image. electronic equipment.

Images