Electronic device

Abstract

The present invention provides an electronic device that does not degrade the image quality of a captured image and the display quality of a display unit. The electronic device includes: a display unit; a photographing unit disposed on a side opposite to a display surface of the display unit; and a control unit that synchronizes a display timing of the display unit with a photographing timing of the photographing unit so that the photographing unit performs photographing when the display unit is not displaying.

Description

Technical Field

[0001] The present invention relates to an electronic device. Background Art

[0002] In recent electronic devices such as smartphones, mobile phones, and PCs (Personal Computers), a camera is disposed on a frame (border) of a display unit, enabling easy video calls and video shooting. Since smartphones and mobile phones are usually carried in a pocket or a bag, it is necessary to make the external dimensions as compact as possible. On the other hand, if the size of the display screen is small, the higher the display resolution, the smaller the size of the displayed characters, making them difficult to recognize. Therefore, research is underway to reduce the border width around the display screen so as to increase the size of the display screen as much as possible without increasing the external dimensions of the electronic device.

[0003] However, since a camera or the like is usually installed on the border of the electronic device, the border width cannot be less than the outer diameter size of the camera.

[0004] In addition, when a camera is disposed on the border, for example, during a video call, the line of sight usually focuses on the vicinity of the center of the display screen. Therefore, the line of sight deviates from the optical axis of the camera, resulting in a sense of incongruity due to the mismatch between the line of sight and the captured image.

[0005] To avoid the above problems, a scheme has been proposed in which a camera module is disposed on the side of the display unit opposite to the display surface, and the camera is used to capture subject light passing through the display unit.

[0006] [Prior Art Documents]

[0007] [Patent Documents]

[0008] Patent Document 1: US Patent Publication Gazette 2018 / 0069060

[0009] [Problems to be Solved by the Invention]

[0010] However, if a camera module is disposed on the side of the display unit opposite to the display surface, the light passing through the display unit will enter the camera module, so the image quality of the captured image may be degraded due to the influence of the display brightness of the display unit.

[0011] In addition, methods such as reducing the pixel density of a partial display area of the display unit that overlaps with the configuration position of the camera module have been considered, but this may cause a local reduction in the display quality of the display unit, resulting in a sense of incongruity. Summary of the Invention

[0012] In one aspect of the present invention, an electronic device is provided that does not degrade the image quality of a captured image and the display quality of a display unit.

[0013] [Solution for Solving the Problem]

[0014] To solve the above problems, an aspect of the present invention provides an electronic device, comprising: a display unit;

[0015] a photographing unit disposed on a side opposite to a display surface of the display unit; and

[0016] a control unit that synchronizes a display timing of the display unit with a photographing timing of the photographing unit so that the photographing unit performs photographing when the display unit does not perform display.

[0017] Compared with a case where the photographing unit does not perform photographing, when the photographing unit performs photographing, the control unit may also make a period during which the display unit does not perform display longer.

[0018] Compared with a case where the photographing unit does not perform photographing, when the photographing unit performs photographing, the control unit may also make a display frequency of at least a part of pixels of the display unit lower.

[0019] Compared with a case where the photographing unit does not perform photographing, when the photographing unit performs photographing, the control unit may also make a frame rate of the display unit lower.

[0020] Compared with a case where the photographing unit does not perform photographing, when the photographing unit performs photographing, the control unit may also make a display frequency of at least a part of horizontal pixel lines of the display unit lower.

[0021] Compared with a case where the photographing unit does not perform photographing, when the photographing unit performs photographing, the control unit may also make a display frequency of the part of horizontal pixel lines including a display area overlapping with the photographing unit lower when looking down on the display unit from a normal direction of the display surface.

[0022] The control unit may also set an exposure time of the photographing unit based on a period during which the display unit does not perform display.

[0023] Compared with a case where the photographing unit does not perform photographing, when the photographing unit performs photographing, the control unit may also make a display pixel density in at least a part of a display area of the display unit lower.

[0024] Compared with a case where the photographing unit does not perform photographing, when the photographing unit performs photographing, the control unit may also make a display pixel density in a part of a display area including a display area overlapping with the photographing unit lower when looking down on the display unit from a normal direction of the display surface.

[0025] When the imaging unit is imaging, as compared with the case where the imaging unit is not imaging, the control unit may also increase the brightness values of the pixels in at least a part of the display area of the display unit.

[0026] When the imaging unit is imaging, as compared with the case where the imaging unit is not imaging, the control unit may also increase the brightness values of the pixels in the part of the display area that includes the display area overlapping with the imaging unit when looking down on the display unit from the normal direction of the display surface.

[0027] When the imaging unit is imaging, as compared with the case where the imaging unit is not imaging, the control unit may also make the display frequency of the pixels in the part of the display area lower and the brightness values of the pixels higher.

[0028] The display unit may also include a first display surface and a second display surface, the first display surface and the second display surface are stacked on each other and are separately controlled for display,

[0029] The first display surface displays the display area of the display unit except for the part of the display area.

[0030] When the second display surface is stacked with the first display surface, the second display surface displays the part of the display area in a manner that does not overlap with the display area of the first display surface.

[0031] It may also include a brightness detection unit for detecting the surrounding brightness.

[0032] The control unit controls the display frequency of at least a part of the pixels of the display unit when the imaging unit is imaging according to the surrounding brightness detected by the brightness detection unit.

[0033] The control unit may also control the display frequency and display pixel density of at least a part of the pixels of the display unit when the imaging unit is imaging according to the surrounding brightness detected by the brightness detection unit.

[0034] It may also include a timing detection unit that detects the timing of a person's blink when looking at the display unit.

[0035] The control unit temporarily sets the display of the display unit to a non-light-emitting state in accordance with the timing detected by the timing detection unit, and performs imaging with the imaging unit.

[0036] The imaging unit may also perform photoelectric conversion on at least one of visible light and infrared light.

[0037] The display unit may also have an opening, and when looking down on the display unit from the normal direction of the display surface, the opening is located at a position overlapping with the imaging unit.

[0038] The opening portion is arranged to penetrate a layer having a light transmittance of less than a specified value for light of a specified wavelength band among a plurality of layers constituting the display portion.

[0039] The display portion may also be constituted by a plurality of layers having a light transmittance of a specified value or more for light in the wavelength band in which the imaging portion can perform photoelectric conversion.

[0040] The control portion may also include:

[0041] a first control portion that performs display control of the display portion; and

[0042] a second control portion that, in cooperation with the first control portion, performs imaging control of the imaging portion. Description of the Drawings

[0043] Figure 1A is a schematic cross-sectional view of the electronic device according to the first embodiment.

[0044] Figure 1B is Figure 1A a cross-sectional view in which an opening portion is provided.

[0045] Figure 2 (a) of Figure 1A is a schematic external view of the electronic device of Figure 2 and (b) of Figure 2 is a cross-sectional view taken along the line A-A of (a) of

[0046] Figure 3 is a cross-sectional view showing an example of the cross-sectional structure of the imaging portion 4.

[0047] Figure 4 is a block diagram showing the internal configuration of the electronic device according to the present embodiment.

[0048] Figure 5 is a flowchart showing the processing operation of the electronic device according to the first embodiment.

[0049] Figure 6 is Figure 5 an operation timing chart corresponding to the flowchart of

[0050] Figure 7 is a flowchart showing the processing operation of the electronic device according to the second embodiment.

[0051] Figure 8 is Figure 7 an operation timing chart corresponding to the flowchart of

[0052] Figure 9 is a flowchart showing the processing operation of the electronic device according to the third embodiment.

[0053] Figure 10 is the action timing diagram corresponding to the flowchart of Figure 9 .

[0054] Figure 11 is a diagram showing an example of the screen display of the display unit 2 displayed by the processing of the flowchart of Figure 10 .

[0055] Figure 12 is a flowchart showing the processing operations of the electronic device according to the fourth embodiment.

[0056] Figure 13 is the action timing diagram corresponding to the flowchart of Figure 12 .

[0057] Figure 14A is a schematic diagram showing a specific example of the second display area according to the fourth embodiment.

[0058] Figure 14B is a schematic diagram showing a specific example of the second display area according to the fourth embodiment.

[0059] Figure 14C is a schematic diagram showing a specific example of the second display area according to the fourth embodiment.

[0060] Figure 15 is a flowchart showing the processing operations of the electronic device according to the fifth embodiment.

[0061] Figure 16 is the action timing diagram corresponding to the flowchart of Figure 15 .

[0062] Figure 17A is a schematic diagram showing a specific example of the second display area according to the fifth embodiment.

[0063] Figure 17B is a schematic diagram showing a specific example of the second display area according to the fifth embodiment.

[0064] Figure 18 is a block diagram showing the internal configuration of the electronic device according to the sixth embodiment.

[0065] Figure 19 is a flowchart showing the processing operations of the electronic device according to the sixth embodiment.

[0066] Figure 20 is the action timing diagram corresponding to the flowchart of Figure 19 .

[0067] Figure 21 is a block diagram showing the internal configuration of the electronic device according to the seventh embodiment.

[0068] Figure 22 It is a flowchart showing the processing operations of the electronic device according to the seventh embodiment.

[0069] Figure 23 It is a diagram showing a cross-sectional structure of the imaging unit of the camera module according to the ninth embodiment.

[0070] Figure 24 It is a diagram showing a cross-sectional structure of the imaging unit in the tenth embodiment.

[0071] Figure 25 It is a perspective view showing an example of the detailed structure of each polarizing element 8b.

[0072] Figure 26 It is a block diagram showing the internal configuration of an electronic device having a function of suppressing the influence of glare and diffracted light.

[0073] Figure 27 It is a plan view when the electronic devices of the first to fourth embodiments are applied to a capsule endoscope.

[0074] Figure 28 It is a rear view when the electronic devices of the first to fourth embodiments are applied to a digital single-lens reflex camera.

[0075] Figure 29A It is a plan view showing an example when the electronic devices of the first to fourth embodiments are applied to an HMD.

[0076] Figure 29B It is a diagram showing an existing HMD.

[0077] Explanation of reference numerals:

[0078] 1 Electronic device; 2 Display unit; 3 Camera module; 4 Imaging unit; 4a Photoelectric conversion unit; 5 Optical system; 11 Semiconductor substrate; 12 Element separation layer; 13 Planarization layer; 14 Color filter layer; 15 On-chip lens; 16 Readout circuit; 17 Interlayer insulating film; 21 Imaging device; 22 Application processor; 23 Resolution conversion unit; 24 A / D conversion unit; 25 Display control unit; 31 A / D conversion unit; 32 Signal processing unit; 33 Imaging control unit; 34 Exposure adjustment unit. Detailed implementation manners

[0079] Hereinafter, embodiments of the electronic device will be described with reference to the drawings. Hereinafter, the description will be centered on the main components of the electronic device, but the electronic device may have components and functions that are not shown or described. The following description does not exclude components and functions that are not shown or described.

[0080] (First embodiment)

[0081] Figure 1A and Figure 1B is a schematic cross-sectional view of the electronic device 1 according to the first embodiment. Figure 1A The electronic device 1 is any electronic device 1 having both a display function and a photographing function, such as a smart phone, a mobile phone, a tablet, a PC, etc. Figure 1A The electronic device 1 is provided with a camera module (photographing unit), and the camera module is disposed on the side opposite to the display surface 1a of the display unit 2. Thus, Figure 1A the electronic device 1 has a camera module 3 provided inside the display surface 1a of the display unit 2. Therefore, the camera module 3 performs photographing through the display unit 2.

[0082] Figure 2 (a) of is Figure 1A a schematic external view of the electronic device 1, Figure 2 (b) of is Figure 2 a cross-sectional view in the direction of line A-A of (a) of. Figure 2 In the example of (a) of, the display surface 1a extends to be close to the outer shape size of the electronic device 1, and the width of the frame 1b around the display surface 1a is several millimeters or less. Usually, a front camera is mostly mounted on the frame 1b, Figure 2 In (a) of, as shown by the dashed line, the camera module 3 is disposed on the inner surface side of the substantially central portion of the display surface 1a to function as a front camera. Thus, by providing the front camera on the inner surface side of the display surface 1a, it is no longer necessary to provide a front camera on the frame 1b, and the width of the frame 1b can be made narrower.

[0083] In addition, in Figure 2 (a) of, the camera module 3 is disposed on the inner surface side of the substantially central portion of the display surface 1a, but in the present embodiment, it is sufficient to be disposed on the inner surface side of the display surface 1a. For example, the camera module 3 may also be disposed on the inner surface side near the peripheral edge portion of the display surface 1a. Thus, the camera module 3 of the present embodiment is disposed at an arbitrary position on the inner surface side overlapping with the display surface 1a.

[0084] As Figure 1AAs shown, the display unit 2 is a laminate in which a protective film 2b, a polyimide substrate 2c, a display layer 2d, a barrier layer 2e, a touch sensor layer 2f, an adhesive layer 2g, a circular polarizing plate 2h, an optical adhesive sheet (OCA: Optical Clear Adhesive) 2i, and a cover glass 2j are laminated in sequence. The display layer 2d can be, for example, an OLED (Organic Light Emitting Device) display layer, a liquid crystal display layer, a MicroLED, or a display layer based on other display principles. The display layer 2d can also be composed of multiple layers. For example, the display layer 2d can include a color filter layer, a backlight layer, etc. The display unit 2 performs display using light in the visible light wavelength range, but the light displayed by the display unit 2 can also include an infrared light component.

[0085] The barrier layer 2e is a layer that prevents oxygen and moisture from entering the display layer 2d. The touch sensor layer 2f incorporates a touch sensor. There are various types of touch sensors such as capacitive type and resistive film type, and any type can be adopted. In addition, the touch sensor layer 2f and the display layer 2d can also be integrated.

[0086] The adhesive layer 2g is provided to bond the circular polarizing plate 2h and the touch sensor layer 2f. The adhesive layer 2g is formed using a material with a high visible light transmittance. The circular polarizing plate 2h is provided to reduce glare or improve the visibility of the display surface 1a in a bright environment. The optical adhesive sheet 2i is provided to improve the adhesion between the circular polarizing plate 2h and the cover glass 2j. The optical adhesive sheet 2i uses a material with a high visible light transmittance. The cover glass 2j is provided to protect the display layer 2d, etc. In addition, the layer structure of the display unit 2 is not necessarily limited to Figure 1A , Figure 2 the structure shown.

[0087] The camera module 3 is disposed on the side opposite to the display surface 1a of the display unit 2, that is, inside the display unit 2. The camera module 3 has a photographing unit 4 and an optical system 5. The optical system 5 is disposed on the light incident surface side of the photographing unit 4, that is, closer to the display unit 2 side, and focuses the light passing through the display unit 2 on the photographing unit 4. The optical system 5 usually consists of multiple lenses. As described below, sometimes multiple camera modules 3 can be disposed on the side opposite to the display surface 1a of the display unit 2. In this case, the focal lengths of the optical systems 5 of the respective camera modules 3 can be different from each other, so that long-distance and wide-angle shooting can be performed from different perspectives.

[0088] The photographing unit 4 includes a photoelectric conversion unit 4a. The photoelectric conversion unit 4a performs photoelectric conversion on the light incident via the display unit 2. The photoelectric conversion unit 4a can be a CMOS (Complementary Metal Oxide Semiconductor) sensor, or a CCD (Charge Coupled Device) sensor. In addition, the photoelectric conversion unit 4a can be a photodiode, or an organic photoelectric conversion film.

[0089] The photoelectric conversion unit 4a has a photoelectric conversion element such as a CMOS sensor in each pixel. The pixels can be arranged in any manner. Specifically, the arrangement of the pixels can be a Bayer arrangement, an interline arrangement, a square arrangement, a striped arrangement, or other arrangements.

[0090] As Figure 1A and Figure 2 shown in (b) of Figure 1A , in the electronic device 1 of the present embodiment, the display unit 2 and the camera module 3 are overlapped and arranged in the front-back direction. Therefore, the camera module 3 photographs the subject light passing through the display unit 2. As Figure 1B shown, the display unit 2 is formed of multiple layers, and it is not a problem if each layer has a high transmittance for light in a wavelength band with sufficient sensitivity for the photographing unit 4. However, in practice, the transmittance of some layers may be low. For example, the visible light transmittance of the polyimide substrate 2c is not high. Thus, as Figure 1B shown, one or more opening portions 2k can be formed on the layer with low transmittance among the multiple layers constituting the display unit 2, and the camera module 3 photographs the light passing through the opening portions 2k. In the example of Figure 1B , a plurality of opening portions 2k are provided that penetrate through the layers of the display unit 2 except for the cover glass 2j. The opening portions 2k are provided at positions overlapping the camera module 3 when the display unit 2 is viewed from above. The diameter and number of each opening portion 2k can be set to appropriate values in consideration of the display of the display unit 2 and the image quality of the photographed image of the photographing unit 4.

[0091] In addition, when a highly transparent substrate is used instead of the polyimide substrate 2c, it is not necessarily required to provide the opening portions 2k.

[0092] Figure 3 is a cross-sectional view showing an example of the cross-sectional structure of the photographing unit 4. Figure 3The photographing unit 4 includes a photoelectric conversion unit 4a formed within the semiconductor substrate 11, and the photoelectric conversion unit 4a is separated by an element isolation layer 12 for each pixel. A planarization layer 13 is disposed on the photoelectric conversion unit 4a, and a color filter layer 14 is disposed on the planarization layer 13. The color filter layer 14 may have RGB three-color filter layers, or may have complementary color cyan, magenta, and red filter layers. Alternatively, it may have a filter layer that transmits light of colors other than visible light such as infrared light, a filter layer with multi-spectral characteristics, or a subtractive color filter layer such as white. By allowing light other than visible light such as infrared light to pass through, sensing information such as depth information can be detected. An on-chip lens 15 is disposed on the color filter layer 14. Light is incident through the on-chip lens 15. In this specification, the side where the on-chip lens 15 is disposed is referred to as the inner surface side of the photographing unit 4.

[0093] On the surface side of the photographing unit 4, a readout circuit 16 is formed on the semiconductor substrate 11, and the periphery of the readout circuit 16 is covered by an interlayer insulating film 17. The readout circuit 16 includes: transfer transistors, reset transistors, amplification transistors, selection transistors, etc. In addition, the cross-sectional structure of the photographing unit 4 is not limited to Figure 3 the structure shown.

[0094] Figure 4 is a block diagram showing the internal configuration of the electronic device 1 according to the present embodiment. As Figure 4 shown, the electronic device 1 includes: a photographing device 21, an application processor (first control unit) 22, a resolution conversion unit 23, an A / D conversion unit 24, a display control unit 25, and a display unit 2.

[0095] The photographing device 21 may be composed of one or more semiconductor devices, and in addition to having the photographing unit 4 (photoelectric conversion unit 4a), the optical system 5, and the IR (Infrared Ray) cut-off filter 6 that constitute the camera module 3, it also has an A / D conversion unit 31, a signal processing unit 32, a photographing control unit (second control unit) 33, and an exposure adjustment unit 34.

[0096] The A / D conversion unit 31 converts the analog pixel signal captured by the photographing unit 4 into digital pixel data. The signal processing unit 32 includes: a clamping unit 32a, a color output unit 32b, a defect correction unit 32c, a linear matrix unit 32d, a gamma correction unit 32e, a luminance-chrominance signal generation unit 32f, a noise reduction unit 32g, an edge enhancement unit 32h, and an output unit 32i.

[0097] The clamping unit 32a performs the process of defining the black level. More specifically, the clamping unit 32a performs the process of subtracting the black level data from the digital pixel data. The color output unit 32b outputs pixel data of each RGB color, for example. The defect correction unit 32c performs a correction process to correct the captured data of a specific pixel that has not been accurately read for some reason, based on the captured data of surrounding pixels. The linear matrix unit 32d performs matrix operations on color information such as RGB to achieve more accurate color reproduction. The gamma correction unit 32e performs gamma correction corresponding to the display characteristics of the display unit 2 to improve the visibility of the display. For example, the gamma correction unit 32e converts from 10 bits to 8 bits while changing the gradient. The luminance-chrominance signal generation unit 32f generates a luminance-chrominance signal for the display of the display unit 2 based on the output data of the gamma correction unit 32e. The noise reduction unit 32g performs a process of reducing the noise contained in the luminance-chrominance signal. The edge enhancement unit 32h performs an enhancement process to enhance the edges of the subject image based on the luminance-chrominance signal. The noise reduction process of the noise reduction unit 32g and the edge enhancement process of the edge enhancement unit 32h may also be performed only when a specified condition is satisfied. The output unit 32i outputs the luminance-chrominance signal after the noise reduction process.

[0098] The shooting control unit 33 sets the shooting frame rate of the shooting unit 4 based on the luminance-chrominance signal. The exposure adjustment unit 34 adjusts the exposure time according to the frame rate set by the shooting control unit 33. The photoelectric conversion unit 4a performs shooting for each pixel according to the exposure time adjusted by the exposure adjustment unit 34.

[0099] The application processor 22 is a semiconductor device separate from the camera module 3 and is mounted on the same substrate or a different substrate as the camera module 3. The inside of the application processor 22 has a CPU (Central Processing Unit) etc., and executes programs such as an operating system and various application software. The application processor 22 may also be equipped with functions for performing image processing, signal processing, etc. of a GPU (Graphics Processing Unit) or a baseband processor. The application processor 22 performs various processes on the input image data and operation results as needed, and controls the display of the image on the display unit 2 of the electronic device 1 or sends it to an external cloud server via a specified network.

[0100] Various control signals are transmitted and received between the application processor 22 and the shooting control unit 33. In addition, the application processor 22 receives the luminance-chrominance signal generated by the luminance-chrominance signal generation unit 32f and provides it to the display control unit 25. More specifically, the application processor 22 receives information on the shooting frame rate of the shooting unit 4 via the shooting control unit 33 and sets the frame rate when the display unit 2 displays an image.

[0101] The application processor 22 and the shooting control unit 33 may also be integrated on one chip. In this specification, the application processor 22 and the shooting control unit 33 are sometimes collectively referred to as the control unit. The control unit of the present embodiment synchronizes the display timing of the display unit 2 with the shooting timing of the shooting unit 4 so that the shooting unit 4 performs shooting when the display unit 2 is not displaying.

[0102] Compared with the case where the shooting unit 4 does not perform shooting, when the shooting unit 4 performs shooting, the control unit makes the period during which the display unit 2 does not display longer. In addition, compared with the case where the shooting unit 4 does not perform shooting, when the shooting unit 4 performs shooting, the control unit may also make the display frequency of at least a part of the pixels of the display unit 2 lower. In addition, compared with the case where the shooting unit 4 does not perform shooting, when the shooting unit 4 performs shooting, the control unit may also make the frame rate of the display unit 2 lower. In addition, compared with the case where the shooting unit 4 does not perform shooting, when the shooting unit 4 performs shooting, the control unit may also make the display frequency of at least a part of the horizontal pixel lines of the display unit 2 lower. In addition, compared with the case where the shooting unit 4 does not perform shooting, when the shooting unit 4 performs shooting, the control unit may also make the display frequency of a part of the horizontal pixel lines included in the display area overlapping with the shooting unit 4 lower when observing the display unit 2 from the normal direction of the display surface 1a. In addition, the control unit may also set the exposure time of the shooting unit 4 based on the period during which the display unit 2 does not display. In addition, compared with the case where the shooting unit 4 does not perform shooting, when the shooting unit 4 performs shooting, the control unit may also make the display pixel density in at least a part of the display area of the display unit 2 lower. In addition, compared with the case where the shooting unit 4 does not perform shooting, when the shooting unit 4 performs shooting, the control unit may also make the display pixel density in a part of the display area including the display area overlapping with the shooting unit 4 lower when observing the display unit 2 from the normal direction of the display surface 1a. In addition, compared with the case where the shooting unit 4 does not perform shooting, when the shooting unit 4 performs shooting, the control unit may also make the brightness value of the pixels in at least a part of the display area of the display unit 2 higher. In addition, compared with the case where the shooting unit 4 does not perform shooting, when the shooting unit 4 performs shooting, the control unit may also make the brightness value of a part of the display area including the display area overlapping with the shooting unit 4 higher when observing the display unit 2 from the normal direction of the display surface 1a. In addition, compared with the case where the shooting unit 4 does not perform shooting, when the shooting unit 4 performs shooting, the control unit may also make the display frequency of at least a part of the pixels of the display unit 2 lower and the brightness value higher.

[0103] The resolution conversion unit 23 converts the resolution of the video signal to be displayed on the display unit 2. The A / D conversion unit 31 converts the video signal whose resolution has been converted by the resolution conversion unit 23 into digital video data. The display control unit 25 controls the digital video data to be displayed on the display unit 2 at the frame rate indicated by the application processor 22.

[0104] Figure 5 is a flowchart showing the processing operations of the electronic device 1 according to the first embodiment, for example, showing the processing operations executed by the application processor 22 and the shooting control unit 33. During the period when the electronic device 1 is powered on, the following flowchart is continuously repeated. Figure 5 of the flowchart.

[0105] First, it is determined whether the user of the electronic device 1 has activated the camera module 3 (step S1). If the camera module 3 is not activated, the normal frame rate is selected (step S2). The normal frame rate is a predetermined frame rate such as 60 Hz, for example. In addition, the frame rate is the frequency required to display the entire display area of the display unit 2 once. Next, the display of the display unit 2 is started at the normal frame rate (step S3).

[0106] On the other hand, when it is determined in step S1 that the camera module 3 has been activated, the low frame rate is selected (step S4). The low frame rate is a frame rate lower than the normal frame rate, such as 30 Hz, for example. Figure 5 In the example of, the low frame rate is set to a fixed frequency, but the frequency of the low frame rate can also be changed. Next, the display of the display unit 2 is started at the low frame rate (step S5).

[0107] When the display of the display unit 2 is performed at a fixed frame rate, the length of the period during which the display unit 2 does not perform display (hereinafter referred to as the display off period) varies according to the frequency of the frame rate. The faster the frame rate, the shorter the display off period. The display off period is set to the period from the end of the display of one horizontal pixel line to the start of the display of the next horizontal pixel line, or from the end of the display of one frame to the start of the display of the next frame.

[0108] After the display is started in step S5, after the display off period, the imaging unit 4 performs exposure during the display off period (step S6). The exposure time of the imaging unit 4 is determined according to the length of the display off period. Since the lower the frame rate, the longer the display off period, the exposure time can be extended when the display is performed at the low frame rate.

[0109] Figure 6 is related to Figure 5 of the flowchart. Figure 6 The waveform w1 of is a signal waveform representing the display period of the normal frame rate, and the waveform w2 is a signal waveform representing the display period of the low frame rate. The high (High) period of the waveforms w1 and w2 represents the display period, and the low (Low) period represents the display off period. The waveform w3 is a signal waveform representing the exposure period, and the high (High) period is the exposure period. As shown in the waveforms w1 to w3, compared with the normal frame rate, the display off period is longer at the low frame rate, so sufficient exposure time can be ensured.

[0110] In the first embodiment, as Figure 5 and Figure 6As shown, the imaging unit 4 performs imaging during the display-off period, so the image quality of the captured image will not be degraded due to the display brightness of the display unit 2. In addition, during imaging, it is displayed at a low frame rate, so the display-off period can be extended, ensuring that the exposure time during imaging is long enough and preventing the captured image from becoming dark due to insufficient exposure time. That is, the imaging sensitivity can be improved.

[0111] (Second Embodiment)

[0112] As shown in the first embodiment, if the frame rate of the display unit 2 is reduced when the camera module 3 is started, since the brightness perceived by the human eye is the averaged brightness, the display appears darker than normal. Therefore, in the second embodiment, the brightness value of the pixel data is increased when the camera module 3 is started.

[0113] The electronic device 1 according to the second embodiment has the same internal configuration as Figure 4 the same. Figure 7 is a flowchart showing the processing operations of the electronic device 1 according to the second embodiment. Hereinafter, the description will focus on aspects different from the Figure 5 flowchart. Figure 7 In this case, when the camera module 3 is not started (NO in step S11), after the process of selecting the normal frame rate (step S12), the normal brightness value is set (step S13). The normal brightness value is displayed according to the original brightness value of the pixel data. Next, pixel data with the set display brightness is provided to the display unit 2 at the normal frame rate, and the display is started (step S14).

[0114] On the other hand, when the camera module 3 is started (YES in step S11), after the process of selecting the low frame rate (step S15), a brightness value higher than normal is set (step S16). The brightness value higher than normal means that the brightness value of the pixel data is adjusted so that the display brightness seen by the human eye is the same as the display brightness when the display unit 2 is displayed at the normal frame rate when the camera module 3 is not started. More specifically, in step S16, the brightness value of the pixel data is made greater than the brightness value set in step S13.

[0115] Next, the display is started with the brightness value set in step S16 (step S17). By increasing the brightness value of the pixel data, the display brightness seen by the human eye is the same as the display brightness when the normal frame is selected even when the frame rate is low. Then, during the display-off period, the imaging unit 4 performs imaging with an exposure time corresponding to the length of the display-off period (step S18).

[0116] Figure 8 is related to Figure 7The timing diagram corresponding to the flowchart. Waveforms w4 to w6 are the same as waveforms w1 to w3. Waveform w7 schematically represents the luminance value of the pixel data displayed on the display unit 2 when the camera module 3 is not activated. Waveform w8 schematically represents the luminance value of the pixel data displayed on the display unit 2 when the camera module 3 is activated.

[0117] From waveforms w7 and w8, it can be seen that when a low frame rate is selected, by increasing the luminance value of the pixel data, the display luminance visible to the human eye can be made equal to the display luminance when the normal frame rate is selected.

[0118] Thus, in the second embodiment, when the camera module 3 is activated, the display unit 2 not only displays at a low frame rate, but also the luminance value of the pixel data is higher than that when the normal frame rate is selected. Therefore, when the camera module 3 is activated, the display of the display unit 2 does not become darker or the display is not coordinated. In addition, as in the first embodiment, since the imaging unit 4 performs imaging during the display off period, the imaging can be performed without being affected by the display of the display unit 2 and with sufficient time.

[0119] (Third Embodiment)

[0120] In the third embodiment, when the camera module 3 is activated, not only the frame rate is reduced, but also the display pixel density is controlled. The electronic device 1 according to the third embodiment has the same internal configuration as Figure 4 the same.

[0121] Figure 9 is a flowchart showing the processing operations of the electronic device 1 according to the third embodiment. Hereinafter, the description will be centered on aspects different from Figure 7 The processing of steps S21 to S24 is the same as the steps S11 to S14 of Figure 7 the same.

[0122] When the camera module 3 is activated, the frame rate and the display pixel density are set (step S25). Here, a low frame rate lower than the normal frame rate is set. In addition, for the display pixel density, a display pixel density lower than the display pixel density when the camera module 3 is not activated is also set. As a more specific example, when the camera module 3 is activated, horizontal pixel lines are selected to be driven in a way of skipping one or two lines, thereby reducing the display pixel density. As a result, the display off period from displaying one horizontal pixel line to displaying the next horizontal pixel line can be extended. Alternatively, multiple pixels on the horizontal pixel line can be driven at intervals. In this case, since the driving intervals of the respective pixels on the horizontal pixel line are large, a display off period can be set within the intervals and the imaging unit 4 can perform imaging.

[0123] Next, based on the set frame rate and display pixel density, the brightness value of the pixel data is set (step S26). The higher the display pixel density, the higher the display brightness seen by the human eye. In addition, the lower the frame rate, the lower the display brightness seen by the human eye. Therefore, in step S26, the brightness value of the pixel data is set considering both the set frame rate and display pixel density to be equal to the display brightness of the display unit 2 when the camera module 3 is not activated.

[0124] Next, display starts with the set brightness value (step S27). Next, during the display off period, the imaging unit 4 performs imaging with an exposure time corresponding to the length of the display off period (step S28).

[0125] Figure 10 is corresponding to Figure 9 is the operation timing chart corresponding to the flowchart. Waveform w9 schematically represents the waveform of the vertical synchronization signal VSYNC at the normal frame rate, waveform w10 schematically represents the waveform of the horizontal synchronization signal HSYNC at the normal frame rate, and waveform w11 schematically represents the waveform of the brightness value of the pixel data displayed on the display unit 2 at the normal frame rate. The high period of the horizontal synchronization signal HSYNC of waveform w10 represents the horizontal pixel line display period. Waveform w12 schematically represents the waveform of the vertical synchronization signal VSYNC at the low frame rate, waveform w13 schematically represents the waveform of the horizontal synchronization signal HSYNC at the low frame rate, waveform w14 schematically represents the waveform of the exposure time for the imaging unit 4 to perform imaging, and waveform w15 schematically represents the waveform of the brightness value of the pixel data displayed on the display unit 2 at the low frame rate.

[0126] As Figure 10 shown, when the camera module 3 is activated, a low frame rate lower than the normal frame rate is selected, and the display pixel density is also reduced. Therefore, the display off period can be further extended, and thus the exposure time when the imaging unit 4 performs imaging can be further extended.

[0127] Figure 11 is a diagram showing Figure 10 the screen display example of the display unit 2 displayed through the processing of the flowchart. In the present embodiment, when the camera module 3 is activated, the frame rate and display pixel density are set for each frame. Therefore, as Figure 11 shown, the entire display area of the display unit 2 is displayed with a display pixel density different from that when the camera module 3 is not activated. However, by making the brightness value of the pixel data when the camera module 3 is activated greater than the brightness value of the pixel data when the camera module 3 is not activated, the display brightness seen by the human eye can be made equal to that when the camera module 3 is not activated.

[0128] Thus, in the third embodiment, when the camera module 3 is activated, the frame rate and the display pixel density are set in units of frames. Therefore, shooting can be performed in a state where the frame rate and the display pixel density in all display areas of the display unit 2 are reduced. By reducing the frame rate and the display pixel density, the exposure time during shooting by the shooting unit 4 can be extended, thereby improving the shooting sensitivity.

[0129] (Fourth Embodiment)

[0130] In the fourth embodiment, the display pixel density is controlled for a part of the horizontal pixel lines.

[0131] The electronic device 1 according to the fourth embodiment has the same internal configuration as Figure 4 However, the display unit 2 in the electronic device 1 has a first display area 2r1 and a second display area 2r2. The first display area 2r1 is an area for displaying at a normal frame rate and a normal display pixel density. The second display area 2r2 is an area for displaying at a low frame rate and a display pixel density lower than normal.

[0132] When the display unit 2 is viewed from above in the normal direction of the display surface 1a, the second display area 2r2 is a display area including the display area overlapping with the camera module 3. The second display area 2r2 of the present embodiment has a strip-shaped area that is an integer multiple of the horizontal pixel lines of the display unit 2. At least a part of the second display area 2r2 overlaps with the arrangement position of the shooting unit 4 in the front-rear direction. By reducing the display pixel density in the second display area 2r2, the display off period of the second display area 2r2 can be set long.

[0133] Figure 12 is a flowchart showing the processing operations of the electronic device 1 according to the fourth embodiment, and shows, for example, the processing operations executed by the application processor 22 and the shooting control unit 33. Hereinafter, the description will focus on aspects different from the Figure 9 flowchart. Figure 12 Steps S31 to S34 of Figure 9 perform the same processing as steps S21 to S24 of Figure 9 When the camera module 3 is activated, the display pixel density of a part of the horizontal pixel lines within one frame is set (step S35). The part of the horizontal pixel lines refers to the horizontal pixel lines included in the second display area 2r2 described above. The number of horizontal pixel lines is not limited, and considering the size of the camera module 3, a plurality of horizontal pixel lines are included in the second display area 2r2. The display pixel density set in step S35 is set, for example, to a value smaller than the display pixel density when the camera module 3 is not activated. More specifically, by driving the plurality of horizontal pixel lines in the second display area 2r2 at intervals, the display pixel density is reduced. By reducing the display pixel density, the display off period of the second display area 2r2 can be extended, thereby setting the exposure time long.

[0134] Next, based on the display pixel density within the second display area 2r2, the brightness value of the pixel data is set (step S36). If the display pixel density decreases, the display becomes darker. Therefore, the brightness value of the pixel data is increased so that the display brightness seen by the human eye is the same as the display brightness when the camera module 3 is not activated. Next, when starting to display a part of the horizontal pixel lines, that is, when displaying the second display area 2r2, the display is performed with the set display pixel density and brightness value (step S37).

[0135] Next, during the period when the display of the second display area 2r2 is turned off, the imaging unit 4 performs imaging with the exposure time determined according to the display pixel density in step S35 (step S38).

[0136] Figure 13 is related to Figure 12 The timing chart corresponding to the flowchart. Waveform w16 schematically represents the waveform of the vertical synchronization signal VSYNC, waveform w17 schematically represents the waveform of the horizontal synchronization signal HSYNC, waveform w18 schematically represents the waveform during the exposure period, and waveform w19 schematically represents the waveform of the brightness value of the pixel data displayed on the display unit 2. The high period of waveform w17 is the display period of each horizontal pixel line.

[0137] In Figure 13 In the example, the display pixel density is adjusted by making only the display period of a part of the horizontal pixel lines within one frame longer than the display periods of other horizontal pixel lines. That is, the horizontal pixel lines within the second display area 2r2 are driven in a way that skips one or two lines to reduce the display pixel density. In addition, within the display period of the horizontal pixel lines in the second display area 2r2, the brightness value of the pixel data is increased.

[0138] Figure 14A , Figure 14B , Figure 14C are schematic diagrams showing specific examples of the second display area 2r2. Figure 14A Shows an example where the camera module 3 is located on the upper end side of the display unit 2, and the second display area 2r2 is set to include the configuration position of the camera module 3 and is in a strip shape along the horizontal direction. Figure 14B Shows an example where Figure 14A the camera module 3 is closer to the center of the display unit 2, and the second display area 2r2 is set to include the configuration position of the camera module 3 and is in a strip shape along the horizontal direction. Figure 14C In

[0139] Thus, in the fourth embodiment, the display pixel density can be adjusted in units of the horizontal pixel lines of the display unit 2. Therefore, when the camera module 3 is activated, the display pixel density of the second display area 2r2, where the configuration position corresponding to the camera module 3 extends in a strip shape in the horizontal direction, can be adjusted, and the exposure time during shooting by the shooting unit 4 can be extended during the period when the display of the second display area 2r2 is turned off.

[0140] (Fifth Embodiment)

[0141] The electronic device 1 according to the fifth embodiment adjusts the display pixel density of the display area according to the outer dimensions of the camera module 3 disposed on the inner surface side of the display unit 2.

[0142] The electronic device 1 according to the fifth embodiment has the same internal configuration as Figure 4 Furthermore, similarly to the fourth embodiment, the display unit 2 has a first display area 2r1 and a second display area 2r2. The second display area 2r2 in the fourth embodiment is an area including a plurality of horizontal pixel lines. In contrast, the second display area 2r2 in this embodiment is a pixel block area including a plurality of pixels arranged adjacent to each other in the horizontal and vertical directions. A part of the pixels among the plurality of horizontal pixel lines is included in the second display area 2r2 in this embodiment.

[0143] Figure 15 is a flowchart showing the processing operation of the electronic device 1 according to the fifth embodiment, for example, showing the processing operation executed by the application processor 22. Hereinafter, the description will focus on aspects different from the Figure 12 flowchart. Figure 15 In steps S41 to S44 of, the same processing as steps S31 to S34 of Figure 12 is executed. When the camera module 3 is activated, the display pixel density of a part of the display area, that is, the second display area 2r2, is set (step S45). Here, the display pixel density can be determined in advance, or can be arbitrarily set according to the surrounding brightness or the like as described below.

[0144] Next, based on the display pixel density set in step S45, the brightness value of the pixel data is set so that the display brightness seen by the human eye is the same as when the camera module 3 is not activated (step S46).

[0145] Next, the display is started, and when the second display area 2r2 is being displayed, the display is performed with the set display pixel density and brightness value (step S47). In this embodiment, even when the camera module 3 is activated, when the first display area 2r1 is being displayed, the display is performed with the same display pixel density and brightness value as when the camera module 3 is not activated. On the other hand, when the second display area 2r2 is being displayed, the display is performed with the display pixel density and brightness value set in steps S45 and S46.

[0146] Next, during the display-off period in which the second display area 2r2 is displayed, the image capturing unit 4 captures the image (step S48). The exposure time at this time is a time corresponding to the length of the display-off period.

[0147] Figure 16 is with Figure 15 The waveform w20 is a waveform representing the horizontal synchronization signal HSYNC, the waveform w21 is a waveform representing the display period of the horizontal pixel line when the camera module 3 is not started, the waveform w22 is a waveform representing the display period of the horizontal pixel line when the camera module 3 is started, the waveform w23 is a waveform representing the exposure period when the camera module 3 is started, and the waveform w24 is a waveform representing the brightness value of the pixel data.

[0148] like Figure 16 As shown, in this embodiment, when displaying the second display area 2r2 arranged at a position overlapping with the arrangement position of the camera module 3, the display pixel density is reduced and the display off period is extended. Thus, the imaging unit 4 can perform imaging during the display off period.

[0149] Figure 17A and Figure 17B 2 is a schematic diagram showing a specific example of the second display area 2r2 when the camera module 3 is activated. Figure 17A FIG. 2 shows an example in which the camera module 3 is arranged on the upper end side of the display unit 2. Figure 17B FIG. 2 shows an example in which the camera module 3 is arranged close to the center of the display unit 2. Figure 17A and Figure 17B In the case of the second display area 2r2 being set according to the shape of the camera module 3, the display pixel density is reduced when the second display area 2r2 is displayed. By adjusting the display brightness, the difference in display brightness seen by human eyes between the second display area 2r2 and the surrounding first display area 2r1 can be suppressed.

[0150] Thus, in the fifth embodiment, the second display area 2r2 is set at a position overlapping with the configuration position of the camera module 3, and the display pixel density and the brightness value of the pixel data of the second display area 2r2 are adjusted, so that the shooting unit 4 can shoot without affecting the display of the display unit 2, thereby improving the image quality of the shot image. In addition, compared with the fourth embodiment, the area of the second display area 2r2 can be reduced, so that the second display area 2r2 becomes less noticeable and the user does not feel that the display is inconsistent.

[0151] (Sixth Embodiment)

[0152] In the sixth embodiment, the frame rate is controlled according to the surrounding brightness.

[0153] Figure 18 is a block diagram showing the internal configuration of the electronic device 1 according to the sixth embodiment. Figure 18 In addition to having Figure 4 the internal configuration of the electronic device 1, the electronic device 1 further includes a brightness detection unit 26. The brightness detection unit 26 detects, for example, the average brightness of the light incident on the photoelectric conversion unit 4a based on the digital pixel data output by the A / D conversion unit 31, or the brightness based on the integrated value of the incident light. The brightness detection unit 26 can detect the brightness around the electronic device 1. Alternatively, the brightness detection unit 26 may be composed of a sensor that separately detects brightness outside the imaging unit 4.

[0154] The shooting control unit 33 and the application processor 22 set the frame rate of the display unit 2 and the exposure time of the shooting unit 4 based on the brightness information detected by the brightness detection unit 26. For example, the darker the surroundings, the lower the frame rate of the display unit 2 is set. Thus, the display off period can be extended, and the exposure time can be extended accordingly, so that the shooting sensitivity of the shooting unit 4 can be improved.

[0155] Figure 19 is a flowchart showing the processing operation of the electronic device 1 according to the sixth embodiment. Next, a description will be given centering on aspects different from Figure 9 the flowchart. Figure 19 In steps S51 to S54 of Figure 9 the same processing as steps S21 to S24 of

[0156] Figure 20 is performed. When the camera module 3 is started, the brightness around is detected by the brightness detection unit 26 (step S55). Next, the frame rate and the exposure time are set based on the detected brightness around (step S56). Next, the display starts at the set frame rate (step S57). Then, during the display off period, the shooting unit 4 shoots at the set exposure time (step S58). Figure 19 is an operation timing diagram corresponding to the flowchart of

[0157] Comparing Figure 20 with Figure 10 it can be seen that the display brightness is not adjusted in this embodiment. This is because, compared with the visibility of the display unit 2, the shooting sensitivity of the shooting unit 4 is given priority.

[0158] In Figure 19 the flowchart, it is assumed that as Figure 11changing the frame rate of all the display areas of the display unit 2 according to the surrounding brightness, but it is also possible to Figure 14A - Figure 14C as shown in Figure 17A - Figure 17B display in the second display area 2r2 including only a part of the horizontal pixel lines, or as shown in

[0159] display in the second display area 2r2 including only a part of the pixel blocks with a display pixel density corresponding to the surrounding brightness.

[0160] (Seventh Embodiment)

[0161] In the seventh embodiment, the blink of a person watching the display unit 2 of the electronic device 1 is detected, and the display of the display unit 2 is turned off during the closed-eye period, and the imaging unit 4 performs imaging.

[0162] Figure 21 is a block diagram showing the internal configuration of the electronic device 1 according to the seventh embodiment. Figure 21 The electronic device 1 includes: an imaging device 21, a timing detection unit 27, a resolution conversion unit 23, an A / D conversion unit 31, a display control unit 25, and a display unit 2. Figure 21 In Figure 4 the application processor 22, the imaging control unit 33, and the exposure adjustment unit 34 are omitted, but the application processor 22, the imaging control unit 33, and the exposure adjustment unit 34 may also be provided.

[0163] Based on the digital pixel data output from the A / D conversion unit 31, the timing detection unit 27 detects the blink timing of a person reflected in the captured image. Then, the timing detection unit 27 notifies the display control unit 25 of the information indicating the closed-eye timing and the open-eye timing of the person. The display control unit 25 performs control to turn off the display of the display unit 2 during the closed-eye period of the person and turn on the display of the display unit 2 during the open-eye period of the person.

[0164] Figure 22 is a flowchart showing the processing operation of the electronic device 1 according to the seventh embodiment. When the camera module 3 is not activated (No in step S61), the display starts at the normal frame rate (step S62). On the other hand, when the camera module 3 is activated (Yes in step S61), the display starts at the normal frame rate (step S63). Then, it is determined whether the person is blinking based on the information of the timing detection unit 27 (step S64). The processing of steps S63 and S64 is repeatedly executed until the person closes their eyes.

[0165] If it is determined that the person has closed eyes, the display of the display unit 2 is turned off (step S65), and the exposure of the imaging unit 4 is started (step S66). Then, it is determined whether the person has opened eyes based on the information of the timing detection unit 27 (step S67). If it is determined that the person has opened eyes, the exposure of the imaging unit 4 is terminated (step S68).

[0166] Thus, in the seventh embodiment, the blinking of a person is detected, the display of the display unit 2 is turned off during the period when the person closes his eyes, and the image capturing unit 4 is performed on the basis of setting the exposure time according to the period when the person closes his eyes. Thus, the image capturing unit 4 can be performed with the display of the display unit 2 turned off, thereby improving the image quality of the captured image. In addition, since the display of the display unit 2 is turned off during the period when the person's eyes are closed, the person's eyes cannot see the turning off of the display, so that the person will not feel the discomfort.

[0167] (Eighth Embodiment)

[0168] In the fourth and fifth embodiments described above, examples of setting the first display area 2r1 and the second display area 2r2 in the display surface 1a of the display unit 2 are shown, but the display surface 1a of the first display area 2r1 (hereinafter referred to as the first display surface) and the display surface 1a of the second display area 2r2 (hereinafter referred to as the second display surface) can also be set in different layers. The layer of the first display surface and the layer of the second display surface are overlapped and stacked in the front and back directions. When the layer of the first display surface and the layer of the second display surface are stacked, each display area is configured in a manner that the respective display areas do not overlap. Therefore, when the line of sight is directed toward the display unit 2, the first display area 2r1 and the second display area 2r2 are integrated, so that the observer does not feel incongruous.

[0169] Thus, by setting the first display area 2r1 and the second display area 2r2 on different display surfaces 1a, the display control of the first display area 2r1 and the display control of the second display area 2r2 can be performed separately, thereby facilitating the processing operation of the display control unit 25.

[0170] (Ninth Embodiment)

[0171] The electronic device 1 according to the ninth embodiment is different from the first to eighth embodiments in the optical system 5 of the camera module 3 .

[0172] Figure 23 It is a diagram showing a cross-sectional structure of an imaging unit 4 of a camera module 3 mounted in an electronic device 1 according to a ninth embodiment. Figure 23 The imaging unit 4 has a microlens array 64 instead of a single lens or a lens group in which single lenses are arranged along the optical axis direction.

[0173] In more detail, Figure 23The photographing unit 4 has: a photoelectric conversion unit 4a disposed along the bottom surface of the housing 63; a microlens array 64 disposed above the photoelectric conversion unit 4a; a plurality of light-shielding bodies 66 disposed between adjacent microlenses 65; and a light guide plate 67 disposed above the microlens array 64. Figure 23 The photographing unit 4 can also be applied to any one of the first to eighth embodiments described above.

[0174] In this way, by providing the microlens array 64 as the optical system 5 of the photographing unit 4, the influence of adjacent pixels can be prevented, and color bleeding can be reduced.

[0175] (Tenth Embodiment)

[0176] A part of the light passing through the display unit 2 is reflected or diffracted and incident on the camera, so it is affected by reflection glare or diffraction, and the image quality of the captured image may be reduced. Therefore, a polarizing element 8b can also be provided in the photographing unit 4 of the first to ninth embodiments described above to obtain polarization information including components of glare and diffracted light, and use this polarization information to generate digital pixel data after removing the components of glare and diffracted light.

[0177] The cross-sectional structure of the photographing unit 4 at this time is represented by, for example, Figure 24 the cross-sectional view shown. Figure 24 In, the same reference numerals are assigned to the parts common to Figure 3 and the following description will focus on the differences. Figure 13 In the photographing unit 4 of, a light-shielding layer 18 and a base insulating layer 19 are disposed on the planarization layer 13, and a plurality of separated polarizing elements 8b are disposed on the base insulating layer 19. The polarizing element 8b is, for example, a wire grid polarizing element 8b having a line-and-space structure disposed on a part of the base insulating layer 19.

[0178] Figure 25 is a perspective view showing an example of the detailed structure of each polarizing element 8b. As Figure 25 shown, each of the plurality of polarizing elements 8b has a plurality of convex-shaped line portions 8d extending in one direction and a space portion 8e between the respective line portions 8d. There are various types of polarizing elements 8b, and the extending directions of the line portions 8d are different from each other. More specifically, in the polarizing element 8b, there are three or more types. For example, the angles formed by the arrangement direction of the photoelectric conversion unit 4a and the extending direction of the line portion 8d can be 0 degrees, 60 degrees, and 120 degrees. Or, the angles formed by the arrangement direction of the photoelectric conversion unit 4a and the extending direction of the line portion 8d can be 0 degrees, 45 degrees, 90 degrees, and 135 degrees, or other angles. Or, the plurality of polarizing elements 8b can also be unidirectionally polarized. The material of the plurality of polarizing elements 8b can be a metal material such as aluminum or tungsten or an organic photoelectric conversion film.

[0179] A plurality of polarizing elements 8b are arranged on the light incident side of at least one of the plurality of photoelectric conversion units 4a. The polarizing element 8b polarizes the light that passes through the display unit 2 and is incident. The light polarized by the polarizing element 8b is incident on the corresponding photoelectric conversion unit 4a for photoelectric conversion. Hereinafter, the output value of the photoelectric conversion unit 4a that performs photoelectric conversion on the light polarized by the polarizing element 8b is referred to as polarization information, and the output value of the photoelectric conversion unit 4a that performs photoelectric conversion on the light that is incident without passing through the polarizing element 8b is referred to as a pixel value or pixel information. In addition, hereinafter, the polarizing element 8b and the photoelectric conversion unit 4a that performs photoelectric conversion on the light polarized by the polarizing element 8b are collectively referred to as polarized pixels, and the photoelectric conversion unit 4a that performs photoelectric conversion on the light that is incident without passing through the polarizing element 8b is referred to as non-polarized pixels.

[0180] Return to Figure 24 , a planarization layer 20 is arranged on the insulating layer 19 on which a plurality of polarizing elements 8b are arranged. A color filter layer 14 is arranged on the planarization layer 20.

[0181] Figure 26 It is a block diagram showing the internal structure of an electronic device having a function of suppressing the influence of glare and diffracted light. Figure 26 Among them, the same reference numerals are used for the parts common to Figure 4 , and the following description will focus on the differences. Figure 26 In addition to having Figure 4 the internal structure of, the electronic device also has a polarization output unit 32j, a glare extraction unit 32k, and a glare correction signal generation unit 32m.

[0182] The polarization output unit 32j outputs polarization information data. The glare extraction unit 32k extracts at least one of a glare component and a diffracted light component from the polarization information data. At least one of the glare component and the diffracted light component extracted by the glare extraction unit 32k is a correction amount. The glare correction signal generation unit 32m subtracts the correction amount extracted by the glare extraction unit 32k from the digital pixel data output by the color output unit 32b, and corrects the digital pixel data through subtraction processing. The output data of the glare correction signal generation unit 32m is digital pixel data from which at least one of the glare component and the diffracted light component has been removed. In this way, the glare correction signal generation unit 32m functions as a correction unit, and corrects the captured image photoelectrically converted by a plurality of non-polarized pixels based on the polarization information.

[0183] The signal level of the digital pixel data at the pixel position of the polarized pixel will decrease after passing through the polarizing element 8b. Therefore, the defect correction unit 32c regards the polarized pixel as a defect and performs a prescribed defect correction process.

[0184] As Figure 24 - Figure 26As shown, by obtaining polarization information through a plurality of polarization elements 8b, it is possible to simply and reliably remove the glare component and diffraction light component contained in the light incident on the plurality of non-polarizing pixels after multiple reflections in the display unit 2, and generate a captured image in this state.

[0185] (Eleventh Embodiment)

[0186] As a specific candidate structure of the electronic device 1 having the configurations described in the first to fourth embodiments above, various structures can be considered. For example, Figure 27 is a plan view when the electronic device 1 of the first to fourth embodiments above is applied to the capsule endoscope 50. Figure 27 The capsule endoscope 50, for example, includes, in a housing 51 having hemispherical end faces and a cylindrical central portion: a camera (ultra-small camera) 52 for capturing an image of the body cavity; a memory 53 for recording the image data captured by the camera 52; and a wireless transmitter 55 that, after the capsule endoscope 50 is discharged from the subject's body, sends the recorded image data to the outside through an antenna 54.

[0187] In addition, a CPU (Central Processing Unit) 56 and a coil (magnetic force or current conversion coil) 57 are provided in the housing 51. The CPU 56 controls the shooting of the camera 52 and the data storage operation to the memory 53, and controls the transmission of data from the memory 53 to a data receiving device (not shown) outside the housing 51 through the wireless transmitter 55. The coil 57 supplies power to the camera 52, the memory 53, the wireless transmitter 55, the antenna 54, and a light source 52b described later.

[0188] Furthermore, a magnetic (wire) switch 58 is provided on the housing 51 for detecting when the capsule endoscope 50 is placed on the data receiving device. The CPU 56 supplies power from the coil 57 to the wireless transmitter 55 at the time point when the wire switch 58 detects that it is placed on the data receiving device and can send data.

[0189] The camera 52, for example, has an imaging element 52a including an optical system 9 for capturing an image of the body cavity and a plurality of light sources 52b for illuminating the body cavity. Specifically, the camera 52 is, for example, composed of a CMOS (Complementary Metal Oxide Semiconductor) sensor, a CCD (Charge Coupled Device), etc. that use an LED (Light Emitting Diode) as the light source 52b.

[0190] The display unit 2 of the electronic device 1 of the first to fourth embodiments includes Figure 27the concept of a light-emitting body such as the light source 52b. Figure 27 In the capsule endoscope 50, for example, there are two light sources 52b, and these light sources 52b can be constituted by a display panel having a plurality of light source parts or an LED module having a plurality of LEDs. In this case, by disposing the photographing part 4 of the camera 52 below the display panel and the LED module, the layout configuration restrictions of the camera 52 can be reduced, and thus a smaller capsule endoscope 50 can be realized.

[0191] In addition, Figure 28 is a rear view when the electronic device 1 of the first to fourth embodiments is applied to a digital single-lens reflex camera 60. The digital single-lens reflex camera 60 and the compact camera have a display part 2 for displaying a preview image on the back surface on the side opposite to the lens. A camera module 3 can be disposed on the side opposite to the display surface of the display part 2, and a face image of the photographer can be displayed on the display surface 1a of the display part 2. In the electronic device 1 according to the first to fourth embodiments, since the camera module 3 can be disposed in the area overlapping with the display part 2, there is no need to provide the camera module 3 in the frame part of the display part 2, and the size of the display part 2 can be made as large as possible.

[0192] Figure 29A is a plan view showing an example in which the electronic device 1 of the first to fourth embodiments is applied to a head-mounted display (hereinafter referred to as HMD) 61. Figure 29A The HMD 61 can be used for VR (Virtual Reality), AR (Augmented Reality), MR (Mixed Reality), or SR (Substitutional Reality), etc. As shown in Figure 29B a camera 62 is mounted on the outer surface, and the wearer of the HMD can see the surrounding images. On the other hand, people around do not know the eyes and facial expressions of the wearer of the HMD.

[0193] Therefore, Figure 29A a display surface of the display part 2 is provided on the outer surface of the HMD 61, and a camera module 3 is provided on the side opposite to the display surface of the display part 2. Thereby, the facial expression of the wearer photographed by the camera module 3 can be displayed on the display surface of the display part 2, and people around the wearer can grasp the facial expression and eye movement of the wearer in real time.

[0194] In Figure 29AIn the example, the camera module 3 is provided on the inner surface side of the display unit 2. Therefore, the installation position of the camera module 3 is not restricted, which can increase the design freedom of the HMD 61. In addition, since the camera can be arranged at the optimal position, it is possible to prevent problems such as the misalignment of the line of sight of the wearer displayed on the display surface.

[0195] Thus, in the eleventh embodiment, the electronic device 1 according to the first to fourth embodiments can be used for various purposes, thereby improving the practical value.

[0196] In addition, the present technology can adopt the following configuration.

[0197] (1) An electronic device, comprising:

[0198] A display unit;

[0199] A photographing unit disposed on the side opposite to the display surface of the display unit; and

[0200] A control unit that synchronizes the display timing of the display unit with the photographing timing of the photographing unit so that the photographing unit performs photographing when the display unit does not perform display.

[0201] (2) The electronic device according to (1), wherein

[0202] Compared with the case where the photographing unit does not perform photographing, when the photographing unit performs photographing, the control unit makes the period during which the display unit does not perform display longer.

[0203] (3) The electronic device according to (1) or (2), wherein

[0204] Compared with the case where the photographing unit does not perform photographing, when the photographing unit performs photographing, the control unit makes the display frequency of at least a part of the pixels of the display unit lower.

[0205] (4) The electronic device according to (3), wherein

[0206] Compared with the case where the photographing unit does not perform photographing, when the photographing unit performs photographing, the control unit makes the frame rate of the display unit lower.

[0207] (5) The electronic device according to (3), wherein

[0208] Compared with the case where the photographing unit does not perform photographing, when the photographing unit performs photographing, the control unit makes the display frequency of at least a part of the horizontal pixel lines of the display unit lower.

[0209] (6) The electronic device according to (5), wherein

[0210] When the photographing unit performs photographing, as compared with a case where the photographing unit does not perform photographing, the control unit makes a display frequency of a part of horizontal pixel lines included in a display region that overlaps with the photographing unit when the display unit is viewed from a normal direction of the display surface lower.

[0211] (7) The electronic device according to any one of (1) to (6), wherein

[0212] the control unit sets an exposure time of the photographing unit based on a period during which the display unit does not perform display.

[0213] (8) The electronic device according to any one of (1) to (7), wherein

[0214] When the photographing unit performs photographing, as compared with a case where the photographing unit does not perform photographing, the control unit makes a display pixel density in at least a part of a display region of the display unit lower.

[0215] (9) The electronic device according to (8), wherein

[0216] When the photographing unit performs photographing, as compared with a case where the photographing unit does not perform photographing, the control unit makes a display pixel density in a part of a display region that includes a display region overlapping with the photographing unit lower when the display unit is viewed from a normal direction of the display surface.

[0217] (10) The electronic device according to (8) or (9), wherein

[0218] When the photographing unit performs photographing, as compared with a case where the photographing unit does not perform photographing, the control unit makes a luminance value of pixels in at least a part of a display region of the display unit higher.

[0219] (11) The electronic device according to (10), wherein

[0220] When the photographing unit performs photographing, as compared with a case where the photographing unit does not perform photographing, the control unit makes a luminance value of pixels in a part of a display region that includes a display region overlapping with the photographing unit higher when the display unit is viewed from a normal direction of the display surface.

[0221] (12) The electronic device according to (11), wherein

[0222] When the photographing unit performs photographing, as compared with a case where the photographing unit does not perform photographing, the control unit makes a display frequency of pixels in the part of the display region lower and a luminance value of the pixels higher.

[0223] (13) The electronic device according to any one of (8) to (12), wherein

[0224] The display unit includes a first display surface and a second display surface. The first display surface and the second display surface are stacked on each other and are separately controlled for display.

[0225] The first display surface displays the display area of the display unit except for a part of the display area.

[0226] When the second display surface is stacked on the first display surface, the second display surface displays the part of the display area in a manner that does not overlap with the display area of the first display surface.

[0227] (14) The electronic device according to any one of (1) to (13),

[0228] The electronic device includes a brightness detection unit that detects the brightness of the surroundings.

[0229] The control unit controls the display frequency of at least a part of the pixels of the display unit when the photographing unit performs photographing according to the brightness of the surroundings detected by the brightness detection unit.

[0230] (15) The electronic device according to (14), wherein

[0231] The control unit controls the display frequency and the display pixel density of at least a part of the pixels of the display unit when the photographing unit performs photographing according to the brightness of the surroundings detected by the brightness detection unit.

[0232] (16) The electronic device according to any one of (1) to (15),

[0233] The electronic device includes a timing detection unit that detects the timing of blinking of a person who directs their line of sight to the display unit.

[0234] The control unit temporarily sets the display of the display unit to a non-luminous state in accordance with the timing detected by the timing detection unit, and performs photographing by the photographing unit.

[0235] (17) The electronic device according to any one of (1) to (16), wherein

[0236] The photographing unit performs photoelectric conversion on at least one of visible light and infrared light.

[0237] (18) The electronic device according to any one of (1) to (17), wherein

[0238] The display unit has an opening, and when the display unit is viewed from above in the normal direction of the display surface, the opening is located at a position overlapping with the photographing unit.

[0239] The opening portion is configured to penetrate a layer having a light transmittance of less than a specified value for light of a specified wavelength band among a plurality of layers constituting the display portion.

[0240] (19) The electronic device according to any one of (1) to (18), wherein

[0241] The display portion is composed of a plurality of layers, and the light transmittance of the plurality of layers for light in the wavelength band in which the imaging portion can perform photoelectric conversion is equal to or higher than a specified value.

[0242] (20) The electronic device according to any one of (1) to (19), wherein

[0243] The control portion includes:

[0244] A first control portion that performs display control of the display portion; and

[0245] A second control portion that, in cooperation with the first control portion, performs imaging control of the imaging portion.

[0246] The mode of the present invention is not limited to the above-described embodiments, but includes various modifications that can be conceived by those skilled in the art, and the effects of the present invention are not limited to the above. That is, it can be derived from the content specified in the claims and equivalents thereof, and various additions, changes, and partial deletions can be made without departing from the concept, idea, and gist of the present invention.

Claims

1. An electronic device, characterized in that, Comprising: A display unit; A photographing unit disposed on a side opposite to the display surface of the display unit, for photographing an analog pixel signal; and A control unit that synchronizes the display timing of the display unit with the photographing timing of the photographing unit, so that the photographing unit performs photographing when the display unit is not displaying, The photographing unit has: A polarization element that polarizes light incident through the display unit; And A photoelectric conversion unit that outputs polarization information after the light polarized by the polarization element is incident on the corresponding photoelectric conversion unit for photoelectric conversion, The electronic device converts the analog pixel signal photographed by the photographing unit into digital pixel data, obtains polarization information including a glare component and a diffracted light component, uses at least one of the glare component and the diffracted light component as a correction amount, and generates the digital pixel data after removing at least one of the glare component and the diffracted light component through subtraction processing.

2. The electronic device according to claim 1, wherein Compared with the case where the photographing unit does not perform photographing, when the photographing unit performs photographing, the control unit makes the period during which the display unit does not display longer.

3. The electronic device according to claim 1, wherein Compared with the case where the photographing unit does not perform photographing, when the photographing unit performs photographing, the control unit makes the display frequency of at least a part of the pixels of the display unit lower.

4. The electronic device according to claim 3, wherein Compared with the case where the photographing unit does not perform photographing, when the photographing unit performs photographing, the control unit makes the frame rate of the display unit lower.

5. The electronic device according to claim 3, wherein Compared with the case where the photographing unit does not perform photographing, when the photographing unit performs photographing, the control unit makes the display frequency of at least a part of the horizontal pixel lines of the display unit lower.

6. The electronic device according to claim 5, wherein Compared with the case where the photographing unit does not perform photographing, when the photographing unit performs photographing, the control unit makes the display frequency of a part of the horizontal pixel lines included in the display area overlapping with the photographing unit lower when observing the display unit from a direction normal to the display surface.

7. The electronic device according to claim 1, wherein The control unit sets the exposure time of the photographing unit based on the period during which the display unit does not display.

8. The electronic device according to claim 1, wherein Compared with the case where the photographing unit does not perform photographing, when the photographing unit performs photographing, the control unit makes the display pixel density in at least a part of the display area of the display unit lower.

9. The electronic device according to claim 8, wherein Compared with the case where the photographing unit does not perform photographing, when the photographing unit performs photographing, the control unit makes the display pixel density in a part of the display area including the display area overlapping with the photographing unit lower when observing the display unit from a direction normal to the display surface.

10. The electronic device according to claim 8, wherein When the imaging unit is imaging, as compared with the case where the imaging unit does not image, the control unit makes the luminance values of the pixels in at least a part of the display area of the display unit higher.

11. The electronic device according to claim 10, wherein When the imaging unit is imaging, as compared with the case where the imaging unit does not image, the control unit makes the luminance values of the pixels in the part of the display area that includes the display area overlapping with the imaging unit higher when observing the display unit from a direction normal to the display surface.

12. The electronic device according to claim 11, wherein When the imaging unit is imaging, as compared with the case where the imaging unit does not image, the control unit makes the display frequency of the pixels in the part of the display area lower and the luminance values of the pixels higher.

13. The electronic device according to claim 8, wherein The display unit includes a first display surface and a second display surface, the first display surface and the second display surface are stacked on each other and are individually controlled for display, The first display surface displays the display area of the display unit except for the part of the display area, When the second display surface is stacked with the first display surface, the second display surface displays the part of the display area in a manner that does not overlap with the display area of the first display surface.

14. The electronic device according to claim 1, wherein The electronic device includes a brightness detection unit that detects the brightness of the surroundings, The control unit controls the display frequency of at least a part of the pixels of the display unit when the imaging unit is imaging according to the brightness of the surroundings detected by the brightness detection unit.

15. The electronic device according to claim 14, wherein The control unit controls the display frequency and the display pixel density of at least a part of the pixels of the display unit when the imaging unit is imaging according to the brightness of the surroundings detected by the brightness detection unit.

16. The electronic device according to claim 1, wherein The electronic device includes a timing detection unit that detects the timing of blinking of a person whose line of sight is directed to the display unit, The control unit temporarily sets the display of the display unit to a non-emitting state in accordance with the timing detected by the timing detection unit, and performs imaging by the imaging unit.

17. The electronic device according to claim 1, wherein The display unit has an opening, and when observing the display unit from a direction normal to the display surface, the opening is located at a position overlapping with the imaging unit, The opening is configured to penetrate a layer among the plurality of layers constituting the display unit and having a transmittance of light of a specified wavelength band less than a specified value.

18. The electronic device according to claim 1, wherein The display unit is composed of a plurality of layers, and the transmittance of light of the wavelength band that the imaging unit can perform photoelectric conversion for the plurality of layers is equal to or higher than a specified value.

19. The electronic device according to claim 1, wherein The control unit includes: A first control unit that performs display control of the display unit; and A second control unit that cooperates with the first control unit to perform imaging control of the imaging unit.

Images