Electronic device

Abstract

An electronic device that suppresses a reduction in the image quality of an image captured by a camera while reducing the bezel width. The electronic device includes: a display unit having a displayable area that has display optical systems arranged in an array along a first direction and a second direction intersecting the first direction; a plurality of imaging optical systems that overlap the displayable area in a third direction intersecting the first direction and the second direction and are arranged on the side of the display unit opposite to the display surface, and that at least include a first imaging optical system and a second imaging optical system, the second imaging optical system having coordinates different from those of the first imaging optical system in at least one of the first direction and the second direction; and an image acquisition unit that acquires image data based on information acquired by the first imaging optical system and the second imaging optical system.

Description

Technical Field

[0001] The present invention relates to an electronic device. Background Art

[0002] Recently, in electronic devices such as smartphones, mobile phones, and PCs (Personal Computers), a camera is arranged on a frame (border) of a display unit, and video calls and video shooting can be easily performed. Since smartphones and mobile phones are usually carried in a pocket or a bag, the outer dimensions must be made as compact as possible. On the other hand, if the size of the display screen is small, the higher the display resolution, the smaller the size of the displayed characters, and it is difficult to recognize. Therefore, research is being conducted to reduce the border width around the display screen so as to increase the size of the display screen as much as possible without increasing the outer dimensions of the electronic device.

[0003] However, since a camera or the like is usually installed on the border of the electronic device, the border width cannot be less than the outer diameter size of the camera. In addition, when a camera is arranged on the border, for example, during a video call, the line of sight usually focuses on the vicinity of the center of the display screen. Therefore, if the line of sight deviates from the optical axis of the camera, a captured image that does not match the line of sight and causes discomfort will be obtained.

[0004] To avoid the above problems, it has been proposed to arrange the camera module on the side opposite to the display surface of the display unit and capture the subject light passing through the display unit with the camera.

[0005] [Prior Art Documents]

[0006] [Patent Documents]

[0007] Patent Document 1: US Patent Publication No. 2018 / 0069060 Summary of the Invention

[0008] [Problems to be Solved by the Invention]

[0009] However, since a part of the light passing through the display unit is reflected or diffracted and enters the camera, the quality of the captured image is degraded due to the influence of glare caused by reflection and diffraction.

[0010] One aspect of the present invention provides an electronic device that can suppress a decrease in the quality of an image captured by a camera while reducing the border width.

[0011] [Solutions to the Problems]

[0012] According to an embodiment, an electronic device includes: a display unit having a displayable area which has display optical systems arranged in an array along a first direction and a second direction intersecting the first direction; a plurality of imaging optical systems which overlap with the displayable area and are arranged on the side of the display unit opposite to the display surface in a third direction intersecting the first direction and the second direction, the plurality of imaging optical systems having at least: a first imaging optical system; and a second imaging optical system having coordinates different from those of the first imaging optical system in at least one of the first direction and the second direction; and an image acquisition unit which acquires image data based on information acquired by the first imaging optical system and the second imaging optical system. In this way, by staggering the positions of at least two imaging optical systems, the position where glare occurs in the captured image can be changed. For example, the first imaging optical system and the second imaging optical system are two optical systems arbitrarily selected from the plurality of imaging optical systems. That is, the configurations listed below can also be features corresponding to at least two of the plurality of imaging optical systems. In addition, one feature and another feature can also be features based on different combinations of a first optical system and a second optical system. In this way, at least two of the plurality of imaging optical systems can have the following features.

[0013] In the electronic device, light can also propagate from the display surface of the display unit to the first imaging optical system and the second imaging optical system through optical systems having different optical characteristics. Moreover, the plurality of imaging optical systems can all have different optical characteristics, or some of them can have the same optical characteristics. In this way, at least two imaging optical systems can have different optical characteristics in the optical path of the light propagating from the display surface. By having different optical characteristics, the characteristics of the generated glare can be changed.

[0014] The electronic device can be provided with an opening on the display unit for propagating the light incident from the display surface, and the light incident from the display surface can propagate to the imaging optical system through the opening. For example, the above-mentioned optical characteristics can be formed through this opening.

[0015] The opening for propagating light to the first imaging optical system and the opening for propagating light to the second imaging optical system can have different layouts. Different optical characteristics can be formed through this layout.

[0016] The opening for propagating light to the first imaging optical system and the opening for propagating light to the second imaging optical system can form diffraction images in different directions. For example, the size of the opening for propagating light to the first imaging optical system in the first direction is larger than that in the second direction, and the size of the opening for propagating light to the second imaging optical system in the second direction is larger than that in the first direction. In this way, diffraction images are formed in different directions through different openings, so that the directions where glare is generated can be different in the two imaging optical systems.

[0017] The electronic device may be provided with a third photographing optical system. The parallax between the third photographing optical system and the first photographing optical system is the same as the parallax between the first photographing optical system and the second photographing optical system. The image acquisition unit acquires image data based on the information obtained from the information acquired from the second photographing optical system and the third photographing optical system and the information acquired from the first photographing optical system. That is, with respect to a combination of any two photographing optical systems, a photographing optical system having the same parallax in the opposite direction is further arranged with any one of the two photographing optical systems as the center.

[0018] In the first direction or the second direction of the display surface, the first photographing optical system is arranged near the center, and the second photographing optical system and the third photographing optical system are arranged near the boundary of the display surface with the first photographing optical system interposed therebetween. Compared with the second photographing optical system and the third photographing optical system, the area of the first photographing optical system facing the display surface in the third direction is small. That is, the second and third photographing optical systems are arranged at both ends of the display surface of the electronic device 1, and the first photographing optical system, which is less conspicuous than these two photographing optical systems, is arranged near the center.

[0019] The first photographing optical system and the second photographing optical system have different coordinates in the first direction and the second direction. In this way, the two photographing optical systems may have different coordinates only in the first direction, only in the second direction, or in both the first direction and the second direction. That is, in the display surface, the two photographing optical systems may be arranged in the horizontal direction, may be arranged in the vertical direction, or may be arranged in an arbitrary direction.

[0020] When the image acquisition unit synthesizes the data acquired from the first photographing optical system and the data acquired from the second photographing optical system, the image acquisition unit acquires a photographing result based on the data with low intensity in the image data as the synthesis result. In the acquired data, glare often has a higher brightness and light intensity than the photographed object. Therefore, when acquiring image data, the image acquisition unit may acquire an image based on the low-intensity signal in the signals acquired from the two photographing optical systems.

[0021] The first photographing optical system and the second photographing optical system each have a direction in which they preferentially reflect. When a difference of a specified value or more occurs in the outputs in each direction, the image acquisition unit may use any result to acquire a photographing result. For example, when the first photographing optical system generates glare in the first direction and the second photographing optical system generates glare in the second direction, the image acquisition unit may acquire image data based on the output of the photographing optical system with less glare.

[0022] Light shielding may be performed between the first photographing optical system and the second photographing optical system. In this way, light shielding can be performed between the first photographing optical system and the second photographing optical system to suppress the mutual influence of the optical systems.

[0023] The image acquisition unit can use the trained model to synthesize the information acquired by the first imaging optical system and the second imaging optical system. In this way, for example, a model generated by machine learning can be used to synthesize the data output from multiple imaging optical systems.

[0024] The trained model can be trained based on the data collected from multiple electronic devices. For example, a model can be generated based on the same model of electronic device 1. In addition, even within the same model, the model can be changed and trained based on the shooting mode, etc.

[0025] The image acquisition unit can perform correction when the parallax in the multiple images acquired by the multiple imaging optical systems exceeds a specified amount. For example, the parallax can be detected from the area where glare is generated, and image data can be obtained by performing image processing on this area.

[0026] At least one imaging optical system can be composed of a microlens array. In this way, a lens system can be provided within one imaging system instead of one lens.

[0027] Multiple imaging optical systems can be arranged in the area where the microlens array is provided. In this way, one microlens array can be used to form multiple imaging optical systems.

[0028] The first imaging optical system and the second imaging optical system can acquire information through the same imaging element. In this way, multiple imaging optical systems can be formed in one imaging element, for example, one chip imaging element. When combined with the above, by providing a microlens array on one chip, multiple imaging optical systems can be provided in each area.

[0029] The image acquisition unit can be arranged on the same chip as the imaging element. For example, an imaging element and a logic circuit are provided on one chip, the data acquired by the imaging element is subjected to DA conversion, and the converted data is processed by the logic circuit. It can also be formed as multiple stacked chips instead of one chip.

[0030] The display unit can include multiple display optical systems with different optical characteristics. For example, OLED, MicroLED, liquid crystal, etc. can be mixed and used as the display optical system. In this case, depending on the imaging optical system, the glare based on the results of reflection, refraction, and diffraction in the display unit can have different characteristics.

[0031] At least one of the multiple imaging optical systems operates when the image acquisition unit requires a correction signal. For example, an electronic device is equipped with an imaging element that performs correction when a strong light source is detected, and whether to perform correction through the imaging optical system can be switched based on the surrounding environment.

[0032] The first imaging optical system and the second imaging optical system can be integrally formed. For example, in the two imaging optical systems, the light receiving element and the optical path reaching the light receiving element can be adjacent to each other.

[0033] The first imaging optical system and the second imaging optical system can be disposed near the boundary of the display surface. The boundary refers to, for example, the end of the display surface, and a plurality of imaging optical systems can be disposed at this end.

[0034] The first imaging optical system and the second imaging optical system are arranged at a distance of 50 mm or more and 80 mm or less, and the image acquisition unit generates parallax image data of the information acquired by the first imaging optical system and the second imaging optical system. In this way, the two imaging optical systems can also be arranged to have a distance equal to the distance between two human eyes.

[0035] The display unit can be provided on two surfaces of the electronic device. In this case, the plurality of display units can each include a plurality of image optical systems. It can be that one side has a plurality of imaging optical systems and the other side does not have an imaging optical system, or one side has one imaging optical system and the other side has a plurality of imaging optical systems.

[0036] It is also possible to have a fourth imaging optical system different from the first imaging optical system and the second imaging optical system, and the fourth imaging optical system and the first imaging optical system or the second imaging optical system have any of the above features. As described above, the first imaging optical system and the second imaging optical system are merely two systems selected from a plurality of imaging optical systems and do not represent a specific imaging optical system. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a schematic cross-sectional view of an electronic device according to an embodiment.

[0038] Figure 2 is a schematic external view of an electronic device according to an embodiment.

[0039] Figure 3A is a schematic diagram showing an example of an imaging optical system according to an embodiment.

[0040] Figure 3B is a schematic diagram showing an example of an imaging optical system according to an embodiment.

[0041] Figure 4 is a block diagram of the imaging operation of an electronic device according to an embodiment.

[0042] Figure 5 is a view showing the imaging optical system according to an embodiment from the display surface side.

[0043] Figure 6A is a diagram showing an example of an image captured by the Figure 5 imaging optical system.

[0044] Figure 6B is a diagram showing an example of an image captured by the Figure 5 imaging optical system.

[0045] Figure 7 is a diagram showing an example of an image acquired by the image acquisition unit according to an embodiment.

[0046] Figure 8 is a diagram showing the imaging optical system according to an embodiment from the display surface side.

[0047] Figure 9 is a diagram showing an example of an image captured by the Figure 8 imaging optical system.

[0048] Figure 10 is a schematic external view of an electronic device according to an embodiment.

[0049] Figure 11 is a diagram showing an example of an image captured by the Figure 10 imaging optical system.

[0050] Figure 12 is a diagram showing an example of an image obtained by synthesizing the images acquired by the imaging optical systems at both ends of Figure 11 .

[0051] Figure 13 is a diagram showing an example of an image captured by the imaging optical system according to an embodiment.

[0052] Figure 14 is a schematic cross-sectional view of an electronic device according to an embodiment.

[0053] Figure 15 is a schematic cross-sectional view of an electronic device according to an embodiment.

[0054] Figure 16 is a schematic cross-sectional view of an electronic device according to an embodiment.

[0055] Figure 17 is a schematic diagram showing an example of the imaging optical system according to an embodiment.

[0056] Figure 18 is a schematic diagram showing an example of the imaging optical system according to an embodiment.

[0057] Figure 19It is a schematic diagram showing an example of an imaging optical system according to an embodiment.

[0058] Figure 20 It is a schematic diagram showing an example of an imaging optical system according to an embodiment.

[0059] Figure 21 It is a schematic diagram showing an example of an imaging optical system according to an embodiment.

[0060] Figure 22 It is a schematic diagram showing an example of an imaging optical system according to an embodiment.

[0061] Figure 23A It is a schematic diagram showing an example of an imaging optical system according to an embodiment.

[0062] Figure 23B It is a schematic diagram showing an example of an imaging optical system according to an embodiment.

[0063] Figure 24 It is a schematic diagram showing an example of a mode of an imaging unit according to an embodiment.

[0064] Figure 25 It is a schematic diagram showing an example of a mode of an imaging unit according to an embodiment.

[0065] Figure 26 It is a schematic diagram showing an example of a mode of an imaging unit according to an embodiment.

[0066] Figure 27 It is a diagram showing an example of the configuration of an imaging optical system according to an embodiment.

[0067] Figure 28 It is a diagram showing an example of the configuration of an imaging optical system according to an embodiment.

[0068] Figure 29 It is a diagram showing an example of the configuration of an imaging optical system according to an embodiment.

[0069] Figure 30 It is a diagram showing an example of the configuration of an imaging optical system according to an embodiment.

[0070] Figure 31A It is a diagram showing an example of the configuration of an imaging optical system according to an embodiment.

[0071] Figure 31B It is a diagram showing an example of the configuration of an imaging optical system according to an embodiment.

[0072] Figure 32 This is a diagram showing an example of the layout of openings according to one embodiment.

[0073] Figure 33 This is a diagram showing an example of the layout of openings according to one embodiment.

[0074] Figure 34 This is a diagram showing an example of the layout of openings according to one embodiment.

[0075] Figure 35 This is a diagram showing an example of the layout of openings according to one embodiment.

[0076] Figure 36 This is a plan view when an electronic device according to one embodiment is applied to a capsule endoscope.

[0077] Figure 37 This is a rear view when an electronic device according to one embodiment is applied to a digital single-lens reflex camera.

[0078] Figure 38A This is a diagram showing an example of applying an electronic device according to one embodiment to an HMD.

[0079] Figure 38B This is a diagram showing an existing HMD.

[0080] Explanation of reference numerals:

[0081] 1 Electronic device; 1a Display screen; 1b Frame; 2 Display unit; 3 Imaging optical system; 3A First imaging optical system; 3B Second imaging optical system; 3C Third imaging optical system; 4 Display panel; 4a Substrate; 4b Light-emitting pixel; 5 Circularly polarized glare; 6 Touch panel; 7 Cover glass; 8 Imaging unit; 8A First imaging unit; 8B Second imaging unit; 9 Optical system; 9A First optical system; 9B Second optical system; 10 Pre-processing unit; 12 Image acquisition unit; 14 Post-processing unit; 16 Output unit; 18 Control unit; 20 Storage unit; 22 Reception interface; 30 Light-shielding unit; 32 Microlens array. Detailed description of the embodiment

[0082] Hereinafter, embodiments of the electronic device will be described with reference to the drawings. Hereinafter, the description will be centered on the main components of the electronic device, but the electronic device may have components and functions that are not shown or described. The following description does not exclude components and functions that are not shown or described. In addition, for the purpose of description, the dimensions, shapes, aspect ratios, etc. may be changed, and appropriate dimensions, shapes, aspect ratios, etc. are provided during installation. In the following description, the acquired signals are described as image information or imaging information, but the image information and imaging information are general concepts that also include still images, moving images, or one frame of video, etc.

[0083] (First Embodiment)

[0084] Figure 1 is a schematic cross-sectional view of the electronic device 1 of the first embodiment. Figure 1 The electronic device 1 is any electronic device having both a display function and a photographing function, such as a smart phone, a mobile phone, a tablet, a PC, etc. As shown in the lower left of the figure, the first direction is toward the right side of the figure, the second direction is perpendicular to the figure, and the third direction is toward the lower side of the figure. That is, the second direction intersects the first direction, and the third direction intersects the first direction and the second direction. In addition, the intersection includes an intersection at an angle of 90°, and may not be strictly 90°. Furthermore, as can be seen from the figure, the first direction and the second direction are distinguished for convenience and can be interchanged with each other.

[0085] Figure 1 The electronic device 1 includes a photographing optical system (such as a camera module), which is arranged on the side opposite to the display surface of the display unit 2. In this way, the electronic device 1 provides the photographing optical system 3 on the inner surface side (opposite side) of the display surface of the display unit 2. Therefore, the photographing optical system 3 performs photographing through the display unit 2.

[0086] As Figure 1 shown, the display unit 2 is a structure formed by sequentially laminating a display panel 4, a circularly polarized light plate 5, a touch panel 6, and a cover glass 7. Figure 1 The lamination is an example, and an adhesive layer or a bonding layer may be provided between the display panel 4, the circularly polarized light plate 5, the touch panel 6, and the cover glass 7 as needed. In addition, the order of the circularly polarized light plate 5 and the touch panel 6 can be appropriately changed according to the design.

[0087] The photographing optical system 3 is provided on the side opposite to the display surface of the display unit 2. The photographing optical system 3 includes, for example, a photoelectric element (light receiving element) that receives light and photoelectrically converts it into an analog signal, and an optical system that propagates the light irradiated on the display surface to the photoelectric element. The optical system can be, for example, an opening provided in the display panel 4. One display unit 2 of the electronic device 1 includes a plurality of the above-mentioned photographing optical systems 3, for example, two as shown in the figure. The light irradiated on the display surface is diffracted at the opening and propagates to the light receiving element as shown by the arrow in the figure. In addition, instead of providing an opening, an optical system having certain optical characteristics, such as adjusting the optical path length or changing the polarization state, can be provided. The photographing optical system 3 includes, for example, a photographing unit 8 and an optical system 9 that condenses, diffuses, etc. the light incident on the photographing unit 8 from the display surface.

[0088] As shown in the figure, multiple imaging optical systems 3, for example, have different coordinates in the second direction, but are not limited thereto. For example, they may also have different coordinates in the first direction, or may have different coordinates in both the first and second directions.

[0089] Although not shown in detail, the display panel 4 may, for example, include an OLED (Organic Light Emitting Device), may also include a liquid crystal such as a TFT, or may further include a MicroLED as an optical system for display (display optical system). The display optical system may also include a light-emitting element based on other display principles. The light-emitting elements of the display optical system may be, for example, arranged in a stripe pattern, a mosaic pattern, arranged in an array in the first and second directions, or arranged obliquely, with partial pixel intervals. In addition, in the display optical system, the light-emitting element may have a stacked filter to change the display color. In the case of including an OLED or the like, the display panel 4 may be composed of multiple layers such as an anode layer and a cathode layer. In addition, these layers may be formed of a material with high transmittance.

[0090] The display panel 4 is sometimes provided with a low-transmittance member such as a color filter layer. In addition, when the display panel 4 includes an OLED, for example, it may include a substrate 4a and an OLED portion. The substrate 4a may be formed of polyimide or the like, for example. When the substrate 4a is formed of a material with low light transmittance such as polyimide, an opening may be formed corresponding to the arrangement position of the imaging optical system 3. If the subject light passing through the opening is incident on the imaging optical system 3, the image quality of the image captured by the imaging optical system 3 can be improved. In addition, instead of forming an opening, a light propagation path formed of a high-transmittance substance may be provided. In this case, the light incident from the display surface of the display unit 2 is also received by the imaging optical system 3 and converted into a signal.

[0091] The circular polarizing plate 5 is provided, for example, to reduce glare or improve the visibility of the display screen 1a even in a bright environment. The touch panel 6 is built with a touch sensor. The touch sensor has various methods such as a capacitive type and a resistive film type, and any method can be adopted. In addition, the touch panel 6 and the display panel 4 may be integrally formed. The cover glass 7 is provided to protect the display panel 4 and the like. As described above, an adhesive layer or a bonding layer such as an OCA (Optical Clear Adhesive) may be provided at an appropriate position. In addition, through design, the order of the circular polarizing plate 5 and the touch panel 6 in the third direction may be interchanged.

[0092] Figure 2 Shows Figure 1 A schematic external view and a cross-sectional view of the electronic device 1 shown. The cross-sectional view shows the cross-section of the display unit 2 including part of the display at the position of the dotted line shown in the figure. The housing of the electronic device 1 and the circuits other than the display unit are omitted.

[0093] In the external view, the display screen 1a extends to be close to the outer diameter size of the electronic device 1, and the width of the frame 1b around the display screen 1a is several millimeters or less. Usually, a front camera is often arranged in the frame 1b. In the present embodiment, for example, as shown by the dashed line in the external view, the front camera is located at approximately the center in the second direction of the display screen 1a as a plurality of imaging optical systems 3. In this way, by arranging the front camera as the imaging optical system 3 on the opposite side of the display surface of the display unit 2, it is not necessary to arrange the front camera in the frame 1b, and the width of the frame 1b can be made narrower.

[0094] In addition, Figure 2 the external view shown represents an example, and the imaging optical system 3, that is, the front camera, can be at any position in the first direction or the second direction in the display screen 1a, and is arranged on the opposite side (rear side) of the display surface of the display unit 2. For example, it can be arranged at the peripheral edge portion (end portion, boundary portion) of the display screen 1a. As Figure 2 shown in the external view, the plurality of imaging optical systems 3 have different coordinates in the first direction, for example. No matter at which position the imaging optical system 3 is arranged, as long as it has different coordinates in at least one of the first direction and the second direction. In addition, although two imaging optical systems 3 are shown, the present invention is not limited thereto, and more imaging optical systems can be arranged on the opposite side of the display surface.

[0095] For example, as shown in the cross-sectional view, the imaging optical system 3 is arranged on the inner surface side of the display unit 2 on the side opposite to the display surface which is the display surface side. In addition, this cross-sectional view is a simplified view. For example, similar to the above, Figure 2 a bonding layer and the like are also provided in the structure of the cross-sectional view shown, but are omitted for simplicity of description.

[0096] Figure 3A is a view showing an example of the imaging optical system 3. The imaging optical system 3 includes, for example, an imaging unit 8 and an optical system 9. The optical system 9 is arranged on the light incident surface side of the imaging unit 8, that is, on the side close to the display unit 2. Light passes through the display surface of the display unit 2 and then propagates to the imaging unit 8 through the optical system 9.

[0097] The photographing unit 8 includes, for example, a light receiving element such as a photodiode and a photoelectric element. Light is condensed, diffused, and propagated through the optical system 9, and is received by the pixel array of the photographing unit 8 to output an analog signal. A color filter such as a Bayer arrangement or a stacked color filter may be provided, for example, on the incident surface side of each photographing element in the pixel array of the photographing unit. In addition, a filter for obtaining a color image may be provided. In addition, although not shown, other elements, circuits, etc. required for receiving light and outputting an analog signal are provided. For example, CMOS (Complementary Metal-Oxide-Semiconductor) elements or CCD (Charge Coupled Device) elements may be used for photoelectric conversion. In addition, the above-mentioned filter and polarizing element may be provided.

[0098] The optical system 9 may include, for example, a lens. In addition, the concept of the optical system 9 includes the opening provided in the display panel 4 described above. For example, as the optical system 9, when the display panel 4 is provided with an opening, a lens is arranged at a position closer to the photographing unit 8 than the opening in the third direction. The opening is provided, for example, on the substrate 4a with a low transmittance, and a lens for propagating the light passing through the opening portion to the photographing unit 8 is provided. For example, the numerical aperture Na (Numerical Aperture), F-number (F-Number), and other optical characteristics of each photographing optical system 3 are defined by the lens and the opening. In addition, other optical characteristics such as different Abbe numbers are further provided through the optical system 9. The lens is shown as a single lens, but is not limited thereto, and the lens system may include various types of lenses.

[0099] In addition, the opening and the lens are an example, and the configuration of the optical system 9 is not limited to this combination. In addition, in the figure, one lens is provided for one opening, but is not limited thereto. For example, as Figure 3B shown, multiple openings may be provided for one lens in the optical system 9. In a region where no opening is provided, for example, light emitting elements of the display panel 4 are provided, and an opening may be formed between these light emitting elements. By configuring in this way, the photographing optical system 3 can be configured without impairing the display.

[0100] The multiple photographing optical systems 3 may have different optical characteristics according to the shape of the opening, the performance of the lens, etc. When there are three or more photographing optical systems 3, the corresponding optical systems 9 may be formed to have different optical characteristics. As another example, the photographing optical system 3 may be divided into multiple groups, and each group has different optical characteristics. For example, the optical system 9 may change its opening shape, orientation, or lens material, etc., so that two photographing optical systems 3 have common optical characteristics and one photographing optical system 3 has different optical characteristics. The opening layout is described as an expression including the shape and orientation of the opening.

[0101] As Figure 3A indicated by the arrow of Figure 3A , light enters from the display surface side of the display unit 2, is refracted by the optical system 9, and is received by the imaging unit 8. At the part where the optical system 9 is not provided, reflection and the like can be appropriately suppressed in the same manner as in a normal display, and the display of the display unit 2 can be adjusted for easy viewing. For example, openings are provided between the light-emitting pixels of the display panel 4, and lenses are provided on the opposite side of the openings from the display surface in the third direction, and the light incident from the display surface is projected onto the imaging unit 8. In addition, openings can be provided separately between consecutive light-emitting pixels. In other words, it can also be configured such that light-emitting pixels are provided between the openings.

[0102] Here, an example of the imaging function of the electronic device 1 will be described.

[0103] Figure 4 FIG. 11 shows an example of a block diagram showing the configuration related to the imaging operation of the electronic device 1 according to the present embodiment. The electronic device 1 includes: a display unit 2, a plurality of imaging optical systems 3, a pre-processing unit 10, an image acquisition unit 12, a post-processing unit 14, an output unit 16, a control unit 18, and a storage unit 20.

[0104] Similar to the drawings described above, a plurality of imaging optical systems 3 are provided on the side opposite to the display surface of one display unit 2. Each imaging optical system 3 includes an imaging unit 8 and an optical system 9.

[0105] The pre-processing unit 10 is a circuit that processes the analog signal output from the imaging unit 8. The pre-processing unit 10 includes, for example, an ADC (Analog to Digital Converter), and converts the input analog signal into digital image data.

[0106] The image acquisition unit 12 acquires the captured image based on the digital image data converted by the pre-processing unit 10. Based on the digital image data acquired from the plurality of imaging optical systems 3, the imaging result is acquired. More specifically, the image acquisition unit 12 acquires and outputs, for example, the following imaging result: the imaging result after suppressing the flare generated by each imaging optical system 3 using the image data acquired by the plurality of imaging optical systems 3.

[0107] The post-processing unit 14 performs appropriate processing on the imaging result output from the image acquisition unit 12 and outputs it. The appropriate processing can be, for example, image processing or signal processing such as pixel defect correction, edge enhancement, denoising, brightness adjustment, color correction, white balance adjustment, distortion correction, autofocus processing, etc. In addition, the appropriate processing can also be processing specified by the user.

[0108] The output unit 16 outputs information to the outside of the electronic device 1. The output unit 16 includes, for example, an output interface. The output interface is, for example, an interface that outputs a digital signal such as a USB (Universal Serial Bus) or a user interface such as a display. In addition, the output interface provided in the output unit 16 may also function as an input interface.

[0109] The control unit 18 controls the processing in the electronic device 1. The control unit 18 may include, for example, a CPU (Central Processing Unit) and may control the processing of the pre-processing unit 10, the image acquisition unit 12, the post-processing unit 14, and the output unit 16. In addition, based on the shooting timing indicated by the user interface, it may execute control for shooting using the shooting optical system 3.

[0110] The storage unit 20 stores data in the electronic device 1. The storage unit 20 is, for example, a memory such as a DRAM (Dynamic Random Access Memory) or a storage device such as an SSD (Solid State Drive). The storage unit 20 may be an internal memory or a memory such as a removable memory card. In addition, the storage unit 20 is not necessarily provided inside the electronic device 1 and may be an external storage device connected through an input / output interface. Information is appropriately input and output from the storage unit 20 or to the storage unit 20 in the electronic device 1 at a necessary timing.

[0111] Part or all of the components described above may be formed on the same substrate. For example, the shooting optical system 3, the pre-processing unit 10, the image acquisition unit 12, the post-processing unit 14, the output unit 16, the control unit 18, and the storage unit 20 may be formed on one chip, or a part of them may be appropriately formed on another chip. In addition, the component configuration formed on the same substrate of one chip and the component configuration formed on another substrate may be laminated in the manufacturing process by technologies such as CoC (Chip on Chip), CoW (Chip on Wafer), and WoW (Wafer on Wafer).

[0112] Next, the operations of the shooting optical system 3 and the image acquisition unit 12 will be described in detail.

[0113] Figure 5 FIG. shows the shooting optical system 3 from the display surface side of the display unit 2. For example, a partial area is shown in the figure when observing the display unit 2 from the display surface side. The display panel 4 can be observed from the display surface, and on the display panel 4, as shown by the dotted line, light-emitting pixels 4b formed by a plurality of light-emitting elements arranged in an array in the first direction and the second direction are provided. In addition, the arrangement and orientation of the light-emitting pixels 4b are an example, and the light-emitting pixels 4b are set to be the same asFigure 5 They have the same arrangement, but can also be set by rotating 45°. In addition, the light-emitting pixels 4b are drawn as squares, but are not limited to this. They can be rectangles extending in any direction or not rectangular.

[0114] A plurality of photographing optical systems 3 that form openings between the light-emitting pixels 4b of the display panel 4 can be provided. For example, Figure 5 shows a first photographing optical system 3A and a second photographing optical system 3B. In each photographing optical system, as an example, an elliptical opening having a major axis in the second direction is formed as a first optical system 9A, and an elliptical opening having a major axis in the first direction is formed as a second optical system 9B. A plurality of openings serving as optical systems can be provided in each photographing optical system. In Figure 5 for example, a first photographing unit 8A is provided below the opening of the first optical system 9A, and a second photographing unit 8B is provided below the opening of the second optical system 9B. In addition, without limitation, the photographing unit 8 can be provided at a position offset from the opening.

[0115] In addition, although not shown, a lens can be provided between each opening and the photographing unit 8 or in the opening as a part of the optical system 9 so that light is appropriately diffused and focused onto the photographing area of the photographing unit 8 after passing through the opening. Instead of a lens, other optical systems can also be provided so that the photographing area of the photographing unit 8 can appropriately receive light. Here, the opening represents an optically transmissive area, and can also be an air gap, or can be filled with a transparent material such as resin. In addition, the material filled in the opening is not limited, and can be a material that transmits a specific wavelength through a color filter or the like. For example, by filling the opening with materials having different transmittances, refractive indexes, etc., the first optical system 9A and the second optical system 9B can have different optical characteristics.

[0116] The photographing optical system 3, for example, Figure 5 has different opening layouts and is arranged at staggered positions like the first photographing optical system 3A and the second photographing optical system 3B shown. Due to the different opening layouts, the optical characteristics of the optical system are different, and light enters each photographing unit 8 from the display surface of the display unit 2 based on different optical characteristics. That is, light enters the first photographing unit 8A and the second photographing unit 8B based on different optical characteristics.

[0117] For example, as Figure 5 shown, the openings in each photographing optical system can be set as openings that change the direction of the same ellipse. In this way, by making the same opening have different directions, an optical system 9 that generates diffraction images in different directions can be formed.

[0118] In addition, Figure 5The state is shown in which two openings are provided between three consecutive light-emitting pixels along the first direction and the second direction, but it is not limited thereto. For example, openings may be provided separately between more consecutive light-emitting pixels. In this way, when viewed from the display surface, one optical system 9 in the display unit 2 including the display panel 4 is not necessarily a continuous area. For example, it may also be configured to include a plurality of separate areas including openings periodically arranged in a manner stitched between light-emitting pixels and light-emitting pixels.

[0119] Moreover, in Figure 5 the major axis of one opening substantially corresponds to two or three display pixels, but it is not limited thereto. The major axis of the opening may be longer or shorter.

[0120] In addition, they may not be the same openings. In order to generate diffraction images with different characteristics, the major axis of the opening that is part of the second optical system 9B may be longer than the major axis of the opening that is part of the first optical system 9A, or may be shorter than the major axis of the opening that is part of the first optical system 9A. In this way, by making the opening layouts different, the influence of glare can be further made different in the signals output by the first imaging optical system 3A and the second imaging optical system 3B. As a result, the glare suppression operation based on the image acquisition unit 12 can be assisted. The influence of the opening layout on glare changes based on the state of the electronic device 1 or the states of the respective components within the electronic device 1, and thus can be appropriately defined by the design of the electronic device 1.

[0121] Figure 6A and Figure 6B show images in which the analog signals acquired by the respective imaging optical systems 3 shown in Figure 5 are converted and the coordinates are adjusted. Figure 6A is an image based on the image acquired by the first imaging optical system 3A, Figure 6B is an image based on the image acquired by the second imaging optical system 3B.

[0122] The image acquisition unit 12 may perform position adjustment. The adjustment of the shooting position can be performed, for example, in the following manner: making the image of the mirror image of the image acquired by each imaging optical system 3 displayed on the display panel 4 overlap with the reflected image when observing the display surface from the front. For example, when a person is reflected at the center of the display surface, the position is adjusted so that the image acquired from the imaging optical system 3 is displayed at the center of the display surface. For example, correction may also be performed based on the deviation from the center position of the display surface of the imaging optical system 3. In addition, the adjustment of the position is not limited thereto, and it can be appropriately controlled, and any method can be executed.

[0123] For example, when performing position adjustment in the above manner, as Figure 6A , Figure 6BGlare is generated in the white area shown. For example, as shown in the figure, glare is generated as an area that is brighter than the image actually intended to be captured.

[0124] In this case, in order to suppress glare generated in different parts, the image acquisition unit 12 acquires pixel values with lower light intensity or luminance (brightness) values after conversion into digital signals in the signals output from the first imaging optical system 3A and the second imaging optical system 3B to correct the image. In addition, when processing the signal before conversion into a digital signal, the pre-processing unit 10 can make a selection based on the signal intensity.

[0125] Figure 7 Represents an image output from the electronic device 1 after being processed by the image acquisition unit 12. After the processing by the image acquisition unit 12 as described above, the post-processing unit 14 adjusts the lightness and the like. By outputting the image processed in this way, as Figure 7 shown, glare can be suppressed and an image with adjusted natural lightness can be obtained.

[0126] For example, Figure 6A , Figure 6B The area where glare is generated, such as that, sometimes depends on each imaging optical system 3. Thus, the glare generation area can be stored in advance in the storage unit 20 or the image acquisition unit 12. As a result, the pixel value selection process or the pixel value synthesis process can be executed quickly. For example, regarding a certain pixel, instead of comparing the pixel values output from each imaging optical system 3, the image is obtained based on the output of a specified imaging optical system 3, and so on.

[0127] As described above, according to the present embodiment, by making the directions in which diffracted light is generated different in a plurality of imaging optical systems, it is possible to suppress glare in an image captured by a front camera disposed so as to overlap with the display, which is provided on the inner surface of the display unit of the display. As a result, in the electronic device, without increasing the border width of the display setting surface, an imaging optical system capable of acquiring a high-precision image can be provided on the front side of the display.

[0128] For example, when light emitted from a strong light source irradiates a part of the display surface, glare is likely to be generated in the imaging optical system 3 near the light irradiation area. Also in this case, it is possible to obtain an image with glare suppressed as described above by using an image acquired by at least one other imaging optical system 3 provided at a different position.

[0129] In addition, the first imaging optical system 3A and the second imaging optical system 3B are selected from two of the plurality of imaging optical systems, and three or more imaging optical systems 3 may also be provided. That is, among the three or more imaging optical systems 3, at least two imaging optical systems 3 may function as the first imaging optical system 3A and the second imaging optical system 3B described above. In this embodiment, the functions and effects of the embodiments described below can be achieved. In addition, among the three or more imaging optical systems 3, by providing two or more sets of imaging optical systems having the characteristics of the group of the first imaging optical system and the second imaging optical system, the degree of freedom in obtaining the captured image can be further improved, and the accuracy of suppressing glare can be further improved. In this case, it is not necessary to use the same combination, and the same imaging optical system can be used for different combinations. For example, when 3X, 3Y, and 3Z are provided as the imaging optical systems, two combinations common to X, (3X, 3Y) and (3X, 3Z), may be provided as the combination of (the first imaging optical system, the second imaging optical system).

[0130] In addition, in the above description, each imaging optical system 3 has an aperture as the optical system 9, but it is not limited thereto. For example, it may be configured such that one has an aperture and the other does not. In this case, similarly to the above, glares are generated at different positions from the plurality of imaging optical systems 3, and thus the influence of glare can be suppressed by the images obtained by the plurality of imaging optical systems 3.

[0131] In addition, the shape of the aperture is an ellipse, but it is not limited thereto. For example, it may be a rectangle or a rectangle with rounded corners. In addition, it may be formed by any closed curve as long as the imaging unit 8 can appropriately receive light. In addition, the shape may not be the same in the thickness direction of the aperture, that is, the third direction. For example, it may be a more complex shape such that the upper part, that is, the side closer to the display surface, is a rectangle and the side closer to the imaging unit 8 is an ellipse.

[0132] In addition, in the above description, the shape of the aperture and the like are described as optical characteristics, but it is not limited thereto. For example, different characteristics may be obtained by changing the material filled in the aperture for each imaging optical system 3. In addition, for example, as the optical system 9, a λ / 4 wave plate may be provided. In this case, the p-wave may be shielded in order to reduce the glare caused by the influence of reflection in the display panel 4. Different wave plates may be provided for the plurality of imaging optical systems 3 as the optical system 9. As a result, the influence of glare caused by reflection, diffraction, etc. in the display unit 2 can be changed for each imaging optical system 3, and various image correction methods can be used.

[0133] (Second Embodiment)

[0134] The electronic device according to this embodiment has a plurality of imaging optical systems, which have openings with the same layout and can reduce the influence of glare.

[0135] Figure 8 In the same manner as Figure 5 the display surface of the electronic device 1 according to this embodiment is shown. In this embodiment, the first imaging optical system 3A and the second imaging optical system 3B have the same opening layout. On the other hand, different from the first embodiment, the first imaging optical system 3A and the second imaging optical system 3B are located at different positions in the second direction. That is, each optical system has an elliptical opening with a major axis formed along the second direction and is arranged at a staggered position in the second direction.

[0136] Figure 9 is an image acquired by the second imaging optical system 3B. For example, the image acquired by the first imaging optical system 3A is Figure 6A . In this way, both of the two imaging optical systems 3 have elliptical openings with a major axis formed along the second direction, so the directions of glare generation are the same. However, since the first imaging optical system 3A and the second imaging optical system 3B are arranged staggeredly in the second direction, glare is generated at staggeredly arranged positions in the second direction.

[0137] Therefore, similar to the above first embodiment, the image acquisition unit 12 can acquire an image with the influence of glare suppressed. In addition, the first imaging optical system 3A and the second imaging optical system 3B are also staggered in their first direction. In this way, when they are staggered in both the first direction and the second direction, the center points in the first direction where glare is generated in each imaging optical system become different positions, so an image with better suppression of glare can be acquired.

[0138] (Third Embodiment)

[0139] In this embodiment, an electronic device having three imaging optical systems will be described.

[0140] Figure 10 is a diagram showing an example of the electronic device 1 according to this embodiment. The electronic device 1 includes a first imaging optical system 3A, a second imaging optical system 3B, and a third imaging optical system 3C on the side opposite to the display surface of the display unit 2. For example, the imaging optical systems are arranged along the first direction. The first imaging optical system 3A is arranged at the center of the screen, and the second imaging optical system 3B and the third imaging optical system 3C are respectively arranged near the ends (boundaries) of the display surface of the display unit 2 with the first imaging optical system 3A interposed therebetween. Nearby, for example, means from one to several pixels of the display element from the end or boundary. As another example, it can be several millimeters from the end of the housing of the electronic device 1, or a few percent of the width or height of the electronic device 1.

[0141] With such a configuration, the parallax between the first imaging optical system 3A and the second imaging optical system 3B is equal to the parallax between the third imaging optical system 3C and the first imaging optical system 3A. That is, it is thus possible to use the image obtained by the second imaging optical system 3B and the image obtained by the third imaging optical system 3C to generate an image with a (nearly) zero parallax with respect to the first imaging optical system 3A.

[0142] This setting is because even if pixel damage occurs at the end of the display surface, the impact on the user is very small. On the other hand, for example, the size of the first imaging optical system 3A near the center of the display surface can be made smaller than the sizes of the other imaging optical systems. Here, the size can refer to, for example, the size of the aperture or the size of the imaging unit 8. By configuring in this way, the image displayed by the display unit 2 can be made more natural.

[0143] In Figure 10 , for example, the first imaging optical system 3A includes an ellipse having a major axis formed along the first direction as the optical system 9, and the second imaging optical system 3B and the third imaging optical system 3C may also include an ellipse having a major axis formed along the second direction as the optical system 9.

[0144] Figure 11 is a diagram showing the images that can be obtained from each imaging optical system when having such an aperture. From the top, the images obtained by the first imaging optical system 3A, the second imaging optical system 3B, and the third imaging optical system 3C are shown in sequence. In this way, flare is generated along the second direction in the first imaging optical system 3A, and flare with a shifted position is generated along the first direction in the second imaging optical system 3B and the third imaging optical system 3C.

[0145] Figure 12 is an image in which the images of the second imaging optical system 3B and the third imaging optical system 3C are superimposed considering parallax. In this way, when obtaining an image with suppressed flare considering parallax, for example, as Figure 12 shows, the flare can be located at both ends. Using this image and the image at the top of Figure 11 , for example, an image with suppressed flare as shown in Figure 7 can be obtained. The synthesis or correction of these images can be performed by the image acquisition unit 12.

[0146] In the above description, the second imaging optical system 3B and the third imaging optical system 3C are particularly provided at both ends, but it is not limited thereto. For example, the second imaging optical system 3B and the third imaging optical system 3C are provided at positions where the same parallax occurs with the first imaging optical system 3A interposed therebetween, and an image with suppressed glare can be acquired in the same manner. In addition, in the present embodiment, it is set to be displaced in the first direction, but it may also be displaced in the second direction. Further, the imaging optical systems 3 may be arranged at positions displaced in both the first direction and the second direction.

[0147] (Fourth Embodiment)

[0148] The intensity of glare generation sometimes varies depending on the direction. For example, it may be strong in the first direction and weak in the second direction. In this case, in the present embodiment, the imaging optical system that is likely to generate glare in the first direction or the second direction is appropriately selected, and an image with suppressed glare is acquired based on the signal obtained from the selected imaging optical system.

[0149] Figure 13 For example, when Figure 5 acquiring an image in the imaging optical system shown, an example of acquiring an image based on the output from the first imaging optical system 3A where glare is strong in the second direction and weak in the first direction is shown. As Figure 13 shown, since the glare intensity in the first direction is weak, an image in which the glare portion is darker than Figure 6A is acquired. On the other hand, since the glare in the second direction is strong, the image output from the second imaging optical system 3B is such that Figure 6B an image with a bright glare portion is acquired as shown.

[0150] In this way, when the strength of glare varies depending on the direction, an image with suppressed glare can be acquired based on the output from the first imaging optical system 3A. Regarding the strength of glare, for example, in the pre-processing unit 10, the variances of the brightness in the first direction and the second direction are calculated from the image data output from each imaging optical system 3. For example, the variance along the first direction is calculated for each row and averaged, and the variance along the second direction is calculated for each column and averaged. It can be known that the direction with a high average value has a strong glare intensity. In addition, instead of based on the variance, it can also be determined based on the difference between the maximum brightness and the minimum brightness of each row or each column.

[0151] As described above, the imaging optical system 3 in which glare is likely to occur in which direction depends on the optical system 9 of each imaging optical system 3. For example, as described above, it is determined based on the direction of the aperture of the optical system 9. Based on the direction of this aperture, the priority imaging optical system 3 can be determined in advance when glare occurs in any direction.

[0152] In addition, the case of whether glare is likely to occur according to the aperture layout is described, but it is not limited thereto. For example, the optical system for controlling glare can be not only the aperture layout but also the entire display panel including circuit wiring. Depending on what pattern type is periodically arranged on the circuit and what kind of interference of light is generated, the glare and diffraction shape will also change. Based on these factors, it is possible to determine in which direction glare is likely to occur.

[0153] The image acquisition unit 12 acquires an image with reduced glare based on the imaging optical system 3 that forms a priority direction different from the glare generation direction. For example, when the following images are output through each imaging optical system 3 Figure 13 and Figure 6B such images, it is possible to select and output an image with less glare Figure 13 . In addition, it is possible to calculate the weighted average of the images of Figure 13 after increasing the weight of the image of Figure 13 and Figure 6B .

[0154] As another example, when strong glare is generated in the second direction, multiple imaging optical systems 3 with the first direction as the priority direction are used, and an image is acquired based on any of the above acquisition (e.g., selection, synthesis) methods.

[0155] In this way, when glare has a polarity, it is possible to acquire an image with reduced glare based on the image output by the imaging optical system 3 with a direction different from this direction as the priority direction.

[0156] (Fifth Embodiment)

[0157] In the electronic device 1 of this embodiment, even when glare is generated, the generation of glare can be reduced in any imaging optical system 3.

[0158] Figure 14 is a diagram showing a cross-sectional view of the electronic device 1 according to this embodiment. As Figure 14 shown, the electronic device 1 has a light shielding portion 30 between the multiple imaging optical systems 3. The light shielding portion 30 can be a light shielding film formed of a material with high light shielding properties, or an absorption film formed of a material with high light absorption rate.

[0159] In this way, on the side of the display panel 4 opposite to the display surface, a light shielding portion 30 that prevents light from propagating between the imaging optical systems 3 can be provided.

[0160] Figure 15 As another example, the light shielding portion 30 is arranged so that the light reflected in the display panel 4 and the circular polarizing plate 5 does not propagate in the first direction and the second direction with respect to the multiple imaging optical systems 3.

[0161] Figure 16 This is another example. The light-shielding portion 30 is provided to penetrate the display panel 4. By setting it in this way, not only the light reflected in the display portion 2 can be blocked, but also the light reflected outside the display portion 2 can be blocked.

[0162] For example, as Figure 15 , Figure 16 shown, by arranging the light-shielding portion 30 in the area including the display panel 4 and the circularly polarized light plate 5, the generation of glare caused by light such as reflection and diffraction in these layers can be suppressed. In a certain area, even if strong glare is generated due to the influence of incident light or the like, and strong glare is generated in the photographing optical system 3 for the light incident on this area, the influence of glare can also be reduced in other photographing optical systems 3 separated by the light-shielding portion 30.

[0163] In addition, for example, as Figure 14 , Figure 16 shown, by reducing the influence of light reflection and diffraction on the opposite side in the third direction of the display surface of the display portion 2, the influence between the photographing portions 8 can be suppressed.

[0164] In addition, in the above description, the case where the light-shielding portion 30 is provided to the circularly polarized light plate 5 has been described. However, for example, the light-shielding portion 30 can also be provided in the touch panel 6. Moreover, it can also be formed so that the very thin light-shielding portion 30 reaches the area of the cover glass 7. In this case, the display of the display portion 2 in the display surface can be appropriately adjusted in size and arranged to form a natural image for the user. Also, in the display panel 4, the brightness of the light-emitting pixels 4b arranged around the light-shielding portion 30 can also be made higher than that of other light-emitting pixels 4b by software.

[0165] In this way, even when glare is generated, by shielding each of the plurality of photographing optical systems 3 from each other, when strong glare is generated in a certain photographing optical system 3, other photographing optical systems 3 can be used to obtain an image with less influence of glare. The image acquisition unit 12 can, for example, compare or synthesize the output values of the plurality of photographing optical systems 3 in the same manner as in the above-described embodiment to obtain an image with less influence of glare.

[0166] (Seventh Embodiment)

[0167] In each of the above-described embodiments, the image acquisition unit 12 obtains an image with less influence of glare through a prescribed operation (including comparison). In contrast, in the present embodiment, a model is used to synthesize the outputs from the plurality of photographing optical systems 3 to obtain an image with suppressed glare.

[0168] For example, the model can be a statistical model. A model is statistically calculated for each imaging optical system 3 to determine what kind of calculation is used for synthesis, and the image acquisition unit 12 inputs the information obtained from the multiple imaging optical systems 3 into this model, thereby obtaining an image with less influence of glare.

[0169] For example, the model can be a neural network model trained through deep learning. The neural network model can be formed by an MLP (Multi-Layer Perceptron), a CNN (Convolutional Neural Network), etc. In this case, parameters trained with multiple teacher data can also be pre-stored in the storage unit 20 or the image acquisition unit 12, and the image acquisition unit 12 forms a neural network model based on the stored parameters. Using the formed trained model, the image acquisition unit 12 uses the data output from the multiple imaging optical systems 3 to obtain an image with reduced glare.

[0170] Moreover, when using the trained model, the electronic device 1 can also use the captured image to further improve the training accuracy. For example, training can also be performed in the control unit 18 etc. of the electronic device 1. As another example, multiple electronic devices 1 can send data to a storage device etc. existing in the cloud etc., perform training in a server etc., and reflect the re-trained parameters to the electronic device 1. In this case, in order not to include privacy information containing the user's facial information, only glare information can be sent. In addition, the data transmission and reception of the electronic device 1 can be set to a state that can be selected by the user, such as opt-in or opt-out.

[0171] In this way, the image acquisition unit 12 can obtain an image not only through linear processing, but also through non-linear processing, especially through operations using various models including the trained model.

[0172] (Eighth Embodiment)

[0173] In the above-described embodiment, the image acquisition unit 12 obtains an image through image synthesis etc. based on pixel values, preferred directions, or a method using a model. In this embodiment, the area where image correction is performed is defined in each imaging optical system 3.

[0174] For example, as Figure 6A etc. show, in each imaging optical system 3, the area where glare is generated mostly determines a bright subject as a bright spot. This is based on the characteristics of the optical system 9 in each imaging optical system 3 or the configuration of the imaging optical system 3 etc. Therefore, when there is a bright subject, the image acquisition unit 12 can predict the glare generation area of each imaging optical system 3 and correct the image based on the prediction result.

[0175] Glare is often generated by adjacent optical elements. Therefore, the parallax in the images obtained by the multiple imaging optical systems 3 is likely to be larger than that of other subjects. Thus, the parallax can be calculated by the image acquisition unit 12 or the like, and the area with a large parallax is predicted as the glare generation area.

[0176] Regarding correction, for example, in an area where glare is likely to be generated in a certain imaging optical system 3, the pixel value is determined based on the information obtained from other imaging optical systems 3. For example, in addition to the comparison operation described in the first embodiment, interpolation processing from other imaging optical systems 3 can also be performed in this area. Without limitation, correction processing can also be performed by other operations such as weighted operations that increase the influence of the pixel values output from other imaging optical systems 3.

[0177] In addition, when there are three or more imaging optical systems 3 and the probability of glare generation in these other multiple imaging optical systems 3 is low in this area, an image with less glare influence can also be obtained based on the information obtained from these other multiple imaging optical systems 3.

[0178] In addition, this area can be updated by reflecting it to the model after training in the above embodiment. By reflecting it as training, for example, the correction accuracy can be improved among different users having the same electronic device 1.

[0179] In addition, as another example, this area can be determined, for example, by making light incident on the display panel 4 without the display panel 4 displaying. Instead of not having the display panel 4 display, the display panel 4 can also be illuminated with a glare generation intensity to predict the area. In this way, the process of determining the area where glare is generated can be performed not at the shooting timing desired by the user but at other timings, such as before and after shooting. This process can be executed by the control unit 18, for example, or by the image acquisition unit 12.

[0180] (Ninth Embodiment)

[0181] In this embodiment, the electronic device 1 includes a microlens array as the optical system 9 of the imaging optical system 3.

[0182] Figure 17 FIG. shows the imaging optical system 3 according to this embodiment. The optical system 9 of the imaging optical system 3 includes a microlens array 32. The light passing through the microlens array 32 is appropriately incident on the imaging unit 8 and converted into a signal and output from the imaging unit 8.

[0183] The preprocessing unit 10 reconstructs an image based on the signal output from the imaging unit 8. The image acquisition unit 12 acquires an image with the influence of glare reduced by the above-described embodiments based on the reconstructed image.

[0184] Figure 18 This is a diagram showing another example of the imaging optical system 3 according to the present embodiment. The first imaging optical system 3A and the second imaging optical system 3B include a common microlens array 32 as the optical system 9. In this way, since the microlens array 32 limits the region where the light converged by each lens is incident to a certain extent, a plurality of imaging optical systems 3 can be formed by arranging the imaging units 8 by region. In this case, for example, the light-shielding unit 30 described in the fifth embodiment can be provided between the first imaging unit 8A and the second imaging unit 8B. Figure 18 In this case, two imaging optical systems are provided, but it is not limited to this. Three or more imaging optical systems 3 can also be formed by the same microlens array 32.

[0185] (Tenth Embodiment)

[0186] In the foregoing embodiment, as an example, a configuration in which a plurality of imaging units 8 are provided in the same microlens array 32 has been described. However, in the present embodiment, the electronic device 1 provides the same imaging unit 8 for a plurality of optical systems 9.

[0187] Figure 19 This is a diagram showing an example of the imaging optical system 3 according to the present embodiment. The electronic device 1 provides one imaging unit 8 for the first imaging optical system 3A and the second imaging optical system 3B. In this imaging unit 8, a first imaging unit 8A and a second imaging unit 8B are defined in its imaging area. Light is incident on the first imaging unit 8A via the first optical system 9A, and light is incident on the second imaging unit 8B via the second optical system 9B. In this way, by providing a plurality of optical systems 9 for the same imaging unit 8, a plurality of imaging optical systems 3 can also be formed.

[0188] Figure 20 This is a diagram showing another mounting example of the present embodiment. That is, a microlens array 32 is provided as the optical system 9, and the first imaging optical system 3A and the second imaging optical system 3B are formed using the same microlens array 32 and imaging unit 8.

[0189] Figure 21 is Figure 20 a modification of this, in which a light-shielding unit 30 is provided between the first imaging optical system 3A and the second imaging optical system 3B. By providing the light-shielding unit 30 in this way, the light incident on each region of the imaging unit 8 can be controlled more clearly.

[0190] Figure 22 is Figure 19In a modified example, a light-shielding portion 30 is provided between the first optical system 9A and the second optical system 9B. By providing the light-shielding portion 30 in this way, even if the microlens array 32 is not provided, the light incident on the first imaging unit 8A and the second imaging unit 8B can be clearly controlled.

[0191] In addition, in Figure 21 , Figure 22 , the light-shielding portion 30 does not penetrate the display panel 4, but of course, it may also penetrate the display panel 4 or pass through the display panel 4 as in the example of Figure 16 .

[0192] By sharing the imaging unit 8 among the plurality of imaging optical systems 3 in this way, the circuit configuration and semiconductor process of the imaging unit 8 can be simplified.

[0193] For example, by making part or all of them the same elements as shown in the present embodiment, the plurality of imaging optical systems 3 can be integrated. By integrally arranging the plurality of imaging optical systems 3 in this way, the influence of parallax between the imaging optical systems 3 can also be reduced. As a result, the influence of parallax can also be reduced when correcting the images obtained separately, and the correction accuracy can be improved.

[0194] In addition, instead of sharing the imaging unit 8 or the optical system 9 as shown in the present embodiment, in the configurations of the foregoing embodiments, the plurality of imaging optical systems 3 may be integrated so that the optical systems 9 are adjacent to each other. In this case, high-precision correction can also be performed while reducing the influence of parallax.

[0195] Figure 23A is a diagram showing another arrangement example of the microlens array 32. As Figure 23A shown, the microlens array is arranged on the side closer to the imaging unit 8 within the opening in the optical system 9. In this way, the opening and the microlens array 32 form the optical system 9 as separate structures. Of course, it is not necessary to have an opening, and the optical system 9 may also be formed by arranging the microlens array 32 on the side opposite to the surface of the display panel 4.

[0196] Figure 23B shows another example of the arrangement of the microlens array 32 and the display panel 4. One or more microlens arrays 32 may be provided for a plurality of openings. In this case, the openings may be formed by appropriately leaving the regions of the light-emitting pixels of the display panel 4, and the microlens array 32 may be provided between the lower part of the opening and the imaging unit 8.

[0197] When set to Figure 23A , Figure 23B , of course, the light-shielding portion 30 may also be provided at an appropriate position.

[0198] (Eleventh Embodiment)

[0199] In this embodiment, the chip configuration of the imaging unit 8 and the like will be described.

[0200] Figure 24 This shows an example of the chip configuration according to this embodiment. Its configuration is roughly shown without being a limiting example, and the case where the same chip also has other functions is not excluded. That is, in addition to the illustrated configuration, a selector, an I / F, a power supply node, etc. are appropriately provided.

[0201] As shown in this Figure 24 figure, analog circuits such as the pre-processing unit 10 and logic circuits such as the image acquisition unit 12 and the imaging unit 8 can be provided on the same chip. Moreover, a control unit 18 and a storage unit 20 can also be provided. In this way, by forming the imaging unit 8, the pre-processing unit 10, and the image acquisition unit 12 on the same chip, data processing can be performed at high speed and with signal degradation suppressed without the signal and data being transmitted via an interface.

[0202] In addition, the pre-processing unit 10 may not be shared but provided for each imaging unit 8. In this case, it can also be configured to transmit digital image data from each pre-processing unit 10 to the image acquisition unit 12.

[0203] Figure 25 This is a diagram showing another example of the chip configuration. In Figure 25 this figure, the configurations other than the imaging unit 8, the pre-processing unit 10, the image acquisition unit 12, and the interface are omitted. The imaging unit 8 can be provided on a separate chip. One chip is provided with the imaging unit 8, and another chip is also provided with the imaging unit 8. In one chip, the imaging unit 8 is provided with a reception interface 22 for transmitting information to the other chip.

[0204] In the other chip, a signal is received via the reception interface 22 and combined with the signal of the imaging unit 8 provided in the same chip, and the image acquisition unit 12 performs image acquisition. If necessary, the pre-processing unit 10 performs signal processing before outputting data to the image acquisition unit 12. The pre-processing unit 10 can be provided on two chips or only on the other chip.

[0205] Of course, more imaging units 8 are provided on another chip rather than two chips. By adopting this configuration, even when the imaging optical system 3 is arranged at various positions on the opposite side of the display surface of the display unit 2, the area of the chip related to the imaging unit 8 can be reduced.

[0206] Figure 26Another example is that multiple imaging units 8 are on separate chips, and the image acquisition unit 12 is on another chip. In this way, a configuration can be adopted where the logic circuit for processing digital signals and the photoelectric conversion circuit with a light receiving element are on separate chips. By setting it up like this, without increasing the chip area, the degree of freedom in the configuration of the imaging optical system 3 and the like can be further improved.

[0207] In Figures 24 to 26 , the receiving interface 22 can be, for example, an interface capable of receiving data in accordance with the MIPI standard. In addition, high-speed transmission of other standards can be performed. Depending on the signal transmitted, it can be either a standard for transceiving analog signals or a standard for transceiving digital signals. The interface type can be appropriately selected according to the chip mounting of the imaging unit 8 and the pre-processing unit 10.

[0208] (The twelfth embodiment)

[0209] In each of the above embodiments, the light-emitting pixels 4b of the display panel 4 are not particularly described, but it is also possible to change the generation of glare of the multiple imaging optical systems 3 through the light-emitting pixels 4b, and obtain an image with little influence of glare caused by the image acquisition unit 12.

[0210] For example, when setting the first imaging optical system 3A and the second imaging optical system 3B as two imaging optical systems 3, the light-emitting pixels 4b around the first imaging optical system 3A and the light-emitting pixels 4b around the second imaging optical system 3B can be made different. For example, one is an OLED and the other is a MicroLED. In this way, by making them display optical systems with different light-emitting elements, especially with different optical characteristics, they can have different characteristics regarding the influence of glare.

[0211] When glare is generated based on different optical characteristics, in the images obtained from the imaging optical system 3, the glare generated varies depending on the situation. By making the characteristics of the glare different, for example, an image with a strong influence of glare is obtained from one imaging optical system 3, while an image with a weak influence of glare is obtained from the other imaging optical system 3. In this way, by using light-emitting elements with different characteristics, the image synthesis and correction methods in the image acquisition unit 12 can be extended to various methods.

[0212] (The thirteenth embodiment)

[0213] In the above-described embodiments, images output from multiple imaging optical systems 3 are used to obtain an image with little influence of glare. That is, multiple imaging optical systems 3 are activated, and images are obtained based on the information acquired by each. In this embodiment, at least one imaging optical system 3 is activated according to the need to reduce the influence of glare. For example, there is at least one imaging optical system 3 that is activated when image correction is required.

[0214] For example, in the initial state, only the imaging optical system 3 near the center of the display surface may be activated. When it is determined based on the image obtained by the imaging optical system 3 that it is necessary to reduce the influence of glare, other imaging optical systems 3 are then activated, and an image with less influence of glare is obtained by the image acquisition unit 12.

[0215] For example, when strong light irradiates the entire area of the display surface or only irradiates a part of the area, it is determined that the influence of glare is strong. In addition, the influence of glare can also be determined based on other conditions. In the above description, it is based on the output of one imaging optical system 3, but it is not limited thereto. For example, it can be determined based on the magnitudes of signals of several light receiving elements. In addition, as another example, it can be determined based on the distribution of the brightness, etc. of the image displayed on the display panel 4. In this way, the influence of glare is determined by various determination methods.

[0216] Based on the above determination, the control unit 18 causes an appropriate imaging optical system 3 to operate, acquires a signal based on the incident light, and an image is obtained by the image acquisition unit 12. The determination of the above influence of glare can also be executed by the control unit 18. In this way, at least a part of the imaging optical systems 3 are caused to operate as needed, and energy saving can be achieved.

[0217] (The Fourteenth Embodiment)

[0218] In each of the above embodiments, various modes of the imaging optical system 3 and the image acquisition unit 12 have been described. In this embodiment, the configuration of the imaging optical system 3 will be described.

[0219] From Figures 27 to 31B are diagrams showing configuration examples of the imaging optical system 3 as viewed from the display surface. These diagrams each show an example of the configuration and are not limited to the drawings. In addition, it may include a part or all of the configurations in multiple drawings. That is, the electronic device 1 may be provided with three or more imaging optical systems 3 that do not only have the features in one drawing but comprehensively have the features of multiple drawings.

[0220] As Figure 27 shown, the imaging optical system 3 can be provided at both ends (near the boundary) in an arbitrary direction, for example, the first direction, of the display screen 1a. By configuring in this way, the imaging optical system 3 can be provided at a position that is not conspicuous to the user. It can also be provided at the upper and lower ends, which are the two ends in the second direction, respectively, that is Figure 27 in the above.

[0221] As Figure 28As shown, within the display screen 1a, the distance between the two imaging optical systems 3 can be set to 50 mm to 80 mm. By configuring in this way, images with a parallax close to that of the human eye can be obtained from the two imaging optical systems 3. Thus, the influence of glare can be reduced among the multiple imaging optical systems 3, and a stereoscopic image with a natural impression for humans can be generated. Moreover, a third imaging optical system 3 can be arranged at other positions. Based on the parallax between the third imaging optical system 3 and the two imaging optical systems 3, an image with further suppressed glare can also be obtained. Thus, a stereoscopic image can be obtained on the basis of improving the accuracy of glare suppression.

[0222] As Figure 29 shown, the multiple imaging optical systems 3 can also be arranged in a staggered manner in each of the first direction and the second direction. For example, by providing openings with major axes formed in the same direction in each optical system 9, images containing glare deviated from the center in both the first direction and the second direction can be obtained. Thus, the image acquisition unit 12 can obtain an image with highly accurate glare suppression through simpler processing (such as the comparison processing and selection processing shown in the first embodiment).

[0223] As Figure 30 shown, three or more imaging optical systems 3 can also be provided. When arranging three or more imaging optical systems 3, they can be arranged without symmetry as Figure 30 shown, or they can be arranged with a certain symmetry with respect to the first direction, the second direction, or the center point, etc. In addition, the third imaging optical system 3 can be arranged at a position far from the other two imaging optical systems 3. In this case, it is also possible to make the glare intensity caused by the position different in each imaging optical system 3.

[0224] Figure 31A , Figure 31B denotes the same electronic device 1. Figure 31B is an external view observed from the arrow direction of Figure 31A . In this way, multiple display screens 1a can be provided. The imaging optical systems 3 in each display screen can also be arranged at the same position facing the display screen 1a as shown in the drawings, or can be arranged at completely different positions. For example, at least two imaging optical systems 3 are arranged at both ends along the first direction on the surface, and at least two imaging optical systems 3 are arranged at both ends along the second direction on the inner surface. In this case, for example, Figure 31B the imaging optical system 3 in can also be used as a rear camera during normal times, and can also be utilized as a rear camera with dual cameras (or the number is two or more).

[0225] In this way, the imaging optical system 3 can be arranged on the display screen 1a in various ways.

[0226] As shown in the above embodiments, the plurality of photographing optical systems 3 may be arranged at least two apart. The arrangement may be freely selected regardless of the drawings. Figure 2 In the above, by setting a plurality of shooting optical systems 3 staggered along the first direction at the center of the second direction of the display surface, it is possible to obtain an image that is natural to the user's line of sight when the user observes the center of the screen. In addition, it is not limited to this, for example, it can be set slightly above the second direction. For example, when obtaining an image of one's own face, the center of the face is located at the center of the screen, and the shooting optical system can be configured near the displayed eyes. In addition, in addition to the above, it can also be set near the center of the first direction along the second direction. In this case, the same image can be obtained by rotating the screen.

[0227] Next, another example of the layout of the openings included in the optical system 9 is shown. Figure 5 , Figure 8 Rather than the simple shapes shown here, there are more complex layouts.

[0228] Figures 32 to 35 FIG. 1 is a diagram showing an example of the layout of the openings. Figure 32 As shown, for example, relative to the display element, the openings are also staggered along the second direction.

[0229] like Figure 33 As shown, for example, the opening is formed between the display elements in a shape that is a combination of an ellipse whose major axis is formed along the first direction and an ellipse whose major axis is formed along the second direction.

[0230] like Figure 34 As shown, for example, the opening belonging to the first optical system 9A and the opening belonging to the second optical system 9B may not intersect at an angle close to 90°. For example, in this case, the angle of the light emitting element of the display panel 4 may also be partially changed. For example, the configuration angle of the light emitting element provided on the display surface in the third direction of any optical system 9 may be rotated 45° or any meaningful angle.

[0231] Of course, if Figure 35 As shown, you can also Figure 33 The openings shown are configured as Figure 34 Like that.

[0232] In this way, the opening as a part of the optical system 9 can also be arranged to intentionally generate glare in other directions. Figure 5 , Figure 8 , Figures 32 to 35The open layouts shown are combined. In this case, since the direction of glare generation depends on the imaging optical system 3 and further becomes a combination of various directions, the range of image acquisition such as image correction, synthesis, and selection can be further expanded.

[0233] As described above, according to each embodiment, the imaging optical system 3 is disposed on the side of the display unit 2 opposite to the display surface, and light passing through the display unit 2 is acquired by the plurality of imaging units 8. Based on the fact that a part of the light passing through the display unit 2 is repeatedly reflected within the display unit 2, it is incident on the imaging units 8 within the plurality of imaging optical systems 3. According to each of the above embodiments, an image is acquired by signals obtained by the plurality of imaging optical systems 3, and it is possible to simply and reliably suppress the glare component and diffracted light component included in the light incident on the plurality of imaging units 8 (including the integrated case) based on repeated reflection within the display unit 2 in the captured image.

[0234] In addition, for example, the processing in the image acquisition unit 12 or the like may be constituted by a digital circuit, or may be constituted by a programmable circuit such as an FPGA (Field Programmable Gate Array). In addition, it may be that the processing content is described by a program, and hardware resources such as a CPU are used to specifically implement software-based information processing.

[0235] Next, several application examples are listed.

[0236] (Fifteenth Embodiment)

[0237] As a specific candidate configuration of the electronic device 1 having the configuration described in the foregoing embodiments, various configurations can be considered. For example, Figure 36 is a plan view when the electronic device 1 of each embodiment is applied to the capsule endoscope 50. Figure 36 The capsule endoscope 50, for example, includes within a housing 51 having hemispherical shapes at both end faces and a cylindrical shape at the central portion: a camera (ultra-small camera) 52 for photographing an image inside the body cavity; a memory 53 for recording image data photographed by the camera 52; and a wireless transmitter 55 for transmitting the recorded image data to the outside through an antenna 54 after the capsule endoscope 50 is discharged from the subject's body.

[0238] In addition, a CPU (Central Processing Unit) 56 and a coil (magnetic force or current conversion coil) 57 are provided inside the housing 51. The CPU 56 controls the photographing of the camera 52 and the data storage operation to the memory 53, and controls the transmission of data from the memory 53 to a data receiving device (not shown) outside the housing 51 through the wireless transmitter 55. The coil 57 supplies power to the camera 52, the memory 53, the wireless transmitter 55, the antenna 54, and a light source 52b described later.

[0239] Moreover, the housing 51 is provided with a magnetic (wire) switch 58 for detecting the capsule endoscope 50 when it is placed on the data receiving device. When the wire switch 58 detects that the capsule endoscope 50 is placed on the data receiving device and is capable of transmitting data, the CPU 56 supplies power from the coil 57 to the wireless transmitter 55.

[0240] The camera 52 includes, for example, an imaging element 52a including an optical system 9 for capturing an image of the body cavity and a plurality of light sources 52b for illuminating the body cavity. Specifically, the camera 52 is constituted by, for example, a CMOS (Complementary Metal Oxide Semiconductor) sensor, a CCD (Charge Coupled Device), etc. having an LED (Light Emitting Diode) as the light source 52b.

[0241] The display unit 2 in the electronic device 1 of the foregoing embodiment includes Figure 36 the concept of a light-emitting body such as the light source 52b. In Figure 36 the capsule endoscope 50, for example, has two light sources 52b, and these light sources 52b may be constituted by a display panel 4 having a plurality of light source units or an LED module having a plurality of LEDs. In this case, by disposing the imaging unit 8 of the camera 52 below the display panel 4 and the LED module, the restriction on the layout configuration of the camera 52 can be reduced, and thus a smaller capsule endoscope 50 can be realized.

[0242] (Sixteenth Embodiment)

[0243] In addition, Figure 37 is a rear view when the electronic device 1 of the foregoing embodiment is applied to a digital single-lens reflex camera 60. The digital single-lens reflex camera 60 and the compact camera have a display unit 2 for displaying a preview screen on the back surface on the side opposite to the lens. The photographing optical system 3 may be disposed on the side opposite to the display surface of the display unit 2, and the face image of the photographer can be displayed on the display screen 1a of the display unit 2. In the electronic device 1 of each of the foregoing embodiments, since the photographing optical system 3 can be disposed in the region overlapping the display unit 2, there is no need to provide the photographing optical system 3 in the frame portion of the display unit 2, and the size of the display unit 2 can be made as large as possible.

[0244] (Seventeenth Embodiment)

[0245] Figure 38A is a plan view showing an example in which the electronic device 1 of the foregoing embodiment is applied to a head-mounted display (hereinafter referred to as an HMD) 61. Figure 38AThe HMD61 can be used in VR (Virtual Reality), AR (Augmented Reality), MR (Mixed Reality), or SR (Substitutional Reality), etc. Existing HMDs, such as Figure 38B shown, have a camera 62 mounted on the outer surface. The wearer of the HMD can see the surrounding images. On the other hand, people around do not know the eyes and facial expressions of the HMD wearer.

[0246] Therefore, Figure 38A discloses that a display surface of a display unit 2 is provided on the outer surface of the HMD61, and an imaging optical system 3 is provided on the side opposite to the display surface of the display unit 2. Thus, the facial expression of the wearer captured by the imaging optical system 3 can be displayed on the display surface of the display unit 2, and people around the wearer can grasp the facial expression and eye movements of the wearer in real time.

[0247] In Figure 38A 's example, the imaging optical system 3 is provided on the back side of the display unit 2. Therefore, the installation location of the imaging optical system 3 is not restricted, and the design freedom of the HMD61 can be improved. In addition, since the camera can be arranged at the optimal position, it is possible to prevent problems such as misalignment of the line of sight of the wearer displayed on the display surface.

[0248] In this way, in the present embodiment, the electronic device 1 of the embodiment can be used for various purposes and its utilization value can be improved.

[0249] In addition, the present technology can be configured as follows.

[0250] (1) An electronic device, comprising:

[0251] A display unit having a displayable area, the displayable area having display optical systems arranged in an array along a first direction and a second direction intersecting the first direction;

[0252] A plurality of imaging optical systems, in a third direction intersecting the first direction and the second direction, overlapping with the displayable area and arranged on the side of the display unit opposite to the display surface. The plurality of imaging optical systems at least include:

[0253] A first imaging optical system; and

[0254] A second imaging optical system having coordinates different from those of the first imaging optical system in at least one of the first direction and the second direction; and

[0255] The image acquisition unit acquires image data based on information obtained by the first imaging optical system and the second imaging optical system.

[0256] (2) The electronic device according to (1), wherein

[0257] Light passes through optical systems with different optical characteristics and propagates from the display surface of the display unit to the first imaging optical system and the second imaging optical system.

[0258] (3) The electronic device according to (1) or (2), wherein

[0259] The display unit has an opening for propagating light incident from the display surface,

[0260] Light incident from the display surface passes through the opening and propagates to the imaging optical system.

[0261] (4) The electronic device according to (3), wherein

[0262] The opening for propagating light to the first imaging optical system and the opening for propagating light to the second imaging optical system have different layouts.

[0263] (5) The electronic device according to (3) or (4), wherein

[0264] The opening for propagating light to the first imaging optical system and the opening for propagating light to the second imaging optical system form diffraction images in different directions.

[0265] (6) The electronic device according to any one of (1) to (5), wherein

[0266] A third imaging optical system is provided, and the parallax between the third imaging optical system and the first imaging optical system is the same as the parallax between the first imaging optical system and the second imaging optical system.

[0267] The image acquisition unit acquires the image data based on information obtained based on information obtained from the second imaging optical system and the third imaging optical system, and information obtained from the first imaging optical system.

[0268] (7) The electronic device according to (6), wherein

[0269] In the first direction or the second direction of the display surface,

[0270] The first imaging optical system is disposed near the center,

[0271] The second imaging optical system and the third imaging optical system are disposed near the boundary of the display surface with the first imaging optical system therebetween.

[0272] As compared with the second imaging optical system and the third imaging optical system, the area of the first imaging optical system facing the display surface in the third direction is small.

[0273] (8) The electronic device according to any one of (1) to (7), wherein

[0274] The first imaging optical system and the second imaging optical system have different coordinates in the first direction and the second direction.

[0275] (9) The electronic device according to any one of (1) to (8), wherein

[0276] When the image acquisition unit synthesizes the data acquired from the first imaging optical system and the data acquired from the second imaging optical system, the image acquisition unit obtains a shooting result based on the data with low intensity in the image data as a synthesis result.

[0277] (10) The electronic device according to any one of (1) to (9), wherein

[0278] The first imaging optical system and the second imaging optical system each have a direction in which reflection is prioritized.

[0279] When the difference in output in each direction exceeds a specified value, the image acquisition unit obtains a shooting result using any result.

[0280] (11) The electronic device according to any one of (1) to (10), wherein

[0281] Light shielding is performed between the first imaging optical system and the second imaging optical system.

[0282] (12) The electronic device according to any one of (1) to (11), wherein

[0283] The image acquisition unit uses a trained model to synthesize the information acquired by the first imaging optical system and the second imaging optical system.

[0284] (13) The electronic device according to (12), wherein

[0285] The trained model is trained based on data collected from a plurality of electronic devices.

[0286] (14) The electronic device according to any one of (1) to (13), wherein

[0287] When the parallax in a plurality of images acquired by the plurality of imaging optical systems exceeds a specified amount, the image acquisition unit performs correction.

[0288] (15) The electronic device according to any one of (1) to (14), wherein

[0289] At least one of the imaging optical systems is constituted by a microlens array.

[0290] (16) The electronic device according to (15), wherein

[0291] A plurality of the imaging optical systems are provided in a region where the microlens array is provided.

[0292] (17) The electronic device according to any one of (1) to (16), wherein

[0293] The first imaging optical system and the second imaging optical system acquire information through the same imaging element.

[0294] (18) The electronic device according to (17), wherein

[0295] The image acquisition unit and the imaging element are arranged on the same chip.

[0296] (19) The electronic device according to any one of (1) to (18), wherein

[0297] The display unit includes a plurality of display optical systems having different optical characteristics.

[0298] (20) The electronic device according to any one of (1) to (19), wherein

[0299] At least one of the plurality of imaging optical systems operates when the image acquisition unit requires a correction signal.

[0300] (21) The electronic device according to any one of (1) to (20), wherein

[0301] The first imaging optical system and the second imaging optical system are integrally formed.

[0302] (22) The electronic device according to any one of (1) to (21), wherein

[0303] The first imaging optical system and the second imaging optical system are provided near the boundary of the display surface.

[0304] (23) The electronic device according to any one of (1) to (22), wherein

[0305] The first imaging optical system and the second imaging optical system are arranged at a distance of more than 50 mm and less than 80 mm apart.

[0306] The image acquisition unit generates parallax image data of the information acquired by the first imaging optical system and the second imaging optical system.

[0307] (24) The electronic device according to any one of (1) to (23), wherein

[0308] The display unit is provided on two surfaces of the electronic device.

[0309] (25) The electronic device according to any one of (1) to (24), wherein

[0310] The first imaging optical system and the second imaging optical system are any two arbitrarily selected from the plurality of imaging optical systems.

[0311] (26) An electronic device having a fourth imaging optical system different from the first imaging optical system and the second imaging optical system,

[0312] The fourth imaging optical system and the first imaging optical system or the second imaging optical system have any one of the features described in (1) to (24).

[0313] Aspects of the present invention are not limited to the above-described embodiments, but include various modifications conceivable by those skilled in the art, and the effects of the present invention are not limited to the above. That is, it can be derived from the content defined in the claims and their equivalents, and various additions, changes, and partial deletions can be made without departing from the concept, idea, and gist of the present invention.

Claims

1. An electronic device, comprising: A display unit having a displayable area, the displayable area having display optical systems arranged in an array along a first direction and a second direction intersecting the first direction; A plurality of imaging optical systems, in a third direction intersecting the first direction and the second direction, overlapping with the displayable area and arranged on the side of the display unit opposite to the display surface, the plurality of imaging optical systems at least comprising: A first imaging optical system; and A second imaging optical system having coordinates different from those of the first imaging optical system in at least one of the first direction and the second direction; And An image acquisition unit that acquires image data based on information acquired by the first imaging optical system and the second imaging optical system, The display unit has an opening for propagating light incident from the display surface, Light incident from the display surface propagates through the opening to the imaging optical system, and the opening for propagating light to the first imaging optical system and the opening for propagating light to the second imaging optical system have different layouts, The layout of the opening includes the shape and orientation of the opening, The shape of the opening includes not only a shape formed by an arbitrary closed curve in the plane defined by the first direction and the second direction, but also a shape having different shapes in a third direction intersecting the first direction and the second direction, The display unit includes a display panel having a substrate, The substrate is formed of polyimide, and the opening is formed in the substrate in a manner corresponding to the arrangement positions of the plurality of imaging optical systems.

2. The electronic device according to claim 1, wherein Light passes through optical systems with different optical characteristics and propagates from the display surface of the display unit to the imaging elements of the first imaging optical system and the second imaging optical system.

3. The electronic device according to claim 1, wherein The electronic device is provided with a third imaging optical system, the parallax between the third imaging optical system and the first imaging optical system being the same as the parallax between the first imaging optical system and the second imaging optical system, The image acquisition unit acquires the image data based on information obtained based on information acquired from the second imaging optical system and the third imaging optical system, and information acquired from the first imaging optical system.

4. The electronic device according to claim 3, wherein In the first direction or the second direction of the display surface, The first imaging optical system is disposed near the center, The second imaging optical system and the third imaging optical system are disposed near the boundary of the display surface with the first imaging optical system therebetween, Compared with the second imaging optical system and the third imaging optical system, the area of the first imaging optical system facing the display surface in the third direction is smaller.

5. The electronic device according to claim 1, wherein When the image acquisition unit synthesizes the data acquired from the first imaging optical system and the data acquired from the second imaging optical system, it acquires a shooting result based on the data with low intensity in the image data as the synthesis result.

6. The electronic device according to claim 1, wherein the first imaging optical system and the second imaging optical system each have a direction in which they preferentially reflect, when the outputs in each direction produce a difference of more than a specified value, the image acquisition unit uses an arbitrary result to acquire a shooting result.

7. The electronic device according to claim 1, wherein light shielding is performed between the first imaging optical system and the second imaging optical system.

8. The electronic device according to claim 1, wherein the image acquisition unit uses a trained model to synthesize the information acquired by the first imaging optical system and the second imaging optical system.

9. The electronic device according to claim 1, wherein the image acquisition unit corrects when the parallax in the multiple images acquired by the multiple imaging optical systems exceeds a specified amount.

10. The electronic device according to claim 1, wherein at least one of the imaging optical systems is constituted by a microlens array.

11. The electronic device according to claim 10, wherein a plurality of the imaging optical systems are provided in the area where the microlens array is provided.

12. The electronic device according to claim 1, wherein the first imaging optical system and the second imaging optical system acquire information through the same imaging element.

13. The electronic device according to claim 1, wherein the display unit includes a plurality of display optical systems having different optical characteristics.

14. The electronic device according to claim 1, wherein at least one of the plurality of imaging optical systems operates when the image acquisition unit needs a correction signal.

15. The electronic device according to claim 1, wherein the first imaging optical system and the second imaging optical system are integrally formed.

16. The electronic device according to claim 1, wherein the first imaging optical system and the second imaging optical system are provided near the boundary of the display surface.

17. The electronic device according to claim 1, wherein the first imaging optical system and the second imaging optical system are arranged at a distance of 50 mm or more and 80 mm or less, and the image acquisition unit generates parallax image data of the information acquired by the first imaging optical system and the second imaging optical system.

18. The electronic device according to claim 1, wherein the display unit is provided on two surfaces of the electronic device.

Images