Electronic device

Abstract

[Problem] While reducing the width of a bezel, the degradation of the image quality of a captured image of a camera is also suppressed. [Solution] This electronic device (1) comprises: a display unit (2) that has a displayable area in which a display optical system is provided in an array shape in a first direction and a second direction intersecting the first direction; a plurality of imaging optical systems having at least a first imaging optical system (3) that is disposed on a side opposite to a display surface of the display unit to overlap the displayable area in the first direction and a third direction intersecting the second direction; a second imaging optical system (3) that has coordinates different from those of the first imaging optical system in at least one among the first direction and the second direction; and an image acquisition unit that acquires image data on the basis of information acquired by the first imaging optical system and the second imaging optical system.

Description

[Technical field]

[0001] The present disclosure relates to electronic devices. [Background technology]

[0002] Recent electronic devices such as smartphones, mobile phones, and PCs (Personal Computers) are equipped with cameras in the bezel of the display, making it easy to make video calls and take videos. Smartphones and mobile phones are often carried in a pocket or bag, so their external size needs to be as compact as possible. On the other hand, if the display screen size is small, the higher the display resolution, the smaller the displayed text size becomes, making it difficult to see. For this reason, studies are being conducted to reduce the width of the bezel around the display screen, thereby making it possible to make the display screen size as large as possible without increasing the external size of the electronic device.

[0003] However, because cameras and other devices are often mounted on the bezels of electronic devices, the bezel width cannot be made smaller than the outer diameter of the camera. In addition, when a camera is placed on the bezel, for example, when talking on a video phone, the line of sight is often placed near the center of the display screen, which causes the line of sight to deviate from the optical axis of the camera, resulting in an unnatural captured image where the line of sight is not aligned.

[0004] In order to avoid the above-mentioned problems, it has been proposed to place a camera module on the opposite side of the display unit from the display surface, and to use the camera to capture the subject light that passes through the display unit. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] U.S. Patent Publication 2018 / 0069060 Summary of the Invention [Problem to be solved by the invention]

[0006] However, a portion of the light passing through the display unit is reflected or diffracted before entering the camera, which causes a problem of degradation in the quality of the captured image due to the effects of flare and diffraction caused by reflection.

[0007] One aspect of the present disclosure provides an electronic device that can reduce the width of a bezel while suppressing degradation in image quality of an image captured by a camera. [Means for solving the problem]

[0008] According to one embodiment, the electronic device includes a display unit having a displayable area in which display optical systems are provided in an array along a first direction and a second direction intersecting with the first direction, a plurality of imaging optical systems arranged on the opposite side of the display surface of the display unit in a third direction intersecting with the first direction and the second direction, and including at least a first imaging optical system and a second imaging optical system having coordinates different from those of the first imaging optical system in at least one of the first direction and the second direction, and an image acquisition unit that acquires image data based on information acquired by the first imaging optical system and the second imaging optical system. In this way, by shifting the positions of at least two imaging optical systems, the position where flare occurs in the captured image can be changed. For example, the first imaging optical system and the second imaging optical system may be two optical systems arbitrarily selected from the plurality of imaging optical systems. That is, the configurations listed below may be characteristics for at least two of the plurality of imaging optical systems. Also, one characteristic and the other characteristic may be characteristics for different combinations of the first optical system and the second optical system. In this way, the following characteristics may be possessed by at least two of the plurality of imaging optical systems.

[0009] In the electronic device, light may propagate from the display surface of the display unit to the first imaging optical system and the second imaging optical system through optical systems having different optical characteristics. Furthermore, all of the multiple imaging optical systems may have different optical characteristics, or some of them may have the same optical characteristics. In this way, at least two imaging optical systems may have different optical characteristics in the optical path of light propagating from the display surface. By providing different optical characteristics, the characteristics of the generated flare may be changed.

[0010] The electronic device may include an opening through which light incident from a display surface of the display unit propagates, and the light incident from the display surface may propagate through the opening to the imaging optical system. For example, the optical characteristic described above may be formed by the opening.

[0011] The aperture for propagating light to the first imaging optical system and the aperture for propagating light to the second imaging optical system may have different layouts, which may result in different optical characteristics.

[0012] The aperture for propagating light to the first imaging optical system and the aperture for propagating light to the second imaging optical system may form diffraction images in different directions. For example, the aperture for propagating light to the first imaging optical system may have a size in the first direction larger than the size in the second direction, and the aperture for propagating light to the second imaging optical system may have a size in the second direction larger than the size in the first direction. In this way, if the apertures form diffraction images in different directions, the directions in which flare occurs can be different in the two imaging optical systems.

[0013] The electronic device may include a third imaging optical system having the same parallax with respect to the first imaging optical system as the parallax with respect to the first imaging optical system and the second imaging optical system, and the image acquisition unit may acquire image data based on information acquired based on information acquired from the second imaging optical system and the third imaging optical system and information acquired from the first imaging optical system. That is, for a combination of two imaging optical systems, the electronic device may further include an imaging optical system having an equivalent parallax in the opposite direction with one of these two imaging optical systems as the center.

[0014] In the first or second direction of the display surface, the first imaging optical system may be provided near the center, the second imaging optical system and the third imaging optical system may be provided near the boundary of the display surface so as to sandwich the first imaging optical system, and the first imaging optical system may have a smaller area facing the display surface in the third direction than the second imaging optical system and the third imaging optical system. In other words, the second and third imaging optical systems may be provided at both ends of the display surface of the electronic device 1, and the first imaging optical system, which is less noticeable than these imaging optical systems, may be provided near the center.

[0015] The arrangement of the first imaging optical system and the second imaging optical system may have different coordinates in both the first direction and the second direction. In this way, the two imaging optical systems may have different coordinates only in the first direction, only in the second direction, or in both the first direction and the second direction. That is, on the display surface, the two imaging optical systems may be arranged in the horizontal direction, may be arranged in the vertical direction, or may be arranged in any direction.

[0016] When the image acquisition unit synthesizes the data acquired from the first imaging optical system and the data acquired from the second imaging optical system, the image acquisition unit may acquire an imaging result based on data with low intensity at a position in the image data that is the synthesis result. Since flare often has higher brightness and light intensity in the acquired data than the object to be imaged, when the image acquisition unit acquires image data, the image acquisition unit may acquire an image based on a signal with low intensity among the signals acquired from the two imaging optical systems.

[0017] The first imaging optical system and the second imaging optical system each have a direction that is preferentially reflected, and when a difference of a predetermined value or more occurs between the outputs for each direction, the image acquisition unit may acquire an imaging result using one of the results. For example, when the first imaging optical system generates a flare along a first direction and the second imaging optical system generates a flare along a second direction, the image acquisition unit may acquire image data based on the output from the imaging optical system that generates less flare.

[0018] A light shield may be provided between the first imaging optical system and the second imaging optical system. In this manner, the light shield may be provided between the first imaging optical system and the second imaging optical system to suppress the influence of each optical system on the other optical system.

[0019] The image acquisition unit may combine information acquired by the first and second imaging optical systems using a trained model. In this manner, for example, data output from a plurality of imaging optical systems may be combined using a model generated by machine learning.

[0020] This trained model may be trained based on data collected from multiple electronic devices. For example, the model may be generated based on electronic devices 1 manufactured with the same model number. Also, even within a model number, the model may be changed and trained based on the imaging mode, etc.

[0021] The image acquisition unit may perform correction when the parallax in the images acquired by the imaging optical systems exceeds a predetermined amount. For example, the image acquisition unit may detect an area where a flare occurs from the parallax and acquire image data by performing image processing on the area.

[0022] At least one of the imaging optical systems may be formed by a microlens array, and thus, one imaging system may be provided with a lens system rather than one lens.

[0023] A plurality of imaging optical systems may be provided in the area where the microlens array is provided. In this manner, a plurality of imaging optical systems may be formed using one microlens array.

[0024] The first imaging optical system and the second imaging optical system may obtain information in the same imaging element. In this way, a plurality of imaging optical systems may be formed in one imaging element, for example, in a one-chip imaging element. In combination with the above, a microlens array may be provided for one chip, thereby providing a configuration in which a plurality of imaging optical systems are provided for each region.

[0025] The image acquisition unit may be disposed on the same chip as the image sensor. For example, the image sensor and the logic circuit may be provided on one chip, and the data acquired by the image sensor may be converted into digital data and the converted data may be processed by the logic circuit. The image acquisition unit may be formed as multiple stacked chips instead of a single chip.

[0026] The display unit may include a plurality of display optical systems having different optical characteristics. For example, the display optical system may be a mixture of OLED, MicroLED, liquid crystal, etc. In this case, the imaging optical system may produce flares with different characteristics based on the results of reflection, refraction, and diffraction in the display unit.

[0027] At least one of the multiple imaging optical systems may operate when signal correction is required in the image acquisition unit. For example, the imaging optical system may be equipped with an imaging element that performs correction when the electronic device detects a strong light source, and the imaging optical system may switch whether or not to perform correction based on the surrounding environment.

[0028] The first imaging optical system and the second imaging optical system may be integrated together. For example, in the two imaging optical systems, the light receiving elements and the optical paths to the light receiving elements may be adjacent to each other.

[0029] The first imaging optical system and the second imaging optical system may be provided near a boundary of the display surface. The boundary may be, for example, an edge of the display surface, and the multiple imaging optical systems may be provided at this edge.

[0030] The first imaging optical system and the second imaging optical system may be disposed at a distance of 50 mm or more and 80 mm or less, and the image acquisition unit may generate parallax image data of the information acquired by the first imaging optical system and the second imaging optical system. In this manner, the two imaging optical systems may be disposed at a distance equivalent to the distance between both human eyes.

[0031] The display units may be provided on both sides of the device. In this case, each of the display units may be provided with multiple imaging optical systems, one display unit may be provided with multiple imaging optical systems and the other display unit may not be provided with any imaging optical systems, or one display unit may be provided with one imaging optical system and the other display unit may be provided with multiple imaging optical systems.

[0032] The fourth imaging optical system may be different from the first imaging optical system and the second imaging optical system, and the fourth imaging optical system and the first imaging optical system or the second imaging optical system may have any of the above-mentioned characteristics. As described at the beginning, the first imaging optical system and the second imaging optical system are merely two of a plurality of imaging optical systems, and do not indicate a specific imaging optical system. [Brief description of the drawings]

[0033] [Figure 1] FIG. 1 is a schematic cross-sectional view of an electronic device according to an embodiment. [Diagram 2] FIG. 1 is a schematic external view of an electronic device according to an embodiment. [Figure 3A] FIG. 1 is a schematic diagram illustrating an example of an imaging optical system according to an embodiment. [Figure 3B] FIG. 1 is a schematic diagram illustrating an example of an imaging optical system according to an embodiment. [Figure 4] FIG. 2 is a block diagram relating to an imaging operation of the electronic device according to an embodiment. [Diagram 5] FIG. 2 is a diagram showing an imaging optical system according to an embodiment from the display surface side. [Figure 6A] 6 is a diagram showing an example of an image captured by the imaging optical system of FIG. 5. [Figure 6B] 6 is a diagram showing an example of an image captured by the imaging optical system of FIG. 5. [Figure 7] FIG. 4 is a diagram showing an example of an image acquired by an image acquisition unit according to an embodiment. [Figure 8] FIG. 2 is a diagram showing an imaging optical system according to an embodiment from the display surface side. [Figure 9] 9 is a diagram showing an example of an image captured by the imaging optical system of FIG. 8. [Figure 10]FIG. 1 is a schematic external view of an electronic device according to an embodiment. [Figure 11] 11 is a diagram showing an example of an image captured by the imaging optical system of FIG. 10. [Figure 12] 12 is a diagram showing an example of a composite image obtained by combining images acquired by the imaging optical systems at both ends of FIG. 11. [Figure 13] FIG. 2 is a diagram showing an example of an image captured by an imaging optical system according to an embodiment. [Figure 14] FIG. 1 is a schematic cross-sectional view of an electronic device according to an embodiment. [Figure 15] FIG. 1 is a schematic cross-sectional view of an electronic device according to an embodiment. [Figure 16] FIG. 1 is a schematic cross-sectional view of an electronic device according to an embodiment. [Figure 17] FIG. 1 is a schematic diagram illustrating an example of an imaging optical system according to an embodiment. [Figure 18] FIG. 1 is a schematic diagram illustrating an example of an imaging optical system according to an embodiment. [Figure 19] FIG. 1 is a schematic diagram illustrating an example of an imaging optical system according to an embodiment. [Figure 20] FIG. 1 is a schematic diagram illustrating an example of an imaging optical system according to an embodiment. [Figure 21] FIG. 1 is a schematic diagram illustrating an example of an imaging optical system according to an embodiment. [Figure 22] FIG. 1 is a schematic diagram illustrating an example of an imaging optical system according to an embodiment. [Figure 23A] FIG. 1 is a schematic diagram illustrating an example of an imaging optical system according to an embodiment. [Figure 23B] FIG. 2 is a schematic diagram illustrating an example of an imaging optical system according to an embodiment. [Figure 24] FIG. 2 is a schematic diagram showing an example of implementation of an imaging unit according to an embodiment. [Diagram 25] FIG. 2 is a schematic diagram showing an example of implementation of an imaging unit according to an embodiment. [Figure 26] FIG. 2 is a schematic diagram showing an example of implementation of an imaging unit according to an embodiment. [Figure 27] FIG. 2 is a diagram showing an example of the arrangement of an imaging optical system according to an embodiment. [Figure 28] FIG. 2 is a diagram showing an example of the arrangement of an imaging optical system according to an embodiment. [Figure 29] FIG. 2 is a diagram showing an example of the arrangement of an imaging optical system according to an embodiment. [Diagram 30] FIG. 2 is a diagram showing an example of the arrangement of an imaging optical system according to an embodiment. [Figure 31A] FIG. 2 is a diagram showing an example of the arrangement of an imaging optical system according to an embodiment. [Figure 31B] FIG. 2 is a diagram showing an example of the arrangement of an imaging optical system according to an embodiment. [Diagram 32] FIG. 4 is a diagram showing an example of an opening layout according to an embodiment. [Diagram 33] FIG. 4 is a diagram showing an example of an opening layout according to an embodiment. [Diagram 34] FIG. 4 is a diagram showing an example of an opening layout according to an embodiment. [Diagram 35] FIG. 4 is a diagram showing an example of an opening layout according to an embodiment. [Diagram 36] FIG. 1 is a plan view showing a case where an electronic device according to an embodiment is applied to a capsule endoscope. [Figure 37] FIG. 2 is a rear view of the electronic device according to the embodiment applied to a digital single-lens reflex camera. [Figure 38A] FIG. 1 is a diagram showing an example in which an electronic device according to an embodiment is applied to an HMD. [Figure 38B] A diagram showing the current HMD. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0034] Hereinafter, an embodiment of an electronic device will be described with reference to the drawings. Hereinafter, the main components of the electronic device will be mainly described, but the electronic device may have components and functions that are not shown or described. The following description does not exclude components and functions that are not shown or described. In addition, the size, shape, aspect ratio, etc. of some components are changed for the purpose of explanation, but these have appropriate sizes, shapes, aspect ratios, etc. in implementation. In the following description, the acquired signal is described as image information or imaging information, but this image information and imaging information are concepts in a broad sense, and include still images, videos, or one frame of images in a video.

[0035] (First embodiment) FIG. 1 is a schematic cross-sectional view of an electronic device 1 according to a first embodiment. The electronic device 1 in FIG. 1 is any electronic device having both a display function and a photographing function, such as a smartphone, a mobile phone, a tablet, or a PC. As shown in the lower left of the figure, the first direction is the right direction in the figure, the second direction is a direction perpendicular to the figure, and the third direction is a downward direction in the figure. That is, the second direction is a direction intersecting with the first direction, and the third direction is a direction intersecting with the first direction and the second direction. Note that intersecting may include intersecting at an angle of 90°, and may not be strictly 90°. Also, as can be seen from the figure, the first direction and the second direction are distinguished for convenience, and are equivalent even if they are interchanged.

[0036] 1 includes an imaging optical system (camera module, etc.) disposed on the opposite side to the display surface of the display unit 2. In this manner, the electronic device 1 includes an imaging optical system 3 on the rear side (opposite side) of the display surface of the display unit 2. Therefore, the imaging optical system 3 captures an image through the display unit 2.

[0037] As shown in Fig. 1, the display unit 2 is a structure in which a display panel 4, a circular polarizer 5, a touch panel 6, and a cover glass 7 are laminated in this order. The laminate shown in Fig. 1 is shown as an example, and an adhesive layer or bonding layer may be provided between the display panel 4, the circular polarizer 5, the touch panel 6, and the cover glass 7 as necessary. The order of the circular polarizer 5 and the touch panel 6 may be appropriately changed depending on the design.

[0038] The imaging optical system 3 is provided on the opposite side of the display surface of the display unit 2. The imaging optical system 3 includes, for example, a photoelectric element (light receiving element) that receives light and photoelectrically converts it into an analog signal, and an optical system that propagates the light irradiated on the display surface to the photoelectric element. The optical system may be, for example, an opening provided in the display panel 4. A plurality of imaging optical systems 3 are provided for one display unit 2 of the electronic device 1, for example, two are provided as shown in the figure. The light irradiated on the display surface is diffracted at the opening as shown by the arrow in the figure, and propagates to the light receiving element. Alternatively, instead of an opening, an optical system having some optical characteristic, for example, an optical characteristic that adjusts the optical path length or changes the polarization state, may be provided. The imaging optical system 3 includes, for example, an imaging unit 8 and an optical system 9 that collects, diffuses, etc. the light incident on the imaging unit 8 from the display surface.

[0039] The imaging optical systems 3 are arranged to have different coordinates in the second direction, as shown in the figure, but are not limited to this. For example, they may have different coordinates in the first direction, or may have different coordinates in both the first and second directions.

[0040] Although the display panel 4 is not illustrated in detail, it may have, for example, an OLED (Organic Light Emitting Device), liquid crystal such as TFT, or MicroLED as an optical system for display (display optical system). The display optical system may have light-emitting elements based on other display principles. The light-emitting elements as the display optical system may have, for example, a stripe arrangement, a mosaic arrangement, an array arrangement in the first direction and the second direction, or an oblique direction or partial pixel thinning. In addition, the display optical system may have a light-emitting element equipped with a laminated filter to change the display color. When an OLED or the like is provided, the display panel 4 may be composed of a plurality of layers such as an anode layer, a cathode layer, etc. In addition, these layers may be formed of a material with high transmittance.

[0041] The display panel 4 may be provided with a member having low transmittance such as a color filter layer. When the display panel 4 includes an OLED, the display panel 4 may include, for example, a substrate 4a and an OLED section. The substrate 4a may be formed of, for example, polyimide. When the substrate 4a is made of a material having low light transmittance such as polyimide, an opening may be formed in accordance with the location of the imaging optical system 3. If the subject light passing through the opening is made incident on the imaging optical system 3, the quality of the image captured by the imaging optical system 3 can be improved. Alternatively, instead of an opening, a light propagation path made of a material having high transmittance may be provided. In this case, the light incident from the display surface of the display section 2 is received by the imaging optical system 3 and converted into a signal.

[0042] The circular polarizer 5 is provided, for example, to reduce glare or to improve the visibility of the display screen 1a even in a bright environment. A touch sensor is incorporated in the touch panel 6. There are various types of touch sensors, such as a capacitance type and a resistive film type, and any type may be used. The touch panel 6 and the display panel 4 may be integrated. The cover glass 7 is provided to protect the display panel 4 and the like. As described above, an adhesive layer or a pressure-sensitive adhesive layer such as OCA (Optical Clear Adhesive) may be provided at an appropriate location. The order of the circular polarizer 5 and the touch panel 6 in the third direction may be reversed depending on the design.

[0043] Fig. 2 shows a schematic external view and a cross-sectional view of the electronic device 1 shown in Fig. 1. The cross-sectional view shows a cross-section of a display portion including the display unit 2 taken along a dashed line shown in the figure. Circuits and the like other than the housing and display portion of the electronic device 1 are omitted.

[0044] In the external view, the display screen 1a extends close to the outer diameter size of the electronic device 1, and the width of the bezel 1b around the display screen 1a is set to a few mm or less. Usually, a front camera is often mounted on the bezel 1b. In this embodiment, for example, the front camera may be located as a plurality of imaging optical systems 3, approximately at the center of the display screen 1a in the second direction, as shown by the dotted line in the external view. In this way, by arranging the front camera as the imaging optical system 3 on the opposite side to the display surface of the display unit 2, it becomes unnecessary to arrange the front camera in the bezel 1b, and the width of the bezel 1b can be narrowed.

[0045] The external view of FIG. 2 is shown as an example, and the imaging optical system 3, i.e., the front camera, may be disposed on the opposite side (back side) of the display surface of the display unit 2 at any position in the first direction and the second direction on the display screen 1a. For example, it may be disposed on the periphery (edge, boundary part) of the display screen 1a. As shown in the external view of FIG. 2, the multiple imaging optical systems 3 are provided so as to have different coordinates in the first direction, for example. Even when the imaging optical system 3 is disposed at any position, it is sufficient that it is disposed so as to have different coordinates in at least one of the first direction and the second direction. In addition, although two imaging optical systems 3 are depicted, this is not limiting, and more imaging optical systems may be provided on the opposite side of the display surface.

[0046] For example, as shown in the cross-sectional view, the imaging optical system 3 is provided on the back side opposite to the display front side, which is the display surface of the display unit 2. Note that this cross-sectional view is omitted. For example, an adhesive layer, etc., is also provided in the configuration of the cross-sectional view of FIG. 2 as described above, but is omitted for simplicity of explanation.

[0047] 3A is a diagram showing an example of the imaging optical system 3. The imaging optical system 3 includes, for example, an imaging unit 8 and an optical system 9. The optical system 9 is disposed on the light incident surface side of the imaging unit 8, i.e., on the side closer to the display unit 2. Light transmitted through the display surface of the display unit 2 is propagated to the imaging unit 8 by the optical system 9.

[0048] The imaging unit 8 includes, for example, a light receiving element such as a photodiode, and a photoelectric element. The light collected, diffused, and propagated by the optical system 9 is received by an imaging pixel array included in the imaging unit 8, which outputs an analog signal. The imaging pixel array may include, for example, a color filter such as a Bayer array, or a stacked color filter, on the incident surface side of each imaging element. In addition, a filter for acquiring a color image may be included. Although not shown, other elements, circuits, and the like required for receiving light and outputting an analog signal are included. For example, the photoelectric conversion may be a CMOS (Complementary Metal-Oxide-Semiconductor) element or a CCD (Charge Coupled Device) element. In addition, the above-mentioned filter and further a polarizing element, etc. may be included.

[0049] The optical system 9 may include, for example, a lens. The optical system 9 may also be a concept including the above-mentioned opening provided in the display panel 4. For example, the optical system 9 includes an opening provided in the display panel 4 and a lens disposed at a position closer to the imaging unit 8 than the opening in the third direction. The opening may be provided in, for example, a substrate 4a having a low transmittance, and a lens that propagates light transmitted through the opening to the imaging unit 8 may be provided. For example, the lens and the opening define optical characteristics such as a numerical aperture Na and an F-number in each imaging optical system 3. Furthermore, the optical system 9 may provide other optical characteristics such as different Abbe numbers for the imaging optical system 3. Although the lens is shown as one lens, the present invention is not limited thereto, and the lens may be provided as a lens system including a plurality of lenses of various types.

[0050] The apertures and lenses are shown as examples, and the configuration of the optical system 9 is not necessarily limited to these combinations. In addition, in the figure, one lens is provided for one aperture, but this is not limited. For example, as shown in FIG. 3B, multiple apertures may be provided for one lens in the optical system 9. In an area where no aperture exists, for example, light-emitting elements of the display panel 4 may be provided, and apertures may be provided between these light-emitting elements. By arranging in this way, it is possible to provide the imaging optical system 3 without disrupting the display.

[0051] In this way, the multiple imaging optical systems 3 may be formed with different optical characteristics depending on the shape of the aperture, the performance of the lens, etc. When there are three or more imaging optical systems 3, the optical systems 9 corresponding to each of the imaging optical systems 3 may have different optical characteristics. As another example, the imaging optical systems 3 may be divided into multiple groups, and each group may have different optical characteristics. For example, the optical systems 9 may be provided with different shapes, orientations, or lens materials of their apertures so that there are two imaging optical systems 3 having common optical characteristics and one imaging optical system 3 having different optical characteristics. The term "opening layout" is used to express the shape and orientation of the openings.

[0052] As shown by the arrow in FIG. 3A, light incident from the display surface side of the display unit 2 is refracted by the optical system 9 and received by the imaging unit 8. In a location where the optical system 9 is not provided, reflections and the like may be appropriately suppressed and the display on the display unit 2 may be adjusted to be easily visible, as in a normal display. For example, an opening may be provided between the light-emitting pixels of the display panel 4, and a lens may be provided on the opposite side of the opening from the display surface in the third direction, and the light incident from the display surface is projected to the imaging unit 8. Also, an opening may be provided between each pair of consecutive light-emitting pixels. In other words, a configuration in which light-emitting pixels are provided between the openings may be used.

[0053] Here, an example of the imaging function of the electronic device 1 will be described.

[0054] 4 shows an example of a block diagram illustrating a configuration related to an imaging operation of the electronic device 1 according to the present embodiment. The electronic device 1 includes a display unit 2, a plurality of imaging optical systems 3, a pre-processing unit 10, an image acquisition unit 12, a post-processing unit 14, an output unit 16, a control unit 18, and a storage unit 20.

[0055] As in the drawings described above, a plurality of imaging optical systems 3 are provided on the opposite side to the display surface of one display unit 2. Each imaging optical system 3 includes an imaging unit 8 and an optical system 9.

[0056] The pre-processing unit 10 is a circuit that processes an analog signal output by the imaging unit 8. The pre-processing unit 10 includes, for example, an ADC (Analog to Digital Converter), and converts the input analog signal into digital image data.

[0057] The image acquisition unit 12 acquires a captured image from the digital image data converted by the pre-processing unit 10. The image acquisition unit 12 acquires an imaging result based on the digital image data acquired from the multiple imaging optical systems 3. More specifically, the image acquisition unit 12 acquires and outputs an imaging result in which, for example, flare generated in each imaging optical system 3 is suppressed by using the image data acquired by the multiple imaging optical systems 3.

[0058] The post-processing unit 14 performs appropriate processing on the imaging result output by the image acquisition unit 12 and outputs the result. The appropriate processing may be, for example, image processing or signal processing such as pixel defect correction, edge emphasis, noise removal, brightness adjustment, color correction, white balance adjustment, distortion correction, autofocus processing, etc. Furthermore, this appropriate processing may be processing designated by the user.

[0059] The output unit 16 outputs information to the outside of the electronic device 1. The output unit 16 includes, for example, an output interface. The output interface may be, for example, an interface that outputs a digital signal, such as a Universal Serial Bus (USB), or may be a user interface, such as a display. The output interface included in the output unit 16 may also function as an input interface.

[0060] The control unit 18 controls the processing in the electronic device 1. The control unit 18 may include, for example, a CPU (Central Processing Unit), and may control the processing of the pre-processing unit 10, the image acquisition unit 12, the post-processing unit 14, and the output unit 16. The control unit 18 may also execute control to perform shooting by the imaging optical system 3 based on the imaging timing instructed from the user interface.

[0061] The storage unit 20 stores data in the electronic device 1. The storage unit 20 may be, for example, a memory such as a dynamic random access memory (DRAM) or a storage such as a solid state drive (SSD). The storage unit 20 may be a built-in memory or a removable memory such as a memory card. The storage unit 20 is not necessarily provided inside the electronic device 1, and may be an external storage connected via an input / output interface. Information is appropriately input and output to and from the storage unit 20 at the timing required by the electronic device 1.

[0062] Some or all of the above-described components may be formed on the same substrate. For example, the imaging optical system 3, pre-processing section 10, image acquisition section 12, post-processing section 14, output section 16, control section 18, and storage section 20 may be formed on one chip, or a portion of these may be formed as separate chips as appropriate. In addition, a portion of the configuration formed on the same substrate of one chip may be integrated with a portion of the configuration formed on another substrate in the manufacturing process using CoC (Chip on Chip) or CoW (CoW on Chip). Alternatively, the semiconductor device may be formed by laminating the layers using techniques such as Wafer on Wafer (Wafer on Wafer) and WoW (Wafer on Wafer).

[0063] Next, the operations of the imaging optical system 3 and the image acquisition unit 12 will be described in detail.

[0064] FIG. 5 is a diagram showing the imaging optical system 3 from the display surface side of the display unit 2. For example, in the figure, a part of the area of ​​the display unit 2 viewed from the display surface side is shown. A display panel 4 can be seen from the display surface, and this display panel 4 is provided with light-emitting pixels 4b formed by a plurality of light-emitting elements arranged in an array in a first direction and a second direction, as shown by a dotted line. Note that the arrangement and orientation of the light-emitting pixels 4b are shown as an example, and the light-emitting pixels 4b may be arranged in the same manner as in FIG. 5, but may be rotated by 45°. In addition, the light-emitting pixels 4b are drawn as squares, but are not limited to this, and may be rectangular extending in either direction, or may not be rectangular.

[0065] A plurality of imaging optical systems 3 having apertures between the light-emitting pixels 4b of the display panel 4 are provided. For example, FIG. 5 shows a first imaging optical system 3A and a second imaging optical system 3B. As an example, as shown in the figure, each imaging optical system is provided with an elliptical opening having a major axis in the second direction as the first optical system 9A and an elliptical opening having a major axis in the first direction as the second optical system 9B. Each imaging optical system may be provided with a plurality of apertures as an optical system. In FIG. 5, for example, the first imaging unit 8A is provided below the aperture provided in the first optical system 9A, and the second imaging unit 8B is provided below the aperture provided in the second optical system 9B. However, this is not limited to this, and the imaging unit 8 may be provided at a position shifted from the aperture.

[0066] Although not shown, a lens may be provided as part of the optical system 9 between each opening and the imaging unit 8 or at the opening so that light passing through the opening is appropriately diffused and focused in the imaging area of ​​the imaging unit 8. Instead of a lens, other optical systems may be provided so that light can be appropriately received in the imaging area of ​​the imaging unit 8. Here, the opening indicates an optically transparent area, and may be an air gap or may be filled with a transparent material such as a resin. In addition, a material that transmits a specific wavelength by a color filter or the like may be used, and the material filling the opening is not limited. For example, the optical characteristics of the first optical system 9A and the second optical system 9B may be made different by filling the opening with a material having a different transmittance, refractive index, etc.

[0067] The imaging optical system 3 has different aperture layouts and is arranged at shifted positions, for example, as the first imaging optical system 3A and the second imaging optical system 3B shown in Fig. 5. Due to the difference in the aperture layouts, the optical characteristics of the optical systems belonging to each are different, and light is incident from the display surface of the display unit 2 to each imaging unit 8 via different optical characteristics. That is, light is incident on the first imaging unit 8A and the second imaging unit 8B via different optical characteristics.

[0068] For example, the apertures belonging to each imaging optical system may be similar ellipses with different directions, as shown in Fig. 5. In this way, similar apertures having different directions may form an optical system 9 that generates diffraction images in different directions.

[0069] 5 shows a state in which two openings are provided between three consecutive light-emitting pixels along the first and second directions, but the present invention is not limited to this. For example, an opening may be provided between each of a larger number of consecutive light-emitting pixels. In this way, in the display unit 2 including the display panel 4, one optical system 9 is not necessarily one continuous region when viewed from the display surface, but may be configured as, for example, a plurality of separate regions including openings periodically arranged between the light-emitting pixels.

[0070] 5, the major axis of one aperture corresponds to approximately two or three display pixels, but this is not limited thereto. The aperture may have, for example, a longer major axis or a shorter major axis.

[0071] In addition, in order to generate diffraction images having different characteristics rather than similar apertures, the major axis of the aperture that is a part of the second optical system 9B may be longer or shorter than the major axis of the aperture that is a part of the first optical system 9A. By making the layout of the apertures different in this way, the influence of flare can be made more different in the signals output from the first imaging optical system 3A and the second imaging optical system 3B. As a result, it is also possible to assist the flare suppression operation by the image acquisition unit 12. The influence of flare due to this layout of the apertures changes based on the state of the electronic device 1 or the state of each component in the electronic device 1, so it is possible to appropriately define it by the design of the electronic device 1.

[0072] Figures 6A and 6B show images whose coordinates have been adjusted and converted from analog signals acquired by the respective imaging optical systems 3 shown in Figure 5. Figure 6A shows an image acquired based on the first imaging optical system 3A, and Figure 6B shows an image acquired based on the second imaging optical system 3B.

[0073] The image acquisition unit 12 may perform the position adjustment. The image capturing position may be adjusted, for example, so that a mirror image of the image captured by each of the image capturing optical systems 3 displayed on the display panel 4 overlaps with a reflected image when the display surface is viewed from the front. For example, when a person is positioned so as to be reflected at the center of the display surface, the position may be adjusted so that the image captured by the image capturing optical system 3 is displayed at the center of the display surface. For example, correction may be performed based on the positional deviation of the image capturing optical system 3 from the center of the display surface. Note that the position adjustment is not limited to this, and may be performed by any method as long as it is appropriately controlled.

[0074] For example, when the position is adjusted as described above, flare occurs as shown by the white areas in Figures 6A and 6B. For example, flare occurs as an area brighter than the image that is actually desired to be captured, as shown in the figures.

[0075] In such a case, in order to suppress the flare occurring at different locations, the image acquisition unit 12 acquires pixel values ​​having lower light intensity or luminance values ​​after conversion to a digital signal in the signals output from the first imaging optical system 3A and the second imaging optical system 3B, and corrects the image. When processing the signal before conversion to a digital signal, the pre-processing unit 10 may make a selection based on the signal intensity.

[0076] 7 shows an image processed by the image acquisition unit 12 and output from the electronic device 1. After the above-mentioned processing by the image acquisition unit 12, the post-processing unit 14 adjusts, for example, brightness. By outputting the image processed in this manner, it is possible to obtain an image in which flare is suppressed and the brightness is adjusted to a natural level, as shown in FIG.

[0077] For example, the region where the flare as shown in Fig. 6A and Fig. 6B occurs may be determined depending on each of the imaging optical systems 3. In this way, the region where the flare occurs may be stored in advance in the storage unit 20 or the image acquisition unit 12. As a result, the pixel value selection process or the pixel value synthesis process can be executed at high speed. For example, for a certain pixel, an image may be acquired based on the output from a predetermined imaging optical system 3 without comparing the pixel values ​​output from the respective imaging optical systems 3.

[0078] As described above, according to the present embodiment, by making the direction in which diffracted light is generated different in the multiple imaging optical systems, it is possible to suppress flare in an image captured by a camera provided on the back surface of the display unit of a display, i.e., a front camera arranged to overlap the display. As a result, in an electronic device, it is possible to provide an imaging optical system on the front side of a display that can capture an image with high accuracy without increasing the width of the bezel on the surface on which the display is provided.

[0079] For example, when light emitted from a strong light source partially hits the surface of the display, flare components are likely to occur in the imaging optical system 3 close to the area hit by the light. Even in such a case, it is possible to obtain an image with suppressed flare as described above by using an image obtained by at least one other imaging optical system 3 provided at a different position.

[0080] The first imaging optical system 3A and the second imaging optical system 3B are two selected from a plurality of imaging optical systems, and three or more imaging optical systems 3 may be provided. That is, if at least two imaging optical systems 3 function as the first imaging optical system 3A and the second imaging optical system 3B in three or more imaging optical systems 3, the operation and effect of this embodiment and the embodiment described below can be achieved. In addition, by providing two or more imaging optical systems having the characteristics of the above-mentioned first imaging optical system and second imaging optical system in three or more imaging optical systems 3, it is possible to further improve the degree of freedom in acquiring captured images and improve the accuracy of flare suppression. In this case, the same imaging optical systems may be used in different combinations as long as they are not the same combinations. For example, when 3X, 3Y, and 3Z are provided as imaging optical systems, the combination of (first imaging optical system, second imaging optical system) may be two combinations with a common X, such as (3X, 3Y) and (3X, 3Z).

[0081] In the above description, each of the imaging optical systems 3 has an aperture as the optical system 9, but this is not limiting. For example, one may have an aperture and the other may not have an aperture. In this case, as in the above description, flares occur at different positions from the multiple imaging optical systems 3, and therefore it is possible to suppress the influence of the flare by the images acquired by the multiple imaging optical systems 3.

[0082] In addition, the shape of the opening is described as an ellipse, but is not limited thereto. For example, it may be a rectangle, or a rectangle with rounded corners. In addition, it may be formed by any closed curve that allows light to be appropriately received by the imaging unit 8. Also, the opening may not have the same shape in the thickness direction, i.e., the third direction. For example, the upper part, i.e., the part closer to the display surface, may be a rectangle, and the part closer to the imaging unit 8 may be an ellipse, which is a more complicated shape.

[0083] In addition, in the above, the shape of the opening has been described as an optical characteristic, but the present invention is not limited thereto. For example, the material filled in the opening may be changed for each imaging optical system 3 to provide different characteristics. Also, for example, the optical system 9 may include a λ / 4 wave plate. In this case, the p-wave may be cut in order to reduce flare caused by the influence of reflection in the display panel 4. The optical system 9 may include different wave plates for multiple imaging optical systems 3. As a result, it becomes possible to change the influence of flare caused by reflection, diffraction, etc. in the display unit 2 for each imaging optical system 3, and various image correction methods can be used.

[0084] Second embodiment The electronic device according to this embodiment has a plurality of imaging optical systems that are provided with apertures of the same layout and that can reduce the influence of flare.

[0085] Fig. 8 shows the display surface of the electronic device 1 according to this embodiment in the same manner as Fig. 5. In this embodiment, the first imaging optical system 3A and the second imaging optical system 3B have the same aperture layout. However, unlike the first embodiment, the first imaging optical system 3A and the second imaging optical system 3B are located at different positions in the second direction. That is, each optical system has an elliptical aperture with a major axis in the second direction, and is located at a position shifted in the second direction.

[0086] Fig. 9 shows an image acquired by the second imaging optical system 3B. For example, the image acquired by the first imaging optical system 3A is as shown in Fig. 6A. Since both imaging optical systems 3 have elliptical apertures with their major axes in the second direction, the directions in which flare occurs are the same. However, since the first imaging optical system 3A and the second imaging optical system 3B are arranged with a shift in the second direction, flare occurs at a position shifted in the second direction.

[0087] Therefore, similarly to the first embodiment described above, the image acquisition unit 12 can acquire an image in which the influence of flare is suppressed. The first imaging optical system 3A and the second imaging optical system 3B may also be shifted in the first direction. When they are shifted in both the first and second directions in this way, the center points in the first direction of the occurrence of flare in each imaging optical system are at different positions, so that an image in which flare is better suppressed can be acquired.

[0088] Third embodiment In this embodiment, an electronic device having three imaging optical systems will be described.

[0089] FIG. 10 is a diagram showing an example of an electronic device 1 according to this embodiment. The electronic device 1 includes a first imaging optical system 3A, a second imaging optical system 3B, and a third imaging optical system 3C on the opposite side of the display surface of the display unit 2. For example, these imaging optical systems are provided along a first direction. The first imaging optical system 3A may be provided around the center of the screen, and the second imaging optical system 3B and the third imaging optical system 3C may be provided near an end (boundary) of the display surface of the display unit 2 so as to sandwich the first imaging optical system 3A therebetween. The vicinity may be, for example, one pixel to several pixels of a display element from the end or boundary, but as another example, it may be several mm from the end of the housing of the electronic device 1, or several % of the width or height of the electronic device 1.

[0090] By disposing them in this way, it is possible to make the parallax between the first imaging optical system 3A and the second imaging optical system 3B, and the parallax between the third imaging optical system 3C and the first imaging optical system 3A equivalent. In other words, this makes it possible to generate an image having (almost) zero parallax with respect to the first imaging optical system 3A, using an image acquired by the second imaging optical system 3B and an image acquired by the third imaging optical system 3C.

[0091] The edges of the display surface are arranged in this manner because pixel disturbance has little effect on the user. On the other hand, for example, the size of the first imaging optical system 3A located near the center of the display surface may be made smaller than the other imaging optical systems. Here, the size may be, for example, the size of the opening or the size of the imaging unit 8. By arranging in this manner, it is possible to make the image displayed by the display unit 2 look more natural.

[0092] In FIG. 10, for example, the first imaging optical system 3A may have an ellipse having a major axis in the first direction as the optical system 9, and the second imaging optical system 3B and the third imaging optical system 3C may have an ellipse having a major axis in the second direction as the optical system 9.

[0093] 11 is a diagram showing images that can be acquired from each imaging optical system when such an aperture is provided. From the top, the images are acquired by the first imaging optical system 3A, the second imaging optical system 3B, and the third imaging optical system 3C. Thus, in the first imaging optical system 3A, flare occurs along the second direction, and in the second imaging optical system 3B and the third imaging optical system 3C, flare that is displaced along the first direction occurs.

[0094] Fig. 12 shows an image obtained by superimposing the images of the second imaging optical system 3B and the third imaging optical system 3C while taking into consideration the parallax. When an image in which flare is suppressed is obtained while taking into consideration the parallax in this manner, it is possible to move the flare to both ends, for example, as shown in Fig. 12. By combining this image with the top image in Fig. 11, it is possible to obtain an image in which flare is suppressed, for example, as shown in Fig. 7. The synthesis or correction of these images may be performed by the image acquisition unit 12.

[0095] In the above description, the second imaging optical system 3B and the third imaging optical system 3C are located at both ends, but this is not limited thereto. For example, even if the second imaging optical system 3B and the third imaging optical system 3C are provided at positions where the same parallax appears on either side of the first imaging optical system 3A, it is possible to similarly acquire an image with suppressed flare. In addition, in this embodiment, the first imaging optical system 3B and the third imaging optical system 3C are shifted in the first direction, but they may be shifted in the second direction. Furthermore, each imaging optical system 3 may be disposed at a position shifted in both the first direction and the second direction.

[0096] (Fourth embodiment) The intensity of flare may vary depending on the direction. For example, flare may occur strongly in a first direction but weakly in a second direction. In such a case, in this embodiment, an imaging optical system in which flare is likely to occur in the first direction or the second direction may be appropriately selected, and an image in which flare is suppressed may be obtained based on a signal obtained from the selected imaging optical system.

[0097] Fig. 13 shows an example of an image acquired based on the output from the first imaging optical system 3A in the case where, for example, a flare occurs strongly in the second direction and a flare occurs not so strongly in the first direction when an image is acquired in the imaging optical system shown in Fig. 5. As shown in Fig. 13, since the flare in the first direction is weak, an image in which the flare portion is darker than that in Fig. 6A is acquired. In contrast, since a flare occurs strongly in the second direction, an image output from the second imaging optical system 3B is acquired in which the flare portion is brighter as shown in Fig. 6B.

[0098] In this way, when the flare varies in strength depending on the direction, an image in which the flare is suppressed may be acquired based on the output from the first imaging optical system 3A. The strength of the flare may be determined, for example, by calculating the variance of brightness in the first direction and the second direction in the image data output from each imaging optical system 3 in the pre-processing unit 10. For example, the variance along the first direction is calculated for each row and the average value is taken, and the variance along the second direction is calculated for each column and the average value is taken. It can be said that the direction with a high average value has a strong flare intensity. Alternatively, the direction may be determined based on the difference between the maximum brightness and the minimum brightness for each row or column, rather than the variance.

[0099] As described above, in which direction the imaging optical system 3 is likely to cause flare depends on the optical system 9 of each imaging optical system 3. For example, as described above, it is determined based on the direction of the aperture provided in the optical system 9. Depending on the direction of the aperture, it may be possible to determine in advance which imaging optical system 3 is to be given priority when flare occurs in any direction.

[0100] Although it has been described whether flare is likely to occur depending on the layout of the apertures, this is not limiting. For example, the optical system for controlling flare may be not only the aperture layout but also the entire display panel including the wiring of the circuit. The flare and diffraction shape also change depending on what kind of pattern is periodically laid out on the circuit and how the light is interfered. In consideration of this, it may be determined in which direction flare is likely to occur.

[0101] The image acquisition unit 12 acquires an image in which flare is suppressed based on the imaging optical system 3 having a priority direction different from the direction in which the flare occurs. For example, when images such as those in Fig. 13 and Fig. 6B are output through the respective imaging optical systems 3, the image in Fig. 13, which has less flare, may be selected and output. Also, a weighted average of the images in Fig. 13 and Fig. 6B, in which the weight of the image in Fig. 13 is increased, may be calculated.

[0102] As yet another example, when a strong flare occurs in the second direction, an image may be acquired based on any of the acquisition (e.g., selection, synthesis) methods described above using an imaging optical system 3 in which multiple first directions are the priority directions.

[0103] In this way, when the flare has polarity, an image in which the flare is suppressed may be acquired based on an image output via an imaging optical system 3 having a priority direction different from the flare direction.

[0104] Fifth embodiment In the electronic device 1 of this embodiment, even if a flare occurs, any one of the imaging optical systems 3 is arranged so that the occurrence of the flare is weakened.

[0105] Fig. 14 is a diagram showing a cross-sectional view of the electronic device 1 according to this embodiment. As shown in Fig. 14, the electronic device 1 includes a light-shielding portion 30 between a plurality of imaging optical systems 3. The light-shielding portion 30 may be a light-shielding film made of a material with high light-shielding properties, or may be an absorbing film made of a material with high light absorption rate.

[0106] In this way, a light blocking section 30 that prevents light from propagating between the imaging optical systems 3 may be provided on the side opposite to the display surface of the display panel 4.

[0107] FIG. 15 shows another example, in which the light blocking portion 30 may be provided for a plurality of imaging optical systems 3 so that the light reflected by the display panel 4 and the circular polarizer 5 is not propagated in the first and second directions.

[0108] 16 shows yet another example, in which the light-shielding portion 30 may be provided so as to penetrate the display panel 4. By providing the light-shielding portion 30 in this manner, it becomes possible to block not only the light reflected by the display portion 2, but also the light reflected by portions other than the display portion 2.

[0109] 15 and 16, by arranging a light-shielding portion 30 in an area where a display panel 4 and a circular polarizer 5 are provided, it is possible to suppress the occurrence of flare due to light reflected, diffracted, etc. in these layers. Even if strong flare occurs in a certain area due to the influence of incident light, etc. and strong flare occurs in the imaging optical system 3 into which the light of that area is incident, it is possible to reduce the influence of the flare in another imaging optical system 3 separated by the light-shielding portion 30.

[0110] Also, for example, as shown in Figures 14 and 16, by reducing the effects of light reflection and diffraction on the opposite side in the third direction of the display surface of the display unit 2, it is possible to suppress the effects between the imaging units 8.

[0111] In the above description, the light shielding portion 30 includes the circular polarizer 5. However, for example, the light shielding portion 30 may also be provided in the touch panel 6. Furthermore, the light shielding portion 30 may be formed to be sufficiently thin so as to reach the area of ​​the cover glass 7. In this case, the size and arrangement may be appropriately adjusted so that the display of the display unit 2 on the display surface appears as a natural image to the user. Furthermore, in the display panel 4, software may be used to address this issue, such as by increasing the brightness of the light-emitting pixels 4b arranged around the light shielding portion 30 to be higher than the brightness of the other light-emitting pixels 4b.

[0112] In this way, even if flare occurs, by blocking light from the multiple imaging optical systems 3, even if strong flare occurs in one imaging optical system 3, it is possible to acquire an image less affected by the flare using the other imaging optical systems 3. The image acquisition unit 12 can acquire an image less affected by flare by, for example, comparing or combining output values ​​from the multiple imaging optical systems 3, as in the above-described embodiment.

[0113] Seventh embodiment In each of the above-described embodiments, the image acquisition unit 12 acquires an image with little effect of flare by performing a predetermined calculation (including comparison). In contrast, in this embodiment, a model is used to combine outputs from multiple imaging optical systems 3 to acquire an image with reduced flare.

[0114] For example, the model may be a statistical model. A model may be generated by statistically calculating what operation should be used for synthesis for various imaging optical systems 3, and the image acquisition unit 12 may acquire an image with little influence of flare by inputting information acquired from the multiple imaging optical systems 3 to this model.

[0115] For example, the model may be a neural network model trained by deep learning. The neural network model may be formed by MLP (Multi-Layer Perceptron), CNN (Convolutional Neural Network), or the like. In this case, parameters trained in advance by a plurality of teacher data may be stored in the storage unit 20 or the image acquisition unit 12, and the image acquisition unit 12 may form a neural network model based on the stored parameters. Using the formed trained model, the image acquisition unit 12 may acquire an image in which flare is suppressed using data output from a plurality of imaging optical systems 3.

[0116] Furthermore, when a trained model is used, the electronic device 1 may further improve the training accuracy by using a captured image. For example, training may be performed in the control unit 18 of the electronic device 1. As another example, a plurality of electronic devices 1 may transmit data to a storage device or the like in a cloud, training may be performed in a server, and the retrained parameters may be reflected in the electronic device 1. In this case, only flare information may be transmitted so that privacy information including the user's face information is not included. In addition, the transmission and reception of data from the electronic device 1 may be selectable by the user, for example, by opting in or opting out.

[0117] In this way, the image acquisition unit 12 may acquire images not only by linear processing but also by nonlinear processing, in particular by calculations using various models including trained models.

[0118] Eighth embodiment In the above-described embodiment, the image acquisition unit 12 acquires images by combining images based on pixel values ​​or preferred directions, or based on a method using a model. In the present embodiment, an attempt is made to limit the area in each imaging optical system 3 where image correction is performed.

[0119] For example, as shown in Fig. 6A etc., in each imaging optical system 3, the area where flare occurs is often determined to be a bright point of a bright subject. This is based on the characteristics of the optical system 9 in each imaging optical system 3 or the arrangement of the imaging optical system 3. Therefore, when there is a bright subject, the image acquisition unit 12 can predict the area where flare occurs for each imaging optical system 3, and may correct the image based on this prediction result.

[0120] Flare often occurs due to nearby optical elements. Therefore, the parallax in the images acquired by the multiple imaging optical systems 3 is likely to be large compared to other objects. Therefore, the image acquisition unit 12 or the like may calculate the parallax, and an area with large parallax may be predicted as an area where flare occurs.

[0121] For example, in an area where there is a high possibility of flare occurring in a certain imaging optical system 3, the pixel value is determined based on information acquired from another imaging optical system 3. For example, instead of the comparison calculation described in the first embodiment, in this area, an interpolation process from the other imaging optical system 3 may be executed. However, the present invention is not limited to this, and the correction process may be performed by other calculations such as a weighting calculation so that the influence of the pixel value output from the other imaging optical system 3 is increased.

[0122] In addition, if three or more imaging optical systems 3 are provided and the probability of flare occurring in the other imaging optical systems 3 in the region is low, an image with little effect of flare may be obtained based on information obtained from these other imaging optical systems 3.

[0123] This region may be updated by reflecting it in the trained model of the above-mentioned embodiment. By reflecting it as training, for example, it is possible to improve the accuracy of correction between different users who have the same electronic device 1.

[0124] As another example, this region may be determined by, for example, allowing light to enter without displaying on the display panel 4. Instead of not allowing display on the display panel 4, the display panel 4 may be illuminated with an intensity that generates flare, and the region may be predicted. In this way, a process may be performed to determine a region where a flare will occur not at the image capture timing desired by the user, but at other timing, for example, before or after image capture. This process may be performed, for example, by the control unit 18 or the image acquisition unit 12.

[0125] Ninth embodiment In this embodiment, the electronic device 1 includes a microlens array as the optical system 9 of the imaging optical system 3.

[0126] 17 is a diagram showing the imaging optical system 3 according to this embodiment. The optical system 9 of the imaging optical system 3 includes a microlens array 32. Light that has passed through the microlens array 32 is appropriately incident on the imaging unit 8, and is converted into a signal and output from the imaging unit 8.

[0127] The pre-processing unit 10 may reconstruct an image based on the signal output from the imaging unit 8. The image acquisition unit 12 acquires an image in which the influence of flare has been reduced by each of the above-described embodiments based on this reconstructed image.

[0128] FIG. 18 is a diagram showing another example of the imaging optical system 3 according to this embodiment. The first imaging optical system 3A and the second imaging optical system 3B include a common microlens array 32 as the optical system 9. In this way, the microlens array 32 is limited to a certain area where the light collected from each lens is incident, so that it is possible to form a plurality of imaging optical systems 3 by arranging the imaging units 8 for each area. In this case, for example, the light shielding unit 30 described in the fifth embodiment may be provided between the first imaging unit 8A and the second imaging unit 8B. In FIG. 18, two imaging optical systems are provided, but this is not limited thereto, and three or more imaging optical systems 3 may be formed by the same microlens array 32.

[0129] Tenth embodiment In the above-described embodiment, a configuration in which a plurality of imaging units 8 are provided in the same microlens array 32 has been described as an example. However, in the present embodiment, the electronic device 1 has the same imaging unit 8 for a plurality of optical systems 9.

[0130] FIG. 19 is a diagram showing an example of an imaging optical system 3 according to this embodiment. The electronic device 1 is provided with one imaging section 8 for the first imaging optical system 3A and the second imaging optical system 3B. In this imaging section 8, a first imaging section 8A and a second imaging section 8B are defined in an imaging-capable area. Light is incident on the first imaging section 8A via a first optical system 9A, and light is incident on the second imaging section 8B via a second optical system 9B. In this way, a plurality of imaging optical systems 3 may be formed by providing a plurality of optical systems 9 for the same imaging section 8.

[0131] 20 is a diagram showing another implementation example of this embodiment. That is, the optical system 9 may include a microlens array 32, and the first imaging optical system 3A and the second imaging optical system 3B may be formed using the same microlens array 32 and the imaging unit 8.

[0132] Fig. 21 is a modification of Fig. 20, and includes a light shielding section 30 between the first imaging optical system 3A and the second imaging optical system 3B. By including the light shielding section 30 in this manner, it becomes possible to more clearly control the light incident on each region of the imaging section 8.

[0133] Fig. 22 is a modified example of Fig. 19, in which a light shielding section 30 is provided between the first optical system 9A and the second optical system 9B. By providing the light shielding section 30 in this manner, even in the case where a microlens array 32 is not used, it is possible to clearly control the light incident on each of the first imaging section 8A and the second imaging section 8B.

[0134] 21 and 22, the light-shielding portion 30 does not penetrate through the display panel 4. However, of course, as in the example of FIG. 16, the light-shielding portion 30 may be provided so as to penetrate through or through the display panel 4.

[0135] By making the imaging section 8 common to a plurality of imaging optical systems 3 in this way, it is possible to simplify the circuit configuration and semiconductor process of the imaging section 8.

[0136] For example, as in this embodiment, by using the same elements for some or all of the imaging optical systems 3, it is possible to integrate multiple imaging optical systems 3. By integrating and disposing multiple imaging optical systems 3 in this manner, it is also possible to reduce the effect of parallax between the imaging optical systems 3. As a result, even when correcting images acquired from each of the imaging optical systems, it is possible to reduce the effect of parallax, and the accuracy of correction can be improved.

[0137] Furthermore, instead of using a common imaging unit 8 or optical system 9 as in this embodiment, in the configurations of the above-described embodiments, a plurality of imaging optical systems 3 may be integrated so that the optical systems 9 are adjacent to each other. In this case, it is also possible to perform accurate correction while similarly reducing the effect of parallax.

[0138] 23A is a diagram showing another example of the arrangement of the microlens array 32. As shown in this Fig. 23A, the microlens array may be arranged in an opening in the optical system 9, closer to the imaging unit 8. In this manner, the optical system 9 may be formed with the opening and the microlens array 32 as separate components. Of course, the opening is not necessarily required, and the optical system 9 may be formed by arranging the microlens array 32 on the side opposite to the surface of the display panel 4.

[0139] 23B shows yet another example of the arrangement of the microlens array 32 and the display panel 4. One or more microlens arrays 32 may be provided for a plurality of openings. In this case, an opening may be formed leaving an appropriate region for the light-emitting pixels of the display panel 4, and the microlens array 32 may be provided below the opening between the image capture unit 8.

[0140] 23A and 23B, the light shielding portion 30 may be provided at an appropriate position.

[0141] Eleventh embodiment In this embodiment, the configuration of a chip such as the imaging unit 8 will be described.

[0142] 24 shows an example of the configuration of a chip according to this embodiment. This configuration is shown simply as a non-limiting example, and does not exclude the inclusion of other functions in the same chip. That is, in addition to those shown in the figure, a selector, an I / F, a power supply node, etc. may be appropriately provided.

[0143] 24, analog circuits such as a pre-processing unit 10 and logic circuits such as an image acquisition unit 12 may be provided on the same chip as the imaging unit 8. Furthermore, a control unit 18 and a storage unit 20 may also be provided. In this way, by forming the imaging unit 8, the pre-processing unit 10, and the image acquisition unit 12 on the same chip, it becomes possible to process data at high speed and with reduced signal degradation without using an interface for transmitting and receiving signals and data.

[0144] The pre-processing unit 10 does not have to be shared among the imaging units 8, and may be provided for each imaging unit 8. In this case, the digital image data may be transmitted from each pre-processing unit 10 to the image acquisition unit 12.

[0145] Fig. 25 is a diagram showing another example of the chip configuration. In Fig. 25, configurations other than the imaging unit 8, pre-processing unit 10, image acquisition unit 12, and interface are omitted. The imaging unit 8 may be provided on separate chips. One chip is provided with the imaging unit 8, and the other chip is also provided with the imaging unit 8. The imaging unit 8 on one chip is provided with a receiving interface 22 that transmits to the other chip.

[0146] In the other chip, the signal is received via the receiving interface 22, and the image acquisition unit 12 acquires an image together with a signal from the imaging unit 8 provided in the same chip. If necessary, the pre-processing unit 10 processes the signal before outputting the data to the image acquisition unit 12. The pre-processing unit 10 may be provided in both chips, or may be provided only in the other chip.

[0147] Of course, instead of two chips, more imaging units 8 may be provided on another chip. With such a configuration, it is possible to reduce the area of ​​the chip related to the imaging units 8 even when the imaging optical systems 3 are arranged at various positions on the opposite side of the display surface of the display unit 2.

[0148] 26 shows yet another example, in which a plurality of imaging units 8 are on separate chips, and an image acquisition unit 12 is on yet another chip. In this way, a logic circuit for processing digital signals and a photoelectric conversion circuit in which a light receiving element is present can be configured on separate chips. In this way, it is possible to further increase the degree of freedom in the arrangement of the imaging optical system 3, without increasing the chip area significantly.

[0149] 24 to 26, the receiving interface 22 may be, for example, an interface capable of receiving data according to the MIPI standard. It may also be an interface capable of high-speed transfer according to another standard. Depending on the signal to be transmitted, it may be an interface that transmits and receives analog signals, or may be an interface that transmits and receives digital signals. The type of interface can be appropriately selected depending on the chip implementation of the imaging unit 8 and the pre-processing unit 10.

[0150] Twelfth embodiment In each of the above-described embodiments, the light-emitting pixels 4b of the display panel 4 were not specifically described. However, the light-emitting pixels 4b can change the occurrence of flare in the multiple imaging optical systems 3, and enable the image acquisition unit 12 to acquire images that are less affected by flare.

[0151] For example, when two imaging optical systems 3, a first imaging optical system 3A and a second imaging optical system 3B, are provided, the light-emitting pixels 4b around the first imaging optical system 3A and the light-emitting pixels 4b around the second imaging optical system 3B may be different. For example, one may be an OLED and the other may be a MicroLED. In this way, by using different light-emitting elements, particularly display optical systems having different optical characteristics, the influence of flare can be made to have different characteristics.

[0152] If flare occurs through different optical characteristics, the occurrence of flare will differ depending on the situation in the images acquired from each imaging optical system 3. Due to the different flare characteristics, for example, an image that is strongly affected by flare will be acquired from one imaging optical system 3, while an image that is weakly affected by flare will be acquired from the other imaging optical system 3. In this way, by using light-emitting elements having different characteristics, it is possible to expand the method of image synthesis and correction in the image acquisition unit 12 to various methods.

[0153] Thirteenth embodiment In each of the above-described embodiments, an image in which the influence of flare is reduced is obtained by using images output from a plurality of imaging optical systems 3. That is, a plurality of imaging optical systems 3 are activated, and an image is obtained based on information acquired from each of the imaging optical systems 3. In this embodiment, at least one imaging optical system 3 is activated as necessary to reduce the influence of flare. For example, at least one imaging optical system 3 is provided which is activated when image correction is required.

[0154] For example, in the initial state, only the imaging optical system 3 located near the center of the display surface may be activated. When it is determined that it is necessary to reduce the influence of flare based on an image acquired by this imaging optical system 3, the other imaging optical systems 3 may be activated and an image with less influence of flare may be acquired by the image acquisition unit 12.

[0155] For example, it may be determined that the influence of flare is strong when strong light is applied to the entire area of ​​the display surface or only to a part of the area. The influence of flare may also be determined based on other conditions. In the above, it is based on the output from one imaging optical system 3, but this is not limited thereto, and for example, it may be determined based on the magnitude of signals from several light receiving elements. As another example, it may be based on the distribution of luminance, etc. of the image displayed on the display panel 4. In this way, the influence of flare is determined by various determination methods.

[0156] Based on this determination, the control unit 18 operates an appropriate imaging optical system 3, acquires a signal based on the incident light, and acquires an image by the image acquisition unit 12. The control unit 18 may also determine the influence of flare as described above. In this way, by operating at least a part of the imaging optical systems 3 as necessary, it is possible to conserve power.

[0157] Fourteenth embodiment In the above-described embodiments, various implementations of the imaging optical system 3 and the image acquisition unit 12 are described. In the present embodiment, the arrangement of the imaging optical system 3 will be described.

[0158] 27 to 31B are diagrams showing examples of the arrangement of the imaging optical system 3 as viewed from the display surface. Each of these diagrams shows an example of the arrangement, and the arrangement is not limited to the diagram. The arrangement may include some or all of the arrangements in multiple diagrams. In other words, the electronic device 1 may include three or more imaging optical systems 3 that have features in multiple diagrams in a superimposed manner, rather than just one diagram.

[0159] As shown in Fig. 27, the imaging optical system 3 may be provided in either direction of the display screen 1a, for example, at both ends (near the boundary) in the first direction. By arranging in this manner, the imaging optical system 3 can be provided in a position that is not noticeable to the user. This may be provided at both ends in the second direction, that is, at the top and bottom ends in Fig. 27.

[0160] As shown in FIG. 28, two imaging optical systems 3 may be provided in the display screen 1a, spaced apart by 50 mm to 80 mm. By arranging them in this way, it is possible to obtain images having a parallax close to that of the human binocular system from the two imaging optical systems 3. This makes it possible to reduce the influence of flare in the multiple imaging optical systems 3 and generate stereo images that give a natural impression to humans. Furthermore, a third imaging optical system 3 may be arranged at a different position, and an image having parallax and further suppressed flare may be obtained from the third imaging optical system 3 and the two imaging optical systems 3. In this way, it is possible to obtain stereo images with improved accuracy in suppressing flare.

[0161] As shown in Fig. 29, a plurality of imaging optical systems 3 may be arranged so as to have a shift in each of the first and second directions. For example, by providing each optical system 9 with an aperture having a major axis in the same direction, it is possible to acquire an image including a flare whose center is shifted in both the first and second directions. This makes it possible for the image acquisition unit 12 to acquire a highly accurate flare-suppressed image even with simpler processing (for example, the comparison processing and selection processing shown in the first embodiment).

[0162] As shown in Fig. 30, three or more imaging optical systems 3 may be provided. When three or more imaging optical systems 3 are arranged, they may be arranged as not having symmetry as shown in Fig. 30, or may be arranged to have some symmetry with respect to the first direction, the second direction, a central point, or the like. Also, the third imaging optical system 3 may be arranged at a position away from the other two imaging optical systems 3. In this case, the intensity of flare caused by the position can be made different for each imaging optical system 3.

[0163] 31A and 31B show the same electronic device 1. FIG. 31B is an external view seen from the direction of the arrow in FIG. 31A. In this way, a plurality of display screens 1a may be provided. The imaging optical systems 3 in each display screen may be disposed at the same position toward the display screen 1a as shown in these figures, or may be disposed at completely different positions. For example, at least two imaging optical systems 3 may be disposed at both ends along the first direction on the front side, and at least two imaging optical systems 3 may be disposed at both ends along the second direction on the back side. In this case, for example, the imaging optical system 3 in FIG. 31B may be used as a rear camera in normal times, and may also be utilized as a two-lens (or more) rear camera.

[0164] In this way, the imaging optical system 3 can be arranged in various ways on the display screen 1a.

[0165] As shown in each of the above-mentioned embodiments, at least two of the imaging optical systems 3 may be provided at different positions. The arrangement can be freely selected without depending on the drawings. For example, in FIG. 2, etc., by providing a plurality of imaging optical systems 3 with a shift in the first direction around the center of the second direction of the display surface, it is possible to obtain an image or video of a natural line of sight of a user looking at the center of the screen. In addition, this is not limited to this, and for example, the imaging optical systems may be provided slightly above the second direction. For example, when obtaining an image of one's own face, the center of the face may be located around the center of the screen, and the imaging optical system may be located around the displayed eyes. In addition to the above, the imaging optical systems may be similarly provided near the center of the first direction along the second direction. In this case, it is possible to obtain a similar image even if the screen is rotated.

[0166] Next, there will be shown another example of the layout of the openings included in the optical system 9. The layout may be more complicated than the simple shapes shown in Figs.

[0167] 32 to 35 are diagrams showing examples of the layout of the openings. As shown in Fig. 32, for example, the openings may be arranged so as to be shifted in the second direction with respect to the display elements.

[0168] As shown in FIG. 33, for example, the opening may have a shape that is a combination of an ellipse having a major axis in the first direction and an ellipse having a major axis in the second direction, between the display elements.

[0169] 34, for example, the opening belonging to the first optical system 9A and the opening belonging to the second optical system 9B do not have to intersect at an angle close to 90°. In this case, for example, the angle of the light-emitting elements of the display panel 4 may also be partially changed. For example, the angle at which the light-emitting elements provided on the third direction display surface of any of the optical systems 9 are arranged may be rotated to 45° or any other significant angle.

[0170] Of course, as shown in FIG. 35, the openings shown in FIG. 33 may be arranged as shown in FIG.

[0171] In this way, the apertures that are part of the optical system 9 may also be laid out so that flare occurs in significantly different directions. When the electronic device 1 is provided with three or more imaging optical systems 3, the layouts of the apertures shown in Figures 5, 8, and 32 to 35 may be combined. In this case, the direction in which flare occurs becomes a combination of even more directions depending on the imaging optical system 3, so it becomes possible to obtain images with a wider range of options, for example, image correction, composition, and selection.

[0172] As described above, according to each embodiment, the imaging optical system 3 is disposed on the side opposite to the display surface of the display unit 2, and light passing through the display unit 2 is captured by the multiple imaging units 8. A portion of the light passing through the display unit 2 is repeatedly reflected within the display unit 2 and then incident on the imaging units 8 in the multiple imaging optical systems 3. According to each embodiment described above, by acquiring an image from signals acquired by the multiple imaging optical systems 3, it is possible to generate an image in which flare components and diffracted light components contained in light repeatedly reflected within the display unit 2 and then incident on the multiple imaging units 8 (including the case where they are integrated) are simply and reliably suppressed.

[0173] Also, for example, the processing in the image acquisition unit 12 etc. may be implemented by a digital circuit or a programmable circuit such as an FPGA (Field Programmable Gate Array). Also, the processing contents may be written in a program, and information processing by software may be specifically realized using hardware resources such as a CPU.

[0174] Some application examples are given below.

[0175] Fifteenth embodiment There are various possible specific candidates for the electronic device 1 having the configuration described in the above-mentioned embodiments. For example, Fig. 36 is a plan view of the electronic device 1 of each embodiment applied to a capsule endoscope 50. The capsule endoscope 50 in Fig. 36 includes, in a housing 51 having, for example, hemispherical end faces and a cylindrical center, a camera (ultra-miniature camera) 52 for capturing images inside a body cavity, a memory 53 for recording image data captured by the camera 52, and a wireless transmitter 55 for transmitting the recorded image data to the outside via an antenna 54 after the capsule endoscope 50 is discharged from the subject's body.

[0176] Also provided within housing 51 are a CPU (Central Processing Unit) 56 and a coil (magnetic force-current conversion coil) 57. CPU 56 controls the shooting by camera 52 and the data storage operation in memory 53, and also controls the data transmission from memory 53 to a data receiving device (not shown) outside housing 51 by wireless transmitter 55. Coil 57 supplies power to camera 52, memory 53, wireless transmitter 55, antenna 54, and light source 52b described below.

[0177] Furthermore, the housing 51 is provided with a magnetic (reed) switch 58 for detecting when the capsule endoscope 50 is set in the data receiving device. When the reed switch 58 detects that the capsule endoscope 50 is set in the data receiving device and data transmission becomes possible, the CPU 56 supplies power from the coil 57 to the wireless transmitter 55.

[0178] The camera 52 has, for example, an image pickup element 52a including an optical system 9 for capturing an image inside the body cavity, and a plurality of light sources 52b for illuminating the inside of the body cavity. Specifically, the camera 52 has, for example, a CMOS (Complementary Metal Oxide Semiconductor) equipped with an LED (Light Emitting Diode) as the light source 52b. It is composed of a CCD (Charge Coupled Device), a CCD (Crystal Oxide Semiconductor) sensor, etc.

[0179] The display unit 2 in the electronic device 1 of the above-mentioned embodiment is a concept including a light-emitting body such as the light source 52b in Fig. 36. The capsule endoscope 50 in Fig. 36 has, for example, two light sources 52b, but these light sources 52b can be configured with a display panel 4 having a plurality of light source units or an LED module having a plurality of LEDs. In this case, by arranging the imaging unit 8 of the camera 52 below the display panel 4 or the LED module, restrictions on the layout arrangement of the camera 52 are reduced, and a smaller capsule endoscope 50 can be realized.

[0180] Sixteenth embodiment 37 is a rear view of the electronic device 1 of the above-described embodiments applied to a digital single-lens reflex camera 60. The digital single-lens reflex camera 60 and compact cameras are provided with a display unit 2 that displays a preview screen on the rear surface opposite the lens. An imaging optical system 3 may be arranged on the opposite side of the display surface of the display unit 2 so that an image of the photographer's face can be displayed on the display screen 1a of the display unit 2. In the electronic device 1 according to each of the above-described embodiments, the imaging optical system 3 can be arranged in an area overlapping with the display unit 2, so that it is not necessary to provide the imaging optical system 3 in the frame portion of the display unit 2, and the size of the display unit 2 can be made as large as possible.

[0181] Seventeenth embodiment Fig. 38A is a plan view showing an example in which the electronic device 1 of the above-mentioned embodiment is applied to a head mounted display (hereinafter, HMD) 61. The HMD 61 in Fig. 38A is used for VR (Virtual Reality), AR (Augmented Reality), MR (Mixed Reality), SR (Substitutional Reality), or the like. Current HMDs, as shown in Fig. 38B, are equipped with a camera 62 on the outer surface, and while a wearer of the HMD can view images of the surroundings, there is a problem in that people around cannot see the eyes or facial expressions of the wearer of the HMD.

[0182] 38A, the display surface of the display unit 2 is provided on the outer surface of the HMD 61, and the imaging optical system 3 is provided on the opposite side to the display surface of the display unit 2. This allows the facial expression of the wearer captured by the imaging optical system 3 to be displayed on the display surface of the display unit 2, allowing people around the wearer to grasp the facial expression and eye movements of the wearer in real time.

[0183] 38A, since the imaging optical system 3 is provided on the back side of the display unit 2, there are no restrictions on the location of the imaging optical system 3, which increases the freedom of design of the HMD 61. In addition, since the camera can be placed in an optimal position, it is possible to prevent problems such as the wearer's line of sight not being aligned with the display surface.

[0184] In this manner, in this embodiment, the electronic device 1 according to the above-described embodiment can be used for various purposes, thereby increasing its utility value.

[0185] The present technology can be configured as follows.

[0186] (1) Electronic devices include: a display unit having a displayable area in which a display optical system is provided in an array along a first direction and a second direction intersecting the first direction; In a third direction intersecting the first direction and the second direction, the display unit is disposed on the opposite side to the display surface of the display unit while overlapping the displayable area, and at least A first imaging optical system; a second imaging optical system having a different coordinate from that of the first imaging optical system in at least one of the first direction and the second direction; A plurality of imaging optical systems having an image acquisition unit that acquires image data based on information acquired by the first imaging optical system and the second imaging optical system; An electronic device comprising:

[0187] (2) Light propagates from the display surface of the display unit to the first imaging optical system and the second imaging optical system via optical systems having different optical characteristics. (1) An electronic device as described in (1).

[0188] (3) the display unit is provided with an opening through which light incident from the display surface propagates; Light incident from the display surface is propagated through the opening to the imaging optical system. An electronic device as described in (1) or (2).

[0189] (4) the opening for propagating light to the first imaging optical system and the opening for propagating light to the second imaging optical system have different layouts; (3) An electronic device as described in (3).

[0190] (5) the aperture for propagating light to the first imaging optical system and the aperture for propagating light to the second imaging optical system form diffraction images in different directions; An electronic device as described in (3) or (4).

[0191] (6) a third imaging optical system having a parallax with respect to the first imaging optical system that is the same as a parallax with respect to the first imaging optical system and the second imaging optical system, the image acquisition unit acquires the image data based on information acquired based on information acquired from the second imaging optical system and the third imaging optical system and on information acquired from the first imaging optical system. An electronic device described in any one of (1) to (5).

[0192] (7) In the first direction or the second direction of the display surface, The first imaging optical system is provided near the center, the second imaging optical system and the third imaging optical system are provided near a boundary of the display surface so as to sandwich the first imaging optical system, the first imaging optical system has a smaller area facing the display surface in the third direction than the second imaging optical system and the third imaging optical system; (6) An electronic device as described in (6).

[0193] (8) The arrangement of the first imaging optical system and the second imaging optical system has different coordinates in both the first direction and the second direction. An electronic device described in any one of (1) to (7).

[0194] (9) When the data acquired from the first imaging optical system and the data acquired from the second imaging optical system are synthesized, the image acquisition unit acquires an imaging result based on data having a low intensity at a position in the image data that is a synthesis result. An electronic device described in any one of (1) to (8).

[0195] (10) the first imaging optical system and the second imaging optical system each have a direction in which light is preferentially reflected; When a difference of a predetermined value or more occurs between the outputs for each direction, the image acquisition unit acquires an imaging result using one of the results. An electronic device according to any one of (1) to (9).

[0196] (11) A light shield is provided between the first imaging optical system and the second imaging optical system. An electronic device according to any one of (1) to (10).

[0197] (12) The image acquisition unit uses a trained model to combine information acquired by the first imaging optical system and the second imaging optical system. An electronic device according to any one of (1) to (11).

[0198] (13) The trained model is trained based on data collected from a plurality of electronic devices. (12) An electronic device as described in (12).

[0199] (14) the image acquisition unit performs a correction when a parallax between the plurality of images acquired by the plurality of imaging optical systems exceeds a predetermined amount. An electronic device according to any one of (1) to (13).

[0200] (15) At least one of the imaging optical systems is constituted by a microlens array. An electronic device according to any one of (1) to (14).

[0201] (16) A plurality of the imaging optical systems are provided in the area where the microlens array is provided. (15) An electronic device as described in (15).

[0202] (17) The first imaging optical system and the second imaging optical system acquire information at the same imaging element. An electronic device according to any one of (1) to (16).

[0203] (18) The image acquisition unit is disposed on the same chip as the imaging element. (17) An electronic device as described in (17).

[0204] (19) The display unit includes a plurality of display optical systems having different optical characteristics. An electronic device according to any one of (1) to (18).

[0205] (20) At least one of the plurality of imaging optical systems operates when signal correction is required in the image acquisition unit. An electronic device according to any one of (1) to (19).

[0206] (twenty one) The first imaging optical system and the second imaging optical system are integrated together. An electronic device according to any one of (1) to (20).

[0207] (twenty two) The first imaging optical system and the second imaging optical system are provided near the boundary of the display surface. An electronic device according to any one of (1) to (21).

[0208] (twenty three) the first imaging optical system and the second imaging optical system are disposed at a distance of 50 mm or more and 80 mm or less; The image acquisition unit generates parallax image data of information acquired by the first imaging optical system and the second imaging optical system. An electronic device according to any one of (1) to (22).

[0209] (twenty four) The display unit is provided on both sides of the device. An electronic device according to any one of (1) to (23).

[0210] (twenty five) The first imaging optical system and the second imaging optical system are any two of the plurality of imaging optical systems. An electronic device according to any one of (1) to (24).

[0211] (26) a fourth imaging optical system different from the first imaging optical system and the second imaging optical system; The fourth imaging optical system, the first imaging optical system, or the second imaging optical system has any one of the features described in (1) to (24). electronic equipment.

[0212] The aspects of the present disclosure are not limited to the above-mentioned individual embodiments, but include various modifications that may be conceived by a person skilled in the art, and the effects of the present disclosure are not limited to the above-mentioned contents. In other words, various additions, modifications, and partial deletions are possible within the scope of the conceptual idea and intent of the present disclosure derived from the contents defined in the claims and their equivalents. [Explanation of symbols]

[0213] 1: electronic device, 1a: display screen, 1b: bezel, 2: Display section, 3: Imaging optical system, 3A: First imaging optical system, 3B: Second imaging optical system, 3C: Third imaging optical system, 4: display panel, 4a: substrate, 4b: light-emitting pixel, 5: circular polarizer, 6: Touch panel, 7: cover glass, 8: imaging unit, 8A: first imaging unit, 8B: second imaging unit, 9: Optical system, 9A: First optical system, 9B: Second optical system, 10: Pre-processing section; 12: Image acquisition unit, 14: Post-processing section, 16: Output section, 18: control unit, 20: Memory section, 22: receiving interface, 30: light shielding part, 32: Microlens array,

Claims

1. a display unit having a displayable area in which a display optical system is provided in an array along a first direction and a second direction intersecting the first direction; In a third direction intersecting the first direction and the second direction, the display unit is disposed on an opposite side to the display surface of the display unit, overlapping the displayable area, and A first imaging optical system; a second imaging optical system having a different coordinate from that of the first imaging optical system in at least one of the first direction and the second direction; A plurality of imaging optical systems having an image acquisition unit that acquires image data based on information acquired by the first imaging optical system and the second imaging optical system; Equipped with the display unit is provided with an opening through which light incident from the display surface propagates; light incident from the display surface through the opening is propagated to the imaging optical system; the opening for propagating light to the first imaging optical system and the opening for propagating light to the second imaging optical system have different layouts; electronic equipment.

2. a display unit having a displayable area in which a display optical system is provided in an array along a first direction and a second direction intersecting the first direction; In a third direction intersecting the first direction and the second direction, the display unit is disposed on an opposite side to the display surface of the display unit, overlapping the displayable area, and A first imaging optical system; a second imaging optical system having a different coordinate from that of the first imaging optical system in at least one of the first direction and the second direction; A plurality of imaging optical systems having an image acquisition unit that acquires image data based on information acquired by the first imaging optical system and the second imaging optical system; Equipped with a third imaging optical system having a parallax with respect to the first imaging optical system that is the same as a parallax with respect to the first imaging optical system and the second imaging optical system, the image acquisition unit acquires the image data based on information acquired based on information acquired from the second imaging optical system and the third imaging optical system and on information acquired from the first imaging optical system; In the first direction or the second direction of the display surface, The first imaging optical system is provided near the center, the second imaging optical system and the third imaging optical system are provided near a boundary of the display surface so as to sandwich the first imaging optical system, the first imaging optical system has a smaller area facing the display surface in the third direction than the second imaging optical system and the third imaging optical system; electronic equipment.

3. a display unit having a displayable area in which a display optical system is provided in an array along a first direction and a second direction intersecting the first direction; In a third direction intersecting the first direction and the second direction, the display unit is disposed on an opposite side to the display surface of the display unit, overlapping the displayable area, and A first imaging optical system; a second imaging optical system having a different coordinate from that of the first imaging optical system in at least one of the first direction and the second direction; A plurality of imaging optical systems having an image acquisition unit that acquires image data based on information acquired by the first imaging optical system and the second imaging optical system; Equipped with When the data acquired from the first imaging optical system and the data acquired from the second imaging optical system are synthesized, the image acquisition unit acquires an imaging result based on data having a low intensity at a position in the image data that is a synthesis result. electronic equipment.

4. a display unit having a displayable area in which a display optical system is provided in an array along a first direction and a second direction intersecting the first direction; In a third direction intersecting the first direction and the second direction, the display unit is disposed on an opposite side to the display surface of the display unit, overlapping the displayable area, and A first imaging optical system; a second imaging optical system having a different coordinate from that of the first imaging optical system in at least one of the first direction and the second direction; A plurality of imaging optical systems having an image acquisition unit that acquires image data based on information acquired by the first imaging optical system and the second imaging optical system; Equipped with the first imaging optical system and the second imaging optical system each have a direction in which the first imaging optical system and the second imaging optical system are preferentially reflected; When a difference of a predetermined value or more occurs between the outputs for each direction, the image acquisition unit acquires an imaging result using one of the results. electronic equipment.

5. Light propagates from the display surface of the display unit to an image sensor of the first image sensor and an image sensor of the second image sensor via optical systems having different optical characteristics.

5. The electronic device according to claim 1.

6. A light shield is provided between the first imaging optical system and the second imaging optical system.

6. The electronic device according to claim 1.

7. The image acquisition unit combines information acquired by the first imaging optical system and the second imaging optical system using a trained model.

7. The electronic device according to claim 1.

8. The trained model is trained based on data collected from a plurality of electronic devices.

8. The electronic device according to claim 7.

9. the image acquisition unit performs a correction when a parallax between the plurality of images acquired by the plurality of imaging optical systems exceeds a predetermined amount.

9. The electronic device according to claim 1.

10. At least one of the imaging optical systems is constituted by a microlens array. The electronic device according to claim 1 .

11. A plurality of the imaging optical systems are provided in the area where the microlens array is provided. The electronic device according to claim 10.

12. the first imaging optical system and the second imaging optical system acquire information at the same imaging element; The electronic device according to claim 1 .

13. The image acquisition unit is disposed on the same chip as the image sensor.

13. The electronic device according to claim 12.

14. The display unit includes a plurality of display optical systems having different optical characteristics. The electronic device according to any one of claims 1 to 13.

15. At least one of the plurality of imaging optical systems operates when signal correction is required in the image acquisition unit.

15. The electronic device according to claim 1.

16. The first imaging optical system and the second imaging optical system are integrated together.

16. The electronic device according to claim 1.

17. The first imaging optical system and the second imaging optical system are provided near a boundary of the display surface.

17. The electronic device according to claim 1.

18. the first imaging optical system and the second imaging optical system are disposed at a distance of 50 mm or more and 80 mm or less; The image acquisition unit generates parallax image data of information acquired by the first imaging optical system and the second imaging optical system.

18. The electronic device according to claim 1.

19. The display unit is provided on both sides of the device.

19. The electronic device according to claim 1.

20. The first imaging optical system and the second imaging optical system are any two of the plurality of imaging optical systems.

20. The electronic device according to claim 1.

Images