Imaging display apparatus

Abstract

To provide a technique for reducing a temporal difference between an actual event and a displayed image in an imaging display device.SOLUTION: The control section 12 creates a plurality of pieces of image information 3a and 3b from image pickup information 2 from the image pickup section 11, and selects a position at which each of the plurality of pieces of image information 3a and 3b is displayed on the display section 14 according to line-of-sight information obtained by detecting, with the infrared light receiving section, light emitted from the infrared light emitting section and reflected by eyeballs.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an imaging and display device. [Background technology]

[0002] Wearable devices equipped with an imaging and display device, such as head-mounted displays and smart glasses, are known. In one type of wearable device, the scene in front of the user is captured as an image using an imaging device, and the image is displayed on a display device. This method allows the user to feel as if they are looking directly at the outside world, even through the display device. Furthermore, a display technology is known that uses gaze information obtained by gaze detection technology with a near-infrared emitter and a near-infrared receiver. The technology with a near-infrared light emitting unit and a near-infrared light receiving unit is also applied to biometric recognition technology such as fingerprint and iris recognition in smartphones. Patent document 1 describes a technology for providing an OLED panel incorporating a near-infrared organic light sensor that enables biometric authentication without affecting the aperture ratio of the organic light emitting diode (OLED) light emitting unit. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-33071 Summary of the Invention [Problem to be solved by the invention]

[0004] By using technology that provides an OLED panel incorporating a near-infrared organic light sensor, as in the device described in Patent Document 1, it is possible to correct the displayed image using information obtained by gaze detection. Furthermore, correction processing based on gaze information enables, for example, foveated display, where the area being viewed by the eyes is displayed at high resolution and other areas at low resolution. However, this requires a long time to capture a real-world event, such as a landscape, correct the image based on the image information and gaze information, and display it as a display image. This creates a discrepancy between the real-world event and the displayed image, making it impossible to, for example, grasp a moving object.

[0005] An object of the present invention is to provide a technique for reducing the time difference between an actual event and a displayed image in an image capturing and displaying device. [Means for solving the problem]

[0006] The present invention provides an imaging and display device including an imaging unit having a plurality of light receiving elements, a display unit having a plurality of light emitting elements, an infrared emitting unit having an infrared emitting element, an infrared receiving unit having an infrared receiving element, and a control unit that processes imaging information captured by the imaging unit, the control unit creates a plurality of pieces of image information from the imaging information obtained by the imaging unit, The light emitted from the infrared light emitting unit is reflected by the eyeball, and the display unit selects one of the multiple image information depending on the line of sight information obtained by detecting the reflected light with the infrared light receiving unit. [Effects of the Invention]

[0007] According to the present invention, an imaging and display device can be obtained in which the time difference between an actual event and a displayed image is reduced. [Brief explanation of the drawings]

[0008] [Figure 1] 1A is a schematic diagram of the configuration of an imaging and display device of the present invention, and FIG. 1B is a schematic diagram of the configuration of a conventional imaging and display device. [Figure 2]FIG. 1A is a diagram illustrating the operation of the imaging and display device of the present invention, and FIG. 1B is a diagram illustrating the operation of a conventional imaging and display device. [Figure 3] 1A and 1B are diagrams showing imaging information obtained by an imaging device and image information displayed on the display units of conventional imaging display devices and imaging display devices of the present invention. [Figure 4] FIG. 1(a) is a schematic diagram showing the configuration of an imaging and displaying device according to a first embodiment of the present invention, and FIG. 1(b) is a schematic diagram showing the configuration of an imaging and displaying device according to a second embodiment of the present invention. [Figure 5] FIG. 10(a) is a schematic diagram showing the configuration of an imaging and displaying device according to a third embodiment of the present invention, and FIG. 10(b) is a schematic diagram showing the configuration of a display unit and a near-infrared light emitting unit of the imaging and displaying device. [Figure 6] FIG. 10(a) is a schematic diagram showing the configuration of an image capturing and displaying device according to a fourth embodiment of the present invention, and FIG. 10(b) is a schematic diagram showing the configuration of a display unit and a near-infrared light receiving unit of the image capturing and displaying device. [Figure 7] FIG. 10(a) is a schematic diagram showing the configuration of an imaging and displaying apparatus according to a fifth embodiment of the present invention, and FIG. 10(b) is a schematic diagram showing the configuration of an imaging and displaying apparatus according to a sixth embodiment of the present invention. [Figure 8] FIG. 10(a) is a schematic diagram showing the configuration of an imaging and display device according to a seventh embodiment of the present invention, and FIG. 10(b) is a schematic diagram showing the configuration of a display unit, a near-infrared light emitting unit, and a near-infrared light receiving unit of the imaging and display device. [Figure 9] FIG. 10(a) is a schematic diagram showing the configuration of an imaging and display device according to an eighth embodiment of the present invention, and FIG. 10(b) is a schematic diagram showing the configuration of a display unit, a near-infrared light emitting unit, and a near-infrared light receiving unit of the imaging and display device. [Figure 10] 1 is a schematic diagram illustrating a configuration of an image capturing and displaying apparatus of the present invention having a plurality of image capturing sections. [Figure 11] (a) is a schematic diagram of smart glasses, (b) is a schematic diagram of smart contact lenses, (c) is a cross-sectional schematic diagram of the imaging unit and display unit, (d) is a plan view schematic diagram of the imaging unit and display unit as seen from the imaging unit side, and (e) is a plan view schematic diagram of the imaging unit and display unit as seen from the display unit side. DETAILED DESCRIPTION OF THE INVENTION

[0009] The imaging and display device of the present invention includes an imaging unit, a display unit, an infrared light emitting unit, an infrared light receiving unit, and a control unit. The imaging unit has a plurality of light receiving elements, the display unit has a plurality of light emitting elements, the infrared light emitting unit has an infrared light emitting element, and the infrared light receiving unit has an infrared light receiving element. The present invention is characterized in that the control unit creates a plurality of image information from imaging information obtained by the imaging unit, and selects one of the image information according to line of sight information obtained by detecting the reflected light from the infrared light emitting unit reflected by the observer's eyeball.

[0010] Each embodiment will be described below with reference to the drawings. In the description of each embodiment, the description of the same configuration as in other embodiments may be omitted. Furthermore, each embodiment can be modified and combined as appropriate.

[0011] The basic configuration and operation of the imaging device of the present invention will be described with reference to FIGS. Fig. 1(a) is a schematic diagram showing the configuration of an embodiment of an imaging display device of the present invention. In Fig. 1(a), an imaging display device 10 includes an imaging unit 11 that captures imaging information 2, a control unit 12, a line-of-sight detection unit (switch) 13 that detects line-of-sight information 1, and a display unit 14 that displays display information 4.

[0012] The imaging unit 11 has a plurality of light receiving elements (not shown). The light emitting elements are, for example, light receiving conversion elements, which convert incident light (imaging information 2) from the outside into an electrical signal and perform an imaging operation to acquire the imaging information 2. The control unit 12 generates image information 3a, 3b of a plurality of resolutions based on the imaging information 2 from the imaging unit 11. For example, image information of high resolution and low resolution can be generated. Of course, image information of three or more types of resolutions, such as high resolution, medium resolution, and low resolution, may also be generated.

[0013] The line-of-sight detection unit 13 includes an infrared emitter and an infrared receiver. Here, the infrared emitter is a near-infrared emitter (NIR emitter) and the infrared receiver is a near-infrared receiver (NIR receiver) in the following description. In this embodiment, image information 3a', 3b' to be displayed on the display unit 14 is selected based on line-of-sight information 1. Therefore, the display information 4 displayed on the display unit 14 includes a mixture of image information 3a' and 3b'. For example, if the control unit 12 generates high-resolution and low-resolution image information, the pixel area that detects line-of-sight information 1, i.e., near-infrared light, can select high-resolution image information, and the pixel area that does not detect line-of-sight information 1, i.e., near-infrared light, can select low-resolution image information. Retinal retroreflection can be used as one method of line-of-sight detection. Retinal retroreflection is characterized by the phenomenon in which light incident perpendicularly to the retina of the eye is selectively reflected and returned. In other words, light emitted from the NIR emitter is efficiently reflected and efficiently detected by the NIR receiver only in the pixel area along the line of sight (the direction the pupil is facing). The reflection efficiency of near-infrared rays drops significantly in pixel areas outside the line of sight. Therefore, by setting a certain threshold value for the intensity of near-infrared rays associated with retinal retroreflection, it is possible to select whether to turn on or off the line of sight information 1.

[0014] The display unit 14 has a plurality of light-emitting elements. The plurality of light-emitting elements convert electrical signals into light. The display unit 14 displays display information 4 (image) according to information selected by the line-of-sight detection unit 13. The imaging unit 11 and the display unit 14 can also be said to be an array of a plurality of pixels. Each pixel of the imaging unit 11 has at least one light-receiving element, and each pixel of the display unit 14 has at least one light-emitting element.

[0015] Here, the configuration of a conventional imaging and display device will be described with reference to Fig. 1(b). Fig. 1(b) is a schematic diagram showing the configuration of an example of a conventional imaging and display device. The imaging and display device 20 has an imaging unit 21 that captures imaging information 2, a control unit 22, a line-of-sight detection unit 23 that detects line-of-sight information 1, and a display unit 24 that displays display information 4.

[0016] The control unit 22 generates new corrected image information 3' based on the imaging information 2 from the imaging unit 21 and the line-of-sight information 1 obtained from the line-of-sight detection unit 23. For example, the control unit 22 creates new corrected image information 3' by setting a high resolution for the pixel region toward which the line of sight is directed and a low resolution for the pixel region toward which the line of sight is not directed. The control unit 22 causes the display unit 24 to display display information 4 based on the corrected image information 3' created by the control unit 22. Therefore, although the display information displayed in a conventional imaging display device is the same as the display information of the present invention, the longer correction processing time taken by the control unit 22 results in a time difference between the imaging information 2 and the display information 4.

[0017] Next, the operation of the imaging and display device of the present invention and the conventional imaging and display device will be described with reference to Fig. 2. Fig. 2(a) is a diagram for explaining the operation of the imaging and display device of the present invention shown in Fig. 1(a), and Fig. 2(b) is a diagram for explaining the operation of the conventional imaging and display device shown in Fig. 1(b), and is a diagram for explaining the relationship between imaging information, image information or corrected image information, and a display image for one frame at a certain time. In Fig. 2(a) and Fig. 2(b), at time T n Image information 2 in A n , Gaze information 1 to B n The control unit processes the corrected image information 3' and outputs it to A n +B n , display image A' n Let's say.

[0018] As shown in FIG. 2(a), in the present invention, the imaging unit 11 -2 Image information A -2 At time T -1 Imaging information A-1 At time T0, image information A0 is acquired, and at time T +1 Image information A +1 Therefore, at time T -2 Image information A obtained -2 is the time T -2 Display information A' -2 In other words, the delay between obtaining the imaging information 2 and displaying the display information 4 can be almost eliminated.

[0019] On the other hand, as shown in FIG. 2(b), in the conventional imaging display device, the control unit 22 -2 , A -1 , A0, A +1 and gaze information B -2 , B -1 , B0, B +1 Based on this, corrected image information A -2 +B -2 , A -1 +B -1 , A0+B0, and A1+B1. Then, the control unit 22 generates the corrected image information A -2 +B -2 , A -1 +B -1 , A0+B0, A1+B1 to the display unit 24. The display unit 24 outputs the -1 Corrected image information A -2 +B -2 The image based on the corrected image information A -1 +B -1 At time T1, an image based on the corrected image information A0+B0 is displayed, and at time T2, an image based on the corrected image information A1+B1 is displayed. -1 -T -2 , T0-T -1 , T1-T0, T2-T1, there is a delay between obtaining the image information 2 and displaying the display information 4. If there is a difference between the actual event and the displayed image, for example, it will be impossible to grasp a moving object.

[0020] Furthermore, the control unit 12 of the imaging and display device 10 of the present invention may perform additional image processing. Examples of additional image processing include dark-field image processing to increase the brightness of dark images, enlarged image processing to enlarge and display small subjects, and temperature display processing to display temperature images. In this case, a delay will occur due to the time required for the additional image processing, but in the present invention, the timing for displaying display information is preferably within the following range.

[0021] First, the image capturing unit 11 captures an image at an arbitrary time T n Assume that an image is captured at time T mHere, the difference ΔT between the timing of capturing an image and the timing of displaying the image can be expressed by the following formula (1): ΔT=T n -T m (1)

[0022] Here, the display frame rate DFR (fps; frames per second) is the number of images displayed by the display unit per second. The imaging and display device 10 is controlled so that this difference ΔT satisfies the following formula (2). More preferably, the imaging and display device 10 is controlled so that the difference ΔT satisfies the following formula (3). -2 / DFR≦ΔT≦2 / DFR (2) -1 / DFR≦ΔT≦1 / DFR (3)

[0023] For example, if the display frame rate is 240 [fps], the time from capturing an image to displaying it (one frame) is approximately 4 × 10 -3 [sec]. Therefore, the difference ΔT is as follows: -4×10 -3 ≦ΔT≦4×10 -3 (4) By displaying the display information at such timing, it is possible to display a moving image with little delay between the actual image and the displayed image, which can also be called a real-time display.

[0024] Furthermore, when the amount of additional image processing becomes large, delays can be reduced by generating a predicted image at a certain time, for example, by sensing past image information or other environmental information. Although the present invention can be applied to still images, it is more effective to apply it to moving images. According to the image capturing and displaying apparatus of the present invention, real-time display is possible even when the time required for the additional image processing described above is added.

[0025] Next, the display image of the imaging display device of the present invention will be described with reference to Fig. 3. Fig. 3 shows an image of imaging information at T=0, an image displayed by the imaging display device of the present invention, and an image displayed by the conventional imaging display device shown in Fig. 1(b). Each shows the display information at T=0, with image 30 showing the imaging information, image 40 showing the display information of the conventional imaging display device, and image 50 showing the display information of the imaging display device of the present invention. Image 30 can also be said to be imaging information acquired by the imaging unit 11 at a certain time.

[0026] A conventional imaging and display device will now be described. First, the control unit recognizes the presence of a line-of-sight area 44 around a moving object 43 based on imaging information captured before T=0 and line-of-sight detection, and generates corrected image information such that the resolution of the line-of-sight area 44 including the moving object 43 is increased and the resolution outside of that area is reduced. The time at which this corrected image information is displayed is the image at T=0 in the middle of Figure 3. Compared to the image at T=0, the position at which the moving object 43 is displayed is different. In other words, it can be seen that a delay occurs.

[0027] In the imaging and display device of the present invention, high-resolution and low-resolution image information is generated in the control unit using imaging information, and a switching circuit that utilizes retinal retroreflection due to near-infrared rays selects between these in real time to display information. In other words, there is little delay, and an image similar to the imaging information can be displayed at T=0.

[0028] Furthermore, by changing the processing depending on the part, such as increasing the resolution of only the line-of-sight area and decreasing the resolution of other areas, the load on the control unit can be reduced. Also, by displaying moving objects at high resolution and still objects at low resolution, it is possible to provide a natural image that is closer to what the human eye sees. This type of processing makes it possible to reduce the load on the control unit while still providing real-time display.

[0029] Next, a specific structure of the imaging display device of the present invention will be described. First, the light receiving element of the imaging unit can include, for example, a photodiode or photogate, and a photoelectric conversion film. Examples of materials for the photodiode and photogate include silicon, germanium, indium, gallium, and arsenic. Examples of the photodiode form include a PN junction photodiode, a PIN photodiode, and an avalanche photodiode.

[0030] The imaging unit may be, for example, a CMOS image sensor, which may be either a front-illuminated or back-illuminated type. The CMOS image sensor may have a stacked structure in which a semiconductor substrate on which a photodiode is arranged and a semiconductor substrate on which a scanning circuit and a control circuit are arranged are stacked.

[0031] The photoelectric conversion film may be made of organic or inorganic materials. The organic photoelectric conversion film may have a structure including at least one organic layer that performs photoelectric conversion between a pair of electrodes. The organic photoelectric conversion film may have a structure in which multiple organic layers are stacked between a pair of electrodes. The organic layer may be made of a single material or a mixture of multiple materials. The organic layer may be formed using, for example, a vacuum deposition process or a coating process. Examples of inorganic photoelectric conversion films include quantum dot-type films that use a quantum dot thin film layer containing fine semiconductor crystals instead of an organic layer, and perovskite-type films that have a photoelectric conversion layer made of a transition metal oxide with a perovskite structure.

[0032] The line-of-sight detection unit has an infrared light-emitting unit and an infrared light-receiving unit. Specifically, it may be an NIR light-emitting unit and an NIR light-receiving unit, with the NIR light-emitting unit having a plurality of NIR light-emitting elements and the NIR light-receiving unit having a plurality of NIR light-receiving elements. The near-infrared region refers to a wavelength region from 780 nm to 2500 nm. The NIR light-receiving elements are, for example, photodiodes and photogates as described above, and photoelectric conversion films, which have absorption sensitivity in the near-infrared region. The NIR light-emitting elements are, for example, liquid crystal elements (LCDs), inorganic light-emitting diodes (inorganic LEDs), organic light-emitting diodes (OLEDs), inorganic electroluminescence (inorganic ELs), quantum dot diodes (quantum LEDs), etc., which emit light in the near-infrared region.

[0033] The display unit has multiple light-emitting elements. Examples of light-emitting elements include LCDs, inorganic LEDs, OLEDs, inorganic EL devices, and quantum LEDs. Materials used in inorganic LEDs include aluminum, gallium, arsenic, phosphorus, indium, nitrogen, selenium, zinc, diamond, zinc oxide, and perovskite semiconductors. By using these materials to form a pn junction structure, light is emitted with an energy (wavelength) corresponding to the band gap difference between the materials. OLEDs, for example, have a light-emitting layer containing at least one organic light-emitting material between a pair of electrodes. They may also have multiple light-emitting layers, or may have a structure in which multiple organic layers are stacked. The light-emitting layer may be made of a single material or a mixture of multiple materials. The light from the light-emitting layer may be fluorescent or phosphorescent, and may emit monochromatic light (e.g., blue, green, red, etc.) or white light. The organic layer may be formed using, for example, a vacuum deposition process or a coating process.

[0034] In the present invention, the light-emitting elements constituting the line-of-sight detection unit and the display unit are preferably OLEDs. This is because they can be fabricated by laminating and separately painting them on the same substrate. Furthermore, the light-receiving elements constituting the line-of-sight detection unit are preferably organic photoelectric conversion films. This is because they can be fabricated by laminating and separately painting them on the same substrate, and because they have a higher absorption coefficient in the near-infrared region than photodiodes using silicon, etc., and therefore have higher near-infrared light-receiving sensitivity.

[0035] Furthermore, the image capturing and displaying device may have a structure in which at least three chips, an image capturing section, a control section, and a display section, are stacked and electrically connected to each other by a semiconductor process.

[0036] When the imaging and display device of the present invention is used as a wearable device, it is preferable to reduce the amount of data processed by the processing unit. This is because wearable devices need to be as light and thin as possible, and the smaller the data processing load, the smaller the processing chip can be. One method for reducing the data processing load is to perform AI processing on a separate device (such as the cloud). Other methods for reducing the processing volume include lowering the resolution of areas outside the gaze area, turning areas outside the gaze area into still images, and processing areas outside the gaze area in monochrome rather than color.

[0037] Next, the structure of the image pickup and display device of the present invention will be described with reference to FIGS. [First embodiment] FIG. 4(a) is a schematic diagram showing the configuration of a first embodiment of the image capturing and displaying device of the present invention, in which a display unit 60 and an NIR light emitting / receiving unit 66 having both an NIR light emitting unit and an NIR light receiving unit are vertically stacked. A feature of this embodiment is that the number of pixels in the NIR light emitting / receiving unit 66 is smaller than the number of pixels in the display unit 60. For example, the NIR light emitting / receiving unit 66 can be divided into nine sections, and the corresponding high-resolution area can also be divided into nine sections. The small number of pixels in the NIR light emitting / receiving unit 66 allows for a larger aperture ratio of the NIR light emitting elements and NIR light receiving elements, thereby improving the sensitivity to near-infrared light emission and reception.

[0038] In this embodiment, NIR light 62 from the NIR light-emitting element of the NIR light-emitting / receiving unit 66 passes through the display unit 60 and is reflected by the retina of the pupil 61. The NIR reflected light 63 then reaches the NIR light-receiving element of the NIR light-emitting / receiving unit 66 again, turning on the switch of the pixel it reaches. When the switch is turned on, the pixel of the display unit 60 corresponding to the on-state pixel is connected to form the high-resolution region 64. Because the mechanism reflects only light perpendicular to the pupil 61 with high sensitivity, even if all the NIR light-emitting elements of the NIR light-emitting / receiving unit 66 are emitting light, only the NIR light-receiving element in the direction the pupil 61 is facing receives the reflected light with high sensitivity, making it possible to select the on-state pixel. The pixel of the display unit 60 corresponding to the pixel of the NIR light-emitting / receiving unit 66 that is off becomes the low-resolution region 65. The NIR light-emitting / receiving unit 66 and the display unit 60 can be connected using a known circuit. Furthermore, the NIR light 62 must pass through the display unit 60. For example, methods for achieving this include, but are not limited to, providing an area without wiring layer metal within or between pixels of the display unit 60, using wiring layer metal with a composition that reduces the near-infrared absorption rate, or using a near-infrared emission wavelength that reduces the absorption rate of the wiring layer metal. There is no limitation on the layout within one pixel of the NIR light-emitting / light-receiving unit 66, and the NIR light-emitting / light-receiving unit 66 may be composed of a plurality of NIR light-emitting elements or a plurality of NIR light-receiving elements, and these may be on the same plane or may be stacked.

[0039] Second Embodiment 4(b) is a schematic diagram showing the configuration of a second embodiment of the image capturing and displaying device of the present invention, in which a display section 60 and an NIR light emitting and receiving section 66 are vertically stacked as in the first embodiment, but a feature of this embodiment is that the number of pixels in the display section 60 is the same as the number of pixels in the NIR light emitting and receiving section 66. This allows the high resolution region 64 and the low resolution region 65 to be selected with better resolution.

[0040] Third Embodiment FIG. 5(a) is a schematic diagram showing the configuration of a third embodiment of an imaging and display device of the present invention, and FIG. 5(b) is a schematic plan view of a pixel 70 of a display unit 60. This embodiment is the same as the first embodiment, except that the display unit 60 and the NIR light receiving unit 68 are vertically stacked, and the NIR light emitting unit is disposed within the pixel of the display unit 60. The pixel 70 of the display unit 60 may be composed of, for example, an R (red) light emitting pixel, a G (green) light emitting pixel, a B (blue) light emitting pixel, and an NIR light emitting pixel 67a, but the pixel arrangement, emission color of each pixel, pixel area, aperture ratio, etc. are not particularly limited. In this embodiment, the light receiving area of ​​the NIR light receiving unit 68 can be maximized, and the NIR light 62 only needs to pass through the display unit 60 once. This increases the intensity of the near-infrared light received by the NIR light receiving element, leading to improved switching accuracy.

[0041] [Fourth embodiment] FIG. 6(a) is a schematic diagram showing the configuration of a fourth embodiment of the image capture and display device of the present invention, and FIG. 6(b) is a schematic plan view of a pixel 70 of a display unit 60. This embodiment is the same as the first embodiment, except that the display unit 60 and the NIR light-emitting unit 67 are vertically stacked, and the NIR light-receiving unit is disposed within the pixel of the display unit 60. The pixel 70 of the display unit 60 may be composed of, for example, an R-emitting pixel, a G-emitting pixel, a B-emitting pixel, and an NIR light-receiving pixel 68a, but the pixel arrangement, emission color of each pixel, pixel area, aperture ratio, etc. are not particularly limited. A structure in which RGB light-emitting pixels and a near-infrared light-receiving region are stacked may also be used. A feature of this embodiment is that the light-emitting area of ​​the NIR light-emitting unit 67 can be maximized, and the NIR reflected light 63 only needs to pass through the display unit 60 once, thereby increasing the intensity of the near-infrared light received by the NIR light-receiving element and improving switching accuracy.

[0042] Fifth Embodiment 7(a) is a schematic diagram showing the configuration of a fifth embodiment of the image capturing and displaying device of the present invention, which is the same as the first embodiment except that the display unit 60 and the NIR light emitting and receiving units 66 are arranged on the same plane. A feature of this embodiment is that by all being on the same plane, absorption loss due to wiring layer metals and the like in the display unit 60 can be suppressed, thereby improving switching sensitivity by near-infrared rays. Furthermore, in this embodiment, for example, by providing the NIR light emitting and receiving units 66 in nine locations within the display unit 60, the high-resolution region 64 can also be divided into nine regions corresponding to the nine NIR light emitting and receiving units 66.

[0043] Sixth Embodiment 7(b) is a schematic diagram showing the configuration of a sixth embodiment of an image capturing and displaying device of the present invention, which is the same as the fifth embodiment except that the NIR light emitting and receiving units 66 are arranged in the peripheral region of the display unit 60. A feature of this embodiment is that the NIR light emitting and receiving units 66 are arranged in the peripheral region of the display unit 60, thereby maximizing the number of pixels of the display unit 60. In this case, no NIR light emitting and receiving units 66 are present in the central region of the display unit 60, but, for example, if all of the NIR light emitting and receiving units 66 on the periphery are determined to be switched off, the central region can be selected as the high-resolution region 64.

[0044] Seventh Embodiment FIG. 8( a) is a schematic diagram showing the configuration of a seventh embodiment of the image capturing and displaying device of the present invention, and FIGS. 8( b) and 8(c) are schematic plan views of pixels 70 of a display unit 60. This embodiment is the same as the first embodiment except that the NIR light-emitting / receiving unit 66 is on the same plane as the display unit 60 and has the same number of pixels. A feature of this embodiment is that the high-resolution region 64 and the low-resolution region 65 can be selected with higher resolution. The pixel layout of the display unit 60 and the NIR light-emitting / receiving unit 66 may be, for example, as shown in FIG. 8( b), where the pixel 70 is divided into four parts: RGB light-emitting pixels and near-infrared pixels, with NIR light-emitting pixels 67 a and NIR light-receiving pixels 68 a arranged in the near-infrared pixels. Alternatively, as shown in FIG. 8( c), the pixel 70 may be divided into four parts: RGB light-emitting pixels and NIR light-emitting pixels 67 a, with the NIR light-receiving pixels 68 a arranged around these pixels.

[0045] Eighth Embodiment FIG. 9(a) is a schematic diagram showing the configuration of an eighth embodiment of the image capturing and displaying device of the present invention, and FIGS. 9(b) and 9(c) are schematic plan views of pixels 70 of a display unit 60. This embodiment is the same as the seventh embodiment except that it has a medium-resolution region 69. In this embodiment, for example, multiple thresholds related to the sensitivity of the NIR light receiving element can be set to select multiple resolutions. This smooths the boundaries between different resolutions, making it possible to provide higher-definition display information.

[0046] Ninth Embodiment Fig. 10 is a schematic diagram showing the configuration of a ninth embodiment of the imaging and display device of the present invention, which is similar to the configuration shown in Fig. 1(a) except that it has a plurality of imaging units. In this embodiment, imaging unit 11a acquires high-resolution imaging information 2a, and imaging unit 11b acquires low-resolution imaging information 2b, which can be selected by line-of-sight detection unit 13 without going through control unit 12. This further reduces the delay from imaging to display.

[0047] Although the present invention has been described with reference to the above embodiments, it is not limited thereto. The imaging and display device of the present invention can reduce the time difference between a real event and a displayed image, allowing the user to use the device without feeling uncomfortable.

[0048] [Wearable Devices] An example of application of the imaging and display device of the present invention to a wearable device will be described with reference to Fig. 11. The imaging and display device can be applied to wearable devices such as smart glasses, HMDs (Head Mounted Displays), and smart contact lenses.

[0049] FIG. 11(a) is a schematic diagram illustrating smart glasses. Smart glasses are also called eyeglass-type imaging and display devices or glasses. The smart glasses include an eyeglass frame and the imaging and display device of the present invention. Specifically, the smart glasses include at least an imaging unit 101, a processing unit 102, and a display unit 103. The imaging unit 101 is provided on two sides of the eyeglass frame, and the processing unit 102 is housed in the temples of the eyeglasses. The display unit 103 is provided in any position depending on the display format, but may be included in the lens 111. In either case, the display unit 103 displays an image on the lens 111. The processing unit 102 may include an AI unit. The smart glasses may have an external interface, and the processing unit 102 may communicate with an external AI unit.

[0050] The smart glasses in Fig. 11(a) may have two image capturing and display devices, one for the left eye and one for the right eye. In this case, the timing of image capturing and display can be set arbitrarily in the image capturing and display devices for the left eye and the right eye. Specifically, the operation may be to capture images at the same time and display them at different times, or to capture images at different times and display them at the same time.

[0051] FIG. 11(b) is a schematic diagram illustrating a smart contact lens 120. The smart contact lens 120 is also referred to as a contact lens-type imaging and display device or a contact lens. One smart contact lens 120 has one imaging and display device 121 and one control device 122. The control device 122 functions as a power supply unit that supplies power to the imaging and display device 121, and also has an AI unit that supports the processing unit of the imaging and display device 121. The AI ​​unit may be provided in a terminal separate from the smart contact lens 120. The smart contact lens 120 is preferably provided with an optical system for focusing light on the imaging and display device 121. The power supply unit has an interface for connecting to the outside. The power supply unit may be connected to the outside (for charging) via a wired or wireless connection.

[0052] The base material of the lens 111 in FIG. 11(a) and the smart contact lens 120 in FIG. 11(b) is transparent, and the display unit of the imaging display device projects a display image onto the transparent lens portion.

[0053] 11(a) and 11(b), the imaging unit 101 and the display unit 103 may be provided at different positions, or may be provided so that they are stacked in the line of sight. FIG. 11(c) shows a cross-sectional view of the imaging unit 101 and the display unit 103. FIG. 11(d) shows a plan view of the imaging unit 101 and the display unit 103 as viewed from the imaging unit 101 side. FIG. 11(e) shows a plan view of the imaging unit 101 and the display unit 103 as viewed from the display unit 103 side. FIG. 11(d) shows a center of gravity 132 of an imaging region 131 in which the pixels of the imaging unit 101 are arranged. FIG. 11(e) shows a center of gravity 134 of a display region 133 in which the pixels of the display unit 103 are arranged. As shown in FIG. 11(c), the imaging unit 101 and the display unit 103 may be provided so that a line segment A passes through the two centers of gravity. This is because it is possible to reduce discrepancies that arise due to differences in position between captured image information and the displayed image in a wearable device.

[0054] [Included components] The disclosure of this embodiment includes the following configuration. (Configuration 1) An imaging display device including an imaging unit having a plurality of light receiving elements, a display unit having a plurality of light emitting elements, an infrared emitting unit having an infrared emitting element, an infrared receiving unit having an infrared receiving element, and a control unit that processes imaging information captured by the imaging unit, the control unit creates a plurality of pieces of image information from the imaging information from the imaging unit, An imaging display device characterized in that light emitted from the infrared light emitting unit is reflected by the eyeball, and one of the plurality of light sources is selected depending on line of sight information obtained by detecting the reflected light with the infrared light receiving unit. (Configuration 2) 2. The imaging and display device according to configuration 1, wherein the plurality of pieces of image information are pieces of image information with different resolutions. (Configuration 3) 3. The imaging and display device according to configuration 2, wherein the plurality of pieces of image information are three types of image information with different resolutions.

[0055] (Configuration 4) The infrared light emitting unit and the infrared light receiving unit are on the same plane, the display unit, the infrared light emitting unit, and the infrared light receiving unit are stacked together, 4. The image capturing and displaying device according to any one of configurations 1 to 3, wherein the number of pixels of the infrared light emitting section and the infrared light receiving section is smaller than the number of pixels of the display section. (Configuration 5) The infrared light emitting unit and the infrared light receiving unit are on the same plane, the display unit, the infrared light emitting unit, and the infrared light receiving unit are stacked together, 4. The image capturing and displaying device according to any one of configurations 1 to 3, wherein the number of pixels of the display section is equal to the number of pixels of the infrared light emitting section and the infrared light receiving section.

[0056] (Configuration 6) The display unit and the infrared light emitting unit are on the same plane, 4. The imaging display device according to any one of configurations 1 to 3, wherein the infrared receiving section, the display section, and the infrared emitting section are stacked. (Configuration 7) The display unit and the infrared receiving unit are on the same plane, 4. The imaging display device according to any one of configurations 1 to 3, wherein the infrared light emitting section, the display section, and the infrared light receiving section are stacked.

[0057] (Configuration 8) the display unit, the infrared light emitting unit, and the infrared light receiving unit are on the same plane, 4. The image capturing and displaying device according to any one of configurations 1 to 3, wherein the number of pixels of the infrared light emitting section and the infrared light receiving section is smaller than the number of pixels of the display section. (Configuration 9) the display unit, the infrared light emitting unit, and the infrared light receiving unit are on the same plane, 4. The image capturing and displaying device according to any one of configurations 1 to 3, wherein the infrared light emitting section and the infrared light receiving section are arranged around the display section. (Configuration 10) the display unit, the infrared light emitting unit, and the infrared light receiving unit are on the same plane, 4. The image capturing and displaying device according to any one of configurations 1 to 3, wherein the number of pixels of the display section is equal to the number of pixels of the infrared light emitting section and the infrared light receiving section.

[0058] (Configuration 11) 11. The image capturing and displaying device according to any one of configurations 1 to 10, wherein a plurality of the image capturing units are arranged. (Configuration 12) 12. The image capturing and displaying device according to any one of configurations 1 to 11, wherein the display section has organic light emitting diode elements or inorganic electroluminescence elements as light emitting elements. (Configuration 13) 13. The image capturing and displaying device according to any one of configurations 1 to 12, wherein the infrared light emitting section has an organic light emitting diode element as a light emitting element. (Configuration 14) 14. The image capturing and displaying device according to any one of configurations 1 to 13, wherein the infrared receiving section has an organic photoelectric conversion film. (Configuration 15) 15. The image capturing and displaying device according to any one of configurations 1 to 14, wherein the image capturing section has a back-illuminated CMOS image sensor. (Configuration 16) The imaging display device described in any one of configurations 1 to 14, characterized in that the imaging unit is formed by stacking a substrate on which the plurality of light receiving elements are arranged and a substrate on which a circuit for processing signals from the plurality of light receiving elements is arranged. (Configuration 17) an imaging and display device according to any one of configurations 1 to 16; a power supply unit that supplies power to the image capture and display device. (Configuration 18) 18. The wearable device according to claim 17, wherein the power supply unit includes an interface for wirelessly connecting to the outside. [Explanation of symbols]

[0059] 1: line of sight information, 2, 2a, 2b: imaging information, 3, 3a, 3b, 3a', 3b': image information, 4: display information, 10: imaging display device, 11: imaging unit, 12: control unit, 13: line of sight detection unit, 14: display unit

Claims

1. An imaging display device including an imaging unit having a plurality of light receiving elements, a display unit having a plurality of light emitting elements, an infrared emitting unit having an infrared emitting element, an infrared receiving unit having an infrared receiving element, and a control unit that processes imaging information captured by the imaging unit, the control unit creates a plurality of pieces of image information from the imaging information from the imaging unit, An imaging and display device characterized in that light emitted from the infrared light emitting unit is reflected by the eyeball, and the reflected light is detected by the infrared light receiving unit to obtain line-of-sight information, thereby selecting one of the plurality of image information.

2. 2. The image pickup and display device according to claim 1, wherein the plurality of pieces of image information have different resolutions.

3. 3. The image pickup and display device according to claim 2, wherein the plurality of pieces of image information are three types of image information with different resolutions.

4. The infrared light emitting unit and the infrared light receiving unit are on the same plane, the display unit, the infrared light emitting unit, and the infrared light receiving unit are stacked together, 2. The image pickup display device according to claim 1, wherein the number of pixels of the infrared light emitting section and the infrared light receiving section is smaller than the number of pixels of the display section.

5. The infrared light emitting unit and the infrared light receiving unit are on the same plane, the display unit, the infrared light emitting unit, and the infrared light receiving unit are stacked together, 2. The image pickup display device according to claim 1, wherein the number of pixels of the display section is equal to the number of pixels of the infrared light emitting section and the infrared light receiving section.

6. The display unit and the infrared light emitting unit are on the same plane, 2. The image pickup display device according to claim 1, wherein the infrared receiving section, the display section, and the infrared emitting section are stacked.

7. The display unit and the infrared receiving unit are on the same plane, 2. The image pickup display device according to claim 1, wherein the infrared light emitting section, the display section, and the infrared light receiving section are stacked.

8. the display unit, the infrared light emitting unit, and the infrared light receiving unit are on the same plane, 2. The image pickup display device according to claim 1, wherein the number of pixels of the infrared light emitting section and the infrared light receiving section is smaller than the number of pixels of the display section.

9. the display unit, the infrared light emitting unit, and the infrared light receiving unit are on the same plane, 2. The image pickup display device according to claim 1, wherein the infrared light emitting section and the infrared light receiving section are arranged around the display section.

10. the display unit, the infrared light emitting unit, and the infrared light receiving unit are on the same plane, 2. The image pickup display device according to claim 1, wherein the number of pixels of the display section is equal to the number of pixels of the infrared light emitting section and the infrared light receiving section.

11. 2. The image pickup display device according to claim 1, wherein a plurality of the image pickup units are arranged.

12. 2. The image pickup display device according to claim 1, wherein the display unit has organic light emitting diode elements or inorganic electroluminescence elements as light emitting elements.

13. 2. The image pickup display device according to claim 1, wherein the infrared light emitting section has an organic light emitting diode element as a light emitting element.

14. 2. The image pickup display device according to claim 1, wherein the infrared receiving section has an organic photoelectric conversion film.

15. 2. The image pickup display device according to claim 1, wherein the image pickup unit has a back-illuminated CMOS image sensor.

16. 2. The imaging display device according to claim 1, wherein the imaging unit is formed by stacking a substrate on which the plurality of light receiving elements are arranged and a substrate on which a circuit for processing signals from the plurality of light receiving elements is arranged.

17. The imaging and display device according to any one of claims 1 to 16, a power supply unit that supplies power to the image capture and display device.

18. 18. The wearable device according to claim 17, wherein the power supply unit includes an interface for wirelessly connecting to an external device.

Images