Imaging display device

Abstract

An imaging display device 10 comprises an imaging unit 11, a display unit 14, an infrared-light-emitting unit, an infrared-light-receiving unit, and a control unit 12 in order to reduce any temporal difference between an actual event and a displayed image, wherein the control unit 12 creates a plurality of pieces of image information 3a, 3b from imaging information 2 obtained from the imaging unit 11, and, in accordance with line-of-sight information obtained when luminescence from the infrared-light-emitting unit is reflected by eyeballs and the resulting reflected light is detected by the infrared-light-receiving unit, selects positions at which each of the plurality of pieces of image information 3a, 3b are to be displayed by the display unit 14.

Description

Image capture and display device

[0001] The present invention relates to an imaging and display device.

[0002] Wearable devices equipped with an imaging display device, such as head-mounted displays and smart glasses, are known. In one type of wearable device, the scene in front of the user is captured as an image using an imaging device, and the image is displayed on a display device. This method allows the user to feel as if they are looking directly at the outside world, even through the display device. Furthermore, a display technology that uses gaze information obtained by gaze detection technology with a near-infrared emitter and a near-infrared receiver is also known.

[0003] The technology having a near-infrared light emitting unit and a near-infrared light receiving unit is also applied to biometric recognition technology such as fingerprint and iris recognition in smartphones. Patent Document 1 describes a technology for providing an organic light emitting diode (OLED) panel incorporating a near-infrared organic light sensor that enables biometric authentication without affecting the aperture ratio of the OLED light emitting unit.

[0004] Japanese Patent Application Laid-Open No. 2019-33071

[0005] By using technology that provides an OLED panel incorporating a near-infrared organic light sensor, as in the device of Patent Document 1, it is possible to correct the displayed image using information obtained by gaze detection. Furthermore, correction processing based on gaze information enables, for example, foveated display, in which the area being viewed by the eyes is displayed at high resolution and other areas at low resolution. However, it takes a long time to capture a real-world event, such as a landscape, correct the image based on the image information and gaze information, and display the image. This creates a discrepancy between the real-world event and the displayed image, making it impossible to, for example, grasp a moving object.

[0006] An object of the present invention is to provide a technique for reducing the time difference between an actual event and a displayed image in an image capturing and displaying device.

[0007] The present invention is an imaging and display device comprising an imaging unit having a plurality of light receiving elements, a display unit having a plurality of light emitting elements, an infrared emitting unit having an infrared emitting element, an infrared receiving unit having an infrared receiving element, and a control unit that processes imaging information captured by the imaging unit, wherein the control unit creates a plurality of image information from the imaging information obtained by the imaging unit, and the display unit selects one of the plurality of image information in accordance with line of sight information obtained by detecting the reflected light from the infrared emitting unit reflected by the eyeball.

[0008] According to the present invention, an imaging and display device can be obtained in which the time difference between an actual event and a displayed image is reduced.

[0009] FIG. 1 is a schematic diagram of the configuration of an imaging display device of the present invention. FIG. 2 is a schematic diagram of the configuration of a conventional imaging display device. FIG. 3 is a diagram explaining the operation of an imaging display device of the present invention. FIG. 4 is a diagram explaining the operation of a conventional imaging display device. FIG. 5 is a diagram showing imaging information obtained by an imaging device and image information displayed on a display unit of an imaging display device of the past and the present invention. FIG. 6 is a schematic diagram showing the configuration of an imaging display device of a first embodiment of the present invention. FIG. 7 is a schematic diagram showing the configuration of an imaging display device of a second embodiment of the present invention. FIG. 8 is a schematic diagram showing the configuration of an imaging display device of a third embodiment of the present invention. FIG. 9 is a schematic diagram showing the configuration of a display unit and a near-infrared light emitting unit of the imaging display device. FIG. 10 is a schematic diagram showing the configuration of an imaging display device of a fourth embodiment of the present invention. FIG. 11 is a schematic diagram showing the configuration of a display unit and a near-infrared light receiving unit of the imaging display device. FIG. 12 is a schematic diagram showing the configuration of an imaging display device of a fifth embodiment of the present invention. FIG. 13 is a schematic diagram showing the configuration of an imaging display device of a sixth embodiment of the present invention. FIG. 14 is a schematic diagram showing the configuration of an imaging display device of a seventh embodiment of the present invention. FIG. 15 is a schematic diagram showing the configuration of a display unit, a near-infrared light emitting unit, and a near-infrared light receiving unit of the imaging display device. FIG. 16 is a schematic diagram showing the configuration of a display unit, a near-infrared light emitting unit, and a near-infrared light receiving unit of the imaging display device. 1 is a schematic diagram showing the configuration of a display unit, a near-infrared light emitting unit, and a near-infrared light receiving unit of the imaging display device; FIG. 2 is a schematic diagram showing the configuration of a display unit, a near-infrared light emitting unit, and a near-infrared light receiving unit of the imaging display device; FIG. 3 is a schematic diagram showing the configuration of an imaging display device of the present invention having a plurality of imaging units; FIG. 4 is a schematic diagram of smart glasses; FIG. 5 is a schematic diagram of a smart contact lens; FIG. 6 is a schematic cross-sectional view of an imaging unit and a display unit; FIG. 7 is a schematic plan view of an imaging unit and a display unit as viewed from the imaging unit side; and FIG. 8 is a schematic plan view of an imaging unit and a display unit as viewed from the display unit side.

[0010] The imaging and display device of the present invention includes an imaging unit, a display unit, an infrared light emitting unit, an infrared light receiving unit, and a control unit. The imaging unit has a plurality of light receiving elements, the display unit has a plurality of light emitting elements, the infrared light emitting unit has an infrared light emitting element, and the infrared light receiving unit has an infrared light receiving element. The present invention is characterized in that the control unit creates a plurality of image information from imaging information obtained by the imaging unit, and selects one of the image information according to line of sight information obtained by detecting the reflected light from the infrared light emitting unit reflected by the observer's eyeball.

[0011] Each embodiment will be described below with reference to the drawings. In the description of each embodiment, the description of the same configuration as in other embodiments may be omitted. Furthermore, each embodiment can be modified and combined as appropriate.

[0012] The basic configuration and operation of the imaging device of the present invention will be described with reference to FIGS. 1A to 3. FIG.

[0013] 1A is a schematic diagram showing the configuration of an embodiment of an imaging and display device of the present invention. In FIG. 1A, the imaging and display device 10 includes an imaging unit 11 that captures imaging information 2, a control unit 12, a line-of-sight detection unit (switch) 13 that detects line-of-sight information 1, and a display unit 14 that displays display information 4.

[0014] The imaging unit 11 has a plurality of light-receiving elements (not shown). The light-emitting elements are, for example, light-receiving conversion elements that convert incident light (imaging information 2) from the outside into an electrical signal and perform an imaging operation to acquire the imaging information 2. The control unit 12 generates image information 3a, 3b of a plurality of resolutions based on the imaging information 2 from the imaging unit 11. For example, image information of high resolution and low resolution can be generated. Of course, image information of three or more resolutions, such as high resolution, medium resolution, and low resolution, may also be generated.

[0015] The line-of-sight detection unit 13 includes an infrared emitter and an infrared receiver. Here, the infrared emitter is a near-infrared emitter (NIR emitter) and the infrared receiver is a near-infrared receiver (NIR receiver) in the following description. In this embodiment, image information 3a', 3b' to be displayed on the display unit 14 is selected according to line-of-sight information 1. Therefore, the display information 4 displayed on the display unit 14 includes a mixture of image information 3a' and 3b'. For example, if the control unit 12 generates high-resolution and low-resolution image information, the pixel area that detects line-of-sight information 1, i.e., near-infrared rays, can select high-resolution image information, and the pixel area that does not detect line-of-sight information 1, i.e., near-infrared rays, can select low-resolution image information. Retinal retroreflection can be used as one method of line-of-sight detection. Retinal retroreflection is characterized by the phenomenon in which light incident perpendicularly to the retina of the eye is selectively reflected and returned. In other words, light emitted from the NIR emitter is efficiently reflected and efficiently detected by the NIR receiver only in the pixel area along the line of sight (the direction the pupil is facing). The reflection efficiency of near-infrared rays is significantly reduced in pixel areas outside the line of sight. Therefore, by setting the intensity of near-infrared rays associated with retinal retroreflection at a certain threshold, it is possible to select whether to turn on or off the line of sight information 1.

[0016] The display unit 14 has a plurality of light-emitting elements. The plurality of light-emitting elements convert electrical signals into light. The display unit 14 displays display information 4 (image) according to the information selected by the line-of-sight detection unit 13. The imaging unit 11 and the display unit 14 can also be said to be an array of a plurality of pixels. Each pixel of the imaging unit 11 has at least one light-receiving element, and each pixel of the display unit 14 has at least one light-emitting element.

[0017] The configuration of a conventional imaging and display device will now be described with reference to Fig. 1B. Fig. 1B is a schematic diagram showing the configuration of an example of a conventional imaging and display device. The imaging and display device 20 includes an imaging unit 21 that captures imaging information 2, a control unit 22, a line-of-sight detection unit 23 that detects line-of-sight information 1, and a display unit 24 that displays display information 4.

[0018] The control unit 22 generates new corrected image information 3' based on the imaging information 2 from the imaging unit 21 and the line-of-sight information 1 obtained from the line-of-sight detection unit 23. For example, the control unit 22 creates new corrected image information 3' by setting a high resolution for the pixel region toward which the line of sight is directed and a low resolution for the pixel region toward which the line of sight is not directed. The display unit 24 displays display information 4 based on the corrected image information 3' created by the control unit 22. Therefore, in a conventional imaging and display device, although the display information displayed is the same as the display information of the present invention, a time difference occurs between the imaging information 2 and the display information 4 due to the long correction processing time taken by the control unit 22.

[0019] Next, the operation of the imaging display device of the present invention and the conventional imaging display device will be described with reference to Figures 2A and 2B. Figure 2A is a diagram for explaining the operation of the imaging display device of the present invention shown in Figure 1A, and Figure 2B is a diagram for explaining the operation of the conventional imaging display device shown in Figure 1B, and is a diagram for explaining the relationship between imaging information, image information or corrected image information, and a displayed image for one frame at a certain time. In Figures 2A and 2B, at time T ｎ Image information 2 in A ｎ , Gaze information 1 is B ｎ The control unit processes the corrected image information 3'. ｎ +B ｎ , the display image is A' ｎ Let's say.

[0020] As shown in FIG. 2A, in the present invention, the image capturing unit 11 captures an image at time T －２ Imaging information A －２ At time T －１ Imaging information Ａ－１ At time T ０ Imaging information A ０ At time T ＋１ Imaging information A ＋１ Therefore, at time T －２ Image information A obtained －２ is the time T －２ Display information A' －２ In other words, the delay between obtaining the imaging information 2 and displaying the display information 4 can be almost eliminated.

[0021] On the other hand, as shown in FIG. 2B, in the conventional imaging display device, the control unit 22 －２ , A －１ , A ０ , A ＋１ and gaze information B －２ , B －１ , B ０ , B ＋１ Based on this, corrected image information A －２ +B －２ , A －１ +B －１ , A ０ +B ０ , A １ +B １ Then, the control unit 22 generates the corrected image information A －２ +B －２ , A －１ +B －１ , A ０ +B ０ , A １ +B １ , is output to the display unit 24. The display unit 24 outputs the time T －１ Corrected image information A －２ +B －２ The image based on the time T ０ Corrected image information A －１ +B －１ At time T1, the image based on the corrected image information A ０ +B ０ The image based on the time T ２ Corrected image information A １ +B １ Therefore, the display operation is performed to display an image based on the time T －１ -T －２ , T ０ -T －１ , T １ -T ０ , T ２ -T １ There is a delay between obtaining the image information 2 and displaying the display information 4, which is equal to the difference between the actual event and the displayed image. If there is a difference between the actual event and the displayed image, for example, it will be impossible to grasp a moving object.

[0022] The control unit 12 of the imaging and display device 10 of the present invention may also perform additional image processing. Examples of additional image processing include dark-field image processing for increasing the brightness of dark images, enlarged image processing for enlarging and displaying small subjects, and temperature display processing for displaying temperature images. In this case, a delay will occur due to the time required for the additional image processing, but in the present invention, the timing for displaying display information is preferably within the following range.

[0023] First, the image capturing unit 11 captures an image at an arbitrary time T ｎ Assume that an image is captured at time T ｍ Here, the difference ΔT between the timing of capturing an image and the timing of displaying the image can be expressed by the following formula (1): ΔT=T ｎ -T ｍ (1)

[0024] Here, the display frame rate DFR (fps; frame per second) is the number of images displayed by the display unit per second. The imaging and display device 10 is controlled so that this difference ΔT satisfies the following formula (2). More preferably, the imaging and display device 10 is controlled so that the difference ΔT satisfies the following formula (3). -2 / DFR≦ΔT≦2 / DFR (2) -1 / DFR≦ΔT≦1 / DFR (3)

[0025] For example, when the display frame rate is 240 [fps], the time from capturing an image to displaying it (one frame) is approximately 4×10 －３ [sec]. Therefore, the difference ΔT is: −4×10 －３ ≦ΔT≦4×10 －３ (4)

[0026] By displaying the display information at such timing, it is possible to display a moving image with little delay between the actual image and the displayed image, which can also be called a real-time display.

[0027] Furthermore, when the amount of additional image processing becomes large, delays can be reduced by generating a predicted image at a certain time, for example, by sensing past image information or other environmental information.

[0028] Although the present invention can be applied to still images, it is more effective to apply it to moving images.

[0029] According to the image capturing and displaying apparatus of the present invention, real-time display is possible even when the time required for the additional image processing described above is added.

[0030] Next, the display image of the imaging display device of the present invention will be described with reference to Fig. 3. Fig. 3 shows an image of imaging information at T = 0, an image displayed by the imaging display device of the present invention, and an image displayed by the conventional imaging display device shown in Fig. 1B. Each shows the display information at T = 0, with image 30 showing the imaging information, image 40 showing the display information of the conventional imaging display device, and image 50 showing the display information of the imaging display device of the present invention. Image 30 can also be said to be imaging information acquired by the imaging unit 11 at a certain time.

[0031] A conventional imaging and display device will be described. First, a control unit recognizes the presence of a line-of-sight area 44 around a moving object 43 through imaging information captured before T=0 and line-of-sight detection, and generates corrected image information such that the resolution of the line-of-sight area 44 including the moving object 43 is increased and the resolution outside of that area is reduced. The time at which this corrected image information is displayed is the image at T=0 in the middle of Figure 3. Compared to the image at T=0, the position at which the moving object 43 is displayed is different. In other words, it can be seen that a delay occurs.

[0032] In the imaging and display device of the present invention, high-resolution and low-resolution image information is generated in the control unit using imaging information, and a switching circuit utilizing retinal retroreflection due to near-infrared rays selects between these in real time to display information. In other words, there is little delay, and an image similar to the imaging information can be displayed at T=0.

[0033] Furthermore, by changing the processing depending on the part, such as increasing the resolution of only the line-of-sight area and decreasing the resolution of other areas, the load on the control unit can be reduced. Also, by displaying moving objects at high resolution and still objects at low resolution, it is possible to provide a natural image that is closer to what the human eye sees. This type of processing makes it possible to reduce the load on the control unit while still providing real-time display.

[0034] Next, a specific structure of the imaging display device of the present invention will be described. First, the light receiving element of the imaging unit can include, for example, a photodiode or photogate, and a photoelectric conversion film. Examples of materials for the photodiode and photogate include silicon, germanium, indium, gallium, and arsenic. Examples of the photodiode form include a PN junction photodiode, a PIN photodiode, and an avalanche photodiode.

[0035] The imaging unit may be, for example, a CMOS image sensor, which may be either a front-illuminated or back-illuminated type. The CMOS image sensor may have a stacked structure in which a semiconductor substrate on which a photodiode is arranged and a semiconductor substrate on which a scanning circuit and a control circuit are arranged are stacked.

[0036] The photoelectric conversion film may be made of organic or inorganic materials. The organic photoelectric conversion film may have a structure including at least one organic layer that performs photoelectric conversion between a pair of electrodes. The organic photoelectric conversion film may have a structure in which multiple organic layers are stacked between a pair of electrodes. The organic layer may be made of a single material or a mixture of multiple materials. The organic layer may be formed using, for example, a vacuum deposition process or a coating process. Examples of inorganic photoelectric conversion films include quantum dot-type films that use a quantum dot thin film layer containing fine semiconductor crystals instead of an organic layer, and perovskite-type films that have a photoelectric conversion layer made of a transition metal oxide with a perovskite structure.

[0037] The line-of-sight detection unit has an infrared light emitting unit and an infrared light receiving unit. Specifically, it may be an NIR light emitting unit and an NIR light receiving unit, with the NIR light emitting unit having multiple NIR light emitting elements and the NIR light receiving unit having multiple NIR light receiving elements. The near-infrared region refers to a wavelength region from 780 nm to 2500 nm. The NIR light receiving elements are, for example, photodiodes, photogates, and photoelectric conversion films as described above, which have absorption sensitivity in the near-infrared region. The NIR light emitting elements are, for example, liquid crystal devices (LCDs), inorganic light emitting diodes (inorganic LEDs), organic light emitting diodes (OLEDs), inorganic electroluminescence (inorganic ELs), quantum dot diodes (quantum LEDs), etc., which emit light in the near-infrared region.

[0038] The display unit has multiple light-emitting elements. Examples of light-emitting elements include LCDs, inorganic LEDs, OLEDs, inorganic EL devices, and quantum LEDs. Materials used in inorganic LEDs include aluminum, gallium, arsenic, phosphorus, indium, nitrogen, selenium, zinc, diamond, zinc oxide, and perovskite semiconductors. By using these materials to form a pn junction structure, light is emitted with an energy (wavelength) corresponding to the bandgap difference between the materials. OLEDs, for example, have a light-emitting layer containing at least one organic light-emitting material between a pair of electrodes. They may also have multiple light-emitting layers, or may have a structure in which multiple organic layers are stacked. The light-emitting layer may be made of a single material or a mixture of multiple materials. The light from the light-emitting layer may be fluorescent or phosphorescent, and may emit monochromatic light (blue, green, red, etc.) or white light. The organic layer may be formed using, for example, a vacuum deposition process or a coating process.

[0039] In the present invention, the light-emitting elements constituting the line-of-sight detection unit and the display unit are preferably OLEDs. This is because they can be fabricated on the same substrate by laminating and separately painting them. Furthermore, the light-receiving elements constituting the line-of-sight detection unit are preferably organic photoelectric conversion films. This is because they can be fabricated on the same substrate by laminating and separately painting them, and because they have a higher absorption coefficient in the near-infrared region than photodiodes and the like that use silicon, resulting in higher near-infrared light-receiving sensitivity.

[0040] Furthermore, the image capturing and displaying device may have a structure in which at least three chips, an image capturing section, a control section, and a display section, are stacked and electrically connected to each other by a semiconductor process.

[0041] When the imaging and display device of the present invention is used as a wearable device, it is preferable to reduce the amount of data processed by the processing unit. This is because wearable devices need to be as light and thin as possible, and the smaller the data processing load, the smaller the processing chip can be. One method for reducing the data processing load is to perform AI processing on a separate device (such as the cloud). Other methods for reducing the processing volume include lowering the resolution of areas outside the line of sight, turning areas outside the line of sight into still images, and processing areas outside the line of sight in monochrome rather than color.

[0042] Next, the structure of the image capturing and displaying device of the present invention will be described with reference to FIGS. 4A to 9C.

[0043] [First Embodiment] Figure 4A is a schematic diagram showing the configuration of a first embodiment of an imaging display device of the present invention, in which a display unit 60 and an NIR light-emitting / receiving unit 66 including both an NIR light-emitting unit and an NIR light-receiving unit are vertically stacked. A feature of this embodiment is that the number of pixels in the NIR light-emitting / receiving unit 66 is smaller than the number of pixels in the display unit 60. For example, the NIR light-emitting / receiving unit 66 can be divided into nine regions, and the corresponding high-resolution area can also be divided into nine regions. The small number of pixels in the NIR light-emitting / receiving unit 66 allows for a larger aperture ratio of the NIR light-emitting elements and NIR light-receiving elements, thereby improving the sensitivity to near-infrared light emission and reception.

[0044] In this embodiment, NIR light 62 from the NIR light-emitting element of the NIR light-emitting / receiving unit 66 passes through the display unit 60 and is reflected by the retina of the pupil 61. The NIR reflected light 63 then reaches the NIR light-receiving element of the NIR light-emitting / receiving unit 66 again, turning on the switch of the pixel that it reached. When the switch is turned on, the pixel of the display unit 60 corresponding to the on-state pixel is connected to form the high-resolution region 64. Because the mechanism is such that only light perpendicular to the pupil 61 is sensitively reflected, even if all of the NIR light-emitting elements of the NIR light-emitting / receiving unit 66 are emitting light, only the NIR light-receiving element in the direction the pupil 61 is facing is sensitively reflected light, making it possible to select the on-state pixel. The pixel of the display unit 60 corresponding to the pixel of the NIR light-emitting / receiving unit 66 that is off becomes the low-resolution region 65. The NIR light-emitting / receiving unit 66 and the display unit 60 can be connected using a known circuit. Furthermore, the NIR emission 62 must be transmitted through the display unit 60. For example, methods for achieving this include, but are not limited to, providing an area without wiring layer metal within or between pixels of the display unit 60, using a wiring layer metal with a composition that reduces the absorption rate of near-infrared light, or using a near-infrared emission wavelength that reduces the absorption rate of the wiring layer metal. There are no limitations on the layout within one pixel of the NIR emission / reception unit 66, and the NIR emission / reception unit 66 may be composed of multiple NIR light-emitting elements or multiple NIR light-receiving elements, which may be on the same plane or may be stacked.

[0045] 4B is a schematic diagram showing the configuration of a second embodiment of the image capturing and displaying device of the present invention, in which the display unit 60 and the NIR light emitting and receiving unit 66 are vertically stacked as in the first embodiment, but a feature of this embodiment is that the number of pixels of the display unit 60 is the same as the number of pixels of the NIR light emitting and receiving unit 66. This allows the high resolution region 64 and the low resolution region 65 to be selected with better resolution.

[0046] Third Embodiment FIG. 5A is a schematic diagram showing the configuration of a third embodiment of an imaging display device of the present invention, and FIG. 5B is a schematic plan view of a pixel 70 of a display unit 60. This embodiment is the same as the first embodiment, except that the display unit 60 and the NIR light receiving unit 68 are vertically stacked, and the NIR light emitting unit is disposed within the pixel of the display unit 60. The pixel 70 of the display unit 60 may be composed of, for example, an R (red) light emitting pixel, a G (green) light emitting pixel, a B (blue) light emitting pixel, and an NIR light emitting pixel 67a, but the pixel arrangement, the emitted color of each pixel, the pixel area, the aperture ratio, etc. are not particularly limited. In this embodiment, the light receiving area of ​​the NIR light receiving unit 68 can be maximized, and the NIR emitted light 62 only needs to pass through the display unit 60 once, thereby increasing the intensity of the near-infrared light received by the NIR light receiving element and improving switching accuracy.

[0047] Fourth Embodiment FIG. 6A is a schematic diagram showing the configuration of a fourth embodiment of the imaging display device of the present invention, and FIG. 6B is a schematic plan view of a pixel 70 of a display unit 60. This embodiment is the same as the first embodiment, except that the display unit 60 and the NIR light-emitting unit 67 are vertically stacked, and the NIR light-receiving unit is disposed within the pixel of the display unit 60. The pixel 70 of the display unit 60 may be composed of, for example, an R-emitting pixel, a G-emitting pixel, a B-emitting pixel, and an NIR light-receiving pixel 68a, but the pixel arrangement, the emitted color of each pixel, the pixel area, the aperture ratio, and the like are not particularly limited. A structure in which RGB light-emitting pixels and a near-infrared light-receiving region are stacked may also be used. A feature of this embodiment is that the light-emitting area of ​​the NIR light-emitting unit 67 can be maximized, and the NIR reflected light 63 only needs to pass through the display unit 60 once, thereby increasing the intensity of the near-infrared light received by the NIR light-receiving element and improving switching accuracy.

[0048] 7A is a schematic diagram showing the configuration of a fifth embodiment of the image capture display device of the present invention, which is the same as the first embodiment except that the display unit 60 and the NIR light emitting / receiving units 66 are arranged on the same plane. A feature of this embodiment is that by all being on the same plane, absorption loss due to wiring layer metal, etc. in the display unit 60 can be suppressed, thereby improving switching sensitivity by near-infrared rays. Furthermore, in this embodiment, for example, by providing the NIR light emitting / receiving units 66 in nine locations within the display unit 60, the high-resolution region 64 can also be divided into nine regions corresponding to the nine NIR light emitting / receiving units 66.

[0049] 7B is a schematic diagram showing the configuration of a sixth embodiment of the image capture and display device of the present invention, which is the same as the fifth embodiment except that the NIR light emitting / receiving units 66 are arranged in the outer peripheral region of the display unit 60. A feature of this embodiment is that the NIR light emitting / receiving units 66 are arranged in the outer peripheral region of the display unit 60, thereby maximizing the number of pixels of the display unit 60. In this case, no NIR light emitting / receiving units 66 are present in the central region of the display unit 60, but, for example, if all of the NIR light emitting / receiving units 66 on the periphery are determined to be switched off, the central region can be selected as the high-resolution region 64.

[0050] Seventh Embodiment FIG. 8A is a schematic diagram showing the configuration of a seventh embodiment of the imaging display device of the present invention, and FIGS. 8B and 8C are schematic plan views of pixels 70 of a display unit 60. This embodiment is the same as the first embodiment except that the NIR light-emitting / receiving unit 66 is on the same plane as the display unit 60 and has the same number of pixels. A feature of this embodiment is that the high-resolution region 64 and the low-resolution region 65 can be selected with higher resolution. Note that, for example, as shown in FIG. 8B , the pixel layout of the display unit 60 and the NIR light-emitting / receiving unit 66 may be such that the pixel 70 is divided into four, RGB light-emitting pixels and near-infrared pixels, with NIR light-emitting pixels 67 a and NIR light-receiving pixels 68 a arranged in the near-infrared pixels. Alternatively, as shown in FIG. 8C , the pixel 70 may be divided into four, RGB light-emitting pixels and NIR light-emitting pixels 67 a, with the NIR light-receiving pixels 68 a arranged around these pixels.

[0051] Eighth Embodiment Fig. 9A is a schematic diagram showing the configuration of an eighth embodiment of an image capture and display device of the present invention, and Figs. 9B and 9C are schematic plan views of pixels 70 of a display unit 60. This embodiment is the same as the seventh embodiment except that it has a medium-resolution region 69. In this embodiment, for example, multiple thresholds related to the sensitivity of the NIR light receiving element can be set to select multiple resolutions. This smooths the boundaries between different resolutions, making it possible to provide higher-definition display information.

[0052] 10 is a schematic diagram showing the configuration of a ninth embodiment of the image capturing and displaying device of the present invention, which has the same configuration as that shown in FIG. 1A except that it has a feature of having a plurality of image capturing units. In this embodiment, the image capturing unit 11a acquires high-resolution image capturing information 2a, and the image capturing unit 11b acquires low-resolution image capturing information 2b, which can be selected by the line-of-sight detection unit 13 without going through the control unit 12. This further reduces the delay from capturing to display.

[0053] Although the present invention has been described with reference to the above embodiments, it is not limited thereto. The imaging and display device of the present invention can reduce the time difference between a real event and a displayed image, allowing the user to use the device without feeling uncomfortable.

[0054] 11A to 11E, an example of application of the imaging and display device of the present invention to a wearable device will be described. The imaging and display device can be applied to wearable devices such as smart glasses, HMDs (Head Mounted Displays), and smart contact lenses.

[0055] FIG. 11A is a schematic diagram illustrating smart glasses. Smart glasses are also referred to as eyeglass-type imaging and display devices or glasses. The smart glasses include an eyeglass frame and the imaging and display device of the present invention. Specifically, the smart glasses include at least an imaging unit 101, a processing unit 102, and a display unit 103. The imaging unit 101 is provided on two sides of the eyeglass frame, and the processing unit 102 is housed in the temples of the eyeglasses. The display unit 103 is provided in any position depending on the display format, but may be included in the lens 111. In either case, the display unit 103 displays an image on the lens 111. The processing unit 102 may include an AI unit. The smart glasses may have an external interface, and the processing unit 102 may communicate with an external AI unit.

[0056] 11A may have two image capturing and display devices, one for the left eye and one for the right eye. In this case, the timing of image capturing and display can be set arbitrarily in the image capturing and display devices for the left eye and the right eye. Specifically, the operation may involve capturing images at the same time and displaying them at different times, or capturing images at different times and displaying them at the same time.

[0057] FIG. 11B is a schematic diagram illustrating a smart contact lens 120. The smart contact lens 120 is also referred to as a contact lens-type imaging and display device or a contact lens. One smart contact lens 120 includes one imaging and display device 121 and one control device 122. The control device 122 functions as a power supply unit that supplies power to the imaging and display device 121, and also includes an AI unit that assists the processing unit of the imaging and display device 121. The AI ​​unit may be provided in a terminal separate from the smart contact lens 120. The smart contact lens 120 is preferably provided with an optical system for focusing light on the imaging and display device 121. The power supply unit has an interface for connecting to the outside. The power supply unit may be connected to the outside (charging) via a wired or wireless connection.

[0058] The base material of the lens 111 in FIG. 11A and the smart contact lens 120 in FIG. 11B is transparent, and the display unit of the imaging display device projects a display image onto the transparent lens portion.

[0059] 11A and 11B, the imaging unit 101 and the display unit 103 may be provided at different positions, or the imaging unit 101 and the display unit 103 may be provided so as to be stacked in the line of sight. FIG. 11C shows a cross-sectional schematic diagram of the imaging unit 101 and the display unit 103. FIG. 11D shows a plan view schematic diagram of the imaging unit 101 and the display unit 103 as viewed from the imaging unit 101 side. FIG. 11E shows a plan view schematic diagram of the imaging unit 101 and the display unit 103 as viewed from the display unit 103 side. FIG. 11D shows a center of gravity 132 of an imaging region 131 in which the pixels of the imaging unit 101 are arranged. FIG. 11E shows a center of gravity 134 of a display region 133 in which the pixels of the display unit 103 are arranged. As shown in FIG. 11C, the imaging unit 101 and the display unit 103 may be provided so that a line segment A passes through the two centers of gravity. This is because it is possible to reduce discrepancies that arise due to differences in position between captured image information and the displayed image in a wearable device.

[0060] The present invention is not limited to the above-described embodiments, and various modifications and variations can be made without departing from the spirit and scope of the present invention. Therefore, the following claims are appended to apprise the public of the scope of the present invention.

[0061] This application claims priority based on Japanese Patent Application No. 2024-114850, filed July 18, 2024, the entire contents of which are incorporated herein by reference.

[0062] REFERENCE SIGNS LIST 1 Line of sight information 2, 2a, 2b Imaging information 3, 3a, 3b, 3a', 3b' Image information 4 Display information 10 Imaging and display device 11 Imaging section 12 Control section 13 Line of sight detection section 14 Display section

Claims

1. An imaging and display device comprising an imaging unit having a plurality of light receiving elements, a display unit having a plurality of light emitting elements, an infrared emitting unit having an infrared emitting element, an infrared receiving unit having an infrared receiving element, and a control unit that processes imaging information captured by the imaging unit, wherein the control unit creates a plurality of image information from the imaging information from the imaging unit, and selects one of the plurality of image information in accordance with line of sight information obtained by detecting the reflected light from the infrared emitting unit reflected by the eyeball.

2. The imaging and display device according to claim 1, wherein the plurality of pieces of image information are pieces of image information with different resolutions.

3. The imaging and display device according to claim 2, wherein the plurality of pieces of image information are three types of image information with different resolutions.

4. An imaging display device as claimed in any one of claims 1 to 3, characterized in that the infrared light emitting unit and the infrared light receiving unit are on the same plane, the display unit, the infrared light emitting unit and the infrared light receiving unit are stacked, and the number of pixels of the infrared light emitting unit and the infrared light receiving unit is less than the number of pixels of the display unit.

5. An imaging display device as claimed in any one of claims 1 to 4, characterized in that the infrared light emitting unit and the infrared light receiving unit are on the same plane, the display unit, the infrared light emitting unit and the infrared light receiving unit are stacked, and the number of pixels of the display unit is equal to the number of pixels of the infrared light emitting unit and the infrared light receiving unit.

6. An imaging display device according to any one of claims 1 to 5, characterized in that the display unit and the infrared light emitting unit are on the same plane, and the infrared light receiving unit, the display unit and the infrared light emitting unit are stacked.

7. An imaging display device according to any one of claims 1 to 6, characterized in that the display unit and the infrared receiving unit are on the same plane, and the infrared emitting unit, the display unit and the infrared receiving unit are stacked.

8. An imaging display device according to any one of claims 1 to 7, characterized in that the display unit, the infrared light emitting unit, and the infrared light receiving unit are on the same plane, and the number of pixels of the infrared light emitting unit and the infrared light receiving unit is smaller than the number of pixels of the display unit.

9. An imaging display device according to any one of claims 1 to 8, characterized in that the display unit, the infrared light emitting unit and the infrared light receiving unit are on the same plane, and the infrared light emitting unit and the infrared light receiving unit are arranged around the display unit.

10. An imaging display device as claimed in any one of claims 1 to 9, characterized in that the display unit, the infrared light emitting unit and the infrared light receiving unit are on the same plane, and the number of pixels of the display unit is equal to the number of pixels of the infrared light emitting unit and the infrared light receiving unit.

11. The imaging and display device according to any one of claims 1 to 10, characterized in that a plurality of the imaging units are arranged.

12. An image pickup display device according to any one of claims 1 to 11, characterized in that the display section has organic light emitting diode elements or inorganic electroluminescence elements as light emitting elements.

13. The image pickup display device according to any one of claims 1 to 12, characterized in that the infrared light emitting section has an organic light emitting diode element as a light emitting element.

14. An imaging display device according to any one of claims 1 to 13, characterized in that the infrared receiving section has an organic photoelectric conversion film.

15. An image capturing and displaying device according to any one of claims 1 to 14, characterized in that the image capturing section has a back-illuminated CMOS image sensor.

16. An imaging display device as described in any one of claims 1 to 15, characterized in that the imaging unit is formed by stacking a substrate on which the plurality of light-receiving elements are arranged and a substrate on which a circuit for processing signals from the plurality of light-receiving elements is arranged.

17. A wearable device comprising: an imaging and display device according to any one of claims 1 to 16; and a power supply unit that supplies power to the imaging and display device.

18. The wearable device according to claim 17, wherein the power supply unit is provided with an interface for wirelessly connecting to the outside.

Images