Display device and display method

Abstract

The present application provides a display device that enables further improvement in performance of controlling video presentation in accordance with user eye characteristics. A display device is provided that includes a light source; a processor that processes a characteristic distribution of an eye; a monitoring section that monitors a state of the eye; a matching section that matches the characteristic distribution of the eye with the state of the eye; and an illuminator that irradiates video display light emitted from the light source to a specified position on a retina.

Description

Technical Field

[0001] This technology relates to display devices and display methods. Background Technology

[0002] In recent years, technologies that display images by overlaying them onto external scenes, such as real landscapes, have attracted attention. This technology is also known as augmented reality (AR). Head-mounted displays are an example of products that use this technology. Head-mounted displays are used by wearing them on a user's head. In methods for displaying video (images) using head-mounted displays, light from the head-mounted display also reaches the user's eyes, in addition to light from the outside world, allowing the user to perceive it as if a video composed of light from the display is overlaid on an external image.

[0003] For example, Patent Document 1 discloses an image display device that can detect the position of light illuminating the retina from a light source and can adjust the position of light illuminating the retina.

[0004] Citation List

[0005] Patent documents

[0006] Patent Document 1: WO2012 / 169064 Summary of the Invention

[0007] The problem the invention aims to solve

[0008] However, the technology proposed in Patent Document 1 may not be able to further improve the performance of controlling video presentation based on the user's eye characteristics.

[0009] Therefore, this technology was developed in view of the above circumstances, and the main objective of this technology is to provide a display device and display method that can further improve the performance of controlling video presentation based on the user's eye characteristics.

[0010] means for solving problems

[0011] To achieve the above objectives, the inventors conducted in-depth research. Therefore, the inventors achieved surprising success in further improving the performance of controlling video presentation based on user eye characteristics, and thus completed this technology.

[0012] In other words, as a first aspect, this technology provides a display device, comprising:

[0013] light source;

[0014] A processor that processes the feature distribution of the eyeball;

[0015] A monitoring unit that monitors the state of the eyeball;

[0016] A matching unit that matches the feature distribution of the eyeball with the state of the eyeball; and

[0017] An illuminator that illuminates a designated location on the retina with video display light emitted by the light source.

[0018] The display device according to the first aspect of the present technology may further include

[0019] The acquisition unit acquires the feature distribution of the eyeball.

[0020] In this case, the acquisition unit may include at least one selected from the group consisting of a fundus camera, an OCT, a refractometer, and a light detection device that detects the light returned due to IR scanning.

[0021] In the display device according to the first aspect of this technology

[0022] The monitoring unit can use corneal reflection or fundus reflection to monitor the state of the eyeball.

[0023] The display device according to the first aspect of the present technology may further include

[0024] A follower that enables the video display light to follow the movement of the eyeball.

[0025] In this case, the follower may include at least one selected from the group consisting of a combiner, a relay system driver, a mirror driver, and a phase difference panel.

[0026] In the display device according to the first aspect of this technology

[0027] A coordinate system can be defined based on the feature distribution of the eyeball, and

[0028] In this case, the matching unit can use the coordinate system to match the feature distribution of the eyeball with the state of the eyeball to which the video is to be presented.

[0029] The coordinate system may be defined based on at least two selected from the group consisting of the first fovea of ​​the right eye, the first blind spot of the right eye, the second fovea of ​​the left eye, and the second blind spot of the left eye.

[0030] In the display device according to the first aspect of this technology

[0031] The light source can be a laser source.

[0032] The display device according to the first aspect of the present technology may further include

[0033] Scanning mirror, and

[0034] The scanning mirror can project the video display light onto the retina.

[0035] In the display device according to the first aspect of this technology

[0036] The irradiator also includes a component disposed in front of the eye, and

[0037] The component is a transparent component.

[0038] In this case, the component may include a reflective or transmissive volume hologram, a reflective or transmissive relief hologram, or a first optical element on a metasurface, or a second optical element that reflects a specified light and transmits light other than the specified light.

[0039] The display device according to the first aspect of the present technology may further include a component for compensating for wavelength dispersion.

[0040] In this case, the component used to compensate for wavelength dispersion may be a first optical element including a reflective or transmissive volume hologram, a reflective or transmissive relief hologram, or a metasurface.

[0041] Furthermore, as a second aspect, this technology provides a display method, including...

[0042] Processing the characteristic distribution of the eyeball;

[0043] The state of the eyeball is monitored;

[0044] Matching the feature distribution of the eyeball with the state of the eyeball; and

[0045] A video display is created by illuminating a designated area on the retina with light emitted from a light source.

[0046] The display method according to the second aspect of this technology may further include:

[0047] Obtain the feature distribution of the eyeball.

[0048] This technology can further improve the performance of controlling video presentation based on the user's eye characteristics. Note that the effects described herein are not necessarily limiting and may provide any of the effects described in this disclosure. Attached Figure Description

[0049] [ Figure 1 ] Figure 1 This is a diagram illustrating an example of a process for displaying video using a display device that employs this technology.

[0050] [ Figure 2 ] Figure 2 It is a diagram used to describe the changes in the position of the fovea and blind spot when the right eyeball rotates to the left or right.

[0051] [ Figure 3 ] Figure 3 It is a graph used to describe the result of matching a fundus image with the state of the eyeballs to be presented in the video.

[0052] [ Figure 4 ] Figure 4 It is a graph used to describe the result of matching the interpolated fundus image with the state of the eyeballs to be presented in the video.

[0053] [ Figure 5 ] Figure 5 This is another example of a process for displaying video using a display device that applies this technology.

[0054] [ Figure 6 ] Figure 6 It is a diagram used to describe the state of things that can be seen normally, the state of things that are distorted (transphobia), and the state of things that are dark in the center (central scotoma).

[0055] [ Figure 7 ] Figure 7 It is a diagram used to describe the correction of eyeball deformities.

[0056] [ Figure 8 ] Figure 8 It is a diagram used to describe the correction of eyeball markings.

[0057] [ Figure 9 ] Figure 9 This is a top view showing an example configuration of a display device to which this technology is applied.

[0058] [ Figure 10 ] Figure 10 This is a front view showing an example configuration of a display device to which this technology is applied.

[0059] [ Figure 11 ] Figure 11 This is a block diagram illustrating a configuration example of a display device applying the first embodiment of the present technology.

[0060] [ Figure 12 ] Figure 12 This is a block diagram illustrating a configuration example of a display device applying the second embodiment of the present technology.

[0061] [ Figure 13 ] Figure 13 This is a diagram showing an example of a component arranged in front of the eyes and included in a display device to which this technology is applied.

[0062] [ Figure 14 ] Figure 14 This is a diagram showing another example of a component arranged in front of the eyes and included in a display device to which this technology is applied. Detailed Implementation

[0063] Preferred embodiments for implementing this technology will now be described. The embodiments described below are examples of representative embodiments of this technology, and the scope of this technology is not to be construed as limited to these embodiments. Note that in the description with reference to the accompanying drawings, similar or equivalent elements or components are indicated by similar reference numerals to omit repetitive descriptions.

[0064] Please note that the order of explanation is as follows.

[0065] 1. Overview of this technology

[0066] 2. First Embodiment (First Example of a Display Device)

[0067] 3. Second Embodiment (Second Example of a Display Device)

[0068] 4. Third Implementation Method (First Example of the Method)

[0069] 5. Fourth Implementation (Second Example of the Method)

[0070] <1. Overview of this technology>

[0071] First, an overview of this technology will be provided. This technology relates to display devices and display methods.

[0072] According to this technique, the feature distribution (shape and optical features) of the eyeball is acquired, and the coordinates defined on the distribution are matched with the eye sensing results. This allows video to be displayed at a specified location on the retina based on the feature distribution of the eyeball.

[0073] First, examples of other technologies besides this one will be described.

[0074] As a first example of other technologies, there are examples related to extended glasses with flat panel display panels. In this first example of technology, the focal plane is fixed. Therefore, the stimulated area on the retina may not be precisely identified. Furthermore, as a second example of other technologies, there are examples related to using lasers to project images directly onto the retina. In this second example of technology, when using a semi-reflective mirror, perspective performance is reduced, which may result in objects that are not suitable for drawing attention to. Additionally, when using holographic optics (HOEs), the field of view becomes narrower and wavelength dispersion occurs, which may lead to difficulties in the range and accuracy of stimulus presentation. Furthermore, as a third example of other technologies, there are examples related to position adjustment (following). In this third example of technology, position adjustment can be performed using pattern matching (here, the pattern refers to a retinal pattern such as blood). Therefore, the reference data is large, which may lead to cumbersome processing.

[0075] This technology was developed in view of the above circumstances.

[0076] This technology can further improve the performance of controlling video presentation based on the user's eye characteristics. Furthermore, this technology can further improve the performance of controlling video presentation based on the user's eye movements (eye motion).

[0077] Specifically, in this technology, information about the feature distribution of the eyeball is acquired, a coordinate system is defined, and the displayed video and display position are defined within the coordinate system. Therefore, this reduces the amount of data compared to using pattern mapping. Furthermore, the displayed video (presented video) and display position (presentation position) are controlled based on information about eye movement (such as information about eye rotation). Therefore, this reduces processing intensity compared to using pattern matching. For example, the coordinate system is defined based on at least two selected from a group consisting of the right eye fovea (which may be referred to as the first fovea), the right eye blind spot (which may be referred to as the second blind spot), the left eye second fovea (which may be referred to as the second fovea), and the left eye second blind spot (which may be referred to as the second blind spot).

[0078] Furthermore, in this technology, it is preferable that, due to the use of a laser source, stimulation of a specific wavelength can be presented. Furthermore, it is preferable that, since the component arranged in front of the eye is a see-through component, external objects can be viewed and unconsciously stimulated. Furthermore, it is preferable that, since a component for compensating for wavelength dispersion is used, wavelength dispersion can be compensated, thus allowing precise stimulation of a designated point (any point) on the retina.

[0079] Examples of components positioned in front of the eye include a first optical element and a second optical element. The first optical element includes a reflective or transmissive volumetric hologram, a reflective or transmissive relief hologram, or a metasurface. The second optical element is an optical element that reflects a specified light and transmits light other than the specified light. The first optical element, including the metasurface, may, for example, have a structure on its surface periodically arranged with dielectrics or metals at narrow intervals, and may deflect light in a specific frequency band.

[0080] Examples of second optical elements include a semi-reflective mirror (preferably with a light transmittance of 50% or higher), a bandpass filter that reflects only a specific wavelength (the wavelength of the light source) and transmits wavelengths other than that specific wavelength, and a polarization beam splitter that reflects specific polarized light.

[0081] Examples of components for compensating wavelength dispersion include a first optical element comprising a reflective or transmissive volume hologram, a reflective or transmissive relief hologram, or a metasurface. As described above, a first optical element including a metasurface as a component for compensating wavelength dispersion also, for example, has a structure on its surface in which dielectrics or metals are periodically arranged at narrow intervals, and can deflect light of a specific frequency band.

[0082] Preferred embodiments for implementing this technology will now be described in detail with reference to the accompanying drawings. The embodiments described below are examples of representative embodiments of this technology, and the scope of this technology is not to be construed as limited to these embodiments.

[0083] <2. First Embodiment (First Example of a Display Device>

[0084] The display device according to the first embodiment of the present technology (first example of a display device) includes a light source, a processor for processing the feature distribution of the eyeball, a monitoring unit for monitoring the state of the eyeball, a matching unit for matching the feature distribution of the eyeball with the state of the eyeball, and an illuminator for illuminating a designated position on the retina with video display light emitted by the light source. The display device according to the first embodiment of the present technology may further include a follower. The display device according to the first embodiment of the present technology can be applied to, for example, eyeglass displays and head-mounted displays.

[0085] The processor processes information about the characteristic distribution of the eyeball, such as information obtained from external devices (e.g., fundus camera, OCT, light detection device that detects light reflected from IR scans, or refractometer). The monitoring unit monitors the state of the eyeball, for example, using corneal or fundus reflections, and can acquire the optical axis. The matching unit acquires the deviation between the visual axis and the optical axis (gaze calibration) and can generate a map reflecting the offset. The illuminator modulates light based on the laser source, video display (projector), and the characteristic distribution of the eyeball, adjusts the light intensity based on external light, and can perform controls including distortion correction. The follower enables the displayed video to follow eye movements caused by eye tracking, where light is manipulated to change the illuminated video and illumination position based on information about eye rotation.

[0086] First, refer to Figure 1 An example is described of a process for displaying video using a display device according to a first embodiment of the present technology.

[0087] exist Figure 1 In steps S101 to S103, fundus images are created. More specifically, in step S101, a fundus photograph is taken using, for example, a fundus camera, when the eyeball is viewed from the front; in step S102, a fundus photograph is taken using, for example, a fundus camera, when the eyeball is rotated to the left, right, up, or down; and in step S103, the positions of the fovea and blind spot in the respective fundus photographs taken in steps S101 and S102 are detected.

[0088] refer to Figure 2 Steps S101 to S103 are described in detail. Figure 2 This is a diagram used to describe the changes in the position of the fovea and blind spot when the right eyeball rotates to the left or right. Figure 2 middle, Figure 2 A-2 in the figure shows a fundus photograph of the right eye viewed from the front in step S101. Figure 2 B-2 in the diagram is a schematic top view showing the right eyeball 20 when viewed from the front. Figure 2 middle, Figure 2 A-1 in the image shows a fundus photograph of the right eye when the eyeball 20 rotates to the left in step S102, while Figure 2 B-1 in the diagram is a schematic top view of the right eyeball 20 when the eyeball rotates to the left. Figure 2 middle, Figure 2 A-3 in the image shows a fundus photograph of the right eye when the eyeball 20 rotates to the right in step S102, while Figure 2 B-3 in the diagram is a schematic top view of the right eyeball 20 when the eyeball rotates to the right.

[0089] When comparing Figure 2 When A-1 and A-2 are in the range, Figure 2 The central concave 30-1 shown in A-1 is relative to Figure 2 The central concave 30-2 shown in A-2 has moved to the left. Similarly, Figure 2 Blind spot 31-1 shown in A-1 is relative to Figure 2 The blind spot 31-2 shown in A-2 has moved to the left.

[0090] Notice, Figure 2 The positional relationship between the central fovea 30-2 and the blind spot 31-2 shown in A-2 (the distance between the central fovea 30-2 and the blind spot 31-2) and Figure 2 The positional relationship between the central fovea 30-1 and the blind spot 31-1 shown in A-1 (the distance between the central fovea 30-1 and the blind spot 31-1) is basically the same.

[0091] When comparing Figure 2 When A-3 and A-2 are in the middle, Figure 2 The central concave 30-3 shown in A-3 is relative to Figure 2 The central concave 30-2 shown in A-2 has moved to the right. Similarly, Figure 2 Blind spot 31-3 shown in A-3 is relative to Figure 2 The blind spot 31-2 shown in A-2 has moved to the right.

[0092] Notice, Figure 2 The positional relationship between the central fovea 30-2 and the blind spot 31-2 shown in A-2 (the distance between the central fovea 30-2 and the blind spot 31-2) and Figure 2 The central fovea 30-3 and blind spot 31-3 (the distance between the central fovea 30-3 and blind spot 31-4) shown in A-3 are essentially the same as each other.

[0093] exist Figure 1 In steps S104 and S105, the results of eye sensing (eye tracking) are matched with the fundus image. More specifically, in step S104, gaze calibration is performed using the display device according to the first embodiment of the present technology, and in step S105, the deviation between the optical axis center and the fovea of ​​the display system is calculated.

[0094] refer to Figure 3 Steps S104 to S105 are described in detail. Figure 3 This is a graph used to describe the result of matching a fundus image with the state of the eyeballs to be displayed in the video. For example... Figure 3As shown in Figure A, the optical axis (optical vector) 1 is offset by an angle θ from the visual axis (gaze vector) 2. Optical axis 1 is the corneal normal passing through the center of pupil 10, and visual axis 2 is the axis connecting the nodal point (the central posterior surface of the lens) and the fovea 30.

[0095] like Figure 3 As shown in B, the blind spot 31 extends into the first quadrant 51 and the fourth quadrant 54 in the first quadrant 51 (e.g., the region where the X-axis is positive and the Y-axis is positive), the second quadrant 52 (e.g., the region where the X-axis is negative and the Y-axis is positive), the third quadrant 53 (e.g., the region where the X-axis is negative and the Y-axis is negative), and the fourth quadrant 54 (e.g., the region where the X-axis is positive and the Y-axis is negative) that constitute the video display range (viewpoint) 50.

[0096] Figure 1 Steps S106 to S108 are performed in the video presentation state. More specifically, gaze tracking is performed in step S106. For example, in step S106, eye movements are tracked, pupil position is detected, and the viewing angle of the desired point is calculated. Infrared light is aligned with the optical axis used for image projection, and fundus reflection (the reflectivity at the blind spot location differs from that at the fovea location) is detected. In step S107, an interpolated fundus map is generated. The interpolated fundus map is generated by interpolating between leftward, rightward, upward, and downward rotations of the eyeball. For example, the positions of the fovea and blind spot are estimated using the fundus map when the eyeball rotates to the left, right, upward, and downward at a smaller level than the levels of leftward, rightward, upward, and downward rotations used in the fundus maps generated in steps S101 to S103, to generate the interpolated fundus map. Note that information about the position of a specific eyeball or information about viewing angle deviations can be used to generate the interpolated fundus map. Then, in step S108, a video is generated. After the video is generated, the signal of the generated video is sent to a video display unit, which includes, for example, a light source (such as a laser source), a reflector, a scanning reflector (such as a MEMS reflector), a relay system drive unit (projection optical system), and a perspective component (such as a holographic optical element (hereinafter referred to as HOE)) arranged in front of the eye. Examples of holographic optical elements include reflective or transmissive volumetric holograms and reflective or transmissive relief holograms (also known as surface relief holograms).

[0097] refer to Figure 4 Steps S106 to S108 are described in detail. Figure 4It is a graph used to describe the result (correction result) of matching the interpolated fundus image with the state of the eyeball to be presented in the video.

[0098] like Figure 4 As shown, in the first quadrant 51 (e.g., the region where the X-axis is positive and the Y-axis is positive), the second quadrant 52 (e.g., the region where the X-axis is negative and the Y-axis is positive), the third quadrant 53 (e.g., the region where the X-axis is negative and the Y-axis is negative), and the fourth quadrant 54 (e.g., the region where the X-axis is positive and the Y-axis is negative), which constitute the video display range (viewpoint) 50, the blind spot 31 exists in the first quadrant 54, which is closer to the fourth quadrant 54.

[0099] like Figure 1 As shown in reference number P1, repeat steps S106 to S108 until the desired or specified video is generated. Steps S106 (eye tracking) and S107 (generation of interpolated fundus image) can be performed simultaneously.

[0100] refer to Figure 5 Another example of a process for displaying video using a display device according to a first embodiment of the present technology is described.

[0101] exist Figure 5 In step S201, a fundus image is generated. The method for generating the fundus image in step S201 is the same as that described above. Figure 1 The fundus images generated in steps S101 to S103 are generated using a similar method. Therefore, a detailed description is omitted here.

[0102] exist Figure 5 In step S202, ocular features are acquired. These features are acquired using, for example, optical coherence tomography (OCT), an objective refractometry (or a light detection device that detects the light reflected from an IR scan). OCT captures an image of a cross-section of the retina. A refractometry obtains optical features such as aberrations (data on ocular deformation) and measures, for example, the eye's refractive power and corneal curvature. A light detection device that detects the light reflected from an IR scan measures, for example, the shape of the fundus. By acquiring these ocular features, it can be determined, for example, whether a user's eye is affected by age-related macular degeneration (AMD). AMD is a disease that causes difficulty seeing in the central macula of the retina due to age-related changes or light damage. For example, as... Figure 6 As shown, a normal eye can see things normally. Figure 6In A), the retina becomes deformed due to age-related macular degeneration (myopic vision), and things appear distorted, such as distorted 60B (…). Figure 6 (B in the original text), or due to age-related macular degeneration (myopic vision loss), vision is weakened and the central area appears darker, for example, central scotoma 60C ( Figure 6 (C in the middle).

[0103] exist Figure 5 In step S203, the eye sensing (eye tracking) results are matched with the fundus image. The matching method performed in step S203 is the same as that described above. Figure 1 The matching methods performed in S104 and S105 are similar. Therefore, a detailed description is omitted here.

[0104] Figure 5 Steps S204 to S208 are performed in the video presentation state. More specifically, gaze tracking is performed in step S204. For example, in step S204, eye movement is tracked, pupil position is detected, and the viewing angle of the desired point is calculated. Infrared light is aligned with the optical axis used for image projection, and fundus reflection (the reflectivity at the blind spot location differs from that at the fovea location) is detected. In step S205, an interpolated fundus map is generated. The interpolated fundus map is generated by interpolating between leftward, rightward, upward, and downward rotations of the eyeball. For example, the positions of the fovea and blind spot are estimated using the fundus map when rotating leftward, rightward, upward, and downward at smaller levels than the levels of leftward, rightward, upward, and downward rotations used in the fundus map generated in step S201, to generate the interpolated fundus map. Note that information about the position of a specific eyeball or information about viewing angle deviations can be used to generate the interpolated fundus map.

[0105] Next, in step S206, a correction table for correcting eyeball deformities is reconstructed. (Reference) Figure 7 Step S206 will be described in detail. Figure 7 A in the image represents the video (image) to be presented. Figure 7 B in the image shows a video (image) affected by eyeball deformation. Figure 7 C in the figure shows the output video (generated video) obtained by performing correction using a correction table for correcting eye deformation.

[0106] In step S207, the eye markings (mura) are corrected. (See reference) Figure 8 Step S207 will be described in detail. Figure 8 A in the image represents the video (image) to be presented. Figure 8 B in the image shows a video (image) affected by eye markings. Figure 8 C in the figure shows the output video (generated video) obtained by performing correction using eye pattern correction.

[0107] Then, in step S208, a video is generated. After the video is generated, the signal of the generated video is sent to a video display unit, which includes, for example, a light source (such as a laser source), a reflector, a scanning reflector (such as a MEMS reflector), a relay system drive unit (projection optical system), and a perspective component (such as a holographic optical element (hereinafter referred to as HOE)). Examples of holographic optical elements include reflective or transmissive volumetric holograms and reflective or transmissive relief holograms (also known as surface relief holograms).

[0108] like Figure 5 As shown in reference number P5, repeat steps S204 to S208 until the desired or specified video is generated. Steps S204 (eye tracking) and S205 (generation of interpolated fundus map) can be performed simultaneously.

[0109] refer to Figure 9 and Figure 10 A configuration example of a display device according to a first embodiment of the present technology is described. Figure 9 This is a top view of a display device according to this technology worn on the user's head. Figure 10 This is a front view of a display device according to this technology worn on the user's head. Figure 9 The display device shown includes a video display unit (also called an image display unit), a sensor for detecting changes in the position of the display device relative to the head (in this specification, the sensor for detecting changes in the position of the display device relative to the head is also called a "displacement sensor" or "sensor"), a monitoring unit, a projection position adjustment mechanism, a controller, and a memory. These structural components will be described below.

[0110] (Video Display Department)

[0111] like Figure 9As shown, the display device 100 has an eyeglass shape and is configured to project video display light (in some cases referred to as image display light) onto each eye. In other words, the display device 100 includes a video display unit that projects video display light onto the left eye and a video display unit that projects video display light onto the right eye. The video display unit that projects video display light onto the left eye includes a light source unit 101L, a projection optical system 102L, and a holographic optical element (hereinafter also referred to as HOE) 103L serving as an illuminator. Although not shown in the figure, the display device 100 may include a combiner as a follower, and the combiner may structurally include elements such as the holographic optical element 103L and a semi-reflective mirror. The projection optical system 102L may include a relay system drive unit (not shown) as a follower. Although not shown in the figure, a mirror drive unit and a phase difference panel as a follower may be arranged between the light source unit 101L and the projection optical system 102L, or between the projection optical system 101L and the holographic optical element 103L.

[0112] The light source unit 101L emits video display light. As a structure for emitting this video display light, the light source unit 101L may include, for example, a laser source 120L, a reflector 121L, and a scanning reflector 122L. The laser emitted from the laser source 120L is reflected by the reflector 121L and then reaches the scanning reflector 122L. The scanning reflector 122L performs a two-dimensional scan of the laser. The scanning reflector 122L may be, for example, a MEMS reflector. The scanning reflector 122L can rapidly move the direction of the laser to form an image on the retina.

[0113] The projection optics system 102L adjusts the direction of the video display light so that it reaches the desired area and / or position in the HOE 103L. For example, the projection optics system 102L shapes the video display light scanned by the scanning mirror 122L into parallel light.

[0114] The HOE 103L diffracts the video display light to focus it near the user's pupil, thereby illuminating the retina. The HOE 103L can be, for example, a reflective diffraction element. The HOE 103L can have the following optical properties: it acts as a lens for light within the wavelength range of the video display light, and allows light outside that wavelength range to pass through it. This optical property allows the user to recognize the scenery in front of them, for example, in the direction of their gaze, and to recognize the image formed by the video display light. In other words, the image formed by the video display light can be superimposed on the external scenery. Examples of the HOE 103L include holographic lenses, preferably holographic lenses comprising a film shape, and more preferably transparent holographic lenses comprising a film shape. Holographic lenses with a film shape can be used, for example, by being attached to glass. The desired optical properties can be provided to the holographic lens using methods existing in the art. Furthermore, the HOE 103L can be, for example, a volumetric hologram or a surface relief hologram. In addition, commercially available holographic lenses can be used as holographic lenses, or existing methods in this technical field can be used to manufacture holographic lenses.

[0115] As described above, the light source 101L, the projection optical system 102L, and the HOE 103L deliver the video display light to the user's left eye.

[0116] The display device 100 includes a temple portion 109L and a frame portion 108L, which are part of the shape of eyeglasses. A light source portion 101L and a projection optical system 102L are arranged in the temple portion 109L. A HOE 103L is held in the frame portion 108L. More specifically, an inner frame portion 106L is held in the frame portion 108L via a projection position adjustment mechanism 105L-2, which serves as an illuminator, and a HOE 103L is held in the inner frame portion 106L via the projection position adjustment mechanism 105L-1, which serves as an illuminator.

[0117] The video display unit that projects video display light onto the user's right eye includes a light source unit 101R, a projection optical system 102R, and a HOE 103R as an illuminator. Although not shown in the figures, the display device 100 may include a combiner as a follower, and the combiner may structurally include, for example, the HOE 103R and a semi-reflective mirror. The projection optical system 102R may include a relay system drive unit (not shown) as a follower. Although not shown in the figures, a mirror drive unit and a phase difference panel as a follower may be arranged between the light source unit 101R and the projection optical system 102R, or between the projection optical system 101R and the HOE 103R.

[0118] The descriptions of the light source unit 101L, the projection optical system 102L, and the HOE 103L also apply to the light source unit 101R, the projection optical system 102R, and the HOE 101R.

[0119] Similar to the case of the video display unit for the left eye, the light source unit 101R and the projection optical system 102R are arranged in the temple portion 109R. The HOE 103R is held in the frame portion 108R. More specifically, the inner frame portion 106R is held in the frame portion 108R via the projection position adjustment mechanism 105R-2, which serves as the illuminator, and the HOE 103R is held in the inner frame portion 106R via the projection position adjustment mechanism 105R-1, which serves as the illuminator.

[0120] The frame portions 108L and 108R of the display device 100 are connected to each other via a nose bridge portion 110. The nose bridge portion 110 is the portion that crosses the user's nose when the user wears the display device 100. Furthermore, both the frame portions 108L and 108R of the display device 100 are connected to a headband portion 111. The headband portion 111 is the portion that contacts the top of the user's head when the user wears the display device 100, such as... Figure 10 As shown.

[0121] Figure 9 The light source 101L shown includes a laser source 120L. However, the number of laser sources included in the light source 101L can be two or more, and for example, from two to five. The laser sources among the multiple laser sources can output laser light with different wavelengths. Similarly, the light source 101R includes a laser source 120R. However, the number of laser sources included in the light source 101R can be two or more, and for example, from two to five. The laser sources among the multiple laser sources can output laser light with different wavelengths. By using laser sources 120L and 120R, stimulation of a specific wavelength can be presented.

[0122] Although not shown in the figures, the display device 100 may further include components for compensating for wavelength dispersion. Examples of components for compensating for wavelength dispersion include a first optical element comprising a reflective or transmissive volume hologram, a reflective or transmissive relief hologram, or a metasurface. The components for compensating for wavelength dispersion may be arranged around the periphery of mirrors 121L and / or 121R, for example, between mirror 121R and scanning mirror 122L, and / or between mirror 121R and scanning mirror 122R. Applying components for compensating for wavelength dispersion to the display device 100 compensates for wavelength dispersion and thus allows for precise stimulation of any point on the retina (at a designated point).

[0123] (sensor)

[0124] The display device 100 further includes sensors 104L, 104R, 104C, and 104T for detecting changes in the position of the display device 100 relative to the user's head. The position changes detected by these sensors can be, for example, the direction and / or the amount of the position change. Note that in this specification, sensors 104L, 104R, 104C, and 104T may be collectively referred to as sensor 104.

[0125] Sensors 104L and 104R detect the horizontal positional change of the display device 100 relative to the user's head, sensor 104C detects the forward-backward positional change of the display device 100 relative to the user's head, and sensor 104T detects the vertical positional change of the display device 100 relative to the user's head. This allows for three-dimensional monitoring of the wearing offset.

[0126] (Surveillance Department (Line of Sight Detection Device))

[0127] The display device 100 includes monitoring units 107L and 107R for detecting the user's gaze. In this specification, monitoring units 107L and 107R may be collectively referred to as monitoring unit 107. Monitoring unit 107 can monitor the state of the eye using corneal reflection or fundus reflection. By including this monitoring unit, the display device 100 can adjust the position of the image presented to the user to a more suitable location. For example, when an image presented by the display device 100 overlaps with an external image, detecting the user's gaze allows the image to be displayed at a more appropriate position. In other words, including monitoring unit 107 facilitates the presentation of AR information. The monitoring unit may, for example, be a gaze detection device.

[0128] The monitoring unit 107 may be, for example, an imaging type monitoring unit or a photodiode type monitoring unit. This monitoring unit will be described in more detail below.

[0129] The monitoring unit 107L detects the user's left eye gaze. The monitoring unit 107L can be set in any position, for example, in the frame part 108L, and as long as the monitoring unit 107L can detect the left eye gaze, the monitoring unit 107L can be set in any position in other components (such as the inner frame part 106L, etc.).

[0130] The monitoring unit 107L can be, for example, a photodiode-type monitoring unit. A photodiode-type monitoring unit can include, for example, a combination of a light source and a photodiode. The light source is configured to illuminate the left eye. The light source is preferably an infrared illumination source. This prevents the user's recognition of external images and video display light from being affected. The photodiode can be configured to detect the light (specifically, infrared light) emitted from the light source after it is reflected by the eyeball. For example, the photodiode can detect the difference between the amount of light reflected from the iris (pupil) and the amount of light reflected from the sclera (white part of the eye). For example, the photodiode-type monitoring unit can detect the line of sight based on the area ratio of the iris and the sclera detected by the photodiode.

[0131] Even when the display device is worn at an angle, the photodiode-type monitoring unit cannot detect this shift. Therefore, the accuracy of gaze detection performed by this monitoring unit may decrease when the display device is worn at an angle. As described above, the display device of this technology includes a sensor that detects changes in the position of the display device relative to the head, and thereby can detect the wear shift. Based on the wear shift detected by this sensor, the gaze is corrected, thereby improving the accuracy of gaze detection performed by the monitoring unit. The display device of this technology can detect the gaze with an accuracy of, for example, 3 mm or less, particularly 2 mm or less, and even more particularly 1 mm or less. This level of gaze detection accuracy is particularly advantageous for image presentation via Maxwellian viewing.

[0132] Alternatively, the monitoring unit 107L can be an imaging type monitoring unit. An imaging type monitoring unit may include, for example, a combination of a light source and an imaging device. Similar to a photodiode type monitoring unit, the light source is configured to illuminate the left eye. The light source is preferably an infrared illumination source. For example, the imaging device may be configured to obtain a reflected image (a so-called Purkinje image) obtained by the reflection of light from the light source by the eyeball (specifically, the cornea), and an image from which the centroid of the pupil can be obtained. The imaging device may, for example, be an infrared imaging device. For example, the imaging type monitoring unit may estimate the optical axis of the eyeball based on the Purkinje image and the image. The monitoring unit may convert this estimated optical axis into a visual axis to detect the line of sight.

[0133] When the positional relationship between the light source and the eyeball is fixed when detecting gaze based on the Purkinje image and the image, the position of the Purkinje image will be fixed. Wearing offset causes a shift in this positional relationship, and thus a change in the position from which the Purkinje image can be formed. Moreover, this gaze detection is easily affected by, for example, blinking, or hair or eyelashes. Furthermore, in the gaze detection, calibration is usually performed to correct for individual differences, and when the display device experiences wearing offset, calibration needs to be performed again. As described above, the display device of this technology includes a sensor that detects changes in the position of the display device relative to the head, and thereby can detect wearing offset. Therefore, a correction value corresponding to the amount of wearing offset is provided in advance (e.g., by storing it in a memory, etc.), and correction is performed using this correction value in response to the occurrence of wearing offset. This makes it possible to accurately detect gaze. Moreover, the detection of wearing offset is less affected by, for example, blinking, or hair or eyelashes. Furthermore, by performing correction based on the detected wearing offset, the number of calibrations can be reduced.

[0134] (Projection position adjustment mechanism)

[0135] The illuminator included in the display device 100 may also include projection position adjustment mechanisms 105L-1 and 105L-2, and 105R-1 and 105R-2, for adjusting the projection position of the video display light emitted from the display device 100. Note that in this specification, these four projection position adjustment mechanisms may be collectively referred to as projection position adjustment mechanism 105. For example, projection position adjustment mechanism 105 may be configured to adjust the projection position of the video display light by following the line of sight. Projection position adjustment mechanism 105 allows the projection position of the video display light to be adjusted according to wearing offset.

[0136] Furthermore, the projection position adjustment mechanism 105 can adjust the projection position of the video display light according to the rotational movement of the eyeball or the movement of the gaze. For example, by including the projection position adjustment mechanism 105, the display device 100 can adjust the position of the image presented to the user to a more suitable position. For example, when the image presented by the display device 100 overlaps with an external image, the image can be displayed at a more suitable position by detecting the user's gaze. In other words, including the monitoring unit 107 is beneficial for the presentation of AR information. In addition, the projection position adjustment mechanism can also adjust the position where the video display light is focused in image presentation via Maxwellian viewing.

[0137] Projection position adjustment mechanisms 105L-1 and 105L-2 adjust the projection position of the video display light projected onto the left eye. Projection position adjustment mechanism 105L-1 adjusts the positional relationship between the inner lens frame portion 106L and the lens frame portion 108L in the z-axis direction. For example, projection position adjustment mechanism 105L-1 moves the inner lens frame portion 106L relative to the lens frame portion 108L in the z-axis direction. This adjusts the position of HOE 103L in the z-axis direction. Projection position adjustment mechanism 105L-2 adjusts the positional relationship between HOE 103L and the inner lens frame portion 106L in the x-axis direction. For example, projection position adjustment mechanism 105L-2 moves the HOE 103L relative to the inner lens frame portion 106L in the x-axis direction. This adjusts the position of HOE 103L in the x-axis direction.

[0138] The drive element used to adjust the positional relationship between the inner lens frame portion 106L and the lens frame portion 108L in the z-axis direction by the projection position adjustment mechanism 105L-1 can be, for example, a piezoelectric element, an actuator, or a bimetallic element, but is not limited thereto. Similarly, the drive element used to adjust the positional relationship between the HOE 103L and the inner lens frame portion 106L in the x-axis direction by the projection position adjustment mechanism 105L-2 can also be, for example, a piezoelectric element, an actuator, or a bimetallic element, but is not limited thereto.

[0139] For example, the projection position adjustment mechanism 105L-1 can adjust the positional relationship between the inner frame portion 106L and the frame portion 108L in the z-axis direction based on the positional changes of the display device 100 detected by one, two, three, or all of the four sensors 104L, 104R, 104C, and 104T. Furthermore, the projection position adjustment mechanism 105L-1 can adjust the positional relationship based on this positional change and the line of sight detected by the monitoring unit 107L. For example, the projection position adjustment mechanism 105L-2 can adjust the positional relationship between the HOE 103L and the inner frame portion 106L in the x-axis direction based on the positional changes of the display device 100 detected by one, two, three, or all of the four sensors 104L, 104R, 104C, and 104T. Furthermore, the projection position adjustment mechanism 105L-2 can adjust the positional relationship based on this positional change and the line of sight detected by the monitoring unit 107L.

[0140] Projection position adjustment mechanisms 105R-1 and 105R-2 adjust the projection position of the video display light projected onto the right eye. This adjustment can be performed similarly to the adjustments made by projection position adjustment mechanisms 105L-1 and 105L-2.

[0141] (Controller and memory)

[0142] The display device 100 includes a controller 112. For example... Figure 11 As shown, Figure 11 This is a block diagram showing the main structural components of the display device 100. The controller 112 includes an image controller 181, a projection position controller 182, a gaze correction unit 183, a processor 191, and a matching unit 192. Furthermore, as described above, Figure 11 The display device 100 shown includes a light source 101, a sensor 104, a projection position adjustment mechanism 105 as an illuminator, a monitoring unit (line of sight detection mechanism) 107, and an illuminator 201 and a follower 202, which include a perspective member (such as a reflective or transmissive volumetric hologram or a reflective or transmissive relief hologram) arranged in front of the eye. Note that the display device 100 does not necessarily include the follower 202. The memory 184 may be included in the display device 100 or may be included in an external device other than the display device 100.

[0143] The components arranged in front of the eyes and included in the irradiator 201 of the display device 100 have been described above; further details will be provided below. Figure 13 and 14 Examples of using this component.

[0144] Figure 13 The figure shows a second optical element 300, which is an example of a component arranged in front of the eye that reflects a specified light and transmits light other than the specified light. Figure 14 This diagram illustrates a second optical element 400, which is another example of a component arranged in front of the eye that reflects a specified light and transmits light other than the specified light.

[0145] Figure 13 The device shown includes a second optical element 300, a liquid crystal panel 301, and a special liquid crystal lens 302. The second optical element 300 includes a polarizing beam splitter (PBS), and a λ / 4 plate is arranged in the second optical component (polarizing beam splitter) 300, although it is not shown in the figure. Figure 13 The device shown does not necessarily have to include a special liquid crystal lens 302, and alternatively, Figure 13 The device shown may include a lens between the liquid crystal panel 301 and the second optical element 300.

[0146] like Figure 13As shown, a polarized light beam (video display light) L31 from the liquid crystal panel 310 is reflected by region H31 in the second optical element 300 and transmitted through regions T31-1 and T31-2 in the special liquid crystal lens 302 to reach the user's eye 310 (pupil 310-1). A polarized light beam (video display light) L32 from the liquid crystal panel 301 is reflected by region H32 in the second optical element 300 (polarizing beam splitter) and transmitted through regions T32-1 and T32-2 in the special liquid crystal lens 302 to reach the user's eye 310 (pupil 310-1). The user recognizes a virtual video (image) V13 composed of light emitted from the liquid crystal panel 301 (video display light (polarized beams L31 and L32)). Then, light L310 and L320 from the outside world are transmitted through the second optical element 300 (polarizing beam splitter) and through the special liquid crystal lens 302 to reach the user's eye 310 (pupil 310-1). The light is transmitted through a special liquid crystal lens 302 to reach the user's eye 310 (pupil 310-1). The user perceives the real video (image) R13 composed of light L310 and L320 from the outside world. In other words, the user perceives (seems) as if the virtual video (image) V13 is superimposed on the real video (video) R13.

[0147] Figure 14 The diagram illustrates a second optical element 400, an organic light-emitting diode (OLED) panel (organic EL) 401, and a lens 402. The second optical element 400 is an aspherical semi-reflective mirror and includes a first mirror component 400-1 and a second mirror component 400-2. For example, the second optical element 400 may include a combined semi-reflective mirror 400-1 and semi-reflective mirror 400-2, or it may include a combined polarizing beam splitter (PBS) 400-1 and semi-reflective mirror 400-2. When the second optical element 400 includes a combined polarizing beam splitter (PBS) 400-1 and semi-reflective mirror 400-2, a λ / 4 plate may be arranged in the semi-reflective mirror 4002. When the first mirror component 400-1 and the second mirror component 400-2 are semi-reflective mirrors, or when both the first mirror component 400-1 and the second mirror component 400-2 are semi-reflective mirrors, the light transmittance is preferably 50% or higher.

[0148] like Figure 14As shown, light (video display light) L41 from the organic light-emitting diode (OLED) panel (organic EL) 401 is reflected by region H41-1 in the first reflector member 400-1 included in the second optical element 400, and then reflected by region H41-2 in the second reflector member 400-2 included in the second optical element 400, and transmitted through region T41 in the first reflector member 400-1 to reach the user's eye 410 (pupil 410-1). Light (video display light) L42 from the OLED panel 401 is reflected by region H42-1 in the first reflector member 400-1 included in the second optical element 400, and then reflected by region H42-2 in the second reflector member 400-2 included in the second optical element 400, and transmitted through region T42 in the first reflector member 400-1 to reach the user's eye 410 (pupil 410-1). Light rays (video display light) L43 from the OLED panel 401 are reflected by region H43-1 in the first reflector member 400-1 included in the second optical element 400, and then reflected by region H43-2 in the second reflector member 400-2 included in the second optical element 400. The light then passes through region T43 in the first reflector member 400-1 to reach the user's eye 410 (pupil 410-1). The user perceives a virtual video (image) composed of light emitted from the OLED panel 401 (video display light (rays L41, L42, and L43)). Then, although not shown in the figure, light from the outside world passes through the second optical element 400 to reach the user's eye 410 (pupil 410-1). The user perceives a real video (image) composed of light from the outside world. The user perceives it as if the virtual video (image) is superimposed on the real video (image).

[0149] Note that the above references Figure 13 and 14 The description relating to the two examples of components arranged in front of the eyes and included in the irradiator 201 can also be applied to the display device 100-1 described later as a second embodiment of the present technology.

[0150] The controller 112 is described below.

[0151] Image controller 181 controls the projection of video display light executed by the video display unit. For example, image controller 181 drives light source units 101L and 101R. In particular, image controller 180 drives the laser source and scanning mirror included in these light source units and causes the light source units to output video display light. For example, image controller 181 can acquire image data stored in memory 184 and can cause light source units 101L and 101R to output video display light based on the image data. Image controller 181 can correct the image data based on the position change of the display device 100 relative to the user's head detected by sensor 104. Image controller 181 can cause light source units 101L and 101R to output video display light based on the corrected image data. In other words, the display device 100 can correct the image based on the position change detected by the sensor that detects the position change of the display device relative to the user's head.

[0152] The projection position controller 182 controls the projection position adjustment mechanisms 105L-1, 105L-2, 105R-1, and 105R-2, thereby controlling the projection position of the video display light. For example, the projection position controller 182 can drive one, two, three, or four of the projection position adjustment mechanisms 105L-1, 105L-2, 105R-1, and 105R-2 based on the line of sight detected by the monitoring units 107L and 107R to adjust the projection position of the video display light. For example, the projection position of the video display light can be adjusted to follow the line of sight. The projection position controller 182 can also drive one, two, three, or four of the projection position adjustment mechanisms 105L-1, 105L-2, 105R-1, and 105R-2 based on the line of sight corrected by the line-of-sight correction unit 183, which will be described later, to adjust the projection position of the video display light. For example, the projection position of the video display light can be adjusted to follow the corrected line of sight. The projection position controller 182 can drive one, two, three, or four of the projection position adjustment mechanisms 105L-1, 105L-2, 105R-1, and 105R-2 to adjust the projection position of the video display light based on data (hereinafter also referred to as "displacement data") related to the position change of the display device 100 relative to the user's head detected by one, two, three, or four sensors 104L, 104R, 104C, and 104T.

[0153] For example, the projection position controller 182 can calculate the position adjustment amount performed by each projection position adjustment mechanism based on the displacement data and correction coefficients. The projection position controller 182 can drive each projection position adjustment mechanism to change the positional relationship according to the calculated position adjustment amount. For example, the projection position controller 182 can obtain correction coefficients from a correction table pre-stored in the memory 184 and can use the correction coefficients to calculate the position adjustment amount. For example, the correction table may include multiple correction coefficients, and the projection position controller 182 can select a specified correction coefficient from the multiple correction coefficients based on the displacement data. Furthermore, for example, a correction table can be set for each projection position adjustment mechanism. The correction table may be pre-included in the display device 100 or may be updated in response to user use of the display device 100. By selecting or updating the correction table or correction coefficients, the accuracy of projection position control can be improved. To calculate the position adjustment amount, the projection position controller 182 can use the line of sight detected by the monitoring unit or the line of sight correction unit 183 after correction.

[0154] The gaze correction unit 183 corrects the gaze detected by the monitoring units 107L and 107R based on the displacement data. Therefore, the gaze correction unit 183 can identify the gaze while taking into account wearing offset, thereby improving the accuracy of gaze detection. This correction can be performed relative to the optical axis of the eye, relative to the visual axis of the eye, or relative to a reference axis other than the optical axis and visual axis. The gaze correction unit 183 can also obtain correction coefficients from, for example, a correction table pre-stored in the memory 184, and can use these correction coefficients to perform the gaze correction. For example, the correction table may include multiple correction coefficients, and the gaze correction unit 183 can select a specific correction coefficient from these multiple correction coefficients based on the displacement data. The correction table may be pre-included in the display device 100 or can be updated in response to user use of the head-mounted display device 100. The selection or updating of the correction table or correction coefficients allows for improved accuracy of gaze correction.

[0155] The display device 100 may further include a memory 184. The memory 184 may store data related to the video display light projected by the video display unit, a calibration table for controlling the projection position by the projection position controller 122, and a calibration table for correcting the line of sight by the line of sight correction unit 123.

[0156] Processor 191 performs processing of information about the feature distribution of the eyeball. The feature distribution of the eyeball is acquired, for example, using a fundus camera, or using an OCT, refractometer, or light detection device that detects the light returned due to an IR scan. The characteristic distribution of the eyeball can be obtained from shape features (e.g., the size and shape of the eyeball, the size and shape of the macula (e.g., concave shape), the size and shape of the blind spot (e.g., concave shape), the cross-sectional shape of the retina (e.g., concave-convex shape), essential features (e.g., the location of the fovea, the location of the blind spot, features of lesions, etc.), and optical features (e.g., refraction within the eyeball, aberrations, etc.). The processor 191 can define a coordinate system based on the aforementioned characteristic distribution of the eyeball. Note that the coordinate system can be defined independently of the processor 191, for example, by setting a definition unit for defining the coordinate system in the controller 112, separate from the processor 191. The coordinate system is defined based on at least two elements selected from the group consisting of the first fovea of ​​the right eye, the first blind spot of the right eye, the second fovea of ​​the left eye, and the second blind spot of the left eye. Since a coordinate system is used to define the displayed video and display position, the amount of data can be reduced compared to using a pattern map. Furthermore, the displayed video and display position can be controlled in the processor 191 based on information about eyeball rotation. This reduces processing intensity compared to using pattern matching.

[0157] The matching unit 192 matches the feature distribution of the eyeballs with the state of the eyeballs. For example, by using the aforementioned coordinate system, the matching unit 192 matches the feature distribution of the eyeballs with the state of the eye to which video is to be presented (the eye to which video is to be presented can be referred to as the attention eye). The matching unit 192 acquires the deviation between the visual axis and the optical axis (gaze calibration) and can generate a graph reflecting the offset.

[0158] The display device according to the first embodiment of the present technology (first example of a display device) has been described above. Unless there is a particular technical contradiction, the content of this description can be applied to the display device according to the second embodiment of the present technology (second example of a display device), the display method according to the third embodiment of the present technology (first example of a display method), and the display method according to the fourth embodiment of the present technology (second example of a display method) described below.

[0159] <3. Second Embodiment (Second Example of a Display Device)>

[0160] The display device according to the second embodiment of the present technology (a second example of a display device) is a display device that includes a light source, a processor for processing the feature distribution of the eyeball, a monitoring unit for monitoring the state of the eyeball, a matching unit for matching the feature distribution of the eyeball with the state of the eyeball, and an irradiator for illuminating a designated location on the retina with video display light emitted by the light source, and further includes an acquisition unit for acquiring the feature distribution of the eyeball. In other words, the display device according to the second embodiment of the present technology is a display device obtained by adding the acquisition unit for acquiring the feature distribution of the eyeball to the display device according to the first embodiment of the present technology. The display device according to the second embodiment of the present technology may further include a follower. The display device according to the second embodiment of the present technology can be applied, for example, to eyeglass displays and head-mounted displays.

[0161] The processor, which processes the characteristic distribution of the eyeball, processes information about the characteristic distribution of the eyeball obtained from, for example, an acquisition unit (such as a fundus camera, OCT, a light detection device or refractometer that detects light returned due to IR scanning, etc.). The monitoring unit uses, for example, corneal reflection or fundus reflection to monitor the state of the eyeball and can acquire the optical axis. The matching unit acquires the deviation between the visual axis and the optical axis (line-of-sight calibration) and can generate a map reflecting the offset. The illuminator can modulate light according to the laser source, video display (projector), and the characteristic distribution of the eyeball, adjust the amount of light according to external light, and perform control including distortion correction. The follower enables the displayed video to follow eye movements caused by eye tracking, wherein the light is manipulated to change the illuminated video and the illumination position based on information about eye rotation.

[0162] refer to Figure 12 A configuration example of a display device according to a second embodiment of the present technology is described.

[0163] Figure 12 This is a block diagram showing the main structural elements of a display device (display device 100-1) according to a second embodiment of the present technology.

[0164] Display device 100-1 includes a light source unit 101, a sensor 104, a projection position adjustment mechanism 105 serving as an illuminator, a monitoring unit (line of sight detection mechanism) 107, and an illuminator 201, a follower 202, an acquisition unit 301, and a controller 112, which include a perspective component (such as a reflective or transmissive volumetric hologram or a reflective or transmissive relief hologram) arranged in front of the eyes. Note that display device 100-1 does not necessarily include a follower 202.

[0165] The controller 112 includes an image controller 181, a projection position controller 182, a line-of-sight correction unit 183, a processor 191, and a matching unit 192.

[0166] Image controller 181 controls the projection of video display light executed by the video display unit. For example, image controller 181 drives light source units 101L and 101R. In particular, image controller 180 drives a laser source and scanning mirror included in the light source unit, and causes the light source unit to output video display light. For example, image controller 181 can acquire image data stored in memory 184, and can cause light source units 101L and 101R to output video display light based on the image data. Image controller 181 can correct the image data based on the position change of the display device 100 relative to the user's head detected by sensor 104. Image controller 181 can cause light source units 101L and 101R to output video display light based on the corrected image data. In other words, the display device 100 can correct the image based on the position change detected by the sensor that detects the position change of the display device relative to the user's head.

[0167] The projection position controller 182 controls the projection position adjustment mechanisms 105L-1, 105L-2, 105R-1, and 105R-2, thereby controlling the projection position of the video display light. For example, the projection position controller 182 can drive one, two, three, or four of the projection position adjustment mechanisms 105L-1, 105L-2, 105R-1, and 105R-2 based on the line of sight detected by the monitoring units 107L and 107R to adjust the projection position of the video display light. For example, the projection position of the video display light can be adjusted to follow the line of sight. The projection position controller 182 can also drive one, two, three, or four of the projection position adjustment mechanisms 105L-1, 105L-2, 105R-1, and 105R-2 based on the line of sight corrected by the line-of-sight correction unit 183, which will be described later, to adjust the projection position of the video display light. For example, the projection position of the video display light can be adjusted to follow the corrected line of sight. The projection position controller 182 can drive one, two, three, or four of the projection position adjustment mechanisms 105L-1, 105L-2, 105R-1, and 105R-2 to adjust the projection position of the video display light based on data (hereinafter also referred to as "displacement data") related to the position change of the display device 100 relative to the head detected by one, two, three, or four of the sensors 104L, 104R, 104C, and 104T.

[0168] For example, the projection position controller 182 can calculate the position adjustment amount performed by each projection position adjustment mechanism based on the displacement data and correction coefficients. The projection position controller 182 can drive each projection position adjustment mechanism to change the positional relationship according to the calculated position adjustment amount. For example, the projection position controller 182 can obtain correction coefficients from a correction table pre-stored in the memory 184 and can use the correction coefficients to calculate the position adjustment amount. For example, the correction table may include multiple correction coefficients, and the projection position controller 182 can select a specified correction coefficient from the multiple correction coefficients based on the displacement data. Furthermore, for example, a correction table can be set for each projection position adjustment mechanism. The correction table may be pre-included in the display device 100 or may be updated in response to user use of the display device 100. By selecting or updating the correction table or correction coefficients, the control accuracy of the projection position can be improved. To calculate the position adjustment amount, the projection position controller 182 can use the line of sight detected by the monitoring unit or the line of sight correction unit 183 after correction.

[0169] The gaze correction unit 183 corrects the gaze detected by the monitoring units 107L and 107R based on displacement data. Therefore, the gaze correction unit 183 can identify the gaze while taking into account wearing offset, thereby improving the accuracy of gaze detection. This correction can be performed relative to the optical axis of the eye, relative to the visual axis of the eye, or relative to a reference axis other than the optical axis and visual axis. The gaze correction unit 183 can also obtain correction coefficients from, for example, a correction table pre-stored in the memory 184, and can use these correction coefficients to perform the gaze correction. For example, the correction table may include multiple correction coefficients, and the gaze correction unit 183 can select a specific correction coefficient from these multiple correction coefficients based on displacement data. The correction table may be pre-included in the display device 100 or can be updated in response to user use of the head-mounted display device 100. The selection or updating of the correction table or correction coefficients allows for improved accuracy of gaze correction.

[0170] The display device 100 may also include a memory 184. The memory 184 may store data related to the video display light projected by the video display unit, a calibration table for controlling the projection position by the projection position controller 122, and a calibration table for correcting the line of sight by the line of sight correction unit 123.

[0171] Processor 191 performs processing of information regarding the feature distribution of the eyeball. For example, the feature distribution of the eyeball can be acquired from acquisition unit 301. For example, the feature distribution of the eyeball can be acquired using a fundus camera including acquisition unit 301, or using an OCT, refractometer, or light detection device that detects light returned due to IR scanning, including acquisition unit 301. The characteristic distribution of the eyeball can be obtained from shape features (e.g., the size and shape of the eyeball, the size and shape of the macula (e.g., concave shape), the size and shape of the blind spot (e.g., concave shape), the cross-sectional shape of the retina (e.g., concave-convex shape), essential features (e.g., the location of the fovea, the location of the blind spot, features of lesions, etc.), and optical features (e.g., refraction within the eyeball, aberrations, etc.). The processor 191 can define a coordinate system based on the characteristic distribution of the eyeball. Note that the coordinate system may not be defined by the processor 191 based on the aforementioned characteristic distribution of the eyeball; instead, a definition unit for defining the coordinate system may be set independently of the processor 191 in the controller 112. The coordinate system is defined based on at least two elements selected from the group consisting of the first fovea of ​​the right eye, the first blind spot of the right eye, the second fovea of ​​the left eye, and the second blind spot of the left eye. Since a coordinate system is used to define the displayed video and display position, the amount of data can be reduced compared to using a pattern map. Furthermore, the displayed video and display position can be controlled in the processor 191 based on information about eyeball rotation. This reduces processing intensity compared to using pattern matching.

[0172] The matching unit 192 matches the feature distribution of the eyeballs with the state of the eyeballs. For example, by using the aforementioned coordinate system, the matching unit 192 matches the feature distribution of the eyeballs with the state of the eye to which the video is to be presented (the eye to which the video is to be presented can be referred to as the attention eye). The matching unit 192 acquires the deviation between the visual axis and the optical axis (gaze calibration) and can generate a graph reflecting the offset.

[0173] <4. Third Implementation Method (First Example of the Display Method)>

[0174] The display method according to the third embodiment of the present technology (first example of the display method) is as follows: processing the feature distribution of the eyeball, monitoring the state of the eyeball, matching the feature distribution of the eyeball with the state of the eyeball, and illuminating a designated position on the retina with video display light emitted by a light source.

[0175] The display method according to the third embodiment of the present technology (first example of the display method) can be performed using a display device (e.g., display device 100) according to the first embodiment of the present technology (first example of the display device).

[0176] In addition, the above references Figure 1 and Figure 5 The described content can be applied to the flow of the display method according to the third embodiment of this technology (a first example of the display method). Note that... Figure 1 The steps S101 (taking a fundus photograph when viewing the eyeball from the front), S102 (taking fundus photographs when the eyeball is rotated left, right, up, or down), and S103 (detecting the position of the fovea and blind spot in each fundus photograph) are shown; and Figure 5 Steps S201 (generating a fundus image) and S202 (acquiring eye features) shown can be performed using external devices (e.g., fundus camera, OCT, or refractometer, etc.) other than the display device (e.g., display device 100) according to the first embodiment of the present technology (first example of a display device).

[0177] <5. Fourth Implementation Method (Second Example of the Display Method)>

[0178] The display method according to the fourth embodiment of the present technology (a second example of the display method) is as follows: it includes processing the feature distribution of the eyeball, monitoring the state of the eyeball, matching the feature distribution of the eyeball with the state of the eyeball, and illuminating a designated location on the retina with video display light emitted by a light source, and further includes acquiring the feature distribution of the eyeball. In other words, the display method according to the fourth embodiment of the present technology is a display method obtained by adding the acquired feature distribution of the eyeball to the display method according to the third embodiment of the present technology.

[0179] The display method according to the fourth embodiment of the present technology (a second example of the display method) can be performed using a display device (e.g., display device 100-1) according to the second embodiment of the present technology (a second example of the display device).

[0180] The above reference Figure 1 and Figure 5 The described content can be applied to the flow of the display method according to the fourth embodiment of this technology (a second example of the display method).

[0181] Note that the implementation of this technology is not limited to the above-described implementation and various modifications can be made to it without departing from the scope of this technology.

[0182] Furthermore, the effects described in this article are not limiting but illustrative and may provide other effects.

[0183] In addition, this technology can also be configured as follows.

[0184] (1) A display device, comprising:

[0185] light source;

[0186] A processor that processes the feature distribution of the eyeball;

[0187] A monitoring unit that monitors the state of the eyeball;

[0188] A matching unit that matches the feature distribution of the eyeball with the state of the eyeball; and

[0189] An illuminator that illuminates a designated location on the retina with video display light emitted by the light source.

[0190] (2) The display device according to (1) further includes:

[0191] The acquisition unit acquires the feature distribution of the eyeball.

[0192] (3) The display device according to (2), wherein

[0193] The acquisition unit includes at least one selected from the group consisting of a fundus camera, an OCT, a refractometer, and a light detection device that detects the light returned due to IR scanning.

[0194] (4) The display device according to any one of (1) to (3), wherein

[0195] The monitoring unit uses corneal reflection or fundus reflection to monitor the state of the eyeball.

[0196] (5) The display device according to any one of (1) to (4) further includes

[0197] A follower that causes the video display light to follow the movement of the eyeball.

[0198] (6) The display device according to (5), wherein

[0199] The follower includes at least one selected from the group consisting of a combiner, a relay system driver, a mirror driver, and a phase difference panel.

[0200] (7) The display device according to any one of (1) to (6), wherein

[0201] A coordinate system is defined based on the feature distribution of the eyeball, and

[0202] The matching unit uses the coordinate system to match the feature distribution of the eyeball with the state of the eyeball to which the video is to be presented.

[0203] (8) The display device according to (7), wherein

[0204] The coordinate system is defined based on at least two selected from the group consisting of the first fovea of ​​the right eye, the first blind spot of the right eye, the second fovea of ​​the left eye, and the second blind spot of the left eye.

[0205] (9) The display device according to any one of (1) to (8), wherein

[0206] The light source is a laser source.

[0207] (10) The display device according to any one of (1) to (9) further includes

[0208] Scanning mirror, in which

[0209] The scanning mirror illuminates the video display light onto the retina.

[0210] (11) The display device according to any one of (1) to (10), wherein

[0211] The irradiator also includes a component disposed in front of the eye, and

[0212] The component is a transparent component.

[0213] (12) The display device according to (11), wherein

[0214] The component includes a reflective or transmissive volume hologram, a reflective or transmissive relief hologram, or a first optical element on a metasurface, or a second optical element that reflects a specified light and transmits light other than the specified light.

[0215] (13) The display device according to any one of (1) to (12) further includes

[0216] A component used to compensate for wavelength dispersion.

[0217] (14) The display device according to (13), wherein

[0218] The component used to compensate for wavelength dispersion is a first optical element comprising a reflective or transmissive volume hologram, a reflective or transmissive relief hologram, or a metasurface.

[0219] (15) A display method, comprising:

[0220] Processing the characteristic distribution of the eyeball;

[0221] The state of the eyeball is monitored;

[0222] Matching the feature distribution of the eyeball with the state of the eyeball; and

[0223] A video display is created by illuminating a designated area on the retina with light emitted from a light source.

[0224] (16) The display method according to (15) further includes:

[0225] Obtain the feature distribution of the eyeball.

[0226] (17) According to the display method described in (16), wherein

[0227] The acquisition of the characteristic distribution of the eyeball includes using at least one selected from the group consisting of a fundus camera, an OCT, a refractometer, and a light detection device that detects light returned due to IR scanning.

[0228] (18) The display method according to any one of (15) to (17), wherein

[0229] Monitoring the state of the eyeball includes using corneal or fundus reflexes to monitor the state of the eyeball.

[0230] (19) The display method according to any one of (15) to (18) further includes:

[0231] Make the video display light follow the movement of the eyeball.

[0232] (20) The display method according to (19), wherein

[0233] The method of making the video display light follow the movement of the eyeball includes using at least one selected from the group consisting of a combiner, a relay system driver, a mirror driver, and a phase difference panel to make the video display light follow the movement of the eyeball.

[0234] (21) The display method according to any one of (15) to (20) further includes:

[0235] A coordinate system is defined based on the feature distribution of the eyeball, wherein

[0236] Matching the feature distribution of the eyeball with the state of the eyeball involves using a coordinate system to match the feature distribution of the eyeball with the state of the eyeball to which the video is to be presented.

[0237] (22) According to the display method described in (21), wherein

[0238] The coordinate system is defined based on at least two selected from the group consisting of the first fovea of ​​the right eye, the first blind spot of the right eye, the second fovea of ​​the left eye, and the second blind spot of the left eye.

[0239] (23) The display method according to any one of (15) to (22), wherein

[0240] The light source is a laser source.

[0241] (24) The display method according to any one of (15) to (23), wherein

[0242] This includes using a scanning mirror to illuminate the video display light onto the retina.

[0243] (25) The display method according to any one of (15) to (24) further includes:

[0244] Set transparent components; and

[0245] The perspective component is positioned in front of the eyes.

[0246] (26) The display method according to (25), wherein

[0247] The component includes a reflective or transmissive volume hologram, a reflective or transmissive relief hologram, or a first optical element on a metasurface, or a second optical element that reflects a specified light and transmits light other than the specified light.

[0248] (27) The display method according to any one of (15) to (26) further includes:

[0249] Provide components for compensating for wavelength dispersion.

[0250] (28) The display method according to (27), wherein

[0251] The components used to compensate for wavelength dispersion are first optical elements including reflective or transmissive volume holograms, reflective or transmissive relief holograms, or metasurfaces.

[0252] Reference tag list

[0253] 1. Optical axis (optical vector)

[0254] 2 visual axes (gaze vector)

[0255] 10 pupils

[0256] 20 eyeballs

[0257] 30 Central concave

[0258] 31 blind spots

[0259] 100, 100-1 display devices

[0260] 101 Light Source Department

[0261] 104 sensors

[0262] 105 Projection Position Adjustment Mechanism

[0263] 107 Surveillance Unit (Line of Sight Detection Device)

[0264] 112 controller

[0265] 181 Image Controller

[0266] 182 Projection Position Controller

[0267] 183 Vision Correction Department

[0268] 191 processor

[0269] 192 Matching Department

[0270] 201 Irradiator

[0271] 201 Follower

[0272] 301 Acquisition Department

Claims

1. A display device, comprising: light source; A controller configured to process the characteristic distribution of the eyeball; A monitoring unit configured to monitor the state of the eyeball; A matching unit configured to match the feature distribution of the eyeball with the state of the eyeball; and An illuminator configured to illuminate a designated location on the retina with video display light emitted by the light source. A coordinate system is defined based on the feature distribution of the eyeballs, and the displayed video and the specified location are defined within this coordinate system. This involves performing gaze tracking to obtain information about the rotation of the eyeball. The displayed video and the designated position are controlled based on the information regarding the rotation of the eyeball, and The matching unit performs the matching of the feature distribution of the eyeball with the state of the eyeball by using the coordinate system.

2. The display device according to claim 1, further comprising: The acquisition unit acquires the feature distribution of the eyeball.

3. The display device according to claim 2, wherein The acquisition unit includes at least one selected from the group consisting of a fundus camera, optical coherence tomography (OCT), a refractometer, and a light detection device for detecting light returned due to IR scanning.

4. The display device according to claim 1, wherein The monitoring unit uses corneal reflection or fundus reflection to monitor the state of the eyeball.

5. The display device according to claim 1, further comprising: A follower that causes the video display light to follow the movement of the eyeball.

6. The display device according to claim 5, wherein The follower includes at least one selected from the group consisting of a combiner, a relay system driver, a mirror driver, and a phase difference panel.

7. The display device according to claim 1, wherein The coordinate system is defined based on at least two selected from the group consisting of the first fovea of ​​the right eye, the first blind spot of the right eye, the second fovea of ​​the left eye, and the second blind spot of the left eye.

8. The display device according to claim 1, wherein The light source is a laser source.

9. The display device according to claim 1, further comprising: Scanning mirror, in which The scanning mirror illuminates the video display light onto the retina.

10. The display device according to claim 1, wherein The irradiator also includes a component disposed in front of the eye, and The component is a transparent component.

11. The display device according to claim 10, wherein The component is a first optical element or a second optical element. The first optical element includes a reflective or transmissive volume hologram, a reflective or transmissive relief hologram, or a metasurface. The second optical element is an optical element that reflects a specified light and transmits light other than the specified light.

12. The display device according to claim 1, further comprising: A component used to compensate for wavelength dispersion.

13. The display device according to claim 12, wherein The component used to compensate for wavelength dispersion is a first optical element comprising a reflective or transmissive volume hologram, a reflective or transmissive relief hologram, or a metasurface.

14. A display method, comprising: Processing the characteristic distribution of the eyeball; The state of the eyeball is monitored; Match the feature distribution of the eyeball with the state of the eyeball; as well as A video display illuminating a designated location on the retina is created using light emitted from a light source. A coordinate system is defined based on the feature distribution of the eyeballs, and the displayed video and the specified location are defined within this coordinate system. The process involves performing gaze tracking to obtain information about the rotation of the eyeballs, and controlling the displayed video and the designated position based on the information about the rotation of the eyeballs. The feature distribution of the eyeball is matched with the state of the eyeball by using the coordinate system.

15. The display method according to claim 14, further comprising: Obtain the feature distribution of the eyeball.

Images