Head-mounted display and system

Abstract

To maintain a position of an image projected into the eyeballs of a user even when the gaze of the eyeballs change.SOLUTION: The present invention relates to a head-mounted display used in virtual reality, mixed reality, or augmented reality systems, the head-mounted display including: a display device for displaying an image; a tracking system for determining the gaze of the eyeballs of a user; an adjustable optical device; an optical system for projecting the image into the eyeballs; and a processing circuit including an adjustable optical device and for controlling the adjustable optical device on the basis of the gaze of the eyeballs so that a position of the image projected into the eyeballs do not change when the gaze of the eyeballs change.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a head-mounted display for use in a virtual reality, mixed reality or augmented reality system, such as a telepresence system, a virtual reality system, a mixed reality system and / or an augmented reality system. [Background technology]

[0002] Head-mounted displays, such as virtual reality headsets, are applicable to fields of technology including medical imaging and other medical applications. A head-mounted display may include a display device and one or more optical devices that project the image on the display device onto the retina of a user's eye. Stereoscopic applications may use two display panels, with slightly different images presented to each eye of the user.

[0003] However, the size and / or resolution of the display devices may be limited, which may affect image fidelity and / or user experience. It may be desirable to use one or more high-resolution display devices. However, the computing power and / or bandwidth of the head-mounted display may be limited and insufficient to update the high-resolution display devices at a sufficiently high frame rate. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] US Patent Application Publication No. 2018136720 [Patent Document 2] US Patent Application Publication No. 2021173474 [Patent Document 3] US Patent Application Publication No. 2018081178 Summary of the Invention [Problem to be solved by the invention]

[0005] One of the problems to be solved by the embodiments disclosed in this specification and the drawings is to maintain the position of an image projected within a user's eyeball even when the line of sight of the user's eyeball changes. However, the problems to be solved by the embodiments disclosed in this specification and the drawings are not limited to the above problem. Problems corresponding to the effects of each configuration shown in the embodiments described below can also be positioned as other problems. [Means for solving the problem]

[0006] One embodiment provides a head-mounted display for use in a virtual reality, mixed reality, or augmented reality system, the head-mounted display comprising: a display device for displaying an image; a tracking system for determining a line of sight of a user's eye; an optical system including an adjustable optical device for projecting the image into the user's eye; and processing circuitry for controlling the adjustable optical device based on the line of sight of the user's eye such that a position of the image projected into the user's eye does not change when the line of sight of the user's eye changes.

[0007] One embodiment provides a head-mounted display for use in a virtual reality, mixed reality, or augmented reality system, the head-mounted display comprising: a display device configured to display an image such that a first region of the image has a higher resolution than a second region of the image; a tracking system configured to determine a line of sight of a user's eye; an optical system including an adjustable optical device configured to project the first region of the image onto a central portion of a retina of the user's eye; and a processing circuit configured to control the adjustable optical device based on the line of sight of the user's eye such that a position of the first region of the image projected onto the retina of the user's eye does not change when the line of sight of the user's eye changes.

[0008] One embodiment provides a system comprising a head-mounted display according to one or more embodiments described herein, at least one camera located remotely relative to the head-mounted display, and a control system that controls the position and / or movement of the at least one camera based on the line of sight. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a schematic diagram of a head-mounted display according to an embodiment. [Figure 2] FIG. 2 is a schematic diagram of a portion of the head-mounted display of FIG. [Figure 3] FIG. 3 is a schematic diagram of the portion of the head-mounted display of FIG. 2 including a tracking system for different lines of sight of the user's eyes. [Figure 4] FIG. 4 is a schematic diagram of the portion of the head-mounted display of FIG. 2 at different focal points of the user's eyes. [Figure 5] FIG. 5 is a schematic diagram of a portion of a head-mounted display according to another embodiment. [Figure 6] FIG. 6 is a schematic diagram of the portion of the head-mounted display of FIG. 5 at different lines of sight of the user's eyes. [Figure 7] FIG. 7 is another schematic diagram of the portion of the head-mounted display of FIG. 5 at different lines of sight of the user's eyes. [Figure 8] FIG. 8 is another schematic diagram of the portion of the head-mounted display of FIG. 5 including a tracking system for different lines of sight of the user's eyes. [Figure 9] FIG. 9 is a schematic diagram of the portion of the head-mounted display of FIG. 5 at different focal points of the user's eyes. [Figure 10] FIG. 10 is a schematic diagram of a head mounted display according to another embodiment. [Figure 11] FIG. 11 is a schematic diagram of a head-mounted display according to another embodiment. [Figure 12]FIG. 12 is a schematic diagram of an example in which both the first and second eyes of a user look in the same direction. [Figure 13] FIG. 13 is a schematic diagram of an example in which the first and second eyes of a user look in different directions. [Figure 14] FIG. 14 is a schematic diagram of exemplary movements of a user's head. [Figure 15] FIG. 15 is a schematic diagram of a system including the head-mounted display of FIG. DETAILED DESCRIPTION OF THE INVENTION

[0010] A head-mounted display 10 according to an embodiment is shown schematically in Figure 1. In this embodiment, the head-mounted display 10 comprises a display device 12. The display device 12 may comprise an LCD display, an LED display, etc. The display device 12 displays an image 14. In this embodiment, the display device 12 comprises a display panel 13. The image 14 is displayed on the display panel 13. The image 14 may be an image of a two-dimensional or three-dimensional scene.

[0011] The head mounted display 10 includes a tracking system 16. The tracking system 16 determines the line of sight of the eyes 18 of a user 20.

[0012] The head mounted display 10 includes an optical system 22. As will be explained in more detail below, the optical system 22 projects an image 14 into the eye 18 of the user 20.

[0013] Figure 2 shows a schematic representation of a portion of the head-mounted display 10. For clarity, the tracking system and display panel have been omitted from Figure 2. In the embodiment shown in Figure 2, the eye 18 of the user 20 looks at the center of the display device 12. The line of sight G of the eye 18 of the user 20 is along the optical axis OA of the optical system 22.

[0014] In the embodiment shown in FIGS. 1 and 2 , the optical system 22 includes a first optical device 24 a, which may be implemented as a lens. The first optical device 24 a may also be referred to as an adjustable focusing component. The optical system 22 includes a second optical device 24 b, which is implemented as an adjustable optical device. For example, the second optical device 24 b may include a floating lens or a movable lens. The second optical device 24 b may also be referred to as an adjustable imaging component. While FIG. 2 illustrates the second optical device 24 b as a single optical element, such as a single floating or movable lens, preferably in other embodiments, the second optical device may include two or more optical elements, defining, for example, a compound lens system.

[0015] The first and second optical devices 24a, 24b may be positioned perpendicular to the optical axis OA, e.g., approximately perpendicular. The first and second optical devices 24a, 24b may be positioned such that an intermediate plane 26 containing the intermediate image 14a is located between the first and second optical devices 24a, 24b. The first optical device 24a may image the intermediate image 14a to infinity so that the cornea and lens of the eye 18 of the user 20 can project the intermediate image 14a onto the retina 30. In FIG. 2, the cornea and lens of the eye 18 of the user 20 are indicated as one part of the eye 18 of the user 20 using the reference numeral 28. In FIG. 2, the optical system 22 and the image projected by the eye 18 of the user 20 are indicated by the reference numeral 14b. Preferably, in other embodiments, the implementation and / or position of the first and second optical elements may be different.

[0016] Figure 3 shows schematically the portion of the head mounted display 10 shown in Figure 2 at different lines of sight G of the eye 18 of the user 20. For example, the lines of sight may be at an angle α relative to the optical axis OA. In the embodiment shown in Figure 3, the user 20 may be considered to be looking up. However, preferably in other embodiments, the user may look in other directions, such as down, left, right, or any combination of up, down, left, right, etc.

[0017] 1 and 3, the head mounted display 10 includes a processing unit 32. The processing unit 32 includes processing circuitry 32a. The processing circuitry 32a may include image processing circuitry. The processing unit 32 includes a central processing unit (CPU) and a graphical processing unit (GPU). The processing circuitry 32a may be implemented in a CPU, a GPU, or a combination of a CPU and a GPU.

[0018] In this embodiment, processing circuitry 32a is implemented in the CPU, GPU, or a combination of CPU and GPU of processing unit 32 by a computer program having computer-readable instructions executable to perform one or more operations of processing unit 32. However, in other embodiments, processing circuitry may be implemented in software, hardware, or any suitable combination of hardware and software. In some embodiments, various circuits may be implemented as one or more Application Specific Integrated Circuits (ASICs) or Field Programmable Gate Arrays (FPGAs).

[0019] The processing circuitry 32a controls the second optical device 24b based on the line of sight of the eyeball 18 of the user 20 so that the position of the image projected within the eyeball 18 of the user 20 does not change when the line of sight of the eyeball 18 of the user 20 changes. In other words, the processing circuitry 32a controls the second optical device 24b based on the line of sight G of the eyeball 18 of the user 20 so that the projected image 14b remains unchanged or immobile in the eyeball 18 of the user 20, such as the retina 30 of the eyeball 18 of the user 20. The optical system 22, tracking system 16, and processing device 32 shown in Figures 1-3 may be referred to as a retinal image stabilization system.

[0020] The optical system 22 includes first and second conjugate points 34a, 34b. In FIGS. 2 and 3, the first conjugate point 34a is shown at the center of the display device 12. The first conjugate point 34a may be an object point of the optical system 22. The second conjugate point 34b is shown at the center of the retina 30, which may also be referred to as the orbit 36. The second conjugate point 34b may be an image point of the optical system 22. The processing circuit 32a controls the second optical device 24b based on the line of sight G of the eyeball 18 of the user 20 so that the first conjugate point 34a is located at the center of the display device 12 and the second conjugate point 34b is located at the center of the retina 30 when the line of sight G of the eyeball 18 of the user 20 changes.

[0021] In this embodiment, processing circuitry 32a controls the position of second optical device 24b relative to display device 12. Processing circuitry 32a may control second optical device 24b to move relative to display device 12, for example, in a parallel direction, e.g., approximately parallel to display device 12, and / or in a perpendicular direction, e.g., approximately perpendicular to optical axis OA. In embodiments in which the second optical device includes two or more optical elements, only one of the optical elements may be moved, for example, to minimize the mass of the movable optical elements.

[0022] In the embodiment shown in Figure 3, the processing circuitry 32a controls the second optical device 24b to move upward based on the line of sight G of the eyeball 18 of the user 20. In other words, the processing circuitry 32a controls the second optical device 24b to move from a first position indicated by the dotted outline of the second optical device 24b shown in Figure 3 to a second position indicated by the solid outline of the second optical device 24b shown in Figure 3. When the second optical device 24b is in the first position, the line of sight G is not along the optical axis OA, and therefore the projected image 14b has moved on the retina 30 of the eyeball 18 of the user 20 (although not shown in Figures 2 and 3). However, when the second optical device 24b is in the second position, the line of sight G of the eye 18 of the user 20 is at an angle α with respect to the optical axis OA, but the projected image 14b is at the same location on the retina 30 of the eye 18 of the user 20 as shown in Figure 2. As can be seen in Figure 3, when the second optical device 24b is in the second position, the second conjugate point 34b of the optical system 22 is at the same location on the retina 30 of the eye 18 of the user 20 as shown in Figure 2, such as the orbit 36 ​​of the eye 18 of the user 20.

[0023] Processing circuitry 32a adjusts or modulates the brightness of display device 12. Processing circuitry 32a may increase or decrease the brightness of the display device. In some embodiments, processing circuitry 32a may decrease the brightness of display device 12 to zero. For example, if the display device comprises an LCD display, processing circuitry 32a may adjust or modulate the back illumination of the LCD display. Exemplary back illumination techniques are described in Wonbok Lee et al., "White-LED backlight control for motion-blur reduction and power minimization in large LCD TVs," Journal of the Society for Information Display, Vol. 17, Issue 1, January 2009, Pages 37-45. Preferably, in other embodiments, other methods or techniques may be used to adjust or modulate the brightness of the display device.

[0024] If display device 12 comprises an LED display, such as an OLED display or other LED display, the output of the LED display may be adjusted or modulated. Display device 12 may include a frame rate of approximately 30 frames per second. However, display devices disclosed herein are preferably not limited to this exemplary frame rate. For example, in other embodiments, the display device includes a frame rate lower or higher than 30 frames per second. Adjusting or modulating the brightness of display device 12 may reduce motion blur artifacts and / or smearing of images displayed on the display device, which may include "scan-and-hold" artifacts and / or other artifacts, and / or may improve the user's viewing comfort.

[0025] 3 also shows a tracking system 16. The tracking system 16 may image a portion of the eye 18. In this embodiment, the portion imaged by the tracking system 16 includes a retina 30 of the eye 18 of the user 20.

[0026] In this embodiment, the tracking system 16 includes an emitter 38 that illuminates the retina 30 of the eye 18 of the user 20. The emitter 38 emits light, such as near-infrared light. The near-infrared light may have a wavelength of approximately 820 nm. However, in other preferred embodiments, the near-infrared light may be at another wavelength suitable for illuminating the retina of the user's eye. The emitter 38 may include a light-emitting diode, a laser, or other light source. The emitter 38 may also be referred to as an infrared source.

[0027] In this embodiment, the tracking system 16 includes a sensor 40 that captures images of the retina 30 of the eye 18 of the user 20. The sensor 40 may be implemented as an infrared camera. The sensor 40 may be provided in the form of a charge-coupled device (CCD), a complementary metal-oxide-semiconductor (CMOS) imaging sensor, or other image sensor. The sensor 40 may be selected to capture images of the retina 30 of the eye 18 of the user 20 at high speed and low latency. For example, the sensor 40 may capture approximately 1000 image frames per second. The sensor 40 may also be referred to as an infrared detector or image sensor.

[0028] In this embodiment, the tracking system 16 includes one or more additional optical devices. The one or more additional optical devices may be positioned to direct an image of the retina 30 toward the sensor 40. For example, the one or more additional optical devices may include a reflector 42. The reflector 42 reflects light emitted by the emitter 38. For example, the reflector 42 may be configured to reflect only light having the same wavelength or wavelength band as the emitter 38, e.g., approximately the same wavelength or wavelength band. The reflector 42 may be provided in the form of a dielectric mirror or a Bragg mirror. The reflector 42 may also be referred to as a hot mirror.

[0029] The reflector 42 is positioned so that light emitted by the emitter 38 is reflected onto the retina 30. The reflector 42 is positioned so that an image of the retina 30 is reflected to the sensor 40. In the embodiment shown in FIG. 3, the reflector 42 is positioned so that the sensor 40 is located in a conjugate plane with the display device 12. For example, the center or central portion of the sensor 40 can define the third conjugate point 34c. The center or central portion of the sensor 40 is the third conjugate point to the first and second conjugate points 34a, 34b. In this embodiment, the reflector 42 is positioned so that it intersects the optical axis OA of the optical system 22. The reflector 42 is positioned at an angle β with respect to the optical axis OA. The angle β may be approximately 45 degrees.

[0030] In some embodiments, the one or more further optical devices may comprise a filter element (not shown in FIG. 3 ). The filter element may be positioned such that light or radiation reflected by the reflector 42 passes through the filter element before entering the camera. The transmission wavelength or wavelength band of the filter element may correspond to the wavelength or wavelength band of light emitted by the emitter. The above-described configuration of the reflector 42 and / or filter element may reduce or avoid reflection of light or radiation emitted by the display device 12 onto the sensor 40. This may effectively stabilize the projected image 14b on the retina 30.

[0031] In the embodiment shown in FIG. 3 , the one or more additional optical devices include a third optical device 44 that directs light emitted from the emitter 38 into the optical system 22 to illuminate the retina 30. The third optical device 44 may be positioned between the emitter 38 and the sensor 40. The third optical device 44 includes a beam splitter 44a, a diffuser 44b, and / or a lens 44c. The diffuser 44b and the lens 44c are positioned between the emitter 38 and the beam splitter 44a so that the diffuser 44b appears to be at infinity from the retina 30. This will prevent an image of the diffuser 44b from being formed on the sensor 40. Because the image of the diffuser 44b does not move, if the line of sight G of the eye 18 of the user 20 changes, the image of the diffuser 44b on the camera 40 will reduce the sensitivity of the tracking system 16.

[0032] The sensor 40 may be coupled or connected to the processing unit 32, for example, to transmit an image of the retina 30 from the sensor 40 to the processing unit 32. Based on the position of the image of the retina 30 on the sensor 40 (the image of the retina 30 captured by the sensor 40), the processing circuit 32a may determine the amount of movement to apply to the second optical device 24b. For example, as shown in FIG. 2, if the line of sight G of the eye 18 of the user 20 is along the optical axis OA, the image of the retina 30 on the sensor 40 will be at a first position. The first position of the image of the retina 30 on the sensor may correspond to the center or central portion of the sensor 40. For example, as shown in FIG. 3, if the line of sight G of the eye 18 of the user 20 changes, the image of the retina 30 on the sensor 40 will move to a second position on the sensor 40. The second position of the image of the retina 30 on the sensor 40 indicates a change in the line of sight G of the eye 18 of the user 20. Based on the position of the image of retina 30 on sensor 40, processing circuitry 32a may determine the amount of movement to apply to second optical device 24b to move the image of retina 30 on sensor 40 back from the second position to the first position. In this manner, processing circuitry 32a controls second optical device 24b so that the position of the image projected within eyeball 18 of user 20 does not change regardless of line of sight G of eyeball 18 of user 20.

[0033] In some embodiments, processing circuitry 32a determines a gaze location of eye 18 of user 20 on display device 12 based on line of sight G determined by tracking system 16. In Figure 3, the gaze location of user's eye on display device 12 corresponds to the center of display device 12, for example, when second optical device 24b is in the second position. For example, processing circuitry 32a may determine or calculate the gaze location based on an amount of movement applied to second optical device 24b.

[0034] The processing circuitry 32a may include an image stabilization algorithm or model that determines the amount of movement to apply to the second optical device 24b. The image stabilization algorithm or model may use cross-correlation to determine movement of the image of the retina 30 on the sensor 40. In some embodiments, movement of the image of the retina 30 on the sensor 40, for example from a first position to a second position, may be considered a deviation or error in the position of the image of the retina 30 on the sensor 40. The image stabilization algorithm or model may determine the amount of movement to apply to the second optical device 24b such that the deviation or error in the position of the image of the retina 30 on the sensor 40 is reduced. In this manner, the amount of movement to apply to the second optical device 24b may also be referred to as a correction amount to be applied to the second optical device 24b. The image stabilization algorithm or model may define a feedback or control loop with the sensor 40.

[0035] The processing unit 32 is connected to the optical system 22 and may, for example, send a first signal to the optical system 22. The first signal may indicate the amount of movement to be applied to the second optical unit 24b. The first signal may also be referred to as a correction signal.

[0036] In the embodiment shown in FIG. 3 , the optical system 22 includes an actuator 46. The processing circuitry 32a may send a first signal to the actuator 46. The actuator 46 moves the second optical device 24b based on the first signal from the processing circuitry 32a. The actuator 46 may include an electromagnetic shifting arrangement that moves the second optical device 24b. The electromagnetic shifting arrangement may include one or more electromagnets. Preferably, the optical system 22 may include additional parts or components that limit the range of movement of the second optical device 24b in a direction parallel to the display device 12 and / or in a direction perpendicular to the optical axis OA. The additional parts or components may include one or more bearings, such as one or more magnetic or mechanical bearings.

[0037] 4 schematically illustrates the portion of the head-mounted display 10 shown in FIG. 2. The tracking system 16 may determine the degree of focus of the eye 18 of the user 20. For example, the eye 18 of the user 20 may be focused on the display device 12, in front of the display device 12, or behind the display device 12. The determined degree of focus may represent, for example, the focus in three dimensions and the distance from the eye 18 of the user 20 to the focus. For example, the tracking system 16 may determine the degree of focus of the cornea and lens 28 of the eye 18 of the user 20. Note that the degree of focus is a concept that encompasses focal depth and focal length.

[0038] In this embodiment, processing circuitry 32a controls the focus of image 14b projected into eye 18 of user 20 based on the determined focus. For example, processing circuitry 32a controls the focus of projected image 14b based on the focus determined by tracking system 16. This may compensate for one or more changes in the focus of the cornea and lens 28 of eye 18 of user 20 and / or enable projected image 14b to be accurately focused on retina 30 of eye 18 of user 20.

[0039] The processing circuitry 32a controls the second optical device 24b based on the determined degree of focus. For example, the processing circuitry 32a controls the second optical device 24b to move in a direction parallel to the optical axis OA, such as approximately parallel to the optical axis OA. By moving the second optical device 24b in a direction parallel to the optical axis OA, the image 14b projected onto the retina 30 of the eye 18 of the user 20 will be moved to focus. In other words, the plane of the projected image 14b may be moved so that it lies on the retina 30 of the eye 18 of the user 20. The movement of the second optical device 24b may cause the intermediate image plane 26 and the intermediate image 14a to move, as shown in FIG. 4 .

[0040] The optical system 22, the tracking system 16, and / or the processing device 32 may define an autofocus optical system (not shown). The autofocus optical system may determine the degree of focus of the eye 18 of the user 20. For example, in some embodiments, the autofocus optical system may determine the degree of focus by analyzing one or more spatial frequencies present in the image of the retina 30 on the sensor 40. One or more maximum spatial frequencies may be detected in the image of the retina 30 on the sensor 40 when the cornea and lens 28 precisely focus the projected image 14b onto the retina 30 of the eye 18 of the user 20. If the projected image 14b on the retina 30 falls out of focus, for example, due to a change in the degree of focus of the cornea and lens 28 of the eye 18 of the user 20, the image of the retina 30 on the sensor 40 will be blurred, and the spatial frequencies in the image of the retina 30 on the sensor 40 will change, e.g., decrease.

[0041] The autofocus optical system may determine a direction of movement of the second optical device 24b, e.g., the intermediate image 14a, based on the spatial frequency in the image of the retina 30 on the sensor 40. For example, the autofocus optical system may determine a slope of change in spatial frequency as a function of movement of the second optical device 24b.

[0042] In some embodiments, the autofocus optical system may determine a direction of movement of the second optical device 24b, such as the intermediate image 14a, based on the image of the retina 30 on the sensor 40. For example, the image of the retina 30 on the sensor 40 will appear different depending on where the eye 18 of the user 20 is focused, such as on the display device 12, in front of the display device 12, or behind the display device 12. In this manner, the autofocus optical system may determine a direction of movement of the second optical device 24b based on the appearance of the image of the retina 30 on the sensor 40.

[0043] The above-mentioned direction of movement of the second optical device 24b, e.g., of the intermediate image 14a, may be understood as a direction, e.g., parallel to the optical axis OA, toward or away from the eyeball 18 of the user 20. The processing circuitry 32a controls the direction of movement of the second optical device 24b based on the determined direction of movement.

[0044] In other embodiments, the autofocus optical system may adjust the focal plane based on the convergence of the user's eyes, as described below with respect to FIG. 13 . In such embodiments, the autofocus optical system may determine the direction of movement of the second optical device 24b, such as the intermediate image 14a, based on the subtended angle between a first line of sight of a first eye of the user 20 and a second line of sight of a second eye of the user 20. This angle may also be referred to as the convergence angle. For example, when the first and second eyes of the user 20 are focused on a point in front of the display device 12, the convergence angle will be larger than when the first and second eyes of the user 20 are focused on a point on or behind the display device 12. As such, when the convergence angle is larger, the processing circuit 32a controls the direction of movement of the second optical device 24b so that the intermediate image plane 26 moves closer to the first and second eyes of the user 20. Alternatively, when the convergence angle decreases, the processing circuit 32 a controls the direction of movement of the second optical device 24 b so that the intermediate image plane 26 moves in a direction away from the first and second eyes of the user 20 .

[0045] In still other embodiments, the autofocus optical system may use a phase detection autofocus system, a dual pixel autofocus system, or another autofocus system.

[0046] 3 moves the second optical device 24b in a direction parallel to the optical axis OA. However, preferably in other embodiments, the actuator may be implemented differently, or a different actuator may move the second optical device. In such other embodiments, the actuator or other actuator may comprise a piezoelectric motor, such as an electric motor or an ultrasonic motor, or the like.

[0047] Preferably, in some embodiments, second optical device 24b may only move in a direction perpendicular to the optical axis. In such embodiments, the optical system may further comprise another optical device, such as at least one other lens, that may be configured to move in a direction parallel to the optical axis to focus the projected image onto the retina of the user's eye. Second optical device 24b or other optical devices may each be referred to as an adjustable focusing component.

[0048] The processing circuitry 32a may determine an amount of further movement to apply to the second optical device 24b or other optical devices based on the determined degree of focus. For example, the processing circuitry 32a may include an autofocus algorithm or model that determines an amount of further movement to apply to the second optical device 24b or other optical devices based on the determined degree of focus. The first signal may indicate an amount of further movement to apply to the second optical device 24b or other optical devices.

[0049] Figure 5 schematically shows a portion of another example head-mounted display 10. The portion of the head-mounted display 10 shown in Figure 5 is similar to that shown in Figure 2. Any of the features described above may also be applied to the head-mounted display 10 shown in Figure 5. Only the differences will be described below.

[0050] In this embodiment, the display device 12 including the display panel 13 is implemented as a foveated display. The display device 12 displays the image 14 such that the resolution of the first region 14c of the image 14 is higher than the resolution of the second region 14d of the image 14. The display device 12 may include a first region 12a that displays the first region 14c of the image 14. The display device 12 may include a second region 12b that displays the second region 14d of the image 14. The first region 12a of the display device 12 may also be referred to as the central or foveal region of the display device 12. The second region 12b of the display device 12 may also be referred to as the peripheral region of the display device 12.

[0051] In some embodiments, the pixel density in the first region 12a of the display device 12 is higher than the pixel density in the second region 12b of the display device 12. For example, the display device 12 may have variable spacing between pixels of the display device. The spacing between pixels in the first region 12a of the display device 12 may be smaller compared to the spacing between pixels in the second region 12b of the display device 12.

[0052] In some embodiments, the head mounted display 10 includes rendering circuitry 48. The rendering circuitry 48 renders a first region 14c of the image 14 using a first resolution and a second region 14d of the image 14 using a second resolution that is lower than the first resolution. This is sometimes referred to as foveated rendering.

[0053] In other embodiments, the display device includes a first viewing panel and a second viewing panel combined with a reflective element. For example, the reflective element may be positioned as part of the first viewing panel. The reflective element may be centrally positioned on the first viewing panel. The second viewing panel may be positioned such that the image displayed by the second viewing panel is visible on the reflective element. However, in other preferred embodiments, the first and second viewing panels and the reflective element may be positioned differently.

[0054] 5, the line of sight G of the eye 18 of the user 20 is along the optical axis OA of the optical system 22. The optical system 22 projects the first region 14c of the image 14 onto a central portion, such as the orbit 36, of the retina 30 of the eye 18 of the user 20. For example, the optical system 22 is configured, as described above, such that the center of the first region 12a of the display device 12 and the central portion of the retina 30, such as the orbit 36 ​​of the eye 18 of the user 20, are the first and second conjugate points 34a, 34b of the optical system 22. The orbit 36 ​​has a higher resolution than the periphery of the retina 30 and will therefore benefit most from the higher resolution of the first region 14c of the image 14.

[0055] Figure 6 shows schematically the portion of the head mounted display 10 shown in Figure 5 at different lines of sight G of the eye 18 of the user 20. For example, the lines of sight may be at an angle α with respect to the optical axis OA. In the embodiment shown in Figure 6, the user 20 may be considered to be looking up. However, preferably in other embodiments, the user may look in other directions, such as down, left, right, or any combination of up, down, left, right, etc.

[0056] The processing circuit 32a controls the second optical device 24b based on the line of sight G of the eye 18 of the user 20 so that the position of the first region 14e of the image projected onto the retina 30 of the eye 18 of the user 20 does not change when the line of sight G of the eye 18 of the user 20 changes. This may align the first region 14c of the image 14 with the line of sight G so that, for example, the center of the first region 14c of the image 14 and the center of the retina 30, such as the orbit 36, are conjugate points of the optical system 22. In this way, the center of the field of view of the user 20 may be aligned with the first region 12a of the display device 12.

[0057] By configuring the processing circuit 32a to control the second optical device 24b based on the line of sight G of the eye 18 of the user 20 so that the position of the first region 14e of the image 14b projected onto the retina 30 of the eye 18 of the user 20 does not change when the line of sight G of the eye 18 of the user 20 changes, the user 20 will experience a display device at a high resolution and a high frame rate while allowing for a reduction in the computing power and / or bandwidth of the head-mounted display compared to a head-mounted display that displays the entire image at high resolution.

[0058] As described above, the processing circuit 32a controls the second optical device 24b based on the line of sight G of the eyeball 18 of the user 20, for example, to move the second optical device 24b upward. Referring to FIG. 6 , when the second optical device 24b is in the first position, the line of sight G is at an angle α with respect to the optical axis OA, so that the first region 14e of the projected image 14b is not centered on the orbit 36 ​​of the eyeball 18 of the user 20. However, by moving the second optical device 24b to the second position, for example, as shown in FIG. 6 , the first region 14e of the projected image 14b returns to the center of the retina 30, for example, the orbit 36, even though the line of sight G of the eyeball 18 of the user 20 is at an angle α with respect to the optical axis OA.

[0059] 7 shows a schematic representation of the portion of the head-mounted display 10 shown in FIG. 5 at different lines of sight G of the eyes 18 of the user 20. In this embodiment, the user 20 views different portions of the image 14 displayed by the display device 12. The different portions of the image 14 may also be referred to as focal portions of the image 14. The focal portion of the image 14 is displayed by the second region 12b of the display device 12. The position of the focal portion of the image 14 on the display device 12 is indicated by a dotted line in FIG. 7.

[0060] As shown by the dotted outline of the second optical device 24b in FIG. 7 , when the second optical device 24b is in the first position, the first conjugate point 34a of the optical system 22 is shifted relative to the center of the first region 12a of the display device 12. As described above, the processing circuitry 32a controls the second optical device 24b to move to the second position, as shown by the solid outline of the second optical device 24b in FIG. 7 . By moving the second optical device 24b from the first position to the second position, the first conjugate point 34a returns to the center of the first region 12a of the display device 12. However, the portion of interest of the image 14 remains in the first position on the display device 12, as shown by the dotted line in FIG. 7 .

[0061] Figure 7 also shows, in solid lines, a second position of the portion of interest of image 14 on display device 12. A difference D between the first and second positions of the portion of interest of image 14 on display device 12 is shown in Figure 7. The difference D between the first and second positions of image 14 is sometimes referred to as the amount of pan applied.

[0062] As described above, the processing circuitry 32a determines a gaze location of the eye 18 of the user 20 on the display device 12 based on the line of sight G determined by the tracking system 16. In this example, when the second optical device 24b is in the first position, the gaze location is indicated by the first conjugate point 34a and is shifted relative to the center of the display device 12. In other words, the gaze location may be considered to be a location on the display device 12 that displays the portion of interest of the image 14. As described above, the processing circuitry 32a may determine or calculate the gaze location based on the amount of movement applied to the second optical device 24b.

[0063] As described above, the head-mounted display 10 may include a rendering circuit 48. The rendering circuit 48 may render the image 14 based on the determined gaze location. The rendering circuit 48 may change the position of the image 14 on the display device 12. In this embodiment, the rendering circuit 48 applies a pan amount to the image 14 such that a focal portion of the image 14 moves from a first position to a second position on the display device 12. The rendering circuit 48 may determine a pan amount to apply to the image 14 based on the determined gaze location. The rendering circuit 48 may apply the determined pan amount to the image 14 such that the focal portion of the image 14 corresponds to the first region 14c of the image 14 and / or is displayed by the first region 12a of the display device 12. A change in the gaze G of the eyeball 18 of the user 20 may change the focal portion of the image 14. By configuring a rendering circuit to render image 14 based on a gaze location determined based on line of sight G so that a portion of image 14 of interest corresponds to first region 14c of image 14 and / or is displayed by first region 12a of display device 12, rendering circuit 48 will create the appearance that user 20 is viewing a large display with high resolution when user 20 moves his / her eyes 18 to view image 14 in detail.

[0064] Figure 8 shows a schematic representation of the portion of the head-mounted display 10 shown in Figure 5 at different lines of sight G of the eyes 18 of the user 20, and the tracking system 16. The portion of the head-mounted display 10 shown in Figure 8 is similar to that shown in Figure 3. Any of the features described above may also be applied to the head-mounted display shown in Figure 8. Only the differences will be described below.

[0065] 8 also shows rendering circuitry 48. Rendering circuitry 48 may render image 14 on display device 12 based on the determined gaze location, as described above. Processing circuitry 32a may send a second signal to rendering circuitry 48. The second signal may be indicative of the determined gaze location. Rendering circuitry 48 may determine a pan amount to apply to image 14 based on the second signal. Rendering circuitry 48 may render image 14 based on the determined pan amount to apply to image 14, as described above. Rendering circuitry 48 may send a third signal to display device 12. The third signal may be indicative of the rendered image.

[0066] Figure 9 shows a schematic diagram of a portion of the head mounted display 10 shown in Figure 5. The portion of the head mounted display 10 shown in Figure 9 is similar to that shown in Figure 4. Any of the features described above may also be applied to the head mounted display 10 shown in Figure 9. Only the differences will be described below.

[0067] In this embodiment, the second signal also indicates the determined degree of focus of the eye 18 of the user 20. The rendering circuitry 48 renders the image 14 based on the second signal. For example, the rendering circuitry 48 may be configured such that one or more portions of the first region 14c of the image 14 that are projected within the eye 18 of the user 20 and near the focal point of the eye 18 of the user 20 appear sharper than one or more other portions of the first and / or second regions 14c, 14d of the image 14 that are away from the focal point of the eye 18.

[0068] 1 to 9 show a head mounted display 10 for one eye 18 of a user 20. However, preferably, the head mounted display 10 may include parts for the other eye of the user 20.

[0069] Figure 10 shows a schematic representation of another embodiment of a head mounted display 10. The head mounted display shown in Figure 10 is similar to that described above with respect to Figure 1. Any of the features described above may also be applied to the head mounted display shown in Figure 10. Only the differences will be described below.

[0070] The tracking system 16 may determine the line of sight of each eye of the user 20. In this embodiment, the tracking system 16 includes a first portion 16a that determines the line of sight G1 of a first eye 18a of the user 20. The tracking system 16 includes a second portion 16b that determines the line of sight G2 of a second eye 18b of the user 20. Each of the first and second portions 16a, 16b of the tracking system 16 may include an emitter 38, a sensor 40, a reflector 42, and a third optical device 44, as described above with respect to FIGS.

[0071] In this embodiment, the display device 12 includes a first display panel 13a that displays a first image 14f. The display device 12 includes a second display panel 13b that displays a second image 14g. The first and second images 14f, 14g displayed on the first and second display panels 13a, 13b, respectively, may be different or the same. In this embodiment, the first and second display panels 13a, 13b are each implemented as a foveated display panel, as described above with reference to FIGS. 5-9. However, preferably in other embodiments, the first and second display panels may each be implemented similarly to the display panel 13 of the display device 12 described with reference to FIGS. 1-4.

[0072] The optical system 22 may include a first portion 22a and a second portion 22b. Each of the first and second portions 22a, 22b of the optical system 22 may include a first optical device 24a, a second optical device 24b, and / or other optical devices as described above. The first portion 22a of the optical system 22 projects a first image 14f displayed on the first display panel 13a into the first eyeball 18a of the user 20. The second portion 22b of the optical system 22 projects a second image 14g displayed on the second display panel 13b into the second eyeball 18b of the user 20. The head-mounted display 10 shown in FIG. 10 may also be referred to as a head-mounted stereoscopic display.

[0073] Figure 11 shows a schematic representation of another embodiment of a head mounted display 10. The head mounted display shown in Figure 11 is similar to that described above with respect to Figure 10. Any of the features described above may also be applied to the head mounted display shown in Figure 11. Only the differences will be described below.

[0074] In this embodiment, display device 12 includes a single display panel 13. Display device 12, including display panel 13, is implemented as a foveated display device, as described above with respect to Figures 5-9. However, in other preferred embodiments, the display device may be implemented similarly to display device 12 described with respect to Figures 1-4, respectively.

[0075] In this embodiment, the first and second portions 22a, 22b of the optical system 22 project the image 14 displayed on the display panel 13 into the first and second eyes 18a, 18b, respectively, of the user 20. The head-mounted display 10 shown in Figure 11 is sometimes referred to as a head-mounted binocular display.

[0076] 12 shows an example in which the first and second eyeballs 18a and 18b of the user 20 look in the same direction. The first and second eyeballs 18a and 18b of the user 20 can be considered to be looking at a focal point at a distance.

[0077] 13 illustrates an example in which the first and second eyes 18a, 18b of the user 20 look in different directions. The tracking system 16 may determine a subtended angle γ (the angle γ between the gaze G1 and the gaze G2) for the first and second eyes 18a, 18b of the user 20 based on the gazes G1, G2 of the first and second eyes 18a, 18b of the user 20. The subtended angle γ for the first and second eyes 18a, 18b of the user 20 may indicate the convergence of the first and second eyes 18a, 18b of the user 20. The tracking system 16 may determine the degree of focus of each of the first and second eyes 18a, 18b based on the determined angle γ. As described above with respect to FIGS. 4 and 9 , the optical system 22, for example, focuses the first and second eyes 18a, 18b thereof based on the determined focus of the projected first image, the projected second image, and / or the projected image. This may allow the first image 14f, the second image 14g, and / or the image 14 to be accurately focused within the first and second eyes 18a, 18b of the user 20 and / or may provide a more natural viewing experience for the user 20. For example, the first and second lines of sight G1, G2 of the first and second eyes 18a, 18b of the user 20 may be different in a head-mounted binocular display. This is because the user 20 may choose to view the image 14 on the display panel 13 at an apparent position other than infinity, for example, at an apparent distance of one meter. In such an example, the first and second lines of sight G1, G2 converge to each other as shown in FIG. 13 .

[0078] As described above, the processing circuitry 32a may determine a first gaze location of the first eye 18a of the user 20 and a second gaze location of the second eye 18b of the user 20 based on the first and second gazes G1, G2 at the first and second display panels 13a, 13b, respectively, of the head-mounted display 10 shown in Fig. 10 or at the display panel 13 of the head-mounted display 10 shown in Fig. 11. For example, the processing circuitry 32a may control focusing of the projected first image and the projected second image based on the focus degree determined by the tracking system 16.

[0079] The rendering circuitry 48 may render a first image 14f on a first display panel 13a of the head mounted display 10 shown in Figure 10 and / or a second image 14g on a second display panel 13b, or may render an image 14 on a display panel 13 of the head mounted display 10 shown in Figure 11 as described above based on the determined focus and / or first and second gaze locations. This may provide a stereoscopic or binocular view to the user 20.

[0080] 10 and 11, the head-mounted display 10 includes a sensor 50 for detecting movement of the head 52 of the user 20, the attitude or position of the head 52 of the user 20, and / or changes in the attitude or position of the head 52 of the user 20. For example, the sensor 50 may be implemented as a gyro sensor. Preferably, in other embodiments, one or more other sensors, such as one or more accelerometers, may be used in addition to or instead of the gyro sensor. The sensor 50 may detect the roll, pitch, and / or yaw of the head 52 of the user 20.

[0081] 14 schematically illustrates exemplary movements of the head 52 of the user 20. If the user 20 is interested in the first image 14f, the second image 14g, and / or a portion 14h of the image 14 that is not part of the first image 14f, the second image 14g, and / or the first region 14c of the image 14, the user 20 will move the first eye 18a, the second eye 18b, and / or their head 52. This first image 14f, the second image 14g, and / or the portion 14h of the image 14 will be referred to hereinafter as the portion of interest 14h.

[0082] In this embodiment, processing circuitry 32a may determine a first gaze location of a first eye 18a of user 20 and a second gaze location of a second eye 18b of user 20 at first and second display panels 13a, 13b, respectively, of head-mounted display 10 shown in Fig. 10 or at display panel 13 of head-mounted display 10 shown in Fig. 11 based on the first and second gazes G1, G2 and based on at least one of a detected movement of head 52 of user 20, a detected pose or position of head 52 of user 20, and a detected change in pose or position of head 52 of user 20. Rendering circuitry 48 renders first image 14f, second image 14g, and / or image 14 based on the determined first and second gaze locations to provide user 20 with, for example, a stereoscopic or binocular view of first image 14f, second image 14g, and / or portion of interest 14h of image 14.

[0083] In embodiments in which the display device 12 and / or the first and second display panels 13a, 13b are implemented as foveated display devices, the rendering circuitry 48 may render the first image 14f, the second image 14g, and / or the image 14 based on the determined first and second gaze locations so that the first image 14f, the second image 14g, and / or the focal portion 14h of the image 14 is displayed on the first region 12a of the display device 12 and / or corresponds to the first image 14f, the second image 14g, and / or the first region 14c of the image 14.

[0084] Configuring the rendering circuitry 48 to render the first image 14f, the second image 14g, and / or the image 14 based on the determined first and second gaze locations may provide an immersive and / or more natural viewing experience for the user 20.

[0085] The sensor 50 may transmit a fourth signal to the rendering circuitry 48. The fourth signal may be indicative of a detected movement of the head 52 of the user 20, a detected pose or position of the head 52 of the user 20, and / or a detected change in the pose or position of the head 52 of the user 20. The rendering circuitry 48 may render the first image 14f, the second image 14g, and / or the image 14 based on the fourth signal.

[0086] Figure 15 shows a schematic diagram of a system 54 including a head mounted display 10. The head mounted display 10 shown in Figure 15 may include any of the features of the head mounted display 10 described above with respect to Figure 10. In this embodiment, the system 54 comprises a telepresence system. However, preferably in other embodiments, the system may comprise a virtual reality system, a mixed reality system, or an augmented reality system.

[0087] The system 54 includes at least one camera positioned remotely relative to the head-mounted display 10. The camera may be implemented as a foveated camera. For example, the camera may include an image sensor having a first region with a first resolution and a second region with a second resolution. The first resolution may be higher than the second resolution. The first region of the image sensor may correspond to a central or foveal region of the image sensor. The second region of the image sensor may correspond to a peripheral region of the image sensor.

[0088] 15, the system 54 comprises a first camera 56a and a second camera 56b, but preferably in other embodiments the system may comprise more or less than two cameras.

[0089] The first and second cameras 56a, 56b may be part of a device 58 that is operated remotely from the head mounted display 10. The first and second cameras 56a, 56b may be part of a stereoscopic camera platform 60 of the device 58.

[0090] The first camera 56a may be associated with a first eye 18a of the user 20. The second camera 56b may be associated with a second eye 18b of the user 20. As described above, the display device 12 may include a first display panel 13a and a second display panel 13b. The first display panel 13a may display a first image 14f captured by the first camera 56a. The second display panel 13b may display a second image 14g captured by the second camera 56b.

[0091] System 54 includes a control system 62. Control system 62 may be part of head mounted display 10. Control system 62 may include processing circuitry, which may include any of the features of processing circuitry 32a described above.

[0092] The head mounted display 10 may be connected to the device 58, for example, via a wired or wireless connection. This may allow for signal transmission between the head mounted display 10 and the device 58. For example, the device 58 may transmit a first signal to the head mounted display 10. The first signal may be indicative of first and second images 14f, 14g captured by the first and second cameras 56a, 56b, respectively. The head mounted display 10 may transmit a second signal to the device 58. The second signal may be indicative of the position or attitude, movement, and / or focus of at least one or each of the first and second cameras 56a, 56b. The second signal may also be indicative of the position, attitude, and / or orientation of the device 58, such as the stereoscopic camera platform 60.

[0093] The control system 62 may control at least one or each of the first and second cameras 56a, 56b based on the first and / or second lines of sight G1, G2 of the first and / or second eyes 18a, 18b of the user 20 and / or based on a determined focus of at least one or both of the first and second eyes 18a, 18b of the user 20. For example, the first camera 56a may be controlled based on the first line of sight G1 of the first eye 18a of the user 20 and / or based on a determined focus of the first eye 18a of the user 20. The second camera 56b may be controlled based on the second line of sight G2 of the second eye 18b of the user 20 and / or based on a determined focus of the second eye 18b of the user 20. The control system 62 may control the position or attitude, movement, and / or focus of at least one or both of the first and second cameras 56a, 56b based on the first and / or second line of sight G1, G2 of the first and / or second eyeballs 18a, 18b of the user 20 and / or based on the determined focus of at least one or both of the first and second eyeballs 18a, 18b of the user 20.

[0094] The control system 62 may control the position, attitude, and / or orientation of the device 58, such as the stereoscopic camera platform 60, based on the determined movement of the head 52 of the user 20, the determined pose or position of the head 52 of the user 20, and / or changes in the pose or position of the head 52 of the user 20. In this manner, the position, attitude, and / or orientation of the first and / or second cameras 56a, 56b may be controlled by the control system 62 based on the determined movement of the head 52 of the user 20, the determined pose or position of the head 52 of the user 20, and / or changes in the pose or position of the head 52 of the user 20.

[0095] System 54 could use device 58 to explore hazardous environments that may be unsafe for the user. Alternatively, device 58 may be part of a remotely operated robotic device, facilitating telepresence. In yet other embodiments, device 58 may be part of a robotic medical or surgical device, such as an endoscope, for exploring portions of an organ in a patient or other subject. Preferably, system 54 is not limited to the applications or uses disclosed herein and may be used or adapted in other devices or systems.

[0096] One embodiment provides a head-mounted display for use in a virtual reality system, the head-mounted display including: an eye-tracking system for determining a line of sight of a user's eye; a foveated display for displaying images using a first higher resolution in a central region of the foveated display and a second lower resolution in a peripheral region of the foveated display; and an optical system for projecting an image of the foveated display into the user's eye, the optical system including an adjustable imaging component for adjusting a projection direction of the image depending on the determined line of sight.

[0097] The adjustable imaging component may include a movable lens.

[0098] The gaze tracking system may include an infrared-sensitive camera, a hot mirror that positions the infrared-sensitive camera at a conjugate plane with the foveated display, an infrared source that illuminates a retina of the user's eye so that an image of the retina is formed on the infrared-sensitive camera, and processing circuitry that determines movement of the image of the retina.

[0099] The head-mounted display may further comprise an image stabilization circuit that controls the adjustable imaging component to maintain a fixed position of the image of the retina relative to the infrared-sensitive camera.

[0100] The head-mounted display may further include a rendering circuit for rendering a foveated image for display on the foveated display, the rendering circuit may render the foveated image from a 3D scene using the determined line of sight such that the view presented to the user changes with changes in line of sight.

[0101] The eye-tracking system may determine a degree of focus of the user's eyes, and the optical system may include an adjustable focusing component that adjusts the focus of the projected image in response to the determined degree of focus.

[0102] The rendering circuitry may render the foveated image using the determined focus such that the view presented to the user changes with changes in focus.

[0103] An embodiment provides a head mounted display comprising a first instance and a second instance of a head mounted display according to one or more of the embodiments described above, each of the first and second instances being adapted to provide a stereoscopic effect with respect to a respective eye of a user.

[0104] The gaze tracking system of the first instance may determine the gaze of a first eye of the user, the gaze tracking system of the second instance may determine the gaze of a second eye of the user, and one or both gaze tracking systems may determine a subtended angle between the first eye and the second eye.

[0105] The adjustable focusing component may adjust the focus of the projected image according to the determined subtended angles for the first and second eyes.

[0106] The rendering circuitry may render a pair of foveated images such that a stereoscopic view is presented to a user.

[0107] The head mounted display may further comprise a head motion tracking system for determining the movement and / or posture and / or posture changes of the user's head, the head motion tracking system comprising a gyro sensor for determining the yaw, tilt and roll of the head.

[0108] The rendering circuitry may render the foveated image dependent on the movement and / or pose and / or pose changes of the user's head as determined by the head motion tracking system.

[0109] In one embodiment, a telepresence and / or mixed reality system is provided, comprising a head-mounted display according to one or more of the above-described embodiments, a pair of cameras located remotely relative to the head-mounted display, and a control system for controlling movement of the pair of cameras depending on the line of sight of the first eye, the line of sight of the second eye, and the determined movement and / or posture and / or posture changes of the head.

[0110] A central region of the foveated display may have a higher density of display pixels than a peripheral region of the foveated display.

[0111] The foveated display may comprise two displays combined using a mirror.

[0112] The image displayed by the foveated display may be rendered using a first, higher rendering resolution in the central region of the foveated display and a second, lower rendering resolution in the peripheral region of the foveated display.

[0113] One embodiment provides a head-mounted display comprising a first optical device, a foveated display, a second optical device, an eye-tracking device, and a processing circuit that controls the second optical device by shifting based on movements of a user's eyes tracked by the eye-tracking device.

[0114] The second optical device may be adjusted to image an image onto the user's retina, and the image may be rendered centrally on the foveated display.

[0115] The foveated display may have a first region having a first resolution and a second region surrounding the first region, the second region having a second resolution lower than the first resolution.

[0116] The head-mounted display may further include an infrared source that emits infrared rays, a hot mirror that reflects the infrared rays to the user's retina, and an infrared detector that detects the infrared rays reflected by the user's retina. The gaze tracking device may detect the user's gaze based on the infrared rays detected by the infrared detector.

[0117] One embodiment provides a head-mounted display comprising: a foveated display, wherein the resolution of an image generated at the center of the display is higher than the resolution of the image at the periphery of the display; an optical system capable of projecting an image of the foveated display onto the retina of an eye; an adjustable imaging component within the optical system such that the center of the display is projected onto the center of the retina at any given line of sight of the eye; and an eye-tracking system capable of determining the line of sight of the eye, and adjusting the optical system so that the center of the foveated display remains imaged onto the center of the retina regardless of changes in line of sight.

[0118] The adjustable imaging component of the optical system may include a lens that can be shifted laterally by electronic means. The optical system may further include a hot mirror that positions an infrared-sensitive camera at a conjugate plane with the foveated display. An infrared source may illuminate the retina such that an image of the retina is formed on the infrared camera. The gaze tracking system may include an image processing component that can determine how much the image of the retina on the camera has moved relative to a previous position, so that an image stabilization algorithm can respond by shifting the lens as necessary to compensate for changes in gaze so that the image of the retina remains in a fixed position.

[0119] The head-mounted display may include a rendering system capable of rendering a foveated image for display from the 3D scene when the line of sight is within the scene, and the rendering system may display the foveated image for a current line of sight such that the view presented to the user changes as the line of sight changes.

[0120] An embodiment provides a head mounted binocular display system in which the head mounted display according to one or more embodiments is replicated for each eye of a user to create a stereoscopic effect.

[0121] The system may be configured such that the line of sight of each eye is determined, the subtended angle is used to determine a focal length, and an adjustable focusing component is set according to the focal length to correct the corneal focus of the eye for the current line of sight of the eyes.

[0122] The system may further include a rendering system capable of rendering a foveated image in 3D based on the given line of sight and focus, and the rendering system may display a foveated image for the current line of sight and each eye focus so as to present the user with a stereoscopic view that changes with changes in line of sight or focus.

[0123] The system may further comprise means for determining movement of the head to which the display is attached, the yaw, tilt and roll of the head being determined by a gyro sensor.

[0124] The system may further include a rendering system capable of rendering a foveated image within a 3D scene based on the given gaze, head pose, and focus, and the rendering system may display the foveated image for the current head pose, gaze, and respective eye focus so as to present the user with a stereoscopic view that changes with changes in head pose, gaze, or focus.

[0125] The system may further include a stereoscopic camera platform, wherein the head pose can be used to control the camera platform, the gaze and focus can be used to control the pose and focus of individual cameras corresponding to each eye, and images from each camera can be displayed on a foveated display corresponding to that eye, such that a user can remotely control the camera platform using their head pose and gaze and remotely view the scene as if they were present.

[0126] Although particular circuits are described herein, in alternative embodiments, the functionality of one or more of these circuits may be provided by a single processing resource or other component, or the functionality provided by a single circuit may be provided by a combination of two or more processing resources or other components. A reference to a single circuit encompasses multiple components that provide the functionality of that circuit, whether or not such components are separate from one another. A reference to multiple circuits encompasses a single component that provides the functionality of those circuits.

[0127] Although several embodiments have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, and combinations of embodiments can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, as well as within the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0128] 10 Head-mounted display 12 Display device 12a First Area 12b Second Region 13 Display panel 13a First display panel 13b Second display panel 14 images 14a Intermediate image 14b Projected image 14c First Region 14d Second Area 14e The First Realm 14f First image 14g Second image 14h Featured Part 16 Tracking System 16a First Section 16b Second Part 18 Eyeball 18a First Eye 18b Second Eye 20 users 22 Optical System 22a First Part 22b Second Part 24a First optical device 24b Second optical device 26 Intermediate image plane (intermediate plane) 28 crystalline lens 30 Retina 32 Processing equipment 32a Processing circuit 34a First conjugate point 34b Second conjugate point 34c Third conjugate point 36 Eye socket 38 Emitter 40 sensors 42 Reflector 44 Third Optical Device 44a Beam splitter 44b Diffuser 44c lens 46 Actuator 48 Rendering Circuit 50 sensors 52 Head 54 Systems 56a First Camera 56b Second Camera 58 Equipment 60 Stereoscopic Camera Platform 62 Control System D difference G line of sight G1 First Gaze G2 Second Gaze α, β, γ angles

Claims

1. a display device for displaying an image; a tracking system for determining the user's eye gaze; an optical system including an adjustable optical device for projecting the image into the user's eye; a processing circuit that controls the adjustable optical device based on the line of sight of the user's eye so that the position of the image projected within the user's eye does not change when the line of sight of the user's eye changes; 1. A head-mounted display for use in a virtual reality, mixed reality, or augmented reality system, comprising:

2. the display device displays the image such that a first region of the image has a higher resolution than a second region of the image; the optical system projects the first region of the image onto a central portion of a retina of the user's eye; the processing circuit controls the adjustable optical device based on the line of sight of the user's eyeball such that a position of the first region of the image projected onto a retina of the user's eyeball does not change when the line of sight of the user's eyeball changes. The head-mounted display according to claim 1 .

3. 3. The head-mounted display of claim 2, wherein the display device comprises a first region that displays the first region of the image and a second region that displays the second region of the image, the first region of the display device corresponding to a central or foveal region of the display device, and the second region of the display device corresponding to a peripheral region of the display device.

4. the processing circuitry determines a gaze location of the user's eye on the display device based on the line of sight of the user's eye; the head-mounted display including a rendering circuit for rendering the image displayed on the display device based on the determined gaze location. The head-mounted display of claim 1 .

5. The tracking system includes: an emitter for illuminating a portion of the user's eye; a sensor that captures an image of the portion of the eye of the user; one or more further optical devices positioned to direct the image of the portion of the user's eye towards the sensor; The head mounted display of claim 1 , comprising at least one of:

6. the adjustable optical device comprises a floating or movable lens; the processing circuitry determines an amount of movement to apply to the adjustable optical device based on a position of the image of the portion of the user's eye captured by the sensor. The head mounted display according to claim 5 .

7. the processing circuitry determines a gaze location of the user's eyes on the display device based on the amount of movement applied to the adjustable optical device. The head mounted display according to claim 6.

8. The tracking system determines the degree of focus of the user's eyes; the processing circuitry controls the focus of the image projected into the user's eyeball based on the determined focus degree. The head-mounted display according to claim 1 .

9. the tracking system determines a line of sight for each eye of the user, the tracking system comprising a first portion that determines a first line of sight of a first eye of the user, and a second portion that determines a second line of sight of a second eye of the user. The head-mounted display according to claim 1 .

10. The display device includes: a first display panel that displays a first image and a second display panel that displays a second image; or a single display panel for displaying the image; The head-mounted display according to claim 1 .

11. The optical system comprises: a first portion that projects a first image into a first eye of the user and a second portion that projects a second image into a second eye of the user; or a first portion that projects the image into the first eye of the user, and a second portion that projects the image into the second eye of the user; The present invention provides one of the following: The head-mounted display according to claim 1 .

12. the tracking system determines an angle between the first and second lines of sight of the first and second eyes of the user; The head mounted display of claim 9.

13. the tracking system determines a degree of focus of each of the user's first and second eyes based on the determined angles; the processing circuitry controls focusing of the projected first image, the projected second image, and / or at least one of the projected images based on the determined degree of focus. The head mounted display of claim 12.

14. The processing circuitry determining a first gaze location of the first eye of the user on a first display panel displaying a first image and determining a second gaze location of the second eye of the user on a second display panel displaying a second image; or determining a first gaze location of the first eye of the user and a second gaze location of the second eye of the user on a single display panel; The head mounted display of claim 13.

15. a rendering circuit configured to render at least one of the first image, the second image, and / or the images based on the first and second gaze locations and / or the determined degree of focus; 15. A head mounted display according to claim 14.

16. the head mounted display comprises a sensor for detecting at least one of a movement of the user's head, a position of the user's head, and a change in the position of the user's head; the processing circuitry determines the first gaze location of the first eye of the user on the first display panel and the second gaze location of the second eye of the user on the second display panel, or the first gaze location of the first eye of the user and the second gaze location of the second eye of the user on the single display panel based on at least one of the detected movement of the user's head, the detected position of the user's head, and the detected change in position of the user's head.

15. A head mounted display according to claim 14.

17. the image comprises an image of a two-dimensional or three-dimensional scene; The head-mounted display according to claim 1 .

18. a pixel density in the first region of the display device is greater than a pixel density in the second region of the display device; The head-mounted display according to claim 2 .

19. the processing circuitry determines a gaze location of the user's eye on the display device based on the line of sight of the user's eye; the head mounted display comprising a rendering circuit for rendering the image displayed on the display device based on the determined gaze location; the rendering circuitry renders a first portion of the image using a first resolution and a second portion of the image using a second resolution, the second resolution being lower than the first resolution; The head-mounted display according to claim 2 .

20. The display device comprises two display panels combined with a reflective element. The head-mounted display according to claim 2 .

21. the processing circuitry adjusts or modulates the brightness of the display device; The head-mounted display of claim 1 .

22. A head mounted display according to claim 1; at least one camera located remotely relative to the head mounted display; a control system for controlling the position and / or movement of the at least one camera based on the line of sight; A system comprising:

23. the tracking system determines a degree of focus of each of the first eye and the second eye of the user; the system comprises a first camera and a second camera, the first camera associated with the first eye of the user and the second camera associated with the second eye of the user; the control system controls the position, movement, and / or focus of at least one or both of the first and second cameras based on the line of sight and / or the determined focus of at least one or both of the first and second eyes of the user.

23. The system of claim 22.

24. the head mounted display comprises a sensor for detecting at least one of a movement of the user's head, a position of the user's head, and a change in the position of the user's head; the control system controls the position and / or movement of at least one of the first and second cameras based on at least one of the detected movement of the user's head, the detected position of the user's head, and the detected change in position of the user's head.

24. The system of claim 23.

25. comprising at least one of a telepresence system, a virtual reality system, a mixed reality system, and / or an augmented reality system; 23. The system of claim 22.

Images