Fixed distance virtual and augmented reality system and method

Abstract

A fixed distance display system includes a light source configured to generate a light beam. The system also includes a light guide optical element configured to propagate at least a portion of the light beam by total internal reflection. The system further includes a first inertial measurement unit configured to measure a first value for calculating a head posture of a user. In addition, the system includes a camera configured to capture an image for machine vision optical flow analysis. The display system is configured to display a virtual image only within a tolerance range of a single predetermined optical plane.

Description

[0001] This application is a divisional application of an application with a filing date of July 31, 2017, PCT international application number PCT / US2017 / 044762, Chinese national phase application number 201780054387.0, and invention name “Fixed-Distance Virtual and Augmented Reality Systems and Methods”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the benefit of priority to U.S. Provisional Application No. 62 / 370,117, filed on August 2, 2016, attorney case number ML.30040.00, and entitled “FIXED-DISTANCE VIRTUAL AND AUGMENTED REALITY SYSTEMS AND METHODS.” This application is related to commonly owned U.S. Provisional Application No. 62 / 301,502, filed on February 29, 2016, entitled “VIRTUAL AND AUGMENTED REALISTY SYSTEMS AND METHODS,” and commonly owned U.S. Patent Application Publication No. 15 / 146,296, filed on May 4, 2016, entitled “SEPARATED PUPIL OPTICAL SYSTEMS FOR VIRTUAL AND AUGMENTED REALITY AND METHODS FOR DISPLAYING IMAGES USING SAME,” attorney docket no. ML.20058.00, the entire contents of which are expressly and entirely incorporated herein by reference as if fully set forth. Background Art

[0004] Modern computing and display technologies have facilitated the development of virtual reality ("VR"), augmented reality ("AR"), and mixed reality ("MR") systems, in which digitally reproduced images or portions thereof are presented to users in a manner that they appear to be real or can be perceived as real. Virtual reality "VR" scenarios typically involve presenting digital or virtual image information in a manner that is opaque to other actual real-world visual inputs; augmented reality "AR" scenarios typically involve presenting digital or virtual image information as an enhancement to the visualization of the real world around the user. Mixed reality "MR" systems also introduce simulated objects into real-world environments, but these objects are typically characterized as having a higher level of interactivity in AR systems. Simulated elements can typically interact in real time. Therefore, AR and MR scenarios involve presenting digital or virtual image information in a manner that is at least partially transparent to other actual real-world visual inputs. The human visual perception system is extremely complex, and it is challenging to produce VR / AR / MR technologies that facilitate presenting virtual image elements among other virtual or real-world image elements in a comfortable, natural, and rich manner.

[0005] The visual centers of the brain derive valuable perceptual information from the movement of the eyes and their components relative to each other. The vergence movement of the two eyes relative to each other (i.e., the rotational movement of the pupils toward or away from each other so that the eyes' line of sight converges to fixate on objects at different distances) is closely related to the focusing (or "accommodation") of the eye lenses. Under normal circumstances, changing the focus of the eye lenses or adjusting the eyes to focus on objects at different distances will automatically cause a matching change in vergence to the same distance, according to a relationship known as the "accommodation-vergence reflex." Likewise, under normal circumstances, a change in vergence will trigger a matching change in accommodation. As with most conventional stereoscopic VR / AR / MR configurations, violations of this reflex are known to cause eyestrain, headaches, or other forms of discomfort to users.

[0006] Stereoscopic wearable glasses typically feature two displays for the left and right eye that are configured to display images with slightly different element presentations so that the human visual system perceives a three-dimensional perspective. Such a configuration has been found to be uncomfortable for many users because of the mismatch between vergence and accommodation ("vergence-accommodation conflict"), which must be overcome in order to perceive images in three dimensions. In fact, some users cannot tolerate a stereoscopic configuration. These limitations apply to VR / AR / MR systems. Therefore, most conventional VR / AR / MR systems are not best suited for presenting a rich binocular three-dimensional experience in a manner that is comfortable and most applicable to users, in part because existing systems fail to address some fundamental aspects of the human perceptual system, including vergence-accommodation conflict.

[0007] Full VR / AR / MR systems must also be able to display virtual digital content at a variety of perceived positions and distances relative to the user. The design of VR / AR / MR systems also brings many other challenges, including the speed at which the system delivers virtual digital content, the quality of the virtual digital content, the user's eye relief (solving vergence-accommodation conflict), the size and portability of the system, and other system and optical challenges.

[0008] A possible way to solve these problems (including vergence-accommodation conflict) is to project images on multiple depth planes. In order to realize this type of system, one method is to use a large number of optical elements (e.g., light sources, prisms, gratings, filters, scanning optical elements, beam splitters, reflectors, semi-reflective mirrors, shutters, eyepieces, etc.) to project images at a sufficiently large number of depth planes (e.g., six). The problem with this method is that using a large number of components in this way necessarily requires a larger form factor than required, and limits the extent to which the system size can be reduced. The large number of optical elements in these systems also leads to longer optical paths, which degrade the light and the information contained in the light. These design problems cause the system to be bulky and power-consuming. The systems and methods described here are configured to solve these challenges by presenting a fixed-distance mixed reality optical system. Summary of the invention

[0009] In one embodiment, a fixed distance display system includes a light source configured to generate a light beam. The system also includes a light guide optical element configured to propagate at least a portion of the light beam by total internal reflection. The system further includes a first inertial measurement unit configured to measure a first value for calculating a head posture of a user. In addition, the system includes a camera configured to capture an image for machine vision optical flow analysis. The display system is configured to display a virtual image only within a tolerance range of a single predetermined optical plane.

[0010] In one or more embodiments, the first inertial measurement unit may be disposed adjacent to the user's head. The system may further include a second inertial measurement unit configured to measure a second value for calculating the user's head posture. The second inertial measurement unit may be disposed adjacent to the user's head. The system may further include a third inertial measurement unit configured to measure a third value for calculating the user's body posture. The third inertial measurement unit may be disposed adjacent to the user's waist. The first inertial measurement unit, the second inertial measurement unit, the third inertial measurement unit, and the camera may be configured to match the field of view of the fixed distance display system with the user's body position.

[0011] In one or more embodiments, the tolerance range is within 0.2 diopters to 0.6 diopters of a single predetermined optical plane. The display system can be configured to display a three-dimensional virtual image. The display system can be configured to display a three-dimensional virtual image having a size not exceeding 0.2 diopters to 0.6 diopters.

[0012] In another embodiment, a display method includes measuring a first value associated with a head posture. The method also includes calculating a head posture of a user based on the first value. The method further includes capturing an image. In addition, the method includes performing a machine vision optical flow analysis based on the image. In addition, the method includes generating a light beam. The method also includes propagating at least a portion of the light beam by total internal reflection. The method further includes displaying a virtual image only within a tolerance range of a single predetermined optical plane.

[0013] In one or more embodiments, the method further includes measuring a second value associated with the head posture. The method may further include calculating the user's head posture based on the first value and the second value. In addition, the method may include measuring a third value associated with the body posture. In addition, the method may include calculating the user's body posture based on the third value. The method may also include analyzing the first value, the second value, the third value, and the image to match the user's field of view with the user's body position.

[0014] In one or more embodiments, the tolerance range is within 0.2 diopters to 0.6 diopters of a single predetermined optical plane. The method may also include displaying a three-dimensional virtual image. The displayed three-dimensional virtual image may have a size not exceeding 0.2 diopters to 0.6 diopters.

[0015] In yet another embodiment, a product of a computer program embodied in a non-transitory computer readable medium having stored thereon a series of instructions which, when executed by a processor, cause the processor to perform a method for display, the method comprising generating a light beam. The method further comprises propagating at least a portion of the light beam by total internal reflection. The method further comprises measuring a first value associated with a head posture. Furthermore, the method comprises calculating a head posture of a user based on the first value. Additionally, the method comprises capturing an image. The method further comprises performing a machine vision optical flow analysis based on the image. The method further comprises displaying a virtual image only within a tolerance range of a single predetermined optical plane.

[0016] In yet another embodiment, a body-centric display method includes measuring a first value related to a head posture. The method also includes calculating a user's head posture based on the first value. The method further includes measuring a second value related to the body posture. In addition, the method includes calculating the user's body posture based on the second value. In addition, the method includes generating a virtual image that is configured to be displayed only within a tolerance range of a single predetermined optical plane based on the head posture and the body posture. The method also includes displaying the virtual image only within the tolerance range of the single predetermined optical plane. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings illustrate the design and utility of various embodiments of the present invention. It should be noted that the drawings are not drawn to scale and that elements having similar structure or function are represented by the same reference numerals throughout all drawings. In order to better understand how the above and other advantages and objectives of various embodiments of the present invention are obtained, a more detailed description of the invention briefly described above will be presented by reference to specific embodiments of the present invention. It should be understood that these drawings illustrate only typical embodiments of the present invention and are therefore not to be considered limiting of its scope. The present invention will be described and explained with the aid of additional features and details through the use of the drawings, in which:

[0018] Figure 1 is a diagram showing the focal planes of a multi-plane focusing complete optical system;

[0019] Figure 2 is a diagram illustrating focal planes of a single plane focusing fixed distance optical system according to one embodiment;

[0020] Figure 3 and Figure 4 are schematic diagrams of various fields of view for a user of a single plane focusing fixed distance optical system according to two embodiments;

[0021] Figure 5 is a detailed schematic diagram of an optical system according to one embodiment;

[0022] Figure 6 is a detailed schematic diagram of a light guide optical element of an optical system according to one embodiment;

[0023] Figure 7 is a detailed perspective view of a light guide optical element of an optical system according to one embodiment;

[0024] FIG. 8A to FIG. 8D and 9A to 9D is a schematic diagram of an augmented reality / user identification system according to various embodiments;

[0025] Fig.10 It is a detailed schematic diagram of a complete augmented reality system with multi-plane focusing;

[0026] Fig.11 and Fig.12 is a detailed schematic diagram of a single-plane focused fixed-distance augmented reality system according to two embodiments;

[0027] Fig.13 illustrates the field of view of a user of a single plane focus fixed distance augmented reality system according to one embodiment;

[0028] Fig.14 Shown according to Fig.13 The user's field of view of the single-plane focused fixed-distance augmented reality system of the illustrated embodiment after the user's head has been rotated relative to the user's body;

[0029] Fig.15 and Fig.16 is a flow chart illustrating a method for displaying virtual objects to a user of a single-plane focused fixed-distance augmented reality system in a body-centric manner according to two embodiments. DETAILED DESCRIPTION

[0030] Various embodiments of the present invention are directed to systems, methods, and articles of manufacture for implementing a fixed distance mixed reality optical system in a single embodiment or in multiple embodiments. Other objects, features, and advantages of the present invention are described in the detailed description, drawings, and claims.

[0031] Various embodiments will now be described in detail with reference to the accompanying drawings, and the accompanying drawings are provided as illustrative examples of the present invention so that those skilled in the art can practice the present invention. It should be noted that the following drawings and examples are not meant to limit the scope of the present invention. In the case where some elements of the present invention can be partially or completely realized using known components (or methods or processes), only those parts necessary for understanding the present invention will be described in these known components (or methods or processes), and detailed descriptions of other parts of these known components (or methods or processes) will be omitted to avoid blurring the present invention. In addition, various embodiments include the current and future known equivalents of the components mentioned herein by way of illustration.

[0032] The optical system may be implemented independently of the AR / MR system, but many embodiments are described below only with respect to the AR / MR system for illustrative purposes.

[0033] Summary of Problems and Solutions

[0034] One type of optical system for generating virtual images at various depths includes many optical components (e.g., light sources, prisms, gratings, filters, scanning optical elements, beam splitters, mirrors, semi-reflective mirrors, shutters, eyepieces, etc.), and the number of these optical components increases as the quality of the 3D experience / scene (e.g., the number of imaging planes) and the quality of the image (e.g., the number of image colors) increases, thereby increasing the complexity, size, and cost of the VR / AR / MR system. The increased size of the optical system as the quality of the 3D scene / image increases limits the minimum size of the VR / AR / MR system, resulting in a cumbersome system with reduced optical efficiency.

[0035] The following disclosure describes various embodiments of systems and methods for creating a 3D perception around a single predetermined distance using a single plane focusing optical element for each eye that solves the problem by providing an optical system with increased efficiency with fewer components. In particular, the systems described herein utilize a minimal set of optical system components to provide an optical system for displaying virtual objects at a predetermined distance from a user for a mixed reality scene. The optical system design simplifies and reduces the size of VR / AR / MR systems by limiting the functionality of these systems to displaying virtual objects at a predetermined distance.

[0036] Complete augmented reality system

[0037] Before describing the details of embodiments of fixed-distance mixed reality optical systems, the present disclosure will now provide a brief description of related full AR / MR systems that are configured to present full AR scenes with virtual objects displayed at multiple locations along the user's optical axis (e.g., from infinity to adjacent users). Mixed reality optical systems (full distance and / or fixed distance) can be implemented independently of AR / MR systems, but many systems are described below only with respect to AR / MR systems for illustrative purposes. Various augmented reality display systems have been discussed in commonly owned U.S. utility model patent application 14 / 555,585, entitled “VIRTUAL AND AUGMENTED REALITY SYSTEMS AND METHODS,” filed on November 27, 2014, with attorney docket number ML-30011-US, the contents of which are hereby expressly and entirely incorporated herein by reference as if fully set forth.

[0038] One possible approach to implementing a complete AR / MR system is to use multiple volume phase holograms, surface relief holograms, or light-guide optical elements ("LOEs") with embedded depth plane information to generate images that appear to come from various depth planes for each eye of the user. In other words, a diffraction pattern or diffractive optical element ("DOE") can be embedded in or imprinted on the LOE so that collimated light (a light beam with a substantially planar wavefront) is substantially totally internally reflected along the LOE, intersecting the diffraction pattern at multiple locations and at least partially emerging toward the user's eyes. The DOE is configured so that light emitted from the LOE through it converges so that they appear to originate from a specific depth plane. The collimated light can be generated using an optical focusing lens ("condenser").

[0039] For example, a first LOE can be configured to transmit collimated light to the eye that appears to originate from an optically infinite depth plane (0 diopters). Another LOE can be configured to transmit collimated light that appears to originate from a distance of 2 meters (1 / 2 diopters). Yet another LOE can be configured to transmit collimated light that appears to originate from a distance of 1 meter (1 diopters). By using stacked LOE assemblies, it will be appreciated that multiple depth planes can be formed, wherein each LOE is configured to display an image that appears to originate from a particular depth plane. It should be appreciated that the stack can include any number of LOEs. However, at least N stacked LOEs are required to generate the N depth planes. In addition, N, 2N, or 3N stacked LOEs can be used to produce RGB color images on N depth planes.

[0040] In order to present the virtual content in the complete 3D AR scene to the user, the complete AR / MR system projects images of the virtual content into the user's eyes so that these images appear to originate from various depth planes in the Z direction (i.e., orthogonally away from the user's eyes). In other words, the virtual content can not only change in the X and Y directions (i.e., in the 8D plane orthogonal to the central visual axis of the user's eyes), but also appear to change in the Z direction, so that the user can perceive the object to be very close or at an infinite distance or at any distance between them. In a complete AR / MR system, the user can perceive multiple objects at different depth planes at the same time. For example, the user can see a virtual dragon appearing to run towards the user from infinity. Alternatively, the user can simultaneously see a virtual bird at a distance of 3 meters from the user and a virtual coffee cup at an arm's length (about 1 meter) from the user.

[0041] Multi-plane Focusing A full AR / MR system creates variable depth perception by projecting images on some or all of multiple depth planes at various fixed distances from the user's eyes in the Z direction. Figure 1It should be understood that a multi-plane focusing full AR / MR system is typically performed at a fixed depth plane 202 (e.g., Figure 1 Frames are displayed at six depth planes 202 as shown. An exemplary multi-plane focusing complete AR / MR system has six fixed depth planes 202 in the Z direction. When virtual content is generated at one or more of the six depth planes 202, a 3D perception is formed so that the user perceives one or more virtual objects at different distances from the user's eyes. Assuming that the human eye is more sensitive to objects that are closer than objects that appear to be farther away, more depth planes 202 that are closer to the eye are generated, such as Figure 1 In other systems, the depth planes 202 may be placed at equal distances from each other.

[0042] The depth plane position 202 is typically measured in diopters, which is a unit of optical power that is equal to the reciprocal of the focal length measured in meters. For example, in one system, depth plane 1 may be 1 / 3 diopters away, depth plane 2 may be 0.3 diopters away, depth plane 3 may be 0.2 diopters away, depth plane 4 may be 0.15 diopters away, depth plane 5 may be 0.1 diopters away, and depth plane 6 may represent infinity (i.e., 0 diopters away). It should be understood that other systems may generate depth planes 202 at other distances / diopters. Therefore, when virtual content is generated at strategically placed depth planes 202, the user is able to perceive three-dimensional virtual objects. For example, the user may perceive a first virtual object displayed in depth plane 1 as being close to him, while another virtual object appears to be at depth plane 6 at infinity. Alternatively, a virtual object may be first displayed at depth plane 6, then displayed at depth plane 5, and so on, until the virtual object appears to be very close to the user. It should be understood that the above examples are significantly simplified for illustrative purposes. In another system, all six depth planes can be focused at a specific focal distance away from the user. For example, if the virtual content to be displayed is a coffee cup half a meter away from the user, all six depth planes can be generated at various cross-sections of the coffee cup, giving the user a highly granular 3D view of the coffee cup.

[0043] In one system, the complete AR / MR system can be used as a multi-plane focusing system. In other words, all six LOEs can be illuminated simultaneously, thereby generating images that appear to originate from six fixed depth planes in rapid succession, where the light source quickly delivers image information to LOE 1, then LOE 2, then LOE 3, and so on. For example, a portion of the desired image can be injected at time 1, which includes an image of the sky at optical infinity, and LOE 1090 (e.g., LOE 2090) can be used to maintain light collimation. Figure 1Then, an image of a closer branch can be injected at time 2, and a depth plane configured to appear to originate from 10 meters away (e.g., Figure 1 Then, the pen image can be injected at time 3, and the LOE 1090 configured to generate an image that appears to originate from a depth plane 1 meter away can be utilized. Such examples can be repeated in extremely fast timing (e.g., at 360 Hz) so that the user's eyes and brain (e.g., visual cortex) perceive the inputs as all parts of the same image.

[0044] A complete AR / MR system is required to project images (i.e., via diverging or converging light beams) that appear to originate from various locations along the Z axis (i.e., depth planes) to generate images for a complete 3D experience. As used in this application, a "light beam" includes, but is not limited to, the directional projection of light energy (including visible and invisible light energy) radiated from a light source. Generating images that appear to originate from various depth planes follows, or is synchronized with, the vergence and accommodation of the user's eyes for the image, and minimizes or eliminates vergence-accommodation conflicts.

[0045] The full AR scene capability of projecting images that appear to originate from various depth planes requires minimal complexity, size, and cost constraints on the full AR / MR system. The fixed-distance AR / MR system described here avoids these constraints by reducing the functionality of the system.

[0046] Fixed distance augmented reality system

[0047] Fixed-distance AR / MR systems are configured to display virtual objects close / proximate (i.e., within about 0.2 diopters to about 0.6 diopters) to a single predetermined distance (i.e., "depth plane") from the user in the Z direction for portions of an AR experience / scene. Figure 2 It will be appreciated that a single plane focused fixed distance AR / MR system typically displays frames at one fixed depth plane 202. The visual effect for a user of a fixed distance AR / MR system is that a virtual dome of the displayed image is presented to the user, with the dome being in front of or centered on the user. The dome 206 is positioned at Figure 3 206 is shown as a 2D arc, where the virtual dome 206 of the displayed image is in front of the user. Although the user may have a fixed left to right field of view ("FOV") 208 of approximately 50°, due to head rotation, the effective L / R FOV will be approximately 180°, as shown in FIG. Figure 3In addition, the user's FOV can cover the entire dome 206 by moving the head up or down. Therefore, the fixed distance AR / MR system can present image information to the user over the entire dome 206 surrounding the user 204. In another embodiment where the fixed user can freely rotate the body and head, the effective FOV can be the entire sphere 209, such as Figure 4 As shown. Figure 3 and Figure 4 Dome 206 and sphere 209 represent an image rendered in 2D by a fixed distance AR / MR system, although it should be understood that dome 206 and sphere 209 are three-dimensional surfaces.

[0048] For each eye, a fixed-distance AR / MR system may be implemented using a single volume phase hologram, surface relief hologram, or light-guide optical element (“LOE”) with embedded depth plane information to generate images that appear to originate from a predetermined depth plane within a tolerance range (e.g., within about 0.2 diopters to about 0.6 diopters). The fixed-distance AR / MR system may generate a 3D effect using stereoscopic techniques (i.e., for each eye, respective images displayed at slightly different distances in the Z direction within a tolerance range of a predetermined distance).

[0049] While the human visual system is very accurate with respect to vergence, it has a certain tolerance for accommodation (i.e., focusing) differences before problems associated with the vergence-accommodation reflex occur. Although specific to a particular user, tolerance / uncertainty in accommodation typically occurs between about 0.2 diopters and about 0.6 diopters relative to vergence. Thus, the fixed-distance AR / MR systems disclosed herein exploit the human visual system's tolerance for accommodation differences to display 3D objects near a single predetermined distance in the Z direction, particularly beyond the user's arm's length.

[0050] Because the diopter is the inverse of the distance from the user's eyes (see Figure 1 ), so the accommodation tolerance / uncertainty is constant in diopters and varies in linear distance. For illustrative purposes, at a first distance of 1.00m (=1.0 diopters) from the user, an increase of 0.2 diopters (the lower limit of typical tolerance) will result in a second distance of 0.83m, a change of -17cm. Similarly, 0.2 diopters in the opposite direction corresponds to a distance of 1.25m, and for a total range of 0.83m to 1.25m, if the display is at 1.00m, the content is viewed within an accommodation tolerance of 0.2 diopters (in other words, for a fixed distance AR / MR system displaying content at 1.00m, the 42cm range is within the tolerance of 0.2 diopters).

[0051] On the other hand, for a first distance of 0.50m (=2.0 diopters) from the user, an increase of 0.6 diopters will result in a second distance of 0.38m, a change of -12cm. Moving the corresponding 0.6 diopters from the first distance of 0.50m results in a distance of 0.71m, a change of 21cm, or a total range of 33cm within 0.6 diopters for a fixed AR / MR system to display content at 0.5m. Compared to the 42cm range of a 1.00m fixed-distance AR / MR system with a 0.2 diopters tolerance, tripling the tolerance to 0.6 (the other extreme value of the diopter tolerance) actually reduces the content projection range of the closer 0.5m first distance system. Therefore, a fixed-distance system with a more precise (i.e., lower) adjustment tolerance will have a larger functional range for displaying content at a fixed distance farther from the user.

[0052] A fixed distance AR / MR system that employs the tolerance ranges of the present invention around a single optical plane can greatly simplify functionality if display / rendering of such content is permitted only when the content appears within a certain range from a predetermined optical position of the user (preferably at a larger distance to maximize the tolerance range between different users). Similarly, in some embodiments, if the optical power determining the position of the predetermined optical plane of the fixed distance AR / MR system is known, or the specific preferred tolerance of the user is determined, certain content can be configured to appear only within a certain range.

[0053] Table 1 shows the range relative to a predetermined optical plane that allows display of content within a diopter tolerance according to some embodiments, where the predetermined optical plane is located at a fixed distance from the user. Table 1 shows the tight range (see 0.5m predetermined optical plane distance) for displaying content for a near user optical plane (as opposed to a farther optical plane that allows a larger range of content display) without considering the diopter tolerance.

[0054]

[0055] Table 1

[0056] In some embodiments, a tolerance range for a fixed-distance AR / MR system may set the rendering domain so that content that does not appear within range will not be rendered and thus save battery life, or content may be specifically programmed to appear within a certain distance from the user in order to remain within a tolerance range of a predetermined optical plane.

[0057] In some embodiments, the user may select tolerances by functionality (i.e., selecting a fixed-distance AR embodiment that only actually needs to present content at certain distances, such as a virtual computer screen) or by provision (i.e., recognizing that the user feels discomfort when certain diopter tolerances are exceeded.) This selection is akin to choosing a customized and simplified AR device for a specific setting, rather than a more complex AR / MR system designed to serve all use cases.

[0058] Figure 5 A portion of a fixed distance optical system 100 for projecting an image on a single depth plane is shown. The system 100 includes a light source 120 and a LOE 190 having a diffractive optical element (not shown) and an in-coupling grating 192 ("ICG") associated therewith. The diffractive optical element can be of any type, including volume or surface relief. In one embodiment, the ICG 192 can be a reflective aluminized portion of the LOE 190. In another embodiment, the ICG 192 can be a transmissive diffractive portion of the LOE 190. When the system 100 is in use, a "virtual" light beam 210 from the light source 120 enters the LOE 190 via the ICG 192 and propagates along the LOE 190 by substantially total internal reflection ("TIR") for display to the user's eye. The light beam 210 is "virtual" because it encodes an image of a non-existent "virtual" object or portion thereof as directed by the system 100. It will be appreciated that although Figure 5 Only one beam 210 is shown in FIG, but multiple beams encoding an image may enter the LOE 190 from a wide range of angles through the same ICG 192. The range of angles of entry is related to the FOV achievable for the system 100. A beam "entering" or being "admitted" into the LOE includes, but is not limited to, interacting with the beam so as to propagate along the LOE by substantially TIR. Figure 5 The illustrated system 100 may include various light sources 120 (eg, LEDs, OLEDs, lasers, and masked wide area / broadband emitters). In other embodiments, light from the light sources 120 may also be transmitted to the LOE 190 via a fiber optic cable (not shown).

[0059] like Figure 6As shown, a portion of the LOE 190 in the fixed distance optical system 100 can be used as an exit pupil expander 196 ("EPE") to increase the numerical aperture of the light source 120 in the Y direction, thereby increasing the resolution of the system 100. Since the light source 120 produces light of a small diameter / spot size, the EPE 196 expands the apparent size of the pupil of the light exiting the LOE 190 to increase the system resolution. In addition to the EPE 196, the system 100 can further include an orthogonal pupil expander 194 ("OPE") to expand the light in both the X (OPE) and Y (EPE) directions. More details about the EPE 196 and the OPE 194 are described in the above-referenced U.S. utility model patent application Ser. No. 14 / 555,585, the contents of which have been previously incorporated by reference.

[0060] Figure 6 LOE 190 with ICG 192, OPE 194 and EPE 196 is shown. Figure 6 LOE 190 is shown in a top view similar to the view from the user's eyes. ICG 192, OPE 194, and EPE 196 can be any type of DOE, including volume or surface relief. ICG 192 is a DOE (e.g., a linear grating) that is configured to allow a virtual beam 210 from light source 120 to propagate via TIR. Figure 6 In the illustrated system 100 , the light source 120 is disposed to the side of the LOE 190 .

[0061] OPE 194 is a DOE (e.g., a linear grating) tilted in a transverse plane (i.e., perpendicular to the optical path) such that a virtual light beam 210 propagating through the system 100 is deflected transversely by 90 degrees. OPE 194 is also partially transparent and partially reflective along the optical path such that the light beam 210 partially propagates through OPE 194 to form a plurality of (e.g., 11) small beams 210'. In the illustrated system 100, the optical path is along the X-axis, and OPE 194 is configured to bend the small beams 210' to the Y-axis.

[0062] The EPE 196 is a DOE (e.g., a linear grating) tilted in the Z plane (i.e., perpendicular to the X and Y directions) such that the beamlets 210' propagating through the system 100 are deflected 90 degrees in the Z plane toward the user's eyes. The EPE 196 is also partially transparent and partially reflective along the optical path (Y axis) such that the beamlets 210' partially propagate through the EPE 196 to form a plurality of (e.g., seven) beamlets 210'. For clarity, only the selection beam 210 and the beamlets 210' are labeled.

[0063] OPE 194 and EPE 196 are also at least partially transparent along the Z axis to allow real-world light (e.g., reflected light from real-world objects) to propagate through OPE 194 and EPE 196 in the Z direction to reach the user's eyes. For AR / MR system 100, ICG 192 is at least partially transparent along the Z axis and is also at least partially transparent along the Z axis to allow real-world light to enter.

[0064] Figure 7 Another fixed distance optical system 100 is shown, which includes a LOE 190 having an ICG 192, an OPE 194, and an EPE 196. The system 100 also includes a light source 120, which is configured to direct a virtual light beam 210 into the LOE 190 via the ICG 192. The light beam 210 is split into small beams 210' by the OPE 194 and the EPE 196, as described above with reference to Figure 6 In addition, when the small beams 210' propagate through the EPE 196, they also exit the LOE 190 via the EPE 196 toward the user's eyes. Figure 7 In the illustrated system 100, the light source 120 is arranged orthogonal to the LOE 190. For clarity, only the selected beam 210 and the small beam 210' are labeled. The fixed distance mixed reality optical system can be implemented independently of the AR / MR system, but many systems are described herein with respect to the AR / MR system for illustration purposes only.

[0065] Reference now FIG. 8A to FIG. 8D , which shows some conventional AR / MR system component options. Fig. 8A As shown, a head mounted device user 60 is shown wearing a frame 64 structure, which is coupled to a display system 62 located in front of the eyes of the user 60. The following describes possible components of a conventional AR / MR system. The components described are not required to implement a fixed distance AR / MR system.

[0066] Although not required to implement a fixed distance AR / MR system, a speaker 66 can be coupled to the frame 64 in the configuration shown and positioned near the ear canal of the user 60. In an alternative embodiment, another speaker (not shown) is positioned near the other ear canal of the user 60 to provide stereo / shapeable sound control. In one or more embodiments, the fixed distance AR / MR system can have a display 62, which can be operably coupled to a local processing and data module 70, such as by a wired lead or a wireless connection, and the local processing and data module 70 can be mounted in a variety of configurations, such as fixedly attached to the frame 64, fixedly attached to Figure 8B The helmet or hat 80 shown in the illustrated embodiment is embedded in the earphones to Figure 8CThe backpack configuration shown in the embodiment of the present invention can be removably attached to the torso 82 of the user 60, or Fig.8D The belt coupling configuration shown in the embodiment is removably attached to the hip 84 of the user 60.

[0067] The local processing and data module 70 may include a power-saving processor or controller, and digital memory, such as flash memory, both of which may be used for processing, caching, and storage of auxiliary data. Data may be captured from sensors that may be operably coupled to the frame 64, such as image capture devices (such as cameras) and inertial measurement units (including accelerometers and gyroscopes). Optional sensors that are not required to implement a fixed-distance AR / MR system include compasses, microphones, GPS units, and wireless devices. Alternatively or additionally, a remote processing module 72 and / or a remote data repository 74 may be used to acquire and / or process data, which may be transmitted to the display 62 after such processing or retrieval. The local processing and data module 70 may operably couple 76, 78 to the remote processing module 72 and the remote data repository 74, such as via a wired or wireless communication link, so that these remote modules 72, 74 are operably coupled to each other and may be used as resources for the local processing and data module 70.

[0068] In one embodiment, the remote processing module 72 may include one or more relatively powerful processors or controllers configured to analyze and process data and / or image information. In one embodiment, the remote data repository 74 may include a relatively large-scale digital data storage facility that may be available via the Internet or other network configuration in a "cloud" resource configuration. In one embodiment, all data is stored and all calculations are performed in the local processing and data module, allowing fully autonomous use from any remote module.

[0069] In one or more embodiments, similar to Figures 8A to 8D The head-mounted fixed-distance AR / MR system shown in can be used to display virtual images to a user within a tolerance range of a predetermined distance from the user. It should be understood that although many of the embodiments described below can be implemented in a head-mounted system, other embodiments can also be implemented in a fixed device. For illustrative purposes, the present disclosure will mainly focus on the head-mounted fixed-distance AR / MR system, but it should be understood that the same principles can also be applied to non-head-mounted and non-AR embodiments.

[0070] In one or more embodiments, the fixed-distance AR / MR system is typically adapted to the head of a specific user, and the optical components are aligned with the user's eyes. These configuration steps can be used to ensure that the best augmented reality experience is provided to the user without causing any physiological side effects, such as headaches, nausea, discomfort, etc. Therefore, in one or more embodiments, the user wears a fixed-distance AR / MR system configured (physically or digitally) for each individual user, and a set of programs can be calibrated specifically for the user. In other scenarios, various users can comfortably use loose-fitting AR devices. For example, in some embodiments, the user wears a fixed-distance AR / MR system to know the distance between the user's eyes, the distance between the head-mounted display and the user's eyes, and the curvature of the user's forehead. All of these measurements can be used to provide a head-mounted fixed-distance AR / MR system customized for a given user. In other embodiments, these measurements may not be necessary to perform AR display functions.

[0071] For example, refer to 9A to 9D , a fixed distance AR / MR system can be customized for each user. In one or more embodiments, Fig.9A As shown, the user's head shape 402 can be considered when installing the user identification system worn by the head-mounted user. Fig. 9B As shown, the eye assembly 404 (e.g., the optical device, the structure of the optical device, etc.) can be rotated or adjusted horizontally and vertically to achieve user comfort, or rotated to achieve user comfort. In one or more embodiments, as shown in FIG. Fig. 9C As shown, the rotation point of the head device relative to the user's head can be adjusted based on the structure of the user's head. Fig.9D As shown, the interpupillary distance (IPD) (ie, the distance between the user's eyes) may be compensated.

[0072] Reference now Fig.10 , various components of an exemplary complete AR / MR system will be described. It should be understood that a fixed distance AR / MR system has fewer components than a complete AR / MR system. However, describing an exemplary complete AR / MR system will illustrate the interconnection and functionality of the components of the fixed distance AR / MR system. Fig.10 Provides a basic idea of ​​the various components of an AR / MR system or AR device. For illustration purposes, Fig.10 A simplified version of a head-mounted AR / MR system 62 is shown in the right-hand box diagram.

[0073] refer to Fig.10, where one embodiment of a suitable user display device 62 is shown, which includes a display lens 106 that can be mounted to a user's head or eye via a housing or frame 108. The user display device 62 is a complete AR / MR system that is configured to perform various functions, including displaying to the user 3D virtual objects that appear at various distances from the user. The display lens 106 may include one or more transparent reflectors positioned in front of the user's eye 20 via the housing 84, configured to reflect projected light 38 into the eye 20 and facilitate beam shaping while also allowing transmission of at least some light from the local environment. In the illustrated embodiment, two wide field of view machine vision cameras 16 are coupled to the housing 108 to image the environment surrounding the user; in one embodiment, these cameras 16 are dual-camera visible light / infrared cameras. The system shown also includes a pair of scanning laser shaping wavefront (i.e., for depth) light projector modules 18 (e.g., spatial light modulators such as DLP, fiber scanning device (FSD), LCD, etc.), which, as shown, have display mirrors and optical devices configured to project light 38 into the eye 20.

[0074] Although not required to implement a fixed-distance AR / MR system, the illustrated complete AR / MR system also includes two miniature infrared cameras 24 paired with infrared light sources 26 (such as light emitting diodes or “LEDs”) that are configured to track the user's eyes 20 to support presentation and user input. These infrared cameras 24 are also configured to continuously and dynamically capture images of the user's eyes, particularly the irises of the eyes that can be used for user identification.

[0075] The system also features a sensor assembly 39, which may include X, Y, and Z axis accelerometer functionality and a magnetic compass, and X, Y, and Z axis gyroscope functionality, which preferably provide data at a relatively high frequency (e.g., 200 Hz). An exemplary sensor assembly 39 is an inertial measurement unit (“IMU”). The illustrated system 62 also includes a head pose processor 36 (“image pose processor”), such as an ASIC (application specific integrated circuit), an FPGA (field programmable gate array), and / or an ARM processor (advanced reduced instruction set machine), which can be configured to calculate real-time or near real-time user head pose based on wide field of view image information output from the capture device 16. The head pose processor 36 is operably coupled (90, 92, 94; e.g., via a wired or wireless connection) to the camera 16 and the rendering engine 34.

[0076] Another processor 32 ("sensor gesture processor") is also shown, which is configured to perform digital and / or analog processing to derive gestures based on gyroscope, compass and / or accelerometer data in sensor assembly 39. The system shown also features a GPS (Global Positioning System) subsystem 37 to assist with gestures and positioning. In addition, GPS can also provide cloud-based information about the user's location. This information can be used for user identification purposes. For example, if the user identification algorithm can narrow down the detected user characteristics to two potential user identities, the user's current and historical location data can be used to eliminate one of the potential user identities.

[0077] Finally, the system shown includes a rendering engine 34, which may feature hardware running a software program configured to provide the user with local rendering information to facilitate operation of the scanner and imaging into the user's eyes so that the user can view the world. The rendering engine 34 is operatively coupled 94, 100, 102, 104, 105 (i.e., via a wired or wireless connection) to the image gesture processor 36, the optional eye tracking camera 24, the projection subsystem 18, and the sensor gesture processor 32 to project rendering light 38 using the scanning laser device 18 in a manner similar to a retinal scanning display. The wavefront of the projected light beam 38 can be bent or focused to coincide with a desired focal length of the projected light 38.

[0078] An optional miniature infrared eye tracking camera 24 can be used to track the eyes to support rendering and user input (e.g., where the user is looking, the depth he is focusing on, etc.). As described below, the eye edges can be used to estimate the depth at which the user is focusing. GPS 37 and the gyroscope, compass, and accelerometer in the sensor assembly 39 can be used to provide rough and / or fast pose estimates. The camera 16 image and sensor pose information combined with data from related cloud computing resources can be used to map the local world and share the user's view with a virtual or augmented reality community and / or user recognition system.

[0079] Although Fig.10 Most of the hardware in the display system 62 featured in FIG. 1 is shown as being directly coupled to the housing 108 adjacent to the display 106 and the user's eye 20, but the hardware components shown may be mounted to or housed in other components, such as a belt-mounted component, such as Fig.8D shown.

[0080] In a complete AR / MR system, in addition to the image gesture processor 36, the sensor gesture processor 32 and the rendering engine 34, Fig.10All components of the system 62 characterized by this are directly coupled to the display housing 108, and the communication between the latter three and the remaining components of the system 62 can be wireless communication (such as ultra-wideband) or wired communication. The housing 108 shown is preferably a head-mounted housing and can be worn by the user. It can also feature a speaker, such as a speaker that can be inserted into the user's ear and used to provide sound to the user.

[0081] Already described Fig.10 The conventional components of the complete AR / MR system 62 shown will now be referred to Fig.11 A fixed distance AR / MR system 300 is described. The fixed distance AR / MR system 300 includes a pair of light guide optical elements 302-1, 302-r, which are configured to receive light from respective light sources (not shown) and direct the received light into the left eye and the right eye of the user, respectively. The light from the light source can be modulated by a corresponding spatial light modulator (not shown) to include image data corresponding to a 3D virtual object. For example, light directed to the left and right LOEs 302-1, 302-r can be performed with image data corresponding to the same 3D virtual object from two slightly different viewpoints. These viewpoints correspond to the positions of the left and right LOEs 302-1, 302-r calibrated for the left and right eyes of the user. Therefore, the light transmitted through the left and right LOEs 302-1, 302-r generates an image of the 3D virtual object for the user using stereo technology.

[0082] The fixed distance AR / MR system 300 is configured to generate images of 3D virtual objects that appear close to a fixed distance relative to the user, i.e., a point of vergence of the LOE 302-1, 302-r. As described above, the human visual system has an accommodation tolerance / uncertainty of about 0.2 diopters to about 0.6 diopters. Because the fixed distance AR / MR system 300 is configured to focus on images of 3D virtual objects within a tolerance range of a fixed distance, the system can display images of 3D virtual objects without significant accommodation and vergence issues.

[0083] Although the light source and the spatial light modulator are described as separate components, the spatial light modulator can be part of the respective light source. In another embodiment, a single light source and a single spatial light modulator (whether separate or combined) can provide modulated light beams to both the left and right LOEs 302-1, 302-r in a timed manner. As long as the light beams are provided at a rate of at least about 60 frames / second per LOE (e.g., 120 frames per second for a single light source embodiment), the fixed distance AR / MR system 300 can present 3D virtual objects that move smoothly with minimal motion artifacts.

[0084] To this end, the fixed distance AR / MR system 300 also includes a controller 304 that is operably coupled to other components of the system 300 to coordinate those components. For example, the controller 304 may include a 3D rendering engine for generating image data for the left and right LOEs 302-1, 302-r, which are configured to present 3D virtual objects and send the data to the corresponding light sources and spatial light modulators. In a single light source / spatial light modulator embodiment, when the single light source / spatial light modulator is encoded with light for the image data configured for the left LOE 302-1, the controller 304 can activate the left LOE 302-1 and deactivate the right LOE 302-r. This will minimize crosstalk and artifacts associated with the left and right LOEs 302-1, 302-r. The controller 304 can be a processor in the head-mounted fixed distance AR / MR system 300. In other embodiments, controller 304 may be a processor in a mobile computer (eg, a cell phone) or a portion on a processor that is wirelessly connected to other components of fixed-range AR / MR system 300 .

[0085] The fixed-distance AR / MR system 300 also includes left and right IMUs 306-1, 306-r, which are configured to provide data to determine the head pose of the user / wearer of the head-mounted fixed-distance AR / MR system 300. Although a single IMU is capable of providing data to determine the user's head pose, the two IMUs 306-1, 306-r provide additional data, which improves the determination of the head pose. In one embodiment, once the gyroscopes in the left and right IMUs 306-1, 306-r are calibrated, the system 300 is able to track the direction in which the user's head and eye sockets are pointing. Although the embodiment of the fixed-distance AR / MR system 300 shown does not include an eye tracking camera and IR lights, the left and right IMUs 306-1, 306-r provide for actually estimating the direction the user is looking at using the head pose.

[0086] In addition, the fixed-distance AR / MR system 300 includes an outward-facing image capture device 308 (e.g., a camera) to capture images close to the user's field of view. The image capture device sends the captured image data to the controller 304, which performs machine vision optical flow to correct for drift caused by movement of the system 300 and the image capture device 308 coupled to the head-mounted portion of the system 300. The image capture device 308 can be a visible light and / or infrared light camera, such as a CCD camera.

[0087] The fixed distance AR / MR system 300 uses the left and right LOEs 302-1, 302-r, the controller 304, the left and right IMUs 306-1, 306-r, and the image capture device 308 to generate and present 3D images of virtual objects in the user's field of view, so that the user can view the 3D images without significant accommodation and convergence problems. The left and right LOEs 302-1, 302-r project the images into the left eye and the right eye, respectively. The controller 304 presents the images (to generate 3D images for the user). The left and right IMUs 306-1, 306-r provide the controller 304 with data for determining the user's head posture. The image capture device 308 provides image data for correcting drift in the system 300. The controller 304 uses the data provided by the left and right IMUs 306-1, 306-r and the image capture device 308 to present appropriate images for the user's left and right eyes based on the user's head posture and system drift.

[0088] exist Fig.12 In another embodiment shown, the fixed distance AR / MR system 300 also includes left and right LOEs 302-1, 302-r, a controller 304, left and right IMUs 306-1, 306-r, and an image capture device 308. These components are operably coupled and as described above with respect to Fig.11 The same components in the fixed distance AR / MR system 300 shown function as described. In addition, Fig.12 The fixed-distance AR / MR system 300 shown includes a third or "body" IMU 306-b that is fixed to the user's body (e.g., attached to the user's belt). In other embodiments, the body IMU 306-b is fixed to or is part of the user's cell phone, but the body IMU 306-b is only active when the phone is fixed to the user's belt (e.g., in a holder). The body IMU 306-b is configured to provide data that determines the body posture of the user of the fixed-distance AR / MR system 300. Once the gyroscope in the body IMU 306-b is calibrated, the system 300 is able to track the direction in which the user's body is pointing.

[0089] The controller 304 uses data from the body IMU 306-b to generate a "body-centered" fixed-distance AR scene. For example, the controller 304 can generate a user interface 506 in the user FOV 500 and at a fixed distance from the user, such as Fig.13Unlike a "head-centric" AR scene (which is just an overlay on top of a real-world scene that moves with the user's head movements), a body-centric fixed-distance AR scene "fixes" the user interface 506 at a fixed distance and fixed point relative to the user's body, and only moves to different parts of the user interface 506 as the user's head turns toward the different parts. This is achieved by the controller 304 comparing data from the body IMU 306-b with data from the left and right IMUs 306-1, 306-r to determine the head pose relative to the body pose.

[0090] Example Virtual Object (User Interface)

[0091] Fig.13 Shown wearing such Fig.12 FOV 500 of a user of a head-mounted fixed-distance AR / MR system 300 is shown. The user can see real physical buildings 502 and real bushes 504 in FOV 500. The user can also see a user interface 506 generated and presented by the fixed-distance AR / MR system 300 in FOV 500. The user interface 506 has sufficient opacity to allow the user to see the tree 504 behind the user interface 506. The opacity is illustrated by showing the user interface 506 with dashed lines. The user interface 506 includes six user interface objects 508-1 to 508-6. Due to the limitations of the user's eyes, the FOV 500 is limited to the left side 510-1 and the right side 510-r. The limited FOV 500 is approximately 50°.

[0092] Fig.14 Another FOV 500' of the same user wearing the same head-mounted fixed-distance AR / MR system 300 is shown after the user's head has been turned approximately 25° to the left relative to the user's body. Building 502 and tree 504 have moved to the right side 510-r of the FOV. In fact, the right half of building 502 has moved out of FOV 500'. Because head-mounted fixed-distance AR / MR system 300 is configured to generate and present body-centric user interface 506, after turning their head, the user can see different portions of user interface 506. For example, user interface objects 508-a, 508-b, and 508-c are in the Fig.14 The FOV 500' is shown as visible to the user, but Fig.13 The user is not visible in the illustrated FOV 500. Meanwhile, user interface objects 508-4, 508-5, and 508-6 are not visible in the illustrated FOV 500. Fig.14 The FOV 500' shown is not visible to the user, but they are Fig.13The FOV 500 shown is visible to the user. This ability to scan left and right allows for a larger total effective FOV than the human eye FOV of approximately 50°. The increase in total effective FOV is not limited to the left and right directions, but also includes the up and down directions, thereby forming a dome 206 (see FIG. 20A ) on which the head-mounted fixed-distance AR / MR system 300 can present virtual images. Figure 3 ).Although Fig.13 and Fig.14 A 2D user interface 506 is shown, but it should be noted that the virtual image may be a stereoscopically rendered 3D image as described above.

[0093] The fixed-distance AR / MR system 300 described above can perform the function of displaying virtual objects in a body-centric manner while minimizing the complexity, size, and cost of the system 300. Such economical use of optical system components can result in AR / MR systems (and other optical systems) that are easier to manufacture, more comfortable to use, and cheaper for the average consumer / user.

[0094] Body-centered display of virtual objects

[0095] Having described several embodiments of the fixed distance AR / MR system 300, a method of displaying virtual objects (e.g., such as a virtual object) to a user of the fixed distance AR / MR system 300 in a body-centric manner will now be discussed. Fig.14 Method 600 of the user interface 506 in FIG. Fig.15 As shown, the method begins at step 602 by measuring a first value related to the head posture. For example, the left IMU 306-1 in the fixed-distance AR / MR system 300 (see Fig.14 ) can measure gyroscope data indicating head posture.

[0096] At step 604 , the head pose is calculated based on the first value. For example, the controller 304 in the fixed-distance AR / MR system 300 may calculate the head pose based on the measured gyroscope data.

[0097] At step 606, a second value related to the body posture is measured. For example, the body IMU 306-b in the fixed-distance AR / MR system 300 (see Fig.14 ) can measure gyroscope data that indicates body posture.

[0098] At step 608 , the body posture is calculated based on the second value. For example, the controller 304 in the fixed-distance AR / MR system 300 may calculate the body posture based on the measured gyroscope data.

[0099] At step 610, a virtual image configured to be displayed within a tolerance range of a single predetermined optical plane is generated based on the head pose and the body pose. For example, a 3D rendering engine of controller 304 in fixed-distance AR / MR system 300 may render a virtual image that is configured to be displayed in the user's FOV based on the user's head pose relative to the user's body pose. The 3D rendering engine may change the virtual image generated based on the user's head pose relative to the user's body pose (compare Fig.13 and Fig.14 ). A virtual image can be Fig.13 and Fig.14 Portion of user interface 506 is shown. The virtual image may be a portion of a 3D image (eg, a left image of an image pair configured to present a 3D image when displayed simultaneously to the left and right eyes of a user).

[0100] At step 612, the generated virtual image is displayed within a tolerance range of the single predetermined optical plane. As described above, displaying the virtual image "adjacent" the single predetermined optical plane includes, but is not limited to, displaying the virtual image within about 0.2 diopters to about 0.6 diopters of the single predetermined optical plane. For example, the light source, spatial light modulator, and left LOE 302-1 in the fixed distance AR / MR system 300 (see Fig.14 ) can display the generated virtual image to the user. A single predetermined optical plane can be selected so that the vergence of the user's left eye (based on the angle of light emitted by the left LOE 302-1) corresponds to the focus of the virtual image within the tolerance range of the single predetermined optical plane. This correspondence minimizes or eliminates problems associated with vergence-accommodation conflict.

[0101] Fig.16 FIG. 3 shows a method of displaying virtual objects (e.g., such as a virtual object) to a user of a fixed-distance AR / MR system 300 in a body-centric manner. Fig.14 Another method 600' of the user interface 506 in FIG. 602, 604, 606, 608 and 612 are similar to Fig.15 The corresponding steps in the illustrated method 600 are the same.

[0102] Fig.16 The method 600' shown is different from Fig.15 The method 600 shown in FIG. 6B is that an image is captured at optional step 614. For example, the image acquisition device 308 (see FIG. 6B ) in the fixed distance AR / MR system 300 Fig.14 ) can capture images close to the user's FOV.

[0103] At optional step 616, machine vision optical flow analysis is performed based on the captured images. Optical flow analysis corrects for drift caused by system motion. For example, the controller 304 in the fixed-distance AR / MR system 300 can perform machine vision optical flow analysis based on the captured images.

[0104] At step 610, a virtual image configured to be displayed within a tolerance range of a single predetermined optical plane is generated based on the head pose, body pose, and the results of the machine vision optical flow analysis. For example, a 3D rendering engine of controller 304 in fixed-distance AR / MR system 300 may render a virtual image that is configured to be displayed in the user's FOV based on the head pose, body pose, and the results of the machine vision optical flow analysis. The 3D rendering engine may change the virtual image generated based on the user's head pose relative to the user's body pose (compare Fig.13 and Fig.14 ). Including the results of machine vision optical flow analysis when rendering a virtual image corrects for system drift and results in a more accurately rendered virtual image. The virtual image may be Fig.13 and Fig.14 Portion of user interface 506 is shown. The virtual image may be a portion of a 3D image (eg, a left image of an image pair configured to present a 3D image when displayed simultaneously to the left and right eyes of a user).

[0105] The above-described fixed-distance AR / MR systems are provided as examples of various common systems that can benefit from reduced complexity, size, and cost. Therefore, the use of the fixed-distance optical systems described herein is not limited to the disclosed fixed-distance AR / MR systems, but is applicable to any optical system.

[0106] Various exemplary embodiments of the present invention are described herein. These embodiments are cited in a non-restrictive sense. They are provided to illustrate the more widely applicable aspects of the present invention. Various changes may be made to the described invention without departing from the true spirit and scope of the present invention, and equivalents may be substituted. In addition, many modifications may be made to adapt a particular situation, material, composition of matter, process, one or more process actions or steps to one or more purposes, spirit or scope of the present invention. In addition, as will be appreciated by those skilled in the art, each individual variation described and illustrated herein has discrete components and features that may be easily separated or combined from the features of any other several embodiments without departing from the scope or spirit of the present invention. All of these modifications are intended to fall within the scope of the claims associated with the present disclosure.

[0107] The present invention includes methods that can be performed using subject equipment. The method can include the act of providing such suitable equipment. This provision can be performed by an end user. In other words, the "providing" action only requires the end user to obtain, access, approach, locate, set, activate, power on, or perform other actions to provide the necessary equipment in the subject method. The methods described herein can be performed in any order of the events described that is logically possible, and can also be performed in the described order of events.

[0108] Exemplary aspects of the present invention and details about material selection and manufacturing have been described above. Other details of the present invention can be understood in conjunction with the above patents and disclosures and in a manner generally known or understood by those skilled in the art. This is also true for the method-based aspects of the present invention in terms of additional actions that are generally or logically employed.

[0109] In addition, although the present invention has been described with reference to several examples optionally incorporating various features, the present invention is not limited to those described or indicated with respect to each variation of the present invention. Various changes may be made to the described invention without departing from the spirit and scope of the present invention, and equivalents (whether cited herein or not included for the sake of brevity) may be substituted. In addition, where a range of values ​​is provided, it is understood that each intermediate value between the upper and lower limits of the range, and any other stated or intermediate values ​​within the stated range are included within the present invention.

[0110] Moreover, it is contemplated that any optional features of the described inventive variations may be set forth or claimed independently or in combination with any one or more features described herein. Reference to a singular item includes the possibility of the presence of a plurality of identical items. More specifically, as used herein and in the claims associated therewith, the singular forms "one," "an," "said," and "the" include plural referents unless otherwise specifically stated. In other words, the use of the article allows for "at least one" subject item in the above description and in the claims associated with the present disclosure. It should also be noted that such claims may be drafted to exclude any optional elements. Therefore, this statement is intended to serve as a prior basis for the use of exclusive terms such as "alone," "only," or the use of a "negative" limitation in conjunction with the recitation of claim elements.

[0111] Without using such exclusive terms, the term "comprising" in the claims associated with the present disclosure shall allow the inclusion of any additional elements, regardless of whether a given number of elements are recited in such claims, or the addition of features may be considered to change the nature of the elements set forth in these claims. Unless specifically defined herein, all technical and scientific terms used herein are given as broadly as possible the commonly understood meanings while maintaining the validity of the claims.

[0112] The breadth of the present invention is not limited to the examples provided and / or subject specification, but is limited only by the scope of the claim language associated with the present disclosure.

[0113] In the foregoing description, the invention has been described with reference to specific embodiments thereof. However, it will be apparent that various modifications and changes may be made thereto without departing from the broader spirit and scope of the invention. For example, the above process flow is described with reference to a particular sequence of process actions. However, the sequence of many of the described process actions may be changed without affecting the scope or operation of the invention. Therefore, the description and drawings should be regarded as illustrative rather than restrictive.

Claims

1. A display method using a fixed distance display system, comprising: The processor calculates the user's head posture and body posture; performing, by the processor, machine vision optical flow analysis on the image; displaying, by the fixed distance display system, a body-centric virtual user interface in a field of view of the user at a fixed distance and fixed position relative to the user's body based on the user's head posture relative to the user's body posture; displaying, by the fixed distance display system, the body-centric virtual user interface at a distance and position relative to the user's head, wherein the distance and the position vary with movement of the user's head such that the body-centric virtual user interface moves in the user's field of view as the user's head moves relative to the user's body; as well as A three-dimensional virtual image is displayed by the fixed distance display system at a distance between a first distance from the user's eye corresponding to 0.2 diopters and a second distance from the user's eye corresponding to 0.6 diopters.

2. The method according to claim 1, further comprising: The image is analyzed by the processor to match the user's field of view to the user's body position.

3. The method according to claim 1, wherein: The body-centric virtual user interface includes a plurality of user interface objects displayed in an arc surrounding the user's body at the fixed distance.

4. The method according to claim 3, wherein: The plurality of user interface objects includes a corresponding plurality of icons.

5. A method for displaying, comprising: Calculate the user's head pose and body pose; Perform machine vision optical flow analysis based on images; displaying a body-centric virtual user interface in a field of view of the user at a fixed distance and fixed position relative to the user's body based on the head pose of the user relative to the body pose of the user; as well as displaying a three-dimensional virtual image at a distance between a first distance from the user's eye corresponding to 0.2 diopters and a second distance from the user's eye corresponding to 0.6 diopters, Wherein, the body-centric virtual user interface is displayed at a distance and position relative to the user's head, and the distance and the position change with the movement of the user's head, so that when the user's head moves relative to the user's body, the body-centric virtual user interface moves in the user's field of view.

6. The method according to claim 5, further comprising: The images are analyzed to match the user's field of view to the user's body position.

7. The method according to claim 5, wherein: The body-centric virtual user interface includes a plurality of user interface objects displayed in an arc surrounding the user's body at the fixed distance.

8. The method according to claim 7, wherein: The plurality of user interface objects includes a corresponding plurality of icons.

9. A fixed distance display system comprising: a light source configured to generate a light beam to form a three-dimensional virtual image; as well as A processor configured to: Calculate the user's head pose and body pose; as well as Perform machine vision optical flow analysis based on images; wherein the fixed distance display system is configured to display a body-centric virtual user interface in the user's field of view based on the user's head posture relative to the user's body posture, wherein the body-centric virtual user interface is displayed at a distance and position relative to the user's head, the distance and the position varying with movement of the user's head such that when the user's head moves relative to the user's body, the body-centric virtual user interface moves in the user's field of view, and Wherein, the fixed distance display system is configured to display the three-dimensional virtual image at a distance between a first distance corresponding to 0.2 diopters from the user's eyes and a second distance corresponding to 0.6 diopters from the user's eyes.

10. The system according to claim 9, wherein: The processor is configured to match the user's field of view to the user's body position.

11. The system according to claim 9, wherein: The body-centric virtual user interface includes a plurality of user interface objects displayed in an arc surrounding the user's body at the fixed distance.

12. The system according to claim 11, wherein: The plurality of user interface objects includes a corresponding plurality of icons.

13. A fixed distance display system comprising: a light source configured to generate a light beam to form a three-dimensional virtual image; a first inertial measurement unit configured to measure a first value for calculating a head posture of a user; a third inertial measurement unit configured to measure a third value for calculating a body posture of the user; as well as wherein the fixed distance display system is configured to display a body-centric virtual user interface in the user's field of view based on the user's head posture relative to the user's body posture, wherein the body-centric virtual user interface is displayed at a distance and position relative to the user's head, the distance and the position varying with movement of the user's head such that when the user's head moves relative to the user's body, the body-centric virtual user interface moves in the user's field of view, and Wherein, the fixed distance display system is configured to display the three-dimensional virtual image at a distance between a first distance corresponding to 0.2 diopters from the user's eyes and a second distance corresponding to 0.6 diopters from the user's eyes.

14. The system according to claim 13, wherein: The first inertial measurement unit is positioned adjacent to the user's head. 15 . The system of claim 14 , further comprising a second inertial measurement unit configured to measure a second value for calculating a head posture of the user.

16. The system of claim 15, wherein: The second inertial measurement unit is positioned adjacent to the user's head.

17. The system of claim 13, wherein: The third inertial measurement unit is disposed adjacent to a waist of the user.

18. The system of claim 15, wherein: The first inertial measurement unit, the second inertial measurement unit, the third inertial measurement unit, and a camera are configured to match the field of view of the user to the body position of the user.

19. The system of claim 13, wherein: The body-centric virtual user interface includes a plurality of user interface objects displayed in an arc surrounding the user's body at the fixed distance.

20. The system of claim 13, further comprising a light source and a light guide optical element.

Images