Dynamic field of view variable focus display system

Abstract

An augmented reality (AR) device is described having a display system configured to adjust the apparent distance between a user of the AR device and virtual content presented by the AR device. The AR device includes a first tunable lens that changes shape so as to affect the position of the virtual content. Deformation of real-world content due to changes made to the first tunable lens is prevented by a second tunable lens that changes shape to remain substantially complementary to the optical configuration of the first tunable lens. In this way, virtual content can be located at nearly any distance relative to the user without degrading the view of the outside world or adding a significant amount of bulk to the AR device. The augmented reality device may also include a tunable lens for expanding the field of view of the augmented reality device.

Description

[0001] This application is a divisional application of the patent application with international application number PCT / US2018 / 023847, international application date March 22, 2018, Chinese national application number 201880019632.9, and title “Dynamic Field of View Variable Focus Display System”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims priority to U.S. Provisional Patent Application No. 62 / 475,081, filed on March 22, 2017, entitled “Dynamic Field Of View Variable Focus Display System.” Background Art

[0004] Augmented reality devices are designed to overlay virtual content on the real world. One challenge in merging virtual content naturally with real-world content is to merge the virtual content to an apparent depth that allows the virtual content to interact with real-world objects. Otherwise, the virtual content is more like a two-dimensional display that is not truly integrated into the three-dimensional real-world environment. Unfortunately, augmented reality systems that can display virtual content at different depths are too large or cumbersome to use comfortably or are only capable of displaying virtual content at discrete distances from the user. Another challenge in displaying virtual content to users is that certain types of displays may have a limited field of view that cannot provide truly immersive virtual content. For these reasons, a small form factor device that can accurately position virtual content at any desired distance across an immersive field of view would be needed. Summary of the invention

[0005] The present disclosure describes various embodiments related to an augmented reality device that can accurately display virtual content at a desired location relative to a user and other real-world objects within the user's field of view using tunable optics such as tunable lenses and / or prisms. The present disclosure also discusses using tunable lenses to expand the effective field of view of the augmented reality device.

[0006] To achieve this, each display of an augmented reality (AR) device may include a tunable lens configured to change its optical configuration to adjust the apparent position of virtual content presented to a user without distorting the user's view of real-world objects. A first tunable lens may be located between the user and a waveguide that transmits light representing the virtual content. The first tunable lens may then be reshaped to change the apparent distance between the user of the AR device and the virtual content. A second tunable lens is located between the user and light entering the user's eyes from real-world objects. The second tunable lens is reshaped during operation of the AR device, and the reshaping is synchronized with the reshaping of the first tunable lens so that the second tunable lens can offset any changes made by the first tunable lens that would degrade the user's view of real-world objects.

[0007] Optical manipulation components in the form of one or more tunable prisms may also be used to expand the effective field of view of the augmented reality device. The tunable prism used as a two-dimensional optical manipulation device can be configured to sequentially shift light passing through the tunable prism in a series of different directions to expand the effective field of view of the augmented reality device. In some embodiments, the optical manipulation device may take the form of a liquid crystal prism that is capable of dynamically changing its phase distribution based on an electrical signal applied to the liquid crystal prism.

[0008] An augmented reality device is disclosed, comprising: a first tunable lens; a second tunable lens; a waveguide located between the first tunable lens and the second tunable lens, the waveguide being configured to guide light representing virtual content through the first tunable lens and toward a user of the augmented reality device; and a processor being configured to: direct the first tunable lens to change shape to change the apparent distance between the virtual content and the user of the augmented reality device, and direct the second tunable lens to change shape to maintain the apparent distance between real-world objects and the user of the augmented reality device.

[0009] An augmented reality device is disclosed, comprising: a first tunable lens; a second tunable lens; and a waveguide located between the first tunable lens and the second tunable lens, the waveguide comprising a diffractive optical device, the diffractive optical device configured to guide light representing virtual content through the first tunable lens and toward a user of the augmented reality device. The first tunable lens and the second tunable lens change shape in coordination to adjust the apparent distance between the user of the augmented reality device and the virtual content.

[0010] A wearable display device is disclosed, comprising the following: a first tunable prism; a second tunable prism; a light source configured to emit light representing virtual content; and a waveguide located between the first tunable prism and the second tunable prism, the waveguide comprising a diffractive optical device configured to diffract light emitted from the light source through the first tunable prism and toward a user of the wearable display device. The first tunable prism is configured to displace at least a portion of light received from the light source and exiting the waveguide through a peripheral region of the first tunable prism toward an eye of the user.

[0011] A wearable display device includes: a projector; and an eyepiece including one or more optical manipulation components. The wearable display device also includes a waveguide configured to receive light from the projector and direct it through the one or more optical manipulation components toward a user; an eye gaze tracker configured to detect movement of one or both eyes of the user; and a control circuit communicatively coupled to the one or more optical manipulation components and the eye gaze tracker, the control circuit configured to adjust an optical manipulation pattern of the one or more optical manipulation components based on the detected movement of one or both eyes of the user.

[0012] Other aspects and advantages of the present invention will become apparent from the following detailed description taken in conjunction with the accompanying drawings, which illustrate, by way of example, the principles of the described embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The present disclosure will be readily understood by the following detailed description taken in conjunction with the accompanying drawings, in which like reference numerals denote like structural elements, and in which:

[0014] Figure 1 An exemplary user wearing an augmented reality (AR) device is shown;

[0015] Figure 2A A display system capable of displaying projected virtual content at any apparent distance relative to a user of the display system is shown;

[0016] Figure 2B A display system capable of adjusting the apparent distance to virtual content without affecting the appearance of real-world content according to the described embodiments is shown;

[0017] Figure 3 A top view of one particular configuration according to the described embodiments is shown, in which the diffractive optics of a waveguide are arranged to direct three different colors of light emitted by a projector between tunable lenses and then towards a user.

[0018] Figures 4A-4BA transparent display of an AR device displaying first virtual content and second virtual content according to the described embodiments is shown;

[0019] Figure 5A-5B A side view of a tunable lens is shown according to the described embodiments and how the tunable lens can be adjusted to accommodate different virtual content positions.

[0020] Figure 5C-5D shows how a tunable lens can be adjusted to accommodate the motion of multiple independently moving virtual objects in accordance with the described embodiments;

[0021] Figure 6 A flow chart depicting a method for displaying virtual content at multiple depths using a small form factor AR device is shown;

[0022] Figure 7A-7B Various embodiments configured to direct light from a display device into a user's eyes are shown;

[0023] Figures 8A-8C illustrates an exemplary scanning pattern that may be emitted by a display device to expand the field of view of the display device;

[0024] Figures 9A-9C Shows Fig. 8A how the first scanning pattern shown in can be shifted around the display area;

[0025] Figures 10A-10E Various configurations for optical manipulation devices are shown;

[0026] Figures 11A-11B An optical manipulation device is shown that may include lenses stacked on top of each other to shift incident light vertically and horizontally;

[0027] Figures 11C-11D 10 and 11 respectively show a cross-sectional side view and a top view of a liquid crystal lens 1140 having a Fresnel lens configuration; and

[0028] Figures 12A-12B shows how an optical manipulation device can be incorporated into an augmented reality display device; and

[0029] Figures 12C-12F A display device configured to receive multiple image streams is shown. DETAILED DESCRIPTION

[0030] Representative applications of the methods and apparatus according to the present application are described in this section. These examples are provided merely to increase background and aid in understanding the described embodiments. Therefore, it will be apparent to one skilled in the art that the described embodiments may be practiced without some or all of these specific details. In other cases, well-known process steps are not described in detail to avoid unnecessarily obscuring the described embodiments. Other applications are possible, such that the following examples should not be considered limiting.

[0031] Augmented reality (AR) devices are configured to overlay virtual content on the real world. Virtual content can include information about nearby real-world objects or people. In some cases, virtual content is only applicable to a general area and may not need to be associated with any viewable real-world objects. However, in many cases, it is desirable to merge virtual content with real-world objects. For example, virtual content can include characters that interact with users and / or objects in the real world. In order to merge virtual content in a more realistic way, virtual content can be displayed in a way that makes the virtual content appear to be located at a certain distance away from the user, corresponding to the real-world object that the virtual content is interacting with. This co-location of virtual and real-world content can help improve user immersion. Unfortunately, many AR devices are only configured to display content at a single fixed distance from the user, which may affect the degree to which virtual content is merged into the real-world environment. This limitation may be more obvious when the virtual content is traveling directly toward or away from the user, because the obvious change in depth can limit the object to appear larger or smaller. The ability to accurately depict depth information is also beneficial in the display of a virtual reality (VR) environment, where the virtual content hides the user's view of real-world objects.

[0032] One solution to establishing virtual content at a variable distance from a user of an AR device is to incorporate a tunable lens into the transparent display system of the AR device. The tunable lens can be configured to cooperate to change the apparent position of the virtual content relative to the user. The tunable lens or zoom element can take many forms, including, for example, a liquid crystal lens, a tunable diffractive lens, or a deformable mirror lens. In general, any lens that can be configured to change shape or configuration to adjust the incident light in a manner that changes the apparent depth of the virtual content of the AR device can be applied. The tunable nature of the lens or zoom element advantageously allows the virtual content to appear to be located at almost any distance from the user of the AR device.

[0033] Tunable lenses can be located on the front-facing and rear-facing surfaces of a transparent or translucent display system. A first tunable lens located on the rear-facing or user-facing side of the display can be configured to alter the incident light generated by the AR device so that the incident light displays virtual content that appears to be a desired distance from the AR device. A second tunable lens located on the front-facing or world-facing side of the display can be configured to cooperate with the first tunable lens by adopting a complementary configuration that offsets at least some of the adjustments made by the first tunable lens. In this way, light that is reflected back from real-world objects and passes through the first and second tunable lenses before reaching the user's eyes is not substantially deformed by the first tunable lens.

[0034] In some embodiments, the second tunable lens can allow some of the changes made by the first tunable lens to be applied to light arriving from real-world objects. For example, the tunable lens can be configured to apply myopia, hyperopia, and / or astigmatism correction to a user who would benefit from vision correction. These types of corrections can be applied equally to light associated with virtual content and real-world objects. The correction can take the form of an offset between the first tunable lens and the second tunable lens. In such a configuration, the second tunable lens will not be completely complementary to the first tunable lens, as some of the first tunable lens changes will also be applied to the view of the real-world object.

[0035] In some embodiments, a second tunable lens can be used periodically to distort the real-world view, rather than just counteracting the effects of adjustments made by the first tunable lens. In this way, combinations of tunable lenses can provide enhanced virtuality, mediated reality, and other types of experiences that manipulate real and virtual content.

[0036] In some types of display devices, the refractive index of certain optical components can limit the ability of the display device to generate a large enough field of view to provide an immersive augmented reality experience to the user. One solution to this problem is to equip the display device with a tunable lens. The tunable lens can be used as an optical manipulation device by shaping the lens to shift the light emitted along the periphery of the device toward the user's eyes. In this way, the tunable lens can significantly increase the effective viewing angle. In some embodiments, the position where the light emerges from the display device can be sequentially shifted in a repeating scanning pattern to produce a composite image. The optical manipulation device can be sequentially reshaped to optimize the optical manipulation device with respect to each position in the scanning pattern. For example, the first position in the scanning pattern can be located at the far right of the display device, while another position in the scanning pattern can be located near the bottom of the display device. By changing the optical manipulation device from shifting the light to the left in the first position to shifting the light upward in the second position, the user can enjoy a wider field of view. By continuing to update the optical manipulation device according to the current position of the scanning pattern, portions of the light that would otherwise fall outside the user's field of view become visible.

[0037] Reference below Figure 1-12F These and other embodiments are discussed; however, those skilled in the art will readily appreciate that the detailed description given herein with respect to these figures is for illustration purposes only and should not be construed as limiting.

[0038] Figure 1An exemplary user wearing an augmented reality (AR) device 100 is shown. The AR device 100 can be configured to display virtual content that appears to be located at various locations across the room 102. For example, virtual content 104 can be overlaid on a wall object 106. The wall object 106 can take the form of a picture or a television mounted on the wall of the room 102. In this way, the appearance of the wall object 106 can be changed by the virtual content 104. Similarly, the AR device 100 can be configured to project virtual content 108 on the sofa 110 in a manner that creates the impression that an object or person is resting on the sofa. However, in order to realistically depict virtual content related to other objects in the room 102, it is also important to establish virtual content at a comparable distance from the user. Depth detection sensors can be used to characterize the distance of various objects from the user. Then, the information retrieved by the depth sensor can be used to establish the distance of virtual content associated with objects adjacent to the virtual content. This becomes more complicated when the virtual object changes the distance from the AR device 100. For example, virtual content 112 may take the form of a person walking along a motion path that takes the person around table 114. As virtual content 112 moves from position 112-1 to position 112-2, data retrieved by the depth sensor of AR device 100 may be used to define a motion path that avoids table 114. In order to accurately depict the position of virtual content 112 throughout its motion path, the perceived distance between AR device 100 and virtual content 112 should be continually reduced.

[0039] Figure 2A A display system 200 capable of displaying projected content at any distance is shown. A projector 202 can display virtual content on a tunable lens 204. The tunable lens 204 can then change its optical configuration to adjust the depth at the projected content being displayed. The tunable lens 204 can leverage any of a number of technologies, including, for example, a liquid crystal lens. When the tunable lens 204 is a liquid crystal lens, the lens can be configured to change its phase distribution according to the amount of voltage applied to the liquid crystal lens. Although this configuration works well to adjust the depth of the virtual content, the light arriving from the real-world objects 206 and 208 will be undesirably deformed. For example, as shown by the double-headed arrows, the apparent positions of the real-world objects 206 and 208 can be closer or farther away from the user. Therefore, because one purpose of augmented reality is to enable the user to maintain sight of most of the real world, it may be problematic to use a display system 200 with an AR device due to the undesirable deformation of the light from the real-world objects.

[0040] Figure 2BA display system 250 is shown that is capable of adjusting the apparent distance to virtual content without affecting the appearance of real-world content. This is achieved by projecting virtual data using a waveguide 258 between tunable lenses 254 and 256, which redirects light from a projector 252 between the tunable lenses 254 and 256 and then passes through the tunable lens 254 and toward the user's eyes. In this way, the light emitted by the projector 252 can be adjusted by the tunable lens 254. The tunable lens 256 can be configured to adjust in an opposite manner to the tunable lens 254. The effect of this is that any light originating from a real-world object 206 or 208 can pass through the depth display system 250 essentially unaffected. In this way, the virtual content from the projector 252 can be the only content that experiences a focus shift, resulting in a shift in apparent position limited to the virtual content emitted by the projector.

[0041] While tunable lens 256 may be configured to prevent any changes made by tunable lens 254 from being applied to the perception of real-world objects, in some embodiments, tunable lens 256 may be configured to cooperate with tunable lens 254 to, for example, correct the user's vision. The vision correction may result in a multi-diopter change applied by tunable lens 256, which may be applied equally to real-world objects 206 and 208 because tunable lens 256 may not completely cancel the effects of tunable lens 254. For example, tunable lens 254 may be reconfigured to apply a +2 diopter adjustment. Tunable lens 256 may then apply no diopter adjustment at all, so that both virtual content 210 and real-world objects experience a +2 diopter change, thereby allowing a user who would normally require +2 diopter vision correction to wear display system 250 without any additional vision correction. Using such a vision correction scheme, movement of virtual content 210 may involve changing the diopter adjustment of tunable lens 254 to +3 and changing the diopter adjustment of tunable lens 256 to -1 in order to maintain a +2 diopter shift for vision correction. Similarly, tunable lens 254 may be configured to apply an astigmatism adjustment that is not offset by tunable lens 256.

[0042] Figure 2BThe configuration shown can operate in other ways that allow the tunable lens to apply different effects. In some embodiments, the tunable lens can be configured to purposefully defocus real-world objects to allow the user to focus on the virtual content 210-1. For example, it may be desirable for a software developer to focus the user's attention on a message in a controlled gaming or entertainment environment or even enter a more immersive virtual environment. By defocusing real-world objects, the system will allow the system to mask any distracting real-world stimuli without having to generate light to block the field of view across the entire display. In this way, tunable optics can be used to shape augmented reality experiences.

[0043] Figure 3 A top view of a particular configuration is shown in which the diffractive optics of a waveguide are arranged to direct three different colors of light emitted by a projector between tunable lenses and then direct the light toward a user. In some embodiments, waveguide 302 may include three discrete optical paths 304-1, 304-2, and 304-3 for different colors of light, such as red, green, and blue. Each of the optical paths 304 may utilize diffractive optics to direct light from projector 306 between tunable lenses 308 and 310 and then direct the light out through tunable lens 310 toward the user's eyes. When tunable lens 310 does not apply any changes to the light from waveguide 302, waveguide 302 may be arranged in such a way that the resulting virtual content appears to be located at an infinite distance. In such a configuration, tunable lens 310 may be configured to reduce the apparent distance between the user and the virtual content projected by the diffractive optics by varying the amount based on the desired position of the virtual content relative to the user and other real-world objects. As shown in FIG. Figure 3 As shown, light from real-world objects is substantially unaffected by tunable lenses 308 and 310 , while light passing through waveguide 302 is affected by tunable lens 310 .

[0044] Figures 4A-4B A transparent display 402 of an AR device 400 is shown displaying first virtual content 404 and second virtual content 406 . Figure 4AArrows are depicted showing how virtual content 404 and 406 move across the transparent display 402 over a particular period of time. During this movement, the virtual content 404 moves away from the AR device 400 and the virtual content 406 moves closer to the AR device 400. Because the transparent display 402 includes a tunable lens for adjusting the apparent depth of the virtual content, the upper region 408 of the transparent display 402 can be optically configured to display the virtual content 404 moving away from the AR device 400, and the lower region 410 of the transparent display 402 can be configured to display the virtual content 406 moving closer to the AR device 400. The transition region 412 can take the form of a region in which the shape of the tunable lens is gradually adjusted to accommodate different optical configurations and prevent a visual seam from appearing between the upper region 408 and the lower region 410. Depending on the amount of the difference between the regions 408 and 410, the transition region 412 can be larger or smaller. Although the real-world object 414 is located within the transition region 412, it should be appreciated that when the transparent display 402 includes two tunable lenses that cooperate to prevent deformation of the real-world object, even the transition region 412 has little effect on the appearance of the real-world object 414. Thus, a processor of the AR device 400 that attempts to determine appropriate regions of the display 402 for the upper region 408 and the lower region 410 will only need to consider the motion path of the virtual content when determining how to change the optical configuration for independently moving virtual content.

[0045] Figure 4B An exemplary embodiment is shown in which virtual content 454 is in motion and virtual content 456 remains stationary. In such a configuration, the motion region 458 may occupy a large portion of the viewable area of ​​the transparent display 402, while the stationary region 460 may occupy a much smaller area limited primarily to the virtual content 456. In addition, the motion region 458 may change the apparent distance between the AR device 400 and the virtual content 454, while the stationary region 460 may maintain the apparent distance to the virtual content 456. This narrow stationary region 460 may be even more convenient in situations where the user's head movement is considered unlikely or the position of the virtual content within the transparent display 402 is not controlled by the user's head movement. For example, the virtual content 456 may take the form of status information, such as the time of day, battery charge, or navigation information. This type of information may be distracting if it also moves with any other virtual content that the user is interacting with. It should also be noted again that the real-world content 464 remains unaffected by changes in apparent depth that are affected by the tunable lens of the transparent display 402.

[0046] Figure 5A-5BA side view of the tunable lens 502 is shown and how the tunable lens 502 can be adjusted to accommodate different virtual content positions. Figure 5A 504. The tunable lens 502 is shown as being substantially rectangular in shape and forming a lens element 504 within a rectangular volume. The lens element 504 can be configured to reshape the light emitted from the waveguide so as to establish virtual content at a desired distance from a user of the AR device. When the tunable lens 502 is in the form of a liquid crystal lens, the lens element 504 can change shape to a lens element 506 in response to a voltage applied to the tunable lens 502. The increased depth and curvature of the lens element 506 can cause the virtual content to appear closer to the AR device compared to the lens element 504. In this way, the tunable lens can be configured to change the apparent distance to the virtual content viewed from the AR device.

[0047] Figure 5C-5D 5 shows how to adjust the tunable lens 502 to accommodate the motion of multiple independently moving objects represented by virtual content. In particular, Figure 5C and 5D It can be shown how the tunable lens 502 will move to accommodate Figures 4A-4B The virtual content movement shown in . Figure 5C This may correspond to a situation where virtual contents 404 and 406 start at the same distance from the AR device. Figure 5D 5 shows how tunable lens 502 transitions from forming lens element 508 to lens element 510. Portions of lens element 510 corresponding to upper region 512 may have a thinner effective shape and a smaller effective curvature to give the appearance of virtual content 404 moving farther away from the AR device, while portions of lens element 510 corresponding to lower region 514 may have a thicker effective shape and a larger effective curvature to give the appearance of virtual content 406 moving closer to the AR device. Transition region 516 includes a gradient that smoothly changes the effective thickness of lens element 510 without creating visible lines that affect the real-world view through tunable lens 502.

[0048] Figure 6A flow chart depicting a method for displaying virtual content at multiple depths using a small form factor AR device is shown. At 602, a depth sensor of the AR device characterizes a real-world object within a field of view of a user of the AR device by determining a distance between the user and the real-world object. At 604, a processor of the AR device is configured to determine a position or motion path for a first virtual content relative to the characterized real-world object. At 606, an optical configuration of a first tunable lens of the AR device is configured for initially displaying the first virtual content. At 608, an optical configuration of a second tunable lens of the AR device is configured to prevent the first tunable lens from adversely affecting the view of the real-world object. This is achieved by an optical configuration of the second tunable lens that offsets at least a portion of the optical effect applied by the first tunable lens for the real-world object. It should be noted that in some cases, the second tunable lens can be complementary to the first tunable lens to offset the effect of the first tunable lens on the appearance of the real-world object. In some embodiments, certain visual enhancements can be applied by leaving some adjustments made by the first tunable lens unchanged. In some embodiments, when it is desired to display the second virtual content, the AR device can be configured to check whether the first virtual content and the second virtual content should be at the same distance from the user. At 612, the AR device can maintain the optical configuration by continuing to adjust the tunable lens to track the positions of the first virtual content and the second virtual content.

[0049] At 614, when the first virtual content and the second virtual content are at different distances from the user, the processor may be configured to apply different optical configurations to different areas of the AR device display using a tunable lens. In this way, virtual content at different distances from the user can be presented to the user. In some embodiments, when the user's attention is focused on the first virtual content, the second virtual content can be purposefully defocused. For example, once the user expects to interact with the second virtual content or a piece of software executed by the AR device queues up to interact with the second virtual content, the focus can be transitioned to the second virtual content. In some embodiments, the focus transition between virtual content at different distances from the user can be queued by an eye tracking sensor, which is configured to determine whether the user is focused on a particular virtual object. In other embodiments, the user can manually select virtual content for interaction, at which time the focus can be adjusted to appropriately depict the distance between the selected virtual object and the user. Imaging software can be used to apply a blur effect to any virtual content projected by the AR device that is outside the current depth plane to avoid any impression that all virtual content is at the same distance from the user.

[0050] Figure 7A-7B Various embodiments are shown that are configured to direct light from a display device into the eyes of a user. Fig. 7A 7 shows a top view of a display device 700 that includes a light projector 702 and a waveguide 704 that is configured to redirect light 706 emitted by the projector 702 toward a user's eye 708. While the waveguide 706 can be configured to emit images from thousands or even millions of locations, Fig. 7A Light emitted from five exemplary locations is shown, from which five output cones 710-1 to 710-5 are depicted. Each of the output cones 710 represents light emitted from each location spread across an angle 712, which may be referred to as a service angle. Fig. 7A As shown, the limited size of each output cone prevents light exiting the waveguide 704 along output cones 710-1 and 710-5 from reaching the user's eye 708. In some embodiments, angle 712 is limited below a desired threshold by certain characteristics of the display technology, such as the material refractive index of the waveguide 706. Additional details regarding lightfield and waveguide-based display devices are provided in U.S. Provisional Patent Application No. 62 / 539,934, entitled “HIGH RESOLUTION HIGHFIELD OF VIEW DISPLAY”.

[0051] Figure 7B 7 shows how the display device 700 incorporates one or more optical manipulation components, such as an optical manipulation device 714, which is configured to sequentially displace light 706 exiting the waveguide 704 in different directions to expand the effective viewing angle 712 to an angle 716. In this manner, the user's eye 708 is able to view a wider field of view due to the larger effective angle 716 created by displacing the output cones 710-1-710-5 in different directions. Figure 7B As shown, the expanded effective angle 716 allows at least some of the light from the output cones 710-1 and 710-5 to reach the user's eye 708. In some embodiments, the optical manipulation component 714 can take the form of one or more tunable prisms that can take a variety of different optical configurations (e.g., a liquid crystal lens with a reconfigurable phase distribution). Each optical configuration can be configured to shift the direction of the output cone 710 in a different direction. Although Figure 7B Only light 706 steered in two different directions is shown, it should be understood that light 706 can be steered in many other different directions. In addition, it should be understood that in addition to tunable prisms, other optical elements can also be configured to redirect light toward the user's eyes, and the exemplary prism embodiment should not be interpreted as limiting the scope of the present disclosure.

[0052] Examples of other such optical elements (e.g., time-varying gratings) are described in further detail in U.S. Patent Application No. 14 / 555,585. In some examples, polymer dispersed liquid crystal gratings or other tunable gratings may be implemented as optical manipulation components and used to manipulate output cones 710-1-710-5 by modifying the angle of the TIR waveguide light, the angle at which the light is coupled out through an outcoupling optical element of waveguide 704, or a combination thereof. In some embodiments, one or more metasurfaces (e.g., made of metamaterials) may be implemented as optical manipulation components. Further information on metasurfaces and metamaterials that may be used as optical manipulation components in various embodiments of the present disclosure may be found in U.S. Patent Publication No. 15 / 588,350, U.S. Patent Publication No. 15 / 182,528, and U.S. Patent Publication No. 15 / 182,511. Thus, it should be understood that the optical manipulation components may be switchable or otherwise controllable to operate in a discrete number of different manipulation states, and that the exemplary tunable optical manipulation devices should not be construed as limiting the scope of the present disclosure.

[0053] Figures 8A-8C Exemplary scanning patterns are shown that may be generated by a suitable display device and that may help expand the field of view of the display device. Fig. 8A A first scanning pattern including four different image positions 802, 804, 806, and 808 is shown. Each of the depicted image positions can represent an aggregation of light emitted from the display device at a specific point in time. In some embodiments, light can be delivered to positions 802-808 in a numerical order. The optical manipulation device can then be used to shift the light back to the user's eyes according to the activated image position. For example, when image position 808 is activated, the optical manipulation device can be configured to shift the light downward toward the user's eyes. When the display device presents a video source, the portion of the video frame corresponding to each image position can be sequentially displayed at each of the four positions for each frame of the video. For example, when the video source has a frame rate of 1 / 30 seconds, the corresponding portion of the video frame can be displayed at each position for 1 / 120 seconds. In this way, an expanded field of view can be achieved without reducing the frame rate. In this way, the resulting image created by the scanning pattern maintains a fluid frame rate and also generates a composite image having a significantly higher spatial resolution than that possible using a single static image position. For example, an image projector capable of displaying only 480 lines of vertical resolution can use the scanning techniques described above to reproduce an image or video source having more than 480 lines of resolution.Additional details regarding scanning modes, tiling functions, and tiled display configurations are provided in U.S. Patent Application No. 14 / 555,585.

[0054] Fig. 8AIt also shows how, in some embodiments, portions of adjacent image positions can overlap, e.g. Fig. 8A , as shown in the hash region in . This results in overlapping content within the image position. Overlapping can be used to further improve certain aspects of the composite image generated by the display device. For example, an increase in resolution within the central region 810 can be achieved by applying one or more super-resolution techniques. In particular, the overlapping portion of each image frame can be subsampled and slightly offset, thereby allowing an increase in pixel density instead of stacking pixels on top of each other. This produces a super-resolution effect in the portion of the display with an overlapping region. For example, in an embodiment where the display processor is capable of generating a 4K resolution image (i.e., 2160 lines of vertical resolution), the 4K resolution image can be used to achieve a super-resolution effect by distributing pixels within the overlapping region of the scan pattern and using an image source that is generally only capable of generating significantly lower resolutions. In addition, when a high frame rate video file is being displayed, each sequentially displayed frame can be associated with a different frame of the video. For example, when a video source at 120 frames per second is played back, the portion within the central region 810 of the display can enjoy a full frame rate of 120 frames per second, while the non-overlapping region is updated at a rate of only 30 frames per second. The overlapping region near the center region 810 may be refreshed at a rate of 60 or 90 frames per second, depending on the number of overlapping locations in a particular region.

[0055] Figure 8B A second scanning pattern with a large central area 810 is shown. This second scanning pattern causes one-ninth of the total image to be overlapped by each of the image positions 802-808. In this way, the resolution or frame rate within the central area 810 can be substantially greater than the non-overlapping areas. The depicted scanning pattern can achieve up to a four-fold increase in resolution or frame rate, which generally corresponds to the number of overlapping frames. This type of scanning pattern may be particularly beneficial when the content of interest is located in the central area of ​​the display. In some embodiments, the scanning pattern may be changed to produce an increased amount of overlap in the case where less virtual content is presented in the peripheral area of ​​the display.

[0056] Figure 8CIt is shown how the images located in each of the image positions 802-808 work together to generate a composite image 812, which takes the form of a lamp. In addition to creating a composite image 812 that is larger than any of the image positions 802-808, the sequential display of the images at each of the image positions 802-808 allows the optical properties of the display to be changed depending on which of the image positions 802-808 is currently active during a given scan. For example, when the image is displayed at image position 804, the optical properties can be adjusted to shift the light representing the base of the lamp 812 upward toward the user's eyes. In some embodiments, the location of the scan pattern can be located at a location of the display that places a virtual image such as the lamp 812 within the center area of ​​the scan pattern. This allows the center and / or important features of the image to be displayed at a greater number of image positions.

[0057] Figures 9A-9C Shows Fig. 8A How the first scanning pattern shown in can be shifted within the display area 900. Fig. 9A A first scan pattern is shown located at the upper left corner of the display area 900 at time t0. Fig. 9B A first scanning pattern shifted toward the upper right corner of the display area 900 at time t1 is shown. In some embodiments, the display device may include an eye gaze tracker. Sensor data provided by the eye gaze tracker can be used to shift the scanning pattern to a position within the display area 900 corresponding to the user's current focus point. In some embodiments, the sensor data can help keep the center area 810 in a position covering the user's foveal vision (i.e., the part of the user's vision with the highest visual acuity). By continuously adjusting the scanning pattern in this manner, the user's impression of immersion can be improved. This method can be particularly effective when the prominent content is frequently shifted away from the central portion of the display area 900. Exemplary systems and techniques for performing foveated tracking, rendering foveated virtual content, and displaying foveated virtual content to the user are further described in detail in U.S. Provisional Patent Application No. 62 / 539,934, entitled "HIGHRESOLUTION HIGH FIELD OF VIEW DISPLAY".

[0058] Fig. 9C The first scan pattern is shown again shifted towards the lower portion of the display area 900 . Figures 9A-9CAlso shown is how image positions 802-808 may change and / or better represent the content provided to the user based on the position of the first scanning pattern. For example, the area over which the first scanning pattern extends may be reduced to better represent a virtual image that is much smaller than the size of the standard scanning pattern. In some embodiments, changing the scanning pattern may help optimize the overlapping area of ​​the scanning pattern for a particular representation of virtual content.

[0059] Figures 10A-10E Various phase distributions of optical manipulation devices similar to optical manipulation device 714 are shown. Specifically, Fig. 10A A front view of an exemplary optical manipulation device 1002 is shown in a first optical configuration 1004 for vertically displacing light. Fig. 10B A cross-sectional side view of the optical manipulation device 1002 according to section line AA is shown. The optical manipulation device 1002 can take the form of a liquid crystal lens that is capable of changing its phase distribution depending on the amount of voltage applied to the liquid crystal lens. More specifically, the optical manipulation device 1002 may include two conductive layers having a structure that is capable of generating an electric field in response to application of a voltage thereto. In this way, the optical manipulation device 1002 can switch between multiple different optical configurations. The first optical configuration 1004 can have a phase distribution of multiple refractive indices within the optical manipulation device 1002. In some examples, the local refractive index of the optical manipulation device 1002 can be customized to satisfy a prismatic function or another desired optical function. In at least some of these examples, the optical manipulation device 1002 can exhibit a relatively large phase gradient (e.g., ~πrad / m). In particular, the refractive index can vary according to a sawtooth distribution, such as Fig. 10B As shown, each of the teeth can be configured to receive a portion of light 1006 and emit light 1008 in a different direction. The size and / or spacing of the teeth can be adjusted to reduce or increase the variation in the angle of light passing through the optical manipulation device 1002. For example, as the pattern of wedges approaches the center area of ​​the display, the angle of each wedge can gradually decrease. This first optical configuration 1004 can be used to shift the frame of a scanning pattern located in the lower center area of ​​the display.

[0060] Fig. 10C A front view of the optical manipulation device 1002 in a second optical configuration 1010 is shown. Fig. 10D 1 shows a cross-sectional top view of the optical control device 1002 in a second optical configuration according to the section line BB. In the second optical configuration, the optical manipulation device 1002 is configured to laterally displace the light 1012. Fig. 10CAs shown, optical manipulation device 1002 receives light 1012 and outputs laterally displaced light 1014. While both the first and second illustrated optical configurations displace the direction of light, they do so only vertically or horizontally. It should be understood that in some embodiments, two optical manipulation devices may be layered on top of each other to displace the received light both vertically and horizontally.

[0061] Fig. 10E It is shown how, by orienting a series of toothed ridges diagonally across the optical manipulation device 1002 , light can be displaced both vertically and horizontally by a single optical manipulation device 1002 . Fig. 10E It is also shown how the depicted diagonal configuration can achieve the displacement of light that would otherwise utilize one-dimensional optical manipulation devices 1016 and 1018. The depicted configuration can be employed by optical manipulation device 1002 during a frame of a scanning mode in which light exits the waveguide through the lower right region of the display area. It should be noted that while a number of optical configurations have been depicted, optical manipulation device 1002 can be rearranged in many other configurations not depicted.

[0062] Figures 11A-11B It is shown how an optical manipulation device can include lenses stacked on top of each other to shift incident light vertically and horizontally. Fig.11A A perspective view of the phase shift of an optical manipulation device 1100 is shown, which includes a horizontal shift lens 1102 and a vertical shift lens 1104. The two lenses can be stacked on top of each other to redirect the light incident to the lenses. This optical configuration allows the incident light to be shifted vertically and horizontally. Fig. 11B An optical manipulation device 1120 with reduced thickness is shown by using an array of multiple lenses. In some embodiments, liquid crystal lenses can be used to form an optical configuration equivalent to multiple horizontal shift lenses 1102 and vertical shift lenses 1104, such as Fig. 11B shown.

[0063] Figures 11C-11D A cross-sectional side view and a top view of a liquid crystal lens 1140 having a Fresnel lens configuration are shown, respectively. The Fresnel lens configuration may take the form of an optical manipulation device to magnify or de-magnify selected virtual content. Specifically, Fig. 11C A Fresnel lens configuration is depicted that is configured to vary the magnification of light that is laterally displaced through the liquid crystal lens 1140. This type of configuration can be incorporated into any of the optical manipulation devices described herein. In some embodiments, a user can request to zoom in on a particular area of ​​the screen. In response, the optical manipulation device can be configured to form a Fresnel lens configuration over the particular area in order to zoom in on the content without having to change or update the light that produces the particular image stream that the user is viewing.

[0064] Figures 12A-12B Different ways in which an optical manipulation device may be incorporated into an augmented reality display device are shown. In some embodiments, the optical manipulation device may take the form of a liquid crystal lens that is capable of selectively changing the refractive index to change the direction, perspective, and / or magnification of light incident thereon. Fig. 12A A cross-sectional top view of display device 1200 is shown. Optical manipulation devices 1202 and 1204 are located on opposite sides of a waveguide 1206 of display device 1200. Waveguide 1206 may include one or more discrete paths for delivering light of different colors to an eye 1208 of a user of display device 1200. Optical manipulation devices 1202 and 1204 may have substantially complementary configurations that allow light 1210 reflected from real-world objects to pass through both optical manipulation devices to reach eye 1208 in a substantially undistorted manner. Light 1212 carried by waveguide 1206 may then be configured to visualize virtual content by undergoing beam manipulation according to one or more scanning patterns without adversely deforming light 1210. The scanning patterns and sequential beam manipulation described previously increase the effective field of view of display device 1200. Fig. 12B It is shown how display device 1220 and optical manipulation devices 1202 and 1204 are combined with zoom lens 1209 so that both dynamic focus shift and field of view expansion can be applied to light 1212.

[0065] Figures 12C-12F A display device configured to receive multiple image streams is shown. Fig. 12C A top cross-sectional view of a display device 1240 including waveguide 1206 and waveguide 1244 is shown. Waveguide 1244 may be configured to carry light 1246 that forms a wide field of view, low resolution image stream. Fig. 12CAs shown, the light 1246 does not undergo any scanning pattern before reaching the eye 1208. The waveguide 1206 can be configured to carry light 1212, which forms a narrow field of view, high-resolution image stream. The light 1212 entering the waveguide 1244 can be configured to leave the display device 1240 in the area where the sensor indicates that the user's eyes are focused. In this way, a dynamic foveation of a narrow field of view, a high-resolution image stream can be achieved, resulting in giving the user the impression that the entire display device 1240 is emitting a high-resolution image. The light 1212 can be projected across the surface of the display device, and the optical manipulation device 1202 can dynamically manipulate the light 1212 in a scanning pattern that increases the user's effective field of view. Even when the user is focusing on content near the edge of the display area of ​​the display device, the resulting magnification of the image stream generated by the light 1212 due to the scanning pattern can help the area of ​​the user's field of view that can discern high resolution is completely covered by the light 1212. In this manner, because display 1240 need not display a high-resolution image across the user's entire field of view, significant savings in hardware costs and processing power can be achieved.

[0066] Fig.12D A front view of display device 1240 is shown along with a representation of the image perceived by the user when light 1212 and 1246 is projected onto the retina of the user's eye 1208. Fig.12D The portion of waveguide 1244 that is capable of emitting light and the area of ​​waveguide 1244 that emits light 1212 are shown. Light 1212 is shown to produce a narrow field of view image stream that shifts in a scanning pattern. Light 1246 provides a wide field of view that remains stationary. In some embodiments, the area where light 1246 extends represents the maximum viewable area through which light can be viewed without an optical steering device. Fig.12D As shown, because light 1212 benefits from optical manipulation device 1202 which displaces light 1212 toward eye 1208 , a portion of light 1212 may extend outside of the area covered by light 1246 .

[0067] Fig.12E The optical manipulation devices 1262 and 1264 are shown added to Fig. 12C1206. In this embodiment, light 1212 generating a narrow field of view, high resolution image may be directed through waveguide 1244 and light 1246 generating a wide field of view, low resolution image may be directed through waveguide 1206. Optical manipulator 1262 is configured to independently manipulate light 1212. Optical manipulator 1264 is configured to prevent light 1210 and light 1246 from being deformed by manipulation by optical manipulator 1262. Optical manipulator 1264 may maintain a phase distribution that is substantially complementary to the phase distribution of optical manipulator 1262. In this manner, virtual content generated by light 1246 and 1212 may extend across the expanded field of view, further improving the immersive experience for a user of display device 1260. It should be noted that in some embodiments, the functionality of optical manipulators 1202 and 1264 may be combined into a single optical manipulator capable of presenting a phase distribution that shifts light 1246 in a desired scanning pattern and pre-compensates for any disturbances generated by optical manipulator 1262 .

[0068] In some embodiments, the lights 1246 and 1212 generated by one or more projectors of the display device 1260 can be configured to display the image streams at substantially the same spatial resolution. Then, the optical manipulation devices 1202 and 1262 can be configured to function independently to apply a scanning pattern to the lights 1212 and 1246 so as to maximize the effective field of view of the display device 1260. The spacing between the waveguides 1206 and 1244 can be configured to generate different apparent distances between the user and the virtual content generated by the lights 1212 and 1246. In this way, the depth perception distance can be adjusted without a set of zoom lenses, such as Fig. 12B It should be noted that in some embodiments, waveguides at different distances and varifocal lenses may be used in combination to simultaneously display virtual content at multiple different apparent distances from the eye 1208. The varifocal lens may then be used to change the apparent distance between the eye 1208 and the previously described virtual content.

[0069] Fig.12F A front view of display device 1260 is shown along with a representation of the image perceived by the user when light 1212 and 1246 are projected onto the retina of the user's eye 1208. Specifically, Fig.12F It is shown how light 1212 has a narrow field of view shifted in a first scanning pattern and how light 1246 can provide a wide field of view shifted in a second scanning pattern different from the first scanning pattern. In some embodiments, the scanning patterns can have the same size and / or resolution. Such a configuration can have the benefit of changing the apparent distance between the user and the virtual content.

[0070] The various aspects, embodiments, implementations or features of the described embodiments may be used alone or in any combination. The various aspects of the described embodiments may be implemented by software, hardware, or a combination of hardware and software. The described embodiments may also be embodied as computer-readable code on a computer-readable medium for controlling manufacturing operations or as computer-readable code on a computer-readable medium for controlling a manufacturing line. A computer-readable medium is any data storage device that can store data that can then be read by a computer system. Examples of computer-readable media include read-only memory, random access memory, CD-ROM, HDD, DVD, magnetic tape, and optical data storage devices. Computer-readable media may also be distributed on network-coupled computer systems so that computer-readable code is stored and executed in a distributed manner.

[0071] For the purpose of illustration, the foregoing description uses specific terms to provide a thorough understanding of the described embodiments. However, it will be apparent to those skilled in the art that specific details are not required in order to practice the described embodiments. Therefore, the foregoing descriptions of specific embodiments are provided for the purpose of illustration and description. They are not intended to be exhaustive or to limit the described embodiments to the precise forms disclosed. It will be apparent to those of ordinary skill in the art that many variations and modifications are possible in view of the above teachings.

Claims

1. An augmented reality device, comprising: a first tunable lens; a second tunable lens; a first optical manipulation device; a second optical manipulation device; a waveguide located between the first tunable lens and the second tunable lens and between the first optical manipulation device and the second optical manipulation device, the waveguide configured to direct light representing virtual content through the first tunable lens and the first optical manipulation device and toward a user of the augmented reality device; a light source configured to emit light representing the virtual content according to a scanning pattern, wherein the scanning pattern causes light to be emitted through different peripheral regions of the first optical manipulation device; as well as A processor configured to: directing the first tunable lens to change shape to modify an apparent distance between the virtual content and the user of the augmented reality device, directing the second tunable lens to change shape to maintain an apparent distance between a real-world object and the user of the augmented reality device, The first optical manipulation device is shaped to displace at least a portion of light from the light source received at one or more peripheral regions of the first optical manipulation device toward a central region so as to displace light emitted along a periphery of the augmented reality device toward eyes of a user.

2. The augmented reality device according to claim 1, wherein: The first optical manipulator is located between the waveguide and the first tunable lens.

3. The augmented reality device according to claim 2, wherein: The first tunable lens is configured to differently adjust a plurality of regions such that a first portion of the virtual content may be displayed at a first apparent distance from the user and a second portion of the virtual content may be displayed at a second apparent distance from the user.

4. The augmented reality device according to claim 1, wherein: The waveguide includes diffractive optics that redirect light received from the projector toward the user's eyes.

5. The augmented reality device according to claim 1, wherein: The first tunable lens and the second tunable lens are selected from the group consisting of a diffractive lens, a fluid lens, a mechanical lens, and an electrowetting lens.

6. The augmented reality device according to claim 1, wherein: The waveguide includes a diffractive optic configured to direct light representing the virtual content through the first tunable lens and the first optical manipulation device and toward a user of the augmented reality device.

7. The augmented reality device according to claim 6, wherein: When the first tunable lens and the second tunable lens change shape in coordination, light from a real-world object passes through the first tunable lens and the second tunable lens.

8. The augmented reality device according to claim 6, wherein: The second tunable lens is configured to prevent the first tunable lens from distorting the appearance of real-world objects visible through the waveguide.

9. The augmented reality device according to claim 6, wherein: The first tunable lens and the second tunable lens are configured to correct vision of the user of the augmented reality device.

10. The augmented reality device according to claim 6, wherein: The second tunable lens is further configured to blur a view of real-world objects in an attempt to focus the user's attention on the virtual content.

11. The augmented reality device according to claim 6, wherein: The first tunable lens defines a first region for stationary virtual content at a first distance and a second region for moving virtual content at a second distance different from the first distance.

Images