MIXED REALITY SYSTEM WITH VIRTUAL CONTENT WARPING AND METHOD OF USING SAME TO GENERATE VIRTUAL CONTENT - Patent application

Abstract

To provide mixed reality systems with virtual content warping, and methods of generating a mixed reality experience including warped virtual content using the same.SOLUTION: A computer implemented method for warping virtual content comprises receiving rendered virtual content data including a far depth. The method also comprises receiving movement data indicating a user movement in a direction orthogonal to an optical axis. The method further comprises generating warped rendered virtual content data based on the rendered virtual content data, the far depth, and the movement data.SELECTED DRAWING: None

Description

[Technical field]

[0001] The present disclosure relates to a mixed reality system that involves warping virtual content and a method for using the same to generate a mixed reality experience that includes the warped virtual content. [Background technology]

[0002] Modern computing and display technologies have encouraged the development of "mixed reality" (MR) systems for so-called "virtual reality" (VR) or "augmented reality" (AR) experiences in which digitally reproduced images or parts thereof are presented to a user in a manner that appears or can be perceived as real. VR scenarios typically involve the presentation of digital or virtual image information without transparency to actual real-world visual input. AR scenarios typically involve the presentation of digital or virtual image information as an extension of the visualization of the real world around the user (i.e., transparent to real-world visual input). Thus, AR scenarios involve the presentation of digital or virtual image information with transparency to the real world around the user.

[0003] Various optical systems generate images at various depths for displaying MR (VR and AR) scenarios. Some such optical systems are described in U.S. Utility Patent Application No. 14 / 555,585, filed November 27, 2014 (Attorney Docket No. ML.20011.00), the contents of which are expressly and completely incorporated herein by reference in their entirety as if set forth in their entirety.

[0004] MR systems typically employ a wearable display device (e.g., a head-mounted display, helmet-mounted display, or smart glasses) that is at least loosely coupled to the user's head and thus moves as the user's head moves. When the user's head movement is detected by the display device, the data being displayed can be updated to account for changes in head pose (i.e., the orientation and / or location of the user's head).

[0005] As an example, when a user wearing a head mounted display device views a virtual representation of a virtual object on the display device and walks around the area in which the virtual object appears, the virtual object can be rendered per viewpoint (corresponding to the position and / or orientation of the head mounted display device) to give the user the perception of walking around an object occupying real space. When the head mounted display device is used to present multiple virtual objects at different depths, the head pose measurements can be used to render the scene to match the user's dynamically changing head pose and provide increased immersion. However, there is an unavoidable delay between the steps of rendering the scene and displaying / projecting the rendered scene.

[0006] Head-mounted display devices enable AR to provide simultaneous viewing of both real and virtual objects. With an "optical see-through" display, a user can look through a transparent (or semi-transparent) element in the display system and directly view light from real objects in the real-world environment. The transparent element, often referred to as a "combiner," superimposes the light from the display over the user's view of the real world, and the light from the display projects an image of the virtual content over the see-through view of the real objects in the environment. A camera may be mounted on the head-mounted display device to capture images or video of the scene being viewed by the user.

[0007] MR systems attempt to present photorealistic immersive MR scenarios. However, head movements during the lag time between the generation of virtual content ("rendered virtual content") and the display of the rendered virtual content can result in visual artifacts (e.g., glitches). This problem can be exacerbated by fast head movements during the lag time.

[0008] To address this issue, some optical systems may include warping software / systems / methods that receive rendered virtual content from a source. The warping system then "warps" (i.e., transforms the "frame of reference") the received rendered virtual content for display within the display system's / viewer's frame of reference (the "display frame of reference"). This approach takes the originally rendered virtual content and attempts to transform the way the virtual content is presented so that, when displayed, the virtual content appears from a different perspective.

[0009] Some warping software / systems / methods warp the rendered virtual content to compensate for rotational or coarse positional user movements (i.e., the user's field of view, "FOV"). Such systems are not optimized to compensate for other types of user movements (e.g., substantial lateral user movements along an axis orthogonal to the user's optical axis). Thus, some warping software / systems / methods may introduce artifacts with user movements. Summary of the Invention [Means for solving the problem]

[0010] In one embodiment, a computer-implemented method for warping virtual content includes receiving rendered virtual content data, the rendered virtual content data including a depth of field. The method also includes receiving movement data indicative of user movement in a direction orthogonal to the optical axis. The method further includes generating warped rendered virtual content data based on the rendered virtual content data, the depth of field, and the movement data.

[0011] In one or more embodiments, the rendered virtual content data corresponds to a near virtual object and a far virtual object, and the warped rendered virtual content data corresponds to a warped near virtual object and a warped far virtual object. A first positional shift between the near virtual object and the warped near virtual object may be substantially equal to a second positional shift between the far virtual object and the warped far virtual object.

[0012] In one or more embodiments, generating the warped rendered virtual content data based on the rendered virtual content data, the depth, and the movement data reduces parallax-related artifacts compared to the rendered virtual content data. The parallax-related artifacts may be disocclusion artifacts. The parallax-related artifacts may be smear artifacts or jitter artifacts.

[0013] In one or more embodiments, the rendered virtual content data corresponds to a first depth segment. The method also includes receiving second rendered virtual content data corresponding to a second depth segment, the second rendered virtual content data including a second depth. The method further includes generating warped second rendered virtual content data based on the second rendered virtual content data, the second depth, and the movement data.

[0014] In another embodiment, a computer-implemented method for warping virtual content includes receiving rendered virtual content data, the rendered virtual content data including a far depth and a near depth. The method also includes receiving movement data indicative of user movement in a direction orthogonal to the optical axis. The method further includes generating warped rendered virtual content data based on the rendered virtual content data, the far depth, the near depth, and the movement data.

[0015] In one or more embodiments, generating the warped rendered virtual content data based on the rendered virtual content data, the far depth, the near depth, and the movement data includes averaging the far depth and the near depth.

[0016] In yet another embodiment, a computer-implemented method for warping virtual content includes receiving rendered virtual content data. The method also includes receiving movement data indicative of a user movement in a direction orthogonal to the optical axis. The method further includes receiving gaze data indicative of a user gaze. Further, the method includes determining a depth based on the gaze data. Additionally, the method includes generating warped rendered virtual content data based on the rendered virtual content data, the depth, and the movement data.

[0017] In yet another embodiment, a computer-implemented method for warping virtual content includes receiving rendered virtual content data. The method also includes receiving movement data indicative of user movement in a direction orthogonal to the optical axis. The method further includes generating mesh data from the rendered virtual content data. Further, the method includes determining depth data from the mesh data. Additionally, the method includes generating warped rendered virtual content data based on the rendered virtual content data, the mesh data, the depth data, and the movement data.

[0018] In one or more embodiments, the method also includes adjusting a mesh size when generating the mesh data. The mesh data may be generated using a grid mesh. The depth data may be determined based on a plurality of depths within the mesh and content of the mesh at each of the plurality of depths.

[0019] In another embodiment, a computer-implemented method for determining a depth of a virtual object includes reprojecting a first image of a virtual object corresponding to a first eye over a second image of the virtual object corresponding to a second eye at a plurality of depths to generate a plurality of individual combined images. The method also includes measuring a reprojection error in each of the plurality of individual combined images to generate a plurality of reprojection errors. The method further includes identifying a minimum reprojection error within the plurality of reprojection errors corresponding to the combined image of the plurality of individual combined images. Additionally, the method includes identifying a depth corresponding to the combined image as a depth of the virtual object.

[0020] In one or more embodiments, the depth of the virtual object is used to warp the virtual object from a first perspective to a second perspective. The plurality of depths may consist of 64 depths. Each of the plurality of depths may correspond to a one pixel difference in reprojecting the first image of the virtual object.

[0021] In yet another embodiment, a computer-implemented method for determining a minimum stabilized depth includes obtaining a point of view (POV) movement. The method also includes determining a relationship between the stabilized depth and pixel movement at the stabilized depth based on the POV movement. The method further includes obtaining a maximum undetectable pixel movement for the display system. Furthermore, the method includes identifying a stabilized depth corresponding to the maximum undetectable pixel movement for the display system as the minimum stabilized depth for the display system.

[0022] In one or more embodiments, the method also includes warping only content that is closer than a minimum stabilization depth to the viewer when the viewpoint changes. The relationship between the stabilization depth and pixel movement at the stabilization depth may be an inverse relationship.

[0023] Additional and other objects, features, and advantages of the present disclosure are set forth in the detailed description, drawings, and claims. The present invention provides, for example, the following: (Item 1) 1. A computer-implemented method for warping virtual content, comprising: receiving rendered virtual content data, the rendered virtual content data including depth; receiving movement data indicative of user movement in a direction perpendicular to the optical axis; generating warped rendered virtual content data based on the rendered virtual content data, the depth, and the motion data; A method comprising: (Item 2) the rendered virtual content data corresponds to near and far virtual objects; the warped rendered virtual content data corresponds to a warped near virtual object and a warped far virtual object. The method according to item 1. (Item 3) 3. The method of claim 2, wherein a first positional shift between the near virtual object and the warped near virtual object is substantially equal to a second positional shift between the far virtual object and the warped far virtual object. (Item 4) 2. The method of claim 1, wherein generating the warped rendered virtual content data based on the rendered virtual content data, the depth, and the movement data reduces parallax-related artifacts as compared to the rendered virtual content data. (Item 5) 5. The method of claim 4, wherein the parallax-related artifact is a disocclusion artifact. (Item 6) Item 7. The method according to item 4, wherein the parallax-related artifact is a smear artifact. 5. The method of claim 4, wherein the parallax-related artifact is a jitter artifact. (Item 8) The rendered virtual content data corresponds to a first depth segment, the method further comprising: receiving second rendered virtual content data corresponding to a second depth segment, the second rendered virtual content data including a second far depth; and generating warped second rendered virtual content data based on the second rendered virtual content data, the second depth, and the movement data; The method according to item 1, comprising: (Item 9) 1. A computer-implemented method for warping virtual content, comprising: receiving rendered virtual content data, the rendered virtual content data including a far depth and a near depth; receiving movement data indicative of user movement in a direction perpendicular to the optical axis; generating warped rendered virtual content data based on the rendered virtual content data, the far depth, the near depth, and the motion data; A method comprising: (Item 10) 10. The method of claim 9, wherein generating warped rendered virtual content data based on the rendered virtual content data, the far depth, the near depth, and the movement data includes averaging the far depth and the near depth. (Item 11) 1. A computer-implemented method for warping virtual content, comprising: Receiving rendered virtual content data; receiving movement data indicative of user movement in a direction perpendicular to the optical axis; receiving gaze data indicative of a user's gaze; determining a depth based on the gaze data; and generating warped rendered virtual content data based on the rendered virtual content data, the depth, and the motion data; A method comprising: (Item 12) 1. A computer-implemented method for warping virtual content, comprising: Receiving rendered virtual content data; receiving movement data indicative of user movement in a direction perpendicular to the optical axis; generating mesh data from the rendered virtual content data; determining depth data from the mesh data; generating warped rendered virtual content data based on the rendered virtual content data, the mesh data, the depth data, and the motion data; A method comprising: (Item 13) Item 13. The method of item 12, further comprising adjusting a mesh size when generating the mesh data. (Item 14) Item 13. The method of item 12, wherein the mesh data is generated using a grid mesh. (Item 15) Item 13. The method of item 12, wherein the depth data is determined based on a plurality of depths within the mesh and a content of the mesh at each of the plurality of depths. (Item 16) 1. A computer-implemented method for determining a depth of a virtual object, comprising: reprojecting a first image of the virtual object corresponding to a first eye onto a second image of the virtual object corresponding to a second eye at a plurality of depths to generate a plurality of distinct combined images; measuring a reprojection error in each of the plurality of individual combined images to generate a plurality of reprojection errors; identifying a minimum reprojection error within the plurality of reprojection errors corresponding to a combined image of the plurality of individual combined images; identifying a depth corresponding to the combined image as a depth of the virtual object; A method comprising: (Item 17) Item 17. The method of item 16, wherein the depth of the virtual object is used to warp the virtual object from a first perspective to a second perspective. (Item 18) Item 17. The method of item 16, wherein the plurality of depths consists of 64 depths. (Item 19) Item 17. The method of item 16, wherein each of the multiple depths corresponds to a one pixel difference in reprojecting the first image of the virtual object. (Item 20) 1. A computer-implemented method for determining a minimum stabilization depth, comprising: Obtaining the amount of point of view (POV) movement; determining a relationship between a stabilization depth and pixel movement at the stabilization depth based on the POV movement; Obtaining a maximum undetectable pixel shift for a display system; identifying a stabilization depth corresponding to a maximum undetectable pixel movement for said display system as a minimum stabilization depth for said display system; A method comprising: (Item 21) 21. The method of claim 20, further comprising warping only content that is closer to a viewer than the minimum stabilization depth when the viewpoint changes. (Item 22) 21. The method of claim 20, wherein the relationship between the stabilization depth and pixel movement at the stabilization depth is an inverse correlation. [Brief description of the drawings]

[0024] The drawings illustrate the design and utility of various embodiments of the present disclosure. It should be noted that the drawings are not drawn to scale, and elements of similar structure or function are represented by similar reference numerals throughout the drawings. In order to better understand how to obtain the foregoing and other advantages and objects of the various embodiments of the present disclosure, a detailed description of the present disclosure, briefly described above, will be given by reference to specific embodiments thereof as illustrated in the accompanying drawings. With the understanding that these drawings depict only exemplary embodiments of the present disclosure and therefore should not be considered as limiting its scope, the present disclosure will be described and explained with additional specificity and detail through the use of the accompanying drawings.

[0025] [Figure 1] FIG. 1 depicts a user's view of an augmented reality (AR) through a wearable AR user device, according to some embodiments.

[0026] [Figure 2A] 2A-2D diagrammatically depict an AR system and its subsystems, according to some embodiments. [Figure 2B] 2A-2D diagrammatically depict an AR system and its subsystems, according to some embodiments. [Figure 2C] 2A-2D diagrammatically depict an AR system and its subsystems, according to some embodiments. [Figure 2D] 2A-2D diagrammatically depict an AR system and its subsystems, according to some embodiments.

[0027] [Diagram 3] 3 and 4 illustrate artifacts associated with user movement, according to some embodiments. [Figure 4] 3 and 4 illustrate artifacts associated with user movement, according to some embodiments.

[0028] [Diagram 5]FIG. 5 illustrates warping virtual content to compensate for head rotation, according to some embodiments.

[0029] [Figure 6] 6-9 illustrate warping virtual content to compensate for user movement according to some embodiments. [Figure 7] 6-9 illustrate warping virtual content to compensate for user movement according to some embodiments. [Figure 8] 6-9 illustrate warping virtual content to compensate for user movement according to some embodiments. [Figure 9] 6-9 illustrate warping virtual content to compensate for user movement according to some embodiments.

[0030] [Figure 10] FIG. 10 diagrammatically depicts a graphics processing unit (GPU), according to some embodiments.

[0031] [Figure 11] FIG. 11 depicts a virtual object stored as a primitive, according to some embodiments.

[0032] [Figure 12] 12-14 depict warping two non-overlapping virtual objects located at different depths to compensate for user movement, according to some embodiments. [Figure 13] 12-14 depict warping two non-overlapping virtual objects located at different depths to compensate for user movement, according to some embodiments. [Figure 14] 12-14 depict warping two non-overlapping virtual objects located at different depths to compensate for user movement, according to some embodiments.

[0033] [Figure 15] 15-19 depict dis-occlusion artifacts when warping two overlapping virtual objects located at different depths to compensate for user movement. [Figure 16] 15-19 depict dis-occlusion artifacts when warping two overlapping virtual objects located at different depths to compensate for user movement. [Figure 17] 15-19 depict dis-occlusion artifacts when warping two overlapping virtual objects located at different depths to compensate for user movement. [Figure 18] 15-19 depict dis-occlusion artifacts when warping two overlapping virtual objects located at different depths to compensate for user movement. [Figure 19] 15-19 depict dis-occlusion artifacts when warping two overlapping virtual objects located at different depths to compensate for user movement.

[0034] [Figure 20] 20 and 21 depict smearing artifacts when warping two overlapping virtual objects located at different depths to compensate for user movement. [Figure 21] 20 and 21 depict smearing artifacts when warping two overlapping virtual objects located at different depths to compensate for user movement.

[0035] [Figure 22] 22-24 depict jitter artifacts when warping two overlapping virtual objects located at different depths to compensate for user movement. [Figure 23] 22-24 depict jitter artifacts when warping two overlapping virtual objects located at different depths to compensate for user movement. [Figure 24] 22-24 depict jitter artifacts when warping two overlapping virtual objects located at different depths to compensate for user movement.

[0036] [Diagram 25] 25-27 depict warping two overlapping virtual objects located at different depths to compensate for user movement while minimizing jitter artifacts, according to some embodiments. [Figure 26] 25-27 depict warping two overlapping virtual objects located at different depths to compensate for user movement while minimizing jitter artifacts, according to some embodiments. [Figure 27] 25-27 depict warping two overlapping virtual objects located at different depths to compensate for user movement while minimizing jitter artifacts, according to some embodiments.

[0037] [Figure 28] 28-31 depict using multiple depths to warp two overlapping virtual objects located at different depths to compensate for user movement, according to some embodiments. [Figure 29A] 28-31 depict using multiple depths to warp two overlapping virtual objects located at different depths to compensate for user movement, according to some embodiments. [Figure 29B] 28-31 depict using multiple depths to warp two overlapping virtual objects located at different depths to compensate for user movement, according to some embodiments. [Diagram 30] 28-31 depict using multiple depths to warp two overlapping virtual objects located at different depths to compensate for user movement, according to some embodiments. [Diagram 31]28-31 depict using multiple depths to warp two overlapping virtual objects located at different depths to compensate for user movement, according to some embodiments.

[0038] [Diagram 32] 32-38 depict warping two overlapping virtual objects located at different depths using eye tracking to compensate for user movement, according to some embodiments. [Diagram 33] 32-38 depict warping two overlapping virtual objects located at different depths using eye tracking to compensate for user movement, according to some embodiments. [Diagram 34] 32-38 depict warping two overlapping virtual objects located at different depths using eye tracking to compensate for user movement, according to some embodiments. [Diagram 35] 32-38 depict warping two overlapping virtual objects located at different depths using eye tracking to compensate for user movement, according to some embodiments. [Diagram 36] 32-38 depict warping two overlapping virtual objects located at different depths using eye tracking to compensate for user movement, according to some embodiments. [Figure 37] 32-38 depict warping two overlapping virtual objects located at different depths using eye tracking to compensate for user movement, according to some embodiments. [Figure 38] 32-38 depict warping two overlapping virtual objects located at different depths using eye tracking to compensate for user movement, according to some embodiments.

[0039] [Figure 39]39-45 depict the use of meshes to warp two overlapping virtual objects located at different depths to compensate for user movement, according to some embodiments. [Diagram 40] 39-45 depict the use of meshes to warp two overlapping virtual objects located at different depths to compensate for user movement, according to some embodiments. [Diagram 41] 39-45 depict the use of meshes to warp two overlapping virtual objects located at different depths to compensate for user movement, according to some embodiments. [Diagram 42] 39-45 depict the use of meshes to warp two overlapping virtual objects located at different depths to compensate for user movement, according to some embodiments. [Diagram 43] 39-45 depict the use of meshes to warp two overlapping virtual objects located at different depths to compensate for user movement, according to some embodiments. [Diagram 44] 39-45 depict the use of meshes to warp two overlapping virtual objects located at different depths to compensate for user movement, according to some embodiments. [Diagram 45] 39-45 depict the use of meshes to warp two overlapping virtual objects located at different depths to compensate for user movement, according to some embodiments.

[0040] [Figure 46] FIG. 46 is a block diagram that diagrammatically depicts an illustrative computing system, in accordance with some embodiments.

[0041] [Figure 47] 47-49 diagrammatically depict a method for determining the depth of a virtual object from a stereo image pair of the virtual object, according to some embodiments. [Figure 48] 47-49 diagrammatically depict a method for determining the depth of a virtual object from a stereo image pair of the virtual object, according to some embodiments. [Figure 49] 47-49 diagrammatically depict a method for determining the depth of a virtual object from a stereo image pair of the virtual object, according to some embodiments.

[0042] [Figure 50] 50-52 show various combined images generated by a method for determining the depth of a virtual object from a stereo image pair of the virtual object, according to some embodiments. [Figure 51] 50-52 show various combined images generated by a method for determining the depth of a virtual object from a stereo image pair of the virtual object, according to some embodiments. [Figure 52] 50-52 show various combined images generated by a method for determining the depth of a virtual object from a stereo image pair of the virtual object, according to some embodiments.

[0043] [Diagram 53] FIG. 53 is a flowchart illustrating a method for determining the depth of a virtual object from a stereo image pair of the virtual object, according to some embodiments.

[0044] [Figure 54] FIG. 54 is a graph illustrating the relationship between stabilization depth and apparent movement of a virtual object with POV movement.

[0045] [Figure 55] FIG. 55 is a flowchart illustrating a method for determining stabilization depth, according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0046] Various embodiments of the present disclosure are directed to systems, methods, and articles of manufacture for warping virtual content in a single embodiment or in multiple embodiments. Other objects, features, and advantages of the present disclosure are set forth in the detailed description, drawings, and claims.

[0047] Various embodiments will now be described in detail with reference to the drawings, which are provided as illustrative examples of the present disclosure, to enable those skilled in the art to practice the present disclosure. It should be noted that the following figures and examples are not meant to limit the scope of the present disclosure. When an element of the present disclosure can be partially or completely implemented using known components (or methods or processes), only a part of such known components (or methods or processes) necessary for understanding the present disclosure will be described, and detailed descriptions of other parts of such known components (or methods or processes) will be omitted so as not to obscure the present disclosure. Furthermore, various embodiments encompass current and future known equivalents of the components referenced herein as examples.

[0048] Although the virtual content warping system may be implemented independently of a mixed reality system, some embodiments below are described in the context of an AR system for illustrative purposes only. Additionally, the virtual content warping system described herein may also be used in the same manner as a VR system. (Illustrative Mixed Reality Scenarios and Systems)

[0049] The following description relates to an exemplary augmented reality system in which the warping system may be practiced, however, it should be understood that the embodiments are also suitable for use in other types of display systems (including other types of mixed reality systems) and thus the embodiments are not limited to only the exemplary systems disclosed herein.

[0050] Mixed reality (e.g., VR or AR) scenarios often involve the presentation of virtual content (e.g., images and sounds) that correspond to virtual objects in relation to real-world objects. For example, with reference to FIG. 1, an AR scene 100 is depicted in which a user of the AR technology sees a real-world, physical, park-like setting 102 featuring people, trees, buildings in the background, and a real-world, physical concrete platform 104. In addition to these items, the user of the AR technology also perceives that they "see" a virtual robotic figure 106 standing on the physical concrete platform 104, and a flying virtual cartoon-like avatar character 108 that appears to be an anthropomorphic bumblebee, although these virtual objects 106, 108 do not exist in the real world.

[0051] Like AR scenarios, VR scenarios also take into account the pose used to generate / render the virtual content. Accurately warping the virtual content to a reference AR / VR display frame and warping the warped virtual content can improve the AR / VR scenario, or at least not impair it.

[0052] The following description relates to an exemplary AR system in which the present disclosure may be practiced, however, it should be understood that the present disclosure also lends itself to use in other types of augmented reality and virtual reality systems, and thus the present disclosure is not limited to only the exemplary systems disclosed herein.

[0053] 2A illustrates an AR system 200, according to some embodiments. The AR system 200 may be operated in conjunction with a projection subsystem 208 to provide images of virtual objects mixed with physical objects within the field of view of a user 250. This approach employs one or more at least partially transparent surfaces through which the surrounding environment, including physical objects, can be seen and through which the AR system 200 produces images of the virtual objects. The projection subsystem 208 is stored within a control subsystem 201, which is operably coupled to a display system / subsystem 204 through a link 207. The link 207 may be a wired or wireless communication link.

[0054] For AR applications, it may be desirable to spatially position various virtual objects relative to individual physical objects within the field of view of the user 250. Virtual objects may take any of a wide variety of forms, having any of a variety of data, information, concepts, or logical structures capable of being represented as images. Non-limiting examples of virtual objects may include a virtual text object, a virtual numeric object, a virtual alphanumeric object, a virtual tag object, a virtual field object, a virtual chart object, a virtual map object, a virtual gauge object, or a virtual visual representation of a physical object.

[0055] The AR system 200 includes a frame structure 202 worn by a user 250, a display system 204 carried by the frame structure 202 such that the display system 204 is positioned directly in front of the eye of the user 250, and a speaker 206 incorporated into or connected to the display system 204. In the illustrated embodiment, the speaker 206 is carried by the frame structure 202 such that the speaker 206 is positioned adjacent to (in or around) the ear canal of the user 250 (e.g., ear buds or headphones).

[0056] The display system 204 is designed to present to the eyes of the user 250 a light-based radiation pattern that can be comfortably perceived as an extension to the surrounding environment, including both two-dimensional and three-dimensional content. The display system 204 presents a sequence of frames at a high frequency that provides the perception of a single coherent scene. To this end, the display system 204 includes a projection subsystem 208 and a partially transparent display screen through which the projection subsystem 208 projects images. The display screen is positioned within the field of view of the user 250 between the eyes of the user 250 and the surrounding environment.

[0057] In some embodiments, the projection subsystem 208 takes the form of a scanning-based projection device and the display screen takes the form of a waveguide-based display into which scanned light from the projection subsystem 208 is injected to produce, for example, an image at a single optical viewing distance closer than infinity (e.g., arm's length), images at multiple discrete optical viewing distances or focal planes, and / or image layers stacked at multiple viewing distances or focal planes to represent a stereoscopic 3D object. These layers in the light field may be stacked together closely enough to appear continuous to the human accessory visual system (e.g., one layer is within the cone of confusion of an adjacent layer). Additionally or alternatively, picture elements may be blended across two or more layers to increase the perceived continuity of transitions between layers in the light field even when the layers are more sparsely stacked (e.g., one layer is outside the cone of confusion of an adjacent layer). The display system 204 may be monocular or binocular. The scanning assembly includes one or more light sources that produce light beams (e.g., emit different colored light in a defined pattern). The light sources may take any of a wide variety of forms, such as a set of RGB sources (e.g., laser diodes capable of outputting red, green, and blue light), each operable to produce coherent collimated light of red, green, and blue according to a defined pixel pattern defined within an individual frame of pixel information or data. Laser light provides high color saturation and is highly energy efficient. The optical coupling subsystem includes an optical waveguide input device, such as, for example, one or more reflective surfaces, diffraction gratings, mirrors, dichroic mirrors, or prisms, to optically couple the light into the edge of the display screen. The optical coupling subsystem further includes a collimation element that collimates the light from the optical fiber. Optionally, the optical coupling subsystem includes an optical modulation device configured to converge the light from the collimation element toward a focal point within the center of the optical waveguide input device, thereby allowing the size of the optical waveguide input device to be minimized.Thus, the display system 204 generates a series of composite image frames of pixel information that presents an undistorted image of one or more virtual objects to the user. Further details describing the display subsystem are set forth in "Display System and." No. 14 / 212,961, entitled "Planar Waveguide Apparatus With Diffraction Element and Subsystem Employing Same" (Attorney Docket No. ML.20006.00), and No. 14 / 331,218, entitled "Planar Waveguide Apparatus With Diffraction Element and Subsystem Employing Same" (Attorney Docket No. ML.20020.00), the contents of which are expressly and completely incorporated herein by reference in their entirety as if set forth in their entirety.

[0058] The AR system 200 further includes one or more sensors mounted on the frame structure 202 to detect the position (including orientation) and movement of the user's 250 head and / or the eye position and interocular distance of the user 250. Such sensors may include image capture devices, microphones, inertial measurement units (IMUs), accelerometers, compasses, GPS units, wireless devices, gyroscopes, and the like. For example, in one embodiment, the AR system 200 includes a head-mounted transducer subsystem that includes one or more inertial transducers and captures inertial measurements indicative of the movement of the user's 250 head. Such devices may be used to sense, measure, or collect information about the head movement of the user 250. For example, these devices may be used to detect / measure the movement, velocity, acceleration, and / or position of the user's 250 head. The position (including orientation) of the user's 250 head is also known as the "head pose" of the user 250.

[0059] 2A may include one or more forward-facing cameras. The cameras may be employed for any number of purposes, such as recording images / video from a forward direction of the system 200. In addition, the cameras may be used to capture information about the environment in which the user 250 is located, such as information indicative of the distance, orientation, and / or angular position of the user 250 relative to the environment and specific objects within the environment.

[0060] The AR system 200 may further include a rear-facing camera to track the angular position of the user's 250 eyes (the direction one or both eyes are pointing), eye blinks, and depth of focus (by detecting eye convergence). Such eye tracking information may be determined, for example, by projecting light onto the end user's eyes and detecting the return or reflection of at least a portion of the projected light.

[0061] The augmented reality system 200 further includes a control subsystem 201, which may take any of a wide variety of forms. The control subsystem 201 includes a number of controllers, such as one or more microcontrollers, microprocessors or central processing units (CPUs), digital signal processors, graphics processing units (GPUs), other integrated circuit controllers such as application specific integrated circuits (ASICs), programmable gate arrays (PGAs), such as field programmable gate arrays (FPGAs), and / or programmable logic controllers (PLUs). The control subsystem 201 may include a digital signal processor (DSP), a central processing unit (CPU) 251, a graphics processing unit (GPU) 252, and one or more frame buffers 254. The CPU 251 controls the overall operation of the system, while the GPU 252 renders frames (i.e., converts a three-dimensional scene into a two-dimensional image) and stores these frames in the frame buffer 254. Although not shown, one or more additional integrated circuits may control the reading of frames into and / or from the frame buffer 254 and the operation of the display system 204. Reading into and / or from the frame buffer 254 may employ dynamic addressing, e.g., frames are over-rendered. The control subsystem 201 further includes read-only memory (ROM) and random access memory (RAM). The control subsystem 201 further includes a 3D database 260, from which the GPU 252 can access 3D data of one or more scenes for rendering frames and synthesized sound data associated with virtual sound sources contained within the 3D scenes.

[0062] The control AR augmented reality subsystem 2010 further includes a user orientation detection module 248. The user orientation module 248 detects the instantaneous position of the head of the user 250 and may predict the position of the head of the user 250 based on the position data received from the sensors. The user orientation module 248 also tracks the eyes of the user 250, and in particular the direction and / or distance the user 250 is focused on, based on the tracking data received from the sensors.

[0063] FIG. 2B illustrates an AR system 200′ according to some embodiments. The AR system 200′ illustrated in FIG. 2B is similar to the AR system 200 illustrated in FIG. 2A and described above. For example, the AR system 200′ includes a frame structure 202, a display system 204, a speaker 206, and a control subsystem 201′ operably coupled to the display system 204 through a link 207. The control subsystem 201′ illustrated in FIG. 2B is similar to the control subsystem 201 illustrated in FIG. 2A and described above. For example, the control subsystem 201′ includes a projection subsystem 208, an image / video database 271, a user orientation module 248, a CPU 251, a GPU 252, a 3D database 260, a ROM, and a RAM.

[0064] The difference between the control subsystem 201′, and therefore the AR system 200′ depicted in FIG. 2B, and the corresponding system / system components depicted in FIG. 2A, is the presence of block 290 in the control subsystem 201′ depicted in FIG. 2B. Block 290 is a separate warping block independent of either the GPU 252 or the CPU 251. As illustrated in FIG. 2C, block 290 includes a warping unit 280, a database 292, and a synthesis unit 294. The synthesis unit 294 includes a blending unit 296. As illustrated in FIG. 2D, the warping unit 280 includes a pose estimator 282 and a transformation unit 284.

[0065] Various processing components of the AR system 200, 200' may be contained within a distributed subsystem. For example, the AR system 200, 200' includes a local processing and data module (i.e., control subsystem 201, 201') operably coupled to a portion of the display system 204, such as by wired or wireless connectivity 207. The local processing and data module may be mounted in various configurations, such as fixedly attached to the frame structure 202, fixedly attached to a helmet or hat, built into headphones, removably attached to the torso of the user 250, or removably attached to the waist of the user 250 in a belt-type configuration. The AR system 200, 200' may further include a remote processing module and a remote data repository operably coupled to the local processing and data module, such as by wired or wireless connectivity, such that the remote modules are operably coupled to each other and available as resources to the local processing and data module. The local processing and data module may comprise a power efficient processor or controller and digital memory such as flash memory, both of which may be utilized to aid in the processing, caching, and storage of data captured from the sensor and / or obtained and / or processed using the remote processing module and / or remote data repository, possibly for passing to the display system 204 after processing or retrieval. The remote processing module may comprise one or more relatively powerful processors or controllers configured to analyze and process the data and / or image information. The remote data repository may comprise a relatively large scale digital data storage facility, which may be available through the Internet or other networking configuration in a "cloud" resource configuration. In some embodiments, all data is stored and all calculations are performed within the local processing and data module, allowing for fully autonomous use from any remote module.The couplings between the various components described above may include one or more wired interfaces or ports to provide wire or optical communication, or one or more wireless interfaces or ports to provide wireless communication via RF, microwave, IR, etc. In some implementations, all communications may be wired, while in other implementations, all communications may be wireless, except for optical fiber. (Problem and solution overview)

[0066] When the optical system generates / renders virtual content, it may use a source frame of reference, which may be related to the pose of the system when the virtual content is rendered. In an AR system, the rendered virtual content may have a predetermined relationship to a real physical object. For example, FIG. 3 illustrates an AR scenario 300 including a virtual flower pot 310 positioned on top of a real physical platform 312. The AR system rendered the virtual flower pot 310 based on a source frame of reference, in which the location of the real platform 312 is known, so that the virtual flower pot 310 appears as if it were resting on top of the real platform 312. The AR system may render the virtual flower pot 310 at a first time using the source frame of reference, and display / project the rendered virtual flower pot 310 at a second time after the first time using the output frame of reference. If the source frame of reference and the output frame of reference are the same, the virtual flower pot 310 will appear at its intended location (e.g., on top of the real physical platform 312), for example, as illustrated in FIG. 3.

[0067] However, if the frame of reference changes (e.g., with user head movement) from a first time when the virtual flower pot 310 is rendered to a second time when the rendered virtual flower pot 310 is displayed / projected, the misalignment / difference between the source and output frames of reference may result in visual artifacts / anomalies / glitches. For example, FIG. 4 shows an AR scenario 400 including a virtual flower pot 410 rendered to be positioned on top of an actual physical platform 412. However, because the AR system was moved to the right (e.g., by head rotation to the right) after the virtual flower pot 410 was rendered but before it was displayed / projected, the virtual flower pot 410 is displayed to the right of its intended position 410' (shown in phantom). Thus, the virtual flower pot 410 appears to be floating in the air to the right of the actual physical platform 412. This artifact may be corrected when the virtual flower pot is re-rendered in the next source frame of reference and displayed in the next output frame of reference (assuming the AR system movement ceases). However, the artifact may still be visible to some users and may appear as a glitch, with the virtual flowerpot 410 temporarily jumping to an unexpected position. This glitch and others like it may adversely affect the illusion of continuity of the AR scenario.

[0068] Some optical systems may include a warping system that warps or transforms a frame of reference of the rendered virtual content from a source frame of reference in which the virtual content was generated to an output frame of reference in which the virtual content will be displayed. In the example depicted in FIG. 4, the AR system can detect and / or predict the output frame of reference and / or pose (e.g., using an IMU or eye tracking). The AR system can then warp or transform the rendered virtual content from the source frame of reference to the warped virtual content in the output frame of reference.

[0069] FIG. 5 diagrammatically illustrates warping of virtual content to compensate for head rotation ("rotation warping"), according to some embodiments. Rendered virtual content 512 in a source frame of reference (rendering pose) represented by ray 510 is warped to warped virtual content 512' in an output frame of reference (estimated pose) represented by ray 510'. The warping depicted in FIG. 5 may represent a head rotation to the right. While the rendered virtual content 512 is positioned at the source X,Y location, the warped virtual content 512' is transformed to the output X',Y' location (in a stationary frame of reference). However, due to movement of the user's head and FOV (e.g., a right rotation) between the rendering of the virtual content 512 and the display of the warped virtual content 512', the warped virtual content 512' is displayed at a position / orientation relative to the display frame of reference (pose), represented by rays 510', that is identical to the position / orientation of the rendered virtual content 512 relative to the source frame of reference (rendering pose), represented by rays 510. The warping depicted in Figure 5 results in the warped virtual content 512' being displayed in the display pose, as the rendered virtual content 512 would appear as if it were displayed in the rendering pose.

[0070] 6-9 diagrammatically illustrate warping of virtual content to compensate for head position changes ("position warping"), according to some embodiments. Rendered virtual content 610, 612 from a rendering pose is warped to generate warped virtual content 610', 612' at the display pose. The FOV change from the rendering pose to the display pose is a position change (i.e., translation along some of the six degrees of freedom, in addition to optional rotation).

[0071] FIG. 6 illustrates position warping, according to some embodiments. For example, the AR system may warp all source sub-portions to form a 3-D scenario in the rendered virtual content. For example, primitives forming a chess piece 610 and a cube 612 may be warped to a warped chess piece 610' and a warped cube 612'. The chess piece 610 and the cube 612 are shown in phantom lines in FIG. 6 to indicate that they are in the source frame of reference (i.e., the rendering FOV) and will not be displayed. Instead, the warped chess piece 610' and the warped cube 612' (shown in solid lines) in the output frame of reference (i.e., the display FOV after the position change) will be displayed.

[0072] Some warping systems / methods also depth test all sub-portions (e.g., each primitive that forms the warped chess piece 610' and the warped cube 612') and generate depth data. After depth testing, the AR system stores (e.g., in a list) all warped sub-portions (e.g., primitives) and their relative depths in the source reference frame. The warped virtual content may be stored as a list of all brightness / colors at each X,Y location (e.g., pixel) in the output virtual content, including all instances of conflicting warped virtual data (see below).

[0073] The warping system then resolves all conflicting virtual data at each X,Y location of the stored warped sub-portions and relative depths (e.g., list). For example, as illustrated in FIG. 7, when two or more different sub-portions (e.g., chess piece 610 and cube 612) of a 3-D scenario are warped into the output reference frame (e.g., warped chess piece 610'' and warped cube 612''), some of these sub-portions may be warped / projected (i.e., assigned) into the same pixel of the final display image. For example, area 614'' in FIG. 7 represents the "conflicting" portions of warped chess piece 610'' and warped cube 612''. The warping system resolves / resolves these conflicting portions of the warped virtual content when generating the output virtual content.

[0074] In some embodiments, if distinct pixels of the warped chess piece 610'' and the warped cube 612'' (e.g., first and second virtual content) would be displayed on the same pixel of the display (i.e., colliding pixels), the warping system may compare stored depth data corresponding to the distinct colliding / conflicting pixels. The colliding pixel (e.g., warped virtual content) that is closer to the viewing location in the output reference frame is displayed (assuming the content is opaque). The colliding pixel closest to the viewing location in the output reference frame is used to generate the final display image. The remaining colliding pixels are discarded.

[0075] In one instance shown in FIG. 8, the warped chess piece 610''' is closer to the user than the warped cube 612'''. Thus, when pixels of the warped chess piece 610'''' and the warped cube 612''' collide, the pixels of the warped chess piece 610'''' are displayed in the output content. In another instance shown in FIG. 9, the warped cube 612'''' is closer to the user than the warped chess piece 610''''. Thus, when pixels of the warped chess piece 610'''' and the warped cube 612'''' collide, the pixels of the warped cube 612'''' are displayed in the output content. Because some warping systems generate depth data in a source reference frame, depth comparison may involve multiple transformations and may be more complicated than a direct comparison.

[0076] Various steps of the warping method can be performed in two steps or in a single step, as described in U.S. Utility Application No. 15 / 924,011 (the contents of which have already been incorporated above by reference). Position warping systems / methods (warping, depth testing, depth comparison, compositing, etc.) can be relatively computationally expensive (leading to processor / memory related system limitations) and relatively time consuming (leading to system latency). Therefore, some warping methods focus on rotational warping. However, rotational warping can lead to various artifacts (disocclusion, jitter, smearing, which are described in detail below) with various user FOV movements, such as lateral user FOV movements orthogonal to the optical axis.

[0077] To address these limitations, the systems / methods described herein include various embodiments for warping virtual content when the user FOV is moved (e.g., laterally, orthogonal to the optical axis). Although some embodiments are described with reference to lateral user FOV movement, these embodiments also address artifacts resulting from other types of user FOV movement. This minimizes the required computational cost and time, which increases with the complexity of the 3-D scenario that must be warped for display. Thus, the disclosed embodiments facilitate real-time systems, such as some mixed reality systems, that maintain size, power, thermal, and other processing-related parameters within acceptable limits. (Illustrative Graphics Processing Unit)

[0078] FIG. 10 diagrammatically depicts an exemplary graphics processing unit (GPU) 252 for warping rendered virtual content to an output / display reference frame, according to some embodiments. The GPU 252 includes an input memory 1010 to store the rendered virtual content to be warped. In some embodiments, the virtual content is stored as primitives (e.g., in FIG. 11, triangles 1100). The GPU 252 also includes a command processor 1012, which (1) receives / reads the virtual content from the input memory 1010, (2) divides the virtual content into scheduling units, and (3) sends the scheduling units along the rendering pipeline in waves or warps for parallel processing. The GPU 252 further includes a scheduler 1014, which (1) receives the scheduling units from the command processor 1012, and (2) determines whether "new work" from the command processor 1012 or "old work" returning from downstream in the rendering pipeline (described below) should be sent down the rendering pipeline at any particular time. In effect, the scheduler 1014 determines the sequence in which the GPU 252 processes the various input data.

[0079] The GPU 252 includes one or more GPU cores 1016, each having a number of parallel executable cores / units ("shader cores") 1018 for processing scheduling units in parallel. The command processor 1012 divides the virtual content into a number equal to the number of shader cores 1018 (e.g., 32). The GPU 252 also includes a "first-in, first-out" ("FIFO") memory 1020 to receive output from the GPU cores 1016. From the FIFO memory 1020, the output may be routed back to the scheduler 1014 as "old work" for insertion into additional processing of the rendering pipeline by the GPU cores 1016.

[0080] The GPU 252 further includes a raster operation unit ("ROP") 1022 that receives the output from the FIFO memory 1020 and rasterizes the output for display. For example, primitives of the virtual content may be stored as coordinates of triangle vertices. After processing by the GPU core 1016 (during which the three vertices 1111, 1112, 1114 of the triangle 1100 of FIG. 11 may be warped), the ROP 1022 determines pixels 1116 inside the triangle 1100 defined by the three vertices 1111, 1112, 1114 and fills those pixels 1116 in the virtual content. The ROP 1022 may also perform depth testing on the virtual content.

[0081] The GPU 252 also includes a buffer memory 1024 for temporarily storing the warped virtual content from the ROP 1022. The warped virtual content in the buffer memory 1024 may include brightness / color and depth information at multiple Y positions of the field of view in the output reference frame. The output from the buffer memory 1024 may be routed back to the scheduler 1014 as "old work" for insertion into additional processing of the rendering pipeline by the GPU core 1016 or for display in corresponding pixels of the display system. The GPU core 1016 first processes the vertices 1111, 1112, 1114 of the triangle 1100, and then processes the pixel 1116 inside the triangle 1100. Once all fragments of the virtual content in the input memory 1011 have been warped and depth tested (if necessary), the buffer memory 1024 will contain all of the brightness / color and depth information needed to display the field of view in the output reference frame. Virtual Content Warping System and Method for Lateral Shifting

[0082] With image processing without head pose changes, the results of processing by the GPU 252 are color / brightness and depth values ​​(e.g., at each pixel) at separate X,Y values. However, with lateral head pose changes (i.e., lateral movements of the display FOV orthogonal to the optical axis), different portions of the virtual content viewed from a viewing location in the output frame of reference different from the source frame of reference in which the virtual content was generated may contain various artifacts. As explained above, position warping to correct for lateral movement artifacts after lateral head pose changes may be computationally expensive and slow, making it difficult to use with portable display systems such as mixed reality systems. Furthermore, rotational warping with lateral head pose changes may introduce various other artifacts.

[0083] 12-14 depict position warping in response to lateral FOV movement, according to some embodiments. FIG 12 depicts rendered virtual objects (chess piece 1210 and cube 1212).

[0084] Figure 13 depicts the rendered virtual objects 1210, 1212 and the warped virtual objects 1210', 1212' after a left lateral FOV movement. The rendered virtual objects 1210, 1212 are shown in phantom lines in Figure 13 to indicate that the chess piece 1210 and the cube 1212, which were generated in the source reference frame and are not in the display reference frame, are not displayed. Only after the warping system warps the chess piece 1210 and the cube 1212, are the warped chess piece 1210' and the warped cube 1212' (shown in solid lines) displayed.

[0085] FIG. 14 diagrammatically depicts the rendered virtual objects 1210, 1212 from a top view and illustrates the difference in depth of the rendered virtual objects 1210, 1212. The chess piece 1210 is closer to the viewer at depth D10 (i.e., the depth of the center of the chess piece 1210) and the cube 1212 is farther from the viewer at depth D12 (i.e., the depth of the center of the cube 1212). As a result, with the same left lateral FOV movement, the closer object (chess piece 1210) appears to move further to the right compared to the farther object (cube 1212). This difference is illustrated in FIG. 13 by the different right shifts of the warped chess piece 1210′ and the warped cube 1212′. The difference in right shift depicted in Figure 13 was chosen to illustrate the present principles / effects and is not directly correlated with the depth difference depicted in Figure 14. With a large depth difference, closer objects can appear to shift to the right while more distant objects can appear to be stationary.

[0086] In some embodiments, depth-testing each pixel of each rendered virtual object 1210, 1212 and warping each pixel based on its depth may be computationally expensive, such that such position warping may be impractical to implement in a portable system. (Disocclusion artifacts)

[0087] Figures 15-19 depict disocclusion artifacts associated with position warping, according to some embodiments. Figure 15 depicts rendered virtual objects (chess piece 1510 and cube 1512) with an area of ​​overlap 1514 between the chess piece 1510 and the cube 1512 from a rendering pose. Figure 16 shows that because the chess piece 1510 is closer to the viewer than the cube 1512, the portion of the cube 1512 corresponding to the overlap area 1514 is not rendered, and the portion of the chess piece 1510 corresponding to the overlap area 1514 is rendered.

[0088] FIG. 17 diagrammatically illustrates that with a lateral user head movement to the left, as in FIG. 13, the virtual objects (chess piece 1510 and cube 1512) are warped to the right to generate warped virtual objects (chess piece 1510′ and warped cube 1512′). As discussed with respect to FIGS. 13 and 14, because the chess piece 1510 is closer to the viewer than the cube 1512, the chess piece 1510 will be warped further to the right than the cube 1512 in response to a lateral user head movement to the left. This differential warping of the chess piece 1510′ and the warped cube 1512′ exposes or “disoccludes” the portion of the cube 1512 that corresponds to the overlap area 1514. Referring back to FIG. 16, the portion of the cube 1512 that corresponds to the overlap area 1514 was not rendered. Thus, the now disoccluded portion of the warped cube 1512' that corresponds to the previously occluded overlap area 1514' cannot be displayed because the source virtual object (i.e., cube 1512) did not include the overlap area 1514.

[0089] FIG. 18 illustrates the position warping described in FIG. 17 without including any non-rendered portions. FIG. 18 illustrates that the warped cube 1512′ does not include the previously occluded portions (see 1514′ in FIG. 17). FIG. 19 illustrates the next frame that would be rendered and displayed assuming that the lateral user head movement stops at the frame depicted in FIG. 18. As shown in FIG. 19, with the next rendering without user FOV movement, the newly rendered chess pieces 1510″ and cube 1512″ remain in substantially the same positions as in FIG. 17. However, the newly rendered cube 1512″ includes the previously occluded portions (see 1514′ in FIG. 17), thereby forming a complete cube 1512″.

[0090] A comparison of Figures 18 and 19 shows a disocclusion artifact in which a previously occluded portion of a virtual object (see 1514' in Figure 17) is not initially displayed after warping exposes the previously disoccluded portion as shown in Figure 18. The previously occluded portion then appears when the virtual object is next rendered and displayed as shown in Figure 19. The disocclusion artifact depicted in Figures 17-19 involves significant portions of the virtual object (cube) disappearing and appearing, but in other embodiments, depending on many factors including depth difference, amount of overlap, and speed of FOV movement, the disocclusion artifact may be manifested as cracks or other missing portions in a solid virtual object. Regardless of the manifestation, the disocclusion artifact may affect the user experience and reduce the realism of a mixed reality scenario. Because the embodiment depicted in Figures 15-19 still utilizes position warping, the disocclusion artifact also involves the computational cost of position warping. (Smear artifacts)

[0091] 20 and 21 depict smearing artifacts associated with warping according to some embodiments. To minimize disocclusion artifacts as described above, some warping systems / methods stretch distant virtual objects when disoccluded by warping to fill the space between the differentially warped virtual objects. FIG. 20 illustrates such warping of virtual objects (chess piece 2010 and cube 2012) to the right to generate warped virtual objects (warped chess piece 2010' and warped cube 2012') in response to a lateral user head movement to the left. The unoccluded portions of cube 2012 are stretched to form a filled section 2016 in the space between warped chess piece 2010' and warped cube 2012'.

[0092] FIG. 21 depicts the next rendering of chess pieces 2010'' and cube 2012'' without user FOV movement. With the next rendering, the newly rendered chess pieces 2010'' and cube 2012'' remain in substantially the same positions as in FIG. 20. However, the newly rendered cube 2012'' is based on a full cube (e.g., cube 1514 from FIG. 17) and does not include the infill section 2016 in FIG. 20. Furthermore, any differences in characteristics between the portions of the newly rendered cube 2012'' and the infill section 2016 that was based on a different portion of the previously rendered cube 2012 will be apparent to a user when comparing FIGS. 20 and 21. Such characteristics may include color, texture, and the like. This difference in properties between the newly rendered portion of cube 2012'' and the filled section 2016 may be referred to as a smear artifact, as the filled section 2016 may appear as a smear of different portions of the previously rendered cube 2012 / 2012'. Smear artifacts may also affect the user experience and reduce the realism of mixed reality scenarios. Because the embodiment depicted in FIGS. 20-21 still utilizes position warping, smear artifacts also involve the computational cost of position warping. (single depth warping and jitter artifacts)

[0093] Figures 22-27 depict single depth warping according to two types of embodiments (Figures 22-24 and 25-27). Figures 22-24 depict single depth warping according to some embodiments. Figure 22 depicts rendered virtual objects, namely, a chess piece 2210 at a relatively closer depth and a cube 2212 at a relatively farther depth.

[0094] 23 depicts the steps of warping the rendered virtual objects 2210, 2212 using a single depth. By using a single depth, the relative spatial relationship between the warped virtual objects (the warped chess pieces 2210′ and the warped cube 2212′) is substantially identical to the relative spatial relationship between the rendered virtual objects 2210, 2212. Maintaining substantially the same relative spatial relationship between the warped virtual objects 2210′, 2212′ minimizes disocclusion artifacts because occluded portions of the cube 2212 are not disoccluded until the entire cube 2212″ is re-rendered in FIG.

[0095] In the embodiment depicted in Figures 22-24, the single depth used to warp the rendered virtual objects 2210, 2212 is the closest depth of any virtual object (i.e., the chess piece 2210) within the FOV. This closest depth is contained within the data corresponding to the rendered virtual objects 2210, 2212. Therefore, no depth testing is required. This reduces the computational burden of the present single-depth warping method.

[0096] A comparison of the warped virtual objects 2210′, 2212′ in FIG. 23 with the re-rendered virtual objects 2210″, 2212″ in FIG. 24 (i.e., the next rendered frame without user FOV movement) shows that the warped chess piece 2210′ and the re-rendered chess piece 2210″ are substantially co-located because the depth used to warp the chess piece 2210 is the near depth D10 of the chess piece 2210. On the other hand, the warped cube 2212′ and the re-rendered cube 2212″ are in different positions because the depth D10 used to warp the cube 2212 is less than the actual far depth D12 of the cube 2212. This causes the warped cube 2212′ to be displaced too far to the right in response to a left lateral FOV movement. Thus, sequentially displaying the warped virtual objects 2210', 2212' in Figure 23 followed by the re-rendered virtual objects 2210'', 2212'' in Figure 24 causes the rendered cube 2212 to appear to move to the right and then bounce to the left. This "jitter artifact" (or "wobble artifact") is exacerbated by sustained FOV movement and generally results in a series of orientation changes for more distant objects (i.e., the rendered cube 2212) or objects in the background.

[0097] 25-27 depict single depth warping according to some other embodiments. Figure 25 depicts rendered virtual objects, namely a chess piece 2510 at a relatively closer depth and a cube 2512 at a relatively farther depth.

[0098] 26 depicts warping the rendered virtual objects 2510, 2512 using a single depth and maintaining substantially the same relative spatial relationship between the warped virtual objects (the warped chess piece 2510' and the warped cube 2512'). As explained above, maintaining essentially the same relative spacing between the warped virtual objects 2510', 2512' minimizes disocclusion artifacts because occluded portions of the cube 2512 are not disoccluded until the entire cube 2512'' is re-rendered in FIG.

[0099] In the embodiment depicted in Figures 25-27, the single depth used to warp the rendered virtual objects 2510, 2512 is the farthest depth of any virtual object (i.e., the cube 2512) within the FOV. This farthest depth is contained within the data corresponding to the rendered virtual objects 2510, 2512. Therefore, no depth testing is required. This reduces the computational burden of this single-depth warping method.

[0100] A comparison of the warped virtual objects 2510′, 2512′ in FIG. 26 with the re-rendered virtual objects 2510″, 2512″ in FIG. 27 (i.e., the next rendered frame without user FOV movement) shows that the warped cube 2512′ and the re-rendered cube 2512″ are substantially co-located because the depth used to warp the cube 2512 is the far depth D12 of the cube 2512. On the other hand, the warped chess piece 2510′ and the re-rendered chess piece 2510″ are in different positions because the depth D12 used to warp the chess piece 2510 exceeds the actual near depth D10 of the chess piece 2510. This causes the warped chess piece 2510′ to be under-shifted to the right in response to a left lateral FOV movement. Thus, sequentially displaying the warped virtual objects 2510′, 2512′ in FIG. 26 followed by the re-rendered virtual objects 2510″, 2512″ in FIG. 27 causes the rendered chess piece 2510 to appear to move partially to the right and then further to the right, disoccluding the cube 2512. This artifact has less of an impact on the realism of the mixed reality scenario for a typical user compared to the jitter artifact described above and shown in FIGS. 22-24. This may be due to the lack of directional change in the artifact depicted in FIGS. 25-27, which is perceived as a lag (if not entirely) rather than a jitter or wobble associated with the directional change.

[0101] Although single distance warping using both the farthest and nearest depths of virtual objects in the FOV is described above, other depths within the FOV may also be used to warp the rendered virtual object. For example, in some embodiments, the single distance is a representative value of the distances of the nearest and farthest objects in the FOV. In some embodiments, the single distance is an average / representative value of the distances of all objects in the FOV. In some embodiments, the single distance is a median value of the distances of all objects in the FOV. (Multiple Discrete Depth Warping)

[0102] 28-31 depict multiple depth warping, according to some embodiments. Figure 28 depicts rendered virtual objects, namely, a chess piece 2810 at a relatively closer depth and a cube 2812 at a relatively farther depth.

[0103] FIG. 29A depicts rendered virtual objects 2810, 2812 from a top view and shows the difference in depth of the rendered virtual objects 2810, 2812. The chess piece 2810 is closer to the viewer at depth D10, and the cube 2812 is farther away from the viewer at depth D12. FIG. 29A also shows six discrete depth slices 2814, 2816, 2818, 2820, 2822, 2824 into which the FOV is divided. In some embodiments, these discrete depth slices 2814, 2816, 2818, 2820, 2822, 2824 correspond to the depth slices in which the VR / AR / MR system is configured to display virtual content. The depth slices 2814, 2816, 2818, 2820, 2822, 2824 have limits along the optical axis. For example, a depth slice according to some embodiments may span from 0.37m to 1.5m from the user's eye, while another depth slice according to some embodiments may include all distances greater than 1.5m from the user's eye.

[0104] FIG. 29A depicts six depth slices 2814, 2816, 2818, 2820, 2822, 2824 having similar thicknesses, while in other embodiments, depth slices closer to the user's eye have thinner thicknesses because the user is more sensitive to depth differences closer to the user. For example, FIG. 29B depicts six depth slices 2814', 2816', 2818', 2820', 2822', 2824' according to some other embodiments. In FIG. 29B, each successively more distant depth slice (e.g., 2816' vs. 2814') has a thicker thickness. For example, objects at 1m and 2m may be located within discrete depth slices, while objects at 5m and 10m may be located within the same depth slice. In some embodiments, the farthest depth slice 2824' extends to infinity. In various embodiments, the thickness of the depth slices may increase linearly or logarithmically (as shown in FIG. 29B).

[0105] Although Figure 29A depicts six depth slices 2814, 2816, 2818, 2820, 2822, 2824, in other embodiments, the number of depth slices into which the FOV is divided may correspond to the number of virtual objects within the FOV. For example, if there are three virtual objects within the FOV, the FOV would be divided into three depth slices. Although Figures 29A and 29B include six depth slices, in other embodiments, the number of depth slices can be more or less than six, depending on the virtual scenario and available system resources.

[0106] FIG. 30 depicts warping the rendered virtual objects 2810, 2812 (in FIG. 29A ) using different depths for each rendered virtual object 2810, 2812 to more precisely position the warped virtual objects 2810′, 2812′. In some embodiments, the distance used to warp the relatively closer chess piece 2810 is the far distance (“far clip”) D16 of the depth slice 2816 that contains the chess piece 2810. The distance used to warp the relatively farther cube 2812 is the far clip D22 of the depth slice 2822 that contains most of the cube 2812. In other embodiments, the distance used to warp the objects may be the mid-distance or any other distance of the depth slice that contains all or most or part of the virtual object.

[0107] A comparison of the warped virtual objects 2810', 2812' in FIG. 30 with the re-rendered virtual objects 2810'', 2812'' in FIG. 31 (i.e., the next rendered frame without user FOV movement) shows that the multiple discrete depth warping depicted in FIGS. 28-31 may reduce the computational load compared to full position warping, but is still susceptible to dis-occlusion artifacts.

[0108] Although the virtual objects 2810, 2812 shown in Figure 29A are separated by several depth slices, in other embodiments, the virtual objects may be separated by a small distance so that they may be warped with minimal warping artifacts. For example, in some embodiments, objects less than 0.1 m away from each other (e.g., 1.5 m vs. 1.6 m) do not trigger multiple depth warping. In other embodiments, the threshold for multiple depth warping may be measured in diopters (e.g., only for objects more than 1 diopter apart). (Single Depth Warping Based on Eye Tracking)

[0109] 32-38 depict eye-tracking based single depth warping according to some embodiments. Fig. 32 depicts rendered virtual objects, namely a chess piece 3210 at a relatively closer depth and a cube 3212 at a relatively farther depth.

[0110] Fig. 33 depicts the rendered virtual objects 3210, 3212 from a top view and shows the difference in depth of the rendered virtual objects 3210, 3212. The chess piece 3210 is closer to the viewer at depth D10 and the cube 3212 is farther away from the viewer at depth D12. Fig. 33 also shows that the user's eye 3226 is tracking a direction 3228 that coincides with the location of the chess piece 3210. The direction of the user's eye 3226 can be tracked using, for example, a sensor / camera on the AR system 200.

[0111] Because the direction 3228 of the user's eyes 3226 is tracking the chess piece 3210, the depth used to warp both rendered virtual objects 3210, 3212 is the far clip D16 of the depth slice 3216 that contains the chess piece 3210. The result of this warping is shown in Figure 34, where the warped virtual objects 3210', 3212' are generated using a depth D16 that is more similar to the depth D10 of the closer chess piece 3210 than the depth D12 of the more distant cube 3212.

[0112] As explained above, the warped chess piece 3210' in FIG. 34 and the re-rendered chess piece 3210'' in FIG. 35 (i.e., the next rendered frame without user FOV movement) are substantially in the same location because the depth D16 used to warp the chess piece 3210 approximates the depth D10 of the chess piece 3210. On the other hand, the warped cube 3212' and the re-rendered cube 3212'' are in different positions because the depth D16 used to warp the cube 3212 is less than the actual far depth D12 of the cube 3212. This causes the warped cube 3212' to be displaced too far to the right in response to a left lateral FOV movement. Thus, sequentially displaying the warped virtual objects 3210', 3212' in FIG. 34 followed by the re-rendered virtual objects 3210'', 3212'' in FIG. 35 will cause the rendered cube 3212 to appear to move to the right and then bounce to the left, resulting in jitter artifacts.

[0113] However, these embodiments use depth D16 as the single depth for warping all rendered virtual content because the system is tracking the direction 3228 of the user's eyes 3226 to the closer object (i.e., the chess piece 3212). Thus, while the warped cube 3212' and the re-rendered cube 3212'' are displayed, the user's focus moves away from the cube 3212. The warped cube 3212' and the re-rendered cube 3212'' are shown in dashed lines in Figures 34 and 35 to indicate that the user's focus moves away from the cube 3212. As a result, even if the warping depicted in Figure 34 includes a jitter artifact, the effect of that artifact is minimized by the user's focus moving away from the cube 3212. The user's focus being directed away from the cube 3212 is depicted in Figures 34 and 35 by rendering the warped and re-rendered cubes 3212', 3212'' using dotted lines. On the other hand, the user's focus is directed towards the chess piece 3210, which is warped more precisely to the warped chess piece 3210'.

[0114] Figure 36 depicts the rendered virtual objects 3210, 3212 from a top view, similar to Figure 33. Figure 36 also shows that the user's eyes 3226' are tracking a direction 3228', which coincides with the location of the cube 3212.

[0115] Because the direction 3228' of the user's eyes 3226' is tracking the cube 3212, the depth used to warp both rendered virtual objects 3210, 3212 is the far clip D22 of the depth slice 3222, which contains most of the cube 3212. The result of this warping is shown in Figure 37, where the warped virtual objects 3210''', 3212''' are generated using a depth D22 that is more similar to the depth D12 of the cube 3212 than to the depth D10 of the chess piece 3210.

[0116] As explained above, the warped cube 3212''' in FIG. 37 and the re-rendered cube 3212'''' in FIG. 38 (i.e., the next rendered frame without user FOV movement) are substantially in the same location because the depth D22 used to warp the cube 3212 approximates the far depth D12 of the cube 3212. On the other hand, the warped chess piece 3210''' and the re-rendered chess piece 3210'''' are in different positions because the depth D16 used to warp the chess piece 3210 is less than the actual near depth D10 of the chess piece 3210. This causes the warped chess piece 3210''' to be under-shifted to the right in response to a left lateral FOV movement. Thus, sequentially displaying the warped virtual objects 3210''', 3212''' in FIG. 37 followed by the re-rendered virtual objects 3210'''', 3212'''' in FIG. 38 will cause the rendered chess piece 3210 to appear to move partially to the right and then move further to the right, disoccluding the cube 3212, resulting in a lag artifact.

[0117] However, these embodiments use depth D22 as the single depth for warping all rendered virtual content because the system is tracking the direction 3228' of the user's eye 3226' to the more distant object (i.e., cube 3212). Thus, while the warped chess piece 3210''' and the re-rendered chess piece 3210'''' are displayed, the user's focus moves away from the chess piece 3210. The warped chess piece 3210''' and the re-rendered chess piece 3210'''' are shown with dashed lines in Figures 36 and 37 to indicate that the user's focus moves away from the chess piece 3210. As a result, even if the warping depicted in Figure 36 includes a lag artifact, the effect of that artifact is minimized by the user's focus moving away from the chess piece 3210. The user's focus being directed away from the chess piece 3210 is depicted in FIGS. 37 and 38 by rendering the warped and re-rendered chess pieces 3210', 3210'' using dotted lines. The user's focus, on the other hand, is directed towards the cube 3212, which is precisely warped by the warped cube 3212'''.

[0118] Although a single depth warping using the far clip of the depth slice that contains the virtual object that the user's eyes are tracking is described above, the single depth based on eye tracking can be other depths associated with the virtual object, including the near distance (the "near clip") of the depth slice that contains the virtual object or any distance between the near clip and the far clip. (Mesh-based warping)

[0119] 39-45 depict a step of warping using a mesh of virtual content, e.g., to reduce computational load, according to some embodiments. FIG. 39 diagrammatically depicts rendered virtual objects 3910, 3912 from a top view and shows the depth difference of the rendered virtual objects 3910, 3912. The chess piece 3910 is closer to the viewer at a depth D1, and the cube 3912 is farther away from the viewer at a depth D2. FIG. 39 also shows that the rendered virtual objects 3910, 3912 each include one or more pixels 3914 (see pixel 1116 of the primitive triangle 1100 in FIG. 11). The rendered virtual objects 3910, 3912 may also include one or more primitives. The step of warping each pixel 3914 of each rendered virtual object 3910, 3912 may be computationally expensive.

[0120] FIG. 40 diagrammatically depicts meshes 3910′, 3912′ formed from and approximating rendered virtual objects 3910, 3912 (see FIG. 39). The meshes 3910′, 3912′ do not include individual pixels 3914, which may reduce computational complexity for warping. The meshes can be formed from the rendered virtual objects 3910, 3912 using a grid. Additionally, the meshes 3910′, 3912′ include vertices 3916, which approximate the meshes 3910′, 3912′, as shown in FIG. 41, which may further reduce computational complexity of warping. The meshes 3910′, 3912′ can be reconstructed from the vertices 3916, and the virtual objects 3910, 3912 can be reconstructed from the meshes 3910′, 3912′. However, the simplification and reconstruction process may introduce artifacts into the reconstructed virtual objects 3918, 3920, as shown in FIG.

[0121] FIG. 43 depicts the rendered virtual objects, namely, a chess piece 3910 at a relatively closer depth and a cube 3912 at a relatively farther depth. FIG. 44 depicts the result of mesh-based warping (i.e., a reconstructed chess piece 3918 and a reconstructed cube 3920), and FIG. 45 depicts the next frame of the re-rendered virtual objects (i.e., a re-rendered chess piece 3910'' and a re-rendered cube 3912'' without user FOV movement). Due to the simplification of the rendered virtual objects 3910, 3912 to meshes 3910', 3912' and / or vertices 3916, the simplified virtual objects can be warped using position warping techniques without excessive computational burden. The regularly shaped cube looks similar in both FIGS. 44 and 45, while the less regularly shaped chess piece has some reconstruction artifacts in FIG. 44. The size of the mesh can be adjusted to optimize the reconstruction artifacts while minimizing the computational burden.

[0122] Although various warping techniques are described individually, these techniques can be combined with each other to improve warping effectiveness without departing from the scope and spirit of the present disclosure. (System Architecture Overview)

[0123] 46 is a block diagram of an exemplary computing system 1300 suitable for implementing embodiments of the present disclosure. The computer system 1300 includes a processor 1307, a system memory 1308 (e.g., RAM), a static storage device 1309 (e.g., ROM), a disk drive 1310 (e.g., magnetic or optical), a communication interface 1314 (e.g., modem or Ethernet card), a display 1311 (e.g., CRT or LCD), an input device 1312 (e.g., keyboard), and a bus 1306 or other communication mechanism for communicating information that interconnects subsystems and devices such as cursor control.

[0124] According to one embodiment of the present disclosure, the computer system 1300 performs specific operations by the processor 1307 executing one or more sequences of one or more instructions contained in the system memory 1308. Such instructions may be read into the system memory 1308 from another computer-readable / usable medium, such as the static storage device 1309 or the disk drive 1310. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions to implement the present disclosure. Thus, embodiments of the present disclosure are not limited to any specific combination of hardware circuitry and / or software. In one embodiment, the term "logic" may refer to any combination of software or hardware used to implement all or a portion of the present disclosure.

[0125] The terms "computer-readable medium" or "computer usable medium," as used herein, refer to any medium that participates in providing instructions to the processor 1307 for execution. Such media may take many forms, including but not limited to, non-volatile media and volatile media. Non-volatile media include, for example, optical or magnetic disks, such as the disk drive 1310. Volatile media include dynamic memory, such as the system memory 1308.

[0126] Common forms of computer readable media include, for example, a floppy disk, a flexible disk, a hard disk, a magnetic tape, any other magnetic medium, a CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with a pattern of holes, RAM, PROM, EPROM, FLASH-EPROM (e.g., NAND Flash, NOR Flash), any other memory chip or cartridge, or any other medium from which a computer can read.

[0127] In certain embodiments of the present disclosure, execution of sequences of instructions for practicing the present disclosure is performed by a single computer system 1300. According to other embodiments of the present disclosure, two or more computer systems 1300 coupled by a communications link 1315 (e.g., a LAN, a PTSN, or a wireless network) may cooperate with each other to perform the sequences of instructions required to practice the present disclosure.

[0128] Computer system 1300 may transmit and receive messages, data, and instructions, including programs, e.g., application code, through communications link 1315 and communications interface 1314. Received program code may be executed by processor 1307 as it is received, and / or stored in disk drive 1310 or other non-volatile storage for later execution. A database 1332 in storage medium 1331 may be used to store data accessible by system 1300 via data interface 1333. (Virtual object depth determination)

[0129] The depth of a virtual object is needed to accurately warp that virtual object. However, in some embodiments, the depth of the virtual object is not known. For example, some applications only provide a stereo pair of images of a virtual object (e.g., for projection) and do not provide the depth of the virtual object. In order to accurately warp such virtual objects, a method is needed to determine the depth of such virtual objects while minimizing system resource demands.

[0130] 47-53 depict a method for determining / estimating the depth of a virtual object from a stereo pair of images for that virtual object. The method assumes that the application has rendered both components of the stereo pair using the appropriate point of view (POV), so that the first component of the stereo pair (e.g., left or right eye) is a perfect warping for the other component of the stereo pair (e.g., right or left eye). In general, the method identifies one component of the stereo pair (e.g., right eye) as the "reference eye" and warps / reprojects the second component of the stereo pair (e.g., left eye) to the POV of the first component at one or more depths. The method then compares the reprojected second component to the first component and determines the reprojection error at each of the one or more depths. The depth with the smallest reprojection error is then identified as the determined / estimated depth of the virtual object.

[0131] 53 is a flow chart illustrating a method 5310. In step 5312, the system (e.g., a VR / AR / MR system) reprojects a first image of a stereo pair of a virtual object over a second image of the stereo pair of the virtual object at one or more depths to generate one or more combined images.

[0132] Exemplary combined images are shown in Figures 50-52. In those figures, the second reference image 5012 remains stationary and the first test images are reprojected to different depths 5110, 5110', 5110'' across the second reference image 5112. Because of the different depths used to reproject the first test images, the reprojected first test images 5110, 5110', 5110'' are at different positions relative to the second reference image 5012.

[0133] The number of depths used in the method may be determined by the smaller of (1) the maximum number of depths that the system can evaluate in real time and (2) the maximum disparity in pixels between the viewer's eyes. In other words, even if the system is capable of evaluating 100 different depths in real time, if the maximum distance balance in pixels between the user's eyes corresponds to 50 different depths, then 50 different depths will be used in the method for identifying the depth of the virtual object. In some embodiments, 64 different depths are used in the method. The maximum disparity in pixels between the user's eyes may be a function of interpupillary distance, focal length, and / or near depth plane. In some embodiments, each depth plane is a uniform space such that the corresponding reprojected first test image is shifted by about one pixel interval.

[0134] Referring to Figure 53, in step 5314, the system measures the reprojection error at each depth. Reprojection error occurs when the depth plane used to reproject the first test image is sub-optimally misaligned to the depth of the virtual object (e.g., in the second reference image). The system quantifies the reprojection error at each depth.

[0135] In step 5316, the system analyzes one or more reprojection errors corresponding to one or more depths and identifies a minimum reprojection error. The depth with the minimum reprojection error is then identified in step 5318 as the determined / estimated depth of the virtual object.

[0136] To perform these analyses in real time, the system may split the series of evaluations at each of one or more depth planes into multiple frames (e.g., about 10 frames). The analyses may also be performed on downsampled versions of the image frames (e.g., at 25% in each dimension) to reduce the computational burden. Method 5300 may be repeated several times to improve the accuracy of the method.

[0137] 47-49 diagrammatically depict the method 5300 described above. FIG. 47 shows that a 3D virtual object 4710 is reprojected from the left eye 4714 to the right eye 4716 in one or more depth planes 4712. FIG. 48 shows that the method identifies a best warping plane 4718 (e.g., the combined image with the smallest reprojection error) that corresponds to the depth of the virtual object 4710. FIG. 49 shows that the best warping plane 4718 and its determined / estimated depth may be used by the system to more accurately warp the stereo pair of images 4714, 4716 to a new POV 4720 (e.g., resulting from a movement). (Stabilized plane depth determination)

[0138] For any given amount of POV movement, the apparent movement of the virtual object with the POV movement approaches zero as the virtual object approaches infinity. The "stabilization plane" is defined as the depth (approaching infinity) where the apparent movement of the virtual object with a particular POV movement approaches zero so that the apparent movement is undetectable / imperceptible to a system viewer. For example, system limitations in terms of display, processing, speed, etc. may make small movements (e.g., one or two pixels) undetectable / imperceptible to a typical user. Identifying the depth of the stabilization plane with a particular POV movement allows the system to avoid warping virtual objects farther than the stabilization plane with POV movements less than the particular POV movement. This optimization reduces system demands in terms of processing, memory, communication channels, etc.

[0139] As shown in FIG. 54, in some embodiments, there is an inverse correlation between the apparent movement of a virtual object (in terms of pixels) and the stabilization depth (in terms of diopters, which is the inverse of distance).

[0140] 55 depicts a method for determining stabilization plane depth. In step 5512, a system (e.g., a VR / AR / MR system) obtains a POV movement amount. The POV movement amount may be obtained from a sensor on the system indicative of user / viewer movement, or the POV movement amount may be a predetermined maximum POV movement value.

[0141] In step 5514, the system determines a relationship between stabilization depth and apparent virtual object movement based on the captured POV movement. In some embodiments, a pixel-to-dioptagraph such as that shown in FIG. 54 may be generated.

[0142] In step 5516, the system obtains the maximum undetectable / unperceivable apparent virtual object movement for the display system, as described above. This value may be measured in pixels.

[0143] In step 5518, the system identifies the stabilization depth corresponding to the maximum undetectable / unperceivable apparent virtual object movement as the minimum stabilization depth.

[0144] At this minimum stabilization depth, POV movements up to the POV movement amount obtained in step 5512 will be undetectable / imperceptible to a typical user of the system. Thus, the system may be configured to not warp any virtual objects farther than this minimum stabilization depth as long as the POV movement remains below the POV movement obtained in step 5512. Such warping of these virtual objects improves the efficiency of the system by reducing the system load in terms of processing, memory, communication channels, etc.

[0145] The present disclosure includes methods that may be performed using the subject devices. The methods may include the act of providing such a suitable device. Such providing may be performed by a user. In other words, the act of "providing" merely requires the user to obtain, access, approach, locate, configure, activate, power on, or otherwise act to provide the requisite device in the subject method. The methods described herein may be carried out in any order of the recited events that is logically possible, as well as the recited order of events.

[0146] Exemplary aspects of the present disclosure have been described above, along with details regarding material selection and manufacturing. As for other details of the present disclosure, these can be understood in conjunction with the patents and publications referenced above, and generally understood or understood by those skilled in the art. The same can be true with respect to the method-based aspects of the present disclosure in terms of additional acts as generally or theoretically adopted.

[0147] In addition, although the present disclosure has been described with reference to several embodiments that optionally incorporate various features, the present disclosure is not limited to those described or indicated as being considered with respect to each variation of the present disclosure. Various modifications may be made to the present disclosure as described, and equivalents may be substituted (whether described herein or not included for some brevity) without departing from the true spirit and scope of the present disclosure. In addition, when a range of values ​​is provided, it is to be understood that all intervening values ​​between the upper and lower limits of that range, and any other stated or intervening values ​​within that stated range, are encompassed within the present invention.

[0148] It is also contemplated that any feature of the described inventive variation may be described and claimed independently or in combination with any one or more of the features described herein. Reference to a singular item includes the possibility that there are multiple identical items. More specifically, as used herein and in the claims associated therewith, the singular forms "a," "an," "said," and "the" include multiple referents unless specifically stated otherwise. In other words, the use of articles allows for "at least one" of the subject item in the above description and in the claims associated with this disclosure. Furthermore, it is noted that such claims may be drafted to exclude any optional element. Thus, this statement is intended to serve as a predicate for the use of such exclusive terms, such as "only," "only," and the like, in connection with the description of claim elements, or the use of "negative" limitations.

[0149] Without using such exclusive terms, the term "comprising" in the claims associated with this disclosure is intended to permit the inclusion of any additional elements, regardless of whether a given number of elements are recited in such claims or whether the addition of features can be considered as changing the nature of the elements recited in such claims. Except as specifically defined herein, all technical and scientific terms used herein are to be given their broadest possible commonly understood meanings while maintaining the validity of the claims.

[0150] The scope of the present invention is not intended to be limited to the examples provided and / or this specification, but rather is intended to be limited only by the scope of the claims associated with this disclosure.

[0151] In the foregoing specification, the present disclosure has been described with reference to specific embodiments thereof. However, it will be apparent that various modifications and changes may be made therein without departing from the broader spirit and scope of the present disclosure. For example, the foregoing process flows are described with reference to a particular order of process actions. However, the order of many of the described process actions may be changed without affecting the scope or operation of the present disclosure. The specification and drawings are therefore to be regarded in an illustrative rather than a restrictive sense.

Claims

1. A method for determining a minimum stabilization depth, the method being performed by a computer, the method comprising: Obtaining a point of view (POV) movement amount; determining a relationship between a stabilization depth and pixel movement at the stabilization depth based on the POV movement; obtaining a maximum amount of user-imperceptible pixel movement for the display system; and identifying a stabilization depth corresponding to the maximum amount of pixel movement for the display system as the minimum stabilization depth for the display system; A method comprising:

2. The method of claim 1, further comprising warping only content that is closer to the user than the minimum stabilization depth when the POV changes.

3. The method of claim 2, wherein warping only content closer to the user than the minimum stabilization depth reduces demands on the display system.

4. The method of claim 3, wherein the reduced demand relates to a processor, memory, or communication channels of the display system.

5. The method of claim 1, wherein the relationship between stabilization depth measured in diopters and pixel movement measured in pixels at the stabilization depth is inversely correlated.

6. The method of claim 1, wherein the maximum amount of user-undetectable pixel movement for the display system is determined by limitations of the display system.

7. The method of claim 6, wherein the limits of the display system correspond to a display or processor of the display system.

8. The method of claim 1, wherein obtaining the POV movement amount includes receiving data from a sensor on the display system.

9. The method of claim 1, wherein the POV movement is a predetermined maximum POV movement.

10. The method of claim 1, wherein determining the relationship between a stabilization depth and pixel movement at the stabilization depth based on the POV movement amount includes generating a pixel vs. diopter graph.

11. The method of claim 1, wherein the maximum amount of user-undetectable pixel movement for a display system is measured in pixels.

12. The method described in claim 1, wherein at the stabilization depth from the user, POV movement less than the POV movement is undetectable to the user.

Images