Systems and methods for augmented reality

Abstract

To provide systems and methods for sharing perspective views of virtual content.SOLUTION: A method of presenting a virtual object 1102 to a user #1 via a display comprises determining a first perspective view of the virtual object based on a position of the virtual object and a first position of the user #1. The virtual object is presented to a user #2 via a display according to the first perspective view. An input is received from the first user, and a second perspective view of the virtual object based on the input from the first user is determined. Presenting the virtual object to the second user is presenting a transition from the first perspective view to the second perspective view.SELECTED DRAWING: Figure 10B

Description

[Technical Field]

[0001] (Cross-reference to related applications) This application claims priority to U.S. Provisional Patent Application No. 62 / 736,432, filed on 25 September 2018, the contents of which are incorporated herein by reference in their entirety.

[0002] (Field) This disclosure relates, in general, to systems and methods for sharing and presenting visual signals, and more particularly to systems and methods for sharing and presenting visual signals corresponding to content in a mixed reality environment. [Background technology]

[0003] (background) Modern computing and display technologies are driving the development of systems for so-called “virtual reality” or “augmented reality” experiences, where digitally reproduced images or parts thereof are presented to the user in a manner that makes them appear or be perceived as real. Virtual reality (i.e., “VR”) scenarios typically involve the presentation of digital or virtual imagery information that is not transparent to other real-world visual inputs; augmented reality (i.e., “AR”) scenarios typically involve the presentation of digital or virtual imagery information as an extension of the visualization of the real world around the user.

[0004] For example, referring to Figure 1, an augmented reality scene (4) is depicted, in which a user of AR technology sees a scene (6) resembling a real-world park, featuring people, trees, background buildings, and a concrete platform (1120). In addition to these items, the user of AR technology also perceives seeing a robot statue (1110) standing on the real-world platform (1120), and an anime-like avatar character (2) that appears to be a personification of a bumblebee by flying, even though these elements (2, 1110) do not exist in the real world.

[0005] The correct placement of this virtual image in the real world for realistic augmented reality (or "mixed reality") requires a series of interconnected coordinate frameworks.

[0006] The human visual perception system is extremely complex, making it challenging to create VR or AR technologies that facilitate the comfortable, natural, and rich presentation of virtual image elements within other virtual world image elements or real world image elements.

[0007] For example, a head-mounted AR display (or helmet-mounted display, or smart glasses) is typically at least loosely attached to the user's head, and therefore, when the user's head moves, the head-mounted AR display moves as well. Display components such as eyepieces in a head-mounted display may be positioned asymmetrically with respect to the user's eyes. For example, in a binocular system, one eyepiece may be positioned closer to or further away from a given eye (compared to, for example, a supplemental eyepiece and eye). In a monocular system, the alignment of a single eyepiece may be at an angle such that the left / right eye is not positioned similarly to the other eye.

[0008] Movements of the user's head, or otherwise changes in the user's position, complicate the variations in fit described above.

[0009] For example, if a user wearing a head-mounted display views a virtual representation of a three-dimensional (3D) object on the display and walks around the area where the 3D object appears, the 3D object may be re-rendered for each viewing point, giving the user the perception that they are walking around an object occupying real space. When a head-mounted display is used to present multiple objects in a virtual space (e.g., a rich virtual world), head posture measurement (i.e., the location and orientation of the user's head) may be used to re-render the scene to match the dynamically changing location and orientation of the user's head, providing an increased sense of immersion in the virtual space.

[0010] In an AR system, detecting or calculating head posture can facilitate the display system rendering virtual objects, thereby enabling the virtual objects to appear in a way that occupies space in the real world in a manner understandable to the user.

[0011] In some augmented reality technologies, such as Google Glass®, virtual content is displayed in a fixed position. In such examples, the virtual content and the device share a common coordinate frame so that any movement of the device similarly changes the position of the virtual content.

[0012] In some augmented reality or mixed reality systems, a set of coordinate frames ensures that virtual content appears fixed to the real world or environment in which the device exists, rather than being fixed to the device itself. [Overview of the Initiative] [Means for solving the problem]

[0013] (overview) Examples of this disclosure describe a system and method for sharing viewpoint views of virtual content. In the exemplary method, a virtual object is presented to a first user via a display. A first viewpoint view of the virtual object is determined, and the first viewpoint view is based on the position of the virtual object and the first position of the first user. The virtual object is presented to a second user via a display, and the virtual object is presented to the second user according to the first viewpoint view. Input is received from the first user. A second viewpoint view of the virtual object is determined, and the second viewpoint view is based on input from the first user. The virtual object is presented to the second user via a display, and presenting the virtual object to the second user comprises presenting a transition from the first viewpoint view to the second viewpoint view. The present invention provides, for example, the following: (Item 1) A method, wherein the said method is Presenting a virtual object via the display of a first head-mounted wearable device, Determining a first viewpoint view of the virtual object, wherein the first viewpoint view is based on the position and orientation of the virtual object and the position and orientation of the first head-wearable device. Presenting the virtual object via the display of a second head-mounted wearable device in accordance with the view of the first viewpoint, Receiving an input indicating a change in the view of the first viewpoint, Determining a second viewpoint view of the virtual object, wherein the second viewpoint view is based on the input indicating a change in the first viewpoint view, Presenting the virtual object via the display of the second wearable device in accordance with the view of the second viewpoint. Methods that include... (Item 2) The view of the first perspective presents the virtual object at a first angle with respect to the first head-wearable device, and the view of the second perspective presents the virtual object at a second angle with respect to the first head-wearable device, the method according to item 1. (Item 3) The second angle is different from the first angle, the method according to item 2. (Item 4) The view of the first perspective presents the virtual object at a first size, and the view of the second perspective presents the virtual object at a second size, the method according to item 1. (Item 5) The second size is different from the first size, the method according to item 4. (Item 6) The input indicating a change in the view of the first perspective comprises the first head-wearable device moving from a first position to a second position, the method according to item 1. (Item 7) The input indicating a change in the view of the first perspective comprises moving the virtual object from a first position to a second position, the method according to item 1. (Item 8) The input indicating a change in the view of the first perspective comprises a change in the perspective of the first head-wearable device, the method according to item 1. (Item 9) Presenting the virtual object via the display of the second wearable device according to the view of the second perspective comprises presenting a transition from the view of the first perspective to the view of the second perspective, the method according to item 1. (Item 10) A system, the system comprising a first head-wearable device, and a second head-wearable device, and one or more processors configured to execute the method and The method includes: presenting a virtual object via a display of a first head wearable device; determining a view of a first perspective of the virtual object, the view of the first perspective being based on the position and orientation of the virtual object and the position and orientation of the first head wearable device; presenting the virtual object via a display of a second head wearable device according to the view of the first perspective; receiving an input indicating a change in the view of the first perspective; determining a view of a second perspective of the virtual object, the view of the second perspective being based on the input indicating the change in the view of the first perspective; presenting the virtual object via the display of the second wearable device according to the view of the second perspective A system comprising: (Item 11) The system according to item 10, wherein the view of the first perspective presents the virtual object at a first angle with respect to the first head wearable device, and the view of the second perspective presents the virtual object at a second angle with respect to the first head wearable device. (Item 12) The system according to item 11, wherein the second angle is different from the first angle. (Item 13) The system according to item 10, wherein the view of the first perspective presents the virtual object at a first size, and the view of the second perspective presents the virtual object at a second size. (Item 14) The system according to item 13, wherein the second size is different from the first size. (Item 15) The system according to item 10, wherein the input indicating a change in the view of the first viewpoint comprises the first head-wearable device moving from a first position to a second position. (Item 16) The system according to item 10, wherein the input indicating a change in the view of the first viewpoint comprises moving the virtual object from a first position to a second position. (Item 17) The system according to item 10, wherein the input indicating a change in the view of the first viewpoint comprises a change in the viewpoint of the first head-wearable device. (Item 18) The system according to item 10, wherein presenting the virtual object via the display of the second wearable device in accordance with the view of the second viewpoint comprises presenting a transition from the view of the first viewpoint to the view of the second viewpoint. (Item 19) A non-transient computer-readable medium storing instructions, wherein, when an instruction is executed by one or more processors, the one or more processors cause the one or more processors to execute a method, the method is Presenting a virtual object via the display of a first head-mounted wearable device, Determining a first viewpoint view of the virtual object, wherein the first viewpoint view is based on the position and orientation of the virtual object, and the position and orientation of the first head-wearable device. Presenting the virtual object via the display of a second head-mounted wearable device in accordance with the view of the first viewpoint, Receiving an input indicating a change in the view of the first viewpoint, Determining a second viewpoint view of the virtual object, wherein the second viewpoint view is based on an input indicating a change in the first viewpoint view, Presenting the virtual object via the display of the second wearable device in accordance with the view of the second viewpoint. Non-transient, computer-readable media, including those mentioned above. (Item 20) The non-transient computer-readable medium described in item 19, wherein the first viewpoint view presents the virtual object at a first angle relative to the first head-wearable device, and the second viewpoint view presents the virtual object at a second angle relative to the first head-wearable device. (Item 21) The second angle is different from the first angle, and is a non-transient computer-readable medium as described in item 20. (Item 22) A non-transient, computer-readable medium as described in item 19, wherein the first viewpoint view presents the virtual object at a first size, and the second viewpoint view presents the virtual object at a second size. (Item 23) The second size is a non-transient, computer-readable medium as described in item 22, which differs from the first size. (Item 24) The input indicating a change in the view of the first viewpoint comprises the first head-wearable device moving from a first position to a second position, the non-transient computer-readable medium described in item 19. (Item 25) The non-transient computer-readable medium described in item 19, wherein the input indicating a change in the view of the first viewpoint comprises moving the virtual object from a first position to a second position. (Item 26) The input indicating a change in the view of the first viewpoint is a non-transient, computer-readable medium as described in item 19, comprising a change in the viewpoint of the first head-wearable device. (Item 27) The non-transient computer-readable medium described in item 19, wherein presenting the virtual object via the display of the second wearable device in accordance with the view of the second viewpoint comprises presenting a transition from the view of the first viewpoint to the view of the second viewpoint. [Brief explanation of the drawing]

[0014] [Figure 1] Figure 1 illustrates augmented reality scenarios with specific virtual reality objects according to several embodiments.

[0015] [Figure 2A] Figures 2A-2C illustrate various configurations of components comprising a visual display system according to several embodiments. [Figure 2B] Figures 2A-2C illustrate various configurations of components comprising a visual display system according to several embodiments. [Figure 2C] Figures 2A-2C illustrate various configurations of components comprising a visual display system according to several embodiments.

[0016] [Figure 3] Figure 3 illustrates remote interaction with cloud computing assets in several embodiments.

[0017] [Figure 4] Figure 4 illustrates a single-user coordinate frame system with virtual content in several embodiments.

[0018] [Figure 5] Figure 5 illustrates user device coordinate frames for a binocular rendering camera according to several embodiments.

[0019] [Figure 6]Figure 6 illustrates several embodiments of a multi-user coordinate frame system with virtual content.

[0020] [Figure 7] Figure 7 illustrates several embodiments of a multi-user shared coordinate frame system with virtual content.

[0021] [Figure 8A] Figures 8A-8C illustrate user-to-user world frame selection protocols in several embodiments. [Figure 8B] Figures 8A-8C illustrate user-to-user world frame selection protocols in several embodiments. [Figure 8C] Figures 8A-8C illustrate user-to-user world frame selection protocols in several embodiments.

[0022] [Figure 9A] Figures 9A and 9B illustrate devices for rendering camera frame transformation options according to several embodiments. [Figure 9B] Figures 9A and 9B illustrate devices for rendering camera frame transformation options according to several embodiments.

[0023] [Figure 10A] Figures 10A and 10B illustrate the relationship between the viewer and the resulting virtual content display in several embodiments. [Figure 10B] Figures 10A and 10B illustrate the relationship between the viewer and the resulting virtual content display in several embodiments.

[0024] [Figure 11A] Figures 11A-11C illustrate further angular relationships between a viewer at varying distances and the resulting virtual content display in several embodiments. [Figure 11B] Figures 11A-11C illustrate further angular relationships between a viewer at varying distances and the resulting virtual content display in several embodiments. [Figure 11C] Figures 11A-11C illustrate further angular relationships between a viewer and a composite virtual content display with varying distances, according to several embodiments.

[0025] [Figure 11D] Figures 11D-11E illustrate virtual content perception resulting from angle differences between users in several embodiments. [Figure 11E] Figures 11D-11E illustrate virtual content perception resulting from angle differences between users in several embodiments.

[0026] [Figure 12A] Figures 12A-12C illustrate exemplary mixed reality environments. [Figure 12B] Figures 12A-12C illustrate exemplary mixed reality environments. [Figure 12C] Figures 12A-12C illustrate exemplary mixed reality environments.

[0027] [Figure 13A] Figures 13A–13D illustrate components of an exemplary mixed reality system that could be used to generate and interact with a mixed reality environment. [Figure 13B] Figures 13A–13D illustrate components of an exemplary mixed reality system that could be used to generate and interact with a mixed reality environment. [Figure 13C] Figures 13A–13D illustrate components of an exemplary mixed reality system that could be used to generate and interact with a mixed reality environment. [Figure 13D]Figures 13A–13D illustrate components of an exemplary mixed reality system that could be used to generate and interact with a mixed reality environment.

[0028] [Figure 14A] Figure 14A illustrates an exemplary mixed reality handheld controller that may be used to provide input to a mixed reality environment.

[0029] [Figure 14B] Figure 14B illustrates an exemplary auxiliary unit that may be used in an exemplary mixed reality system.

[0030] [Figure 15] Figure 15 illustrates an exemplary functional block diagram for an exemplary mixed reality system. [Modes for carrying out the invention]

[0031] (Detailed explanation) The present invention relates to virtual content, a head-mounted display (HMD) for presenting AR content to at least one user, and a system and method for generating a plurality of coordinate frames to determine the relative position of the user's eyes.

[0032] Additional embodiments, advantages, and details are described in more detail below with appropriate specific reference to the figures.

[0033] Mixed reality environment

[0034] Like all people, users of mixed reality systems exist within the three-dimensional part of the real environment, or "the real world," and all of its content is perceptible to the user. For example, the user perceives the real environment using their normal human senses (sight, sound, touch, taste, smell) and interacts with the real environment by moving their body within it. A location in the real environment can be described as a coordinate in coordinate space (for example, coordinates may comprise latitude, longitude, and elevation relative to sea level), a distance in three orthogonal dimensions from a reference point, or other appropriate values. Similarly, a vector can describe a quantity that has direction and magnitude in coordinate space.

[0035] A computing device may maintain a representation of a virtual environment, for example, in the memory associated with the device. As used herein, a virtual environment is a computational representation of a three-dimensional space. A virtual environment may include representations of any objects, actions, signals, parameters, coordinates, vectors, or other features associated with that space. In some examples, the circuitry of a computing device (e.g., a processor) may maintain the state of a virtual environment; that is, the processor may determine the state of the virtual environment at a second time t1 based on data associated with the virtual environment and / or inputs provided by the user at a first time t0. For example, if an object in the virtual environment is located at a first coordinate at time t0 and has specific programmed physical parameters (e.g., mass, friction coefficient); and input received from the user indicates that a force should be applied to the object in the direction vector; the processor may apply the laws of kinematics and use basic mechanisms to determine the object's location at time t1. The processor may use any appropriate known information about the virtual environment and / or any appropriate inputs (e.g., real-world parameters) to determine the state of the virtual environment at time t1. In maintaining and updating the state of a virtual environment, the processor may run any appropriate software, including: software related to the creation and deletion of virtual objects in the virtual environment; software for defining the behavior of virtual objects or virtual characters in the virtual environment (e.g., scripts); software for defining the behavior of signals (e.g., acoustic signals) in the virtual environment; software for creating and updating parameters associated with the virtual environment; software for generating acoustic signals in the virtual environment; software for handling inputs and outputs; software for implementing network operations; software for applying asset data (e.g., animation data for moving virtual objects over time); or many other possible software.

[0036] Output devices such as displays or speakers may present some or all aspects of a virtual environment to the user. For example, a virtual environment may include virtual objects that can be presented to the user (including inanimate objects; people; animals; representations of light, etc.). A processor may determine a view of the virtual environment (e.g., corresponding to a "camera" with origin coordinates, view axes, and a frustum of a cone) and render a viewable scene of the virtual environment corresponding to that view on the display. Any suitable rendering technology may be used for this purpose. In some examples, a viewable scene may include only some virtual objects in the virtual environment and exclude certain other virtual objects. Similarly, a virtual environment may include acoustic aspects that can be presented to the user as one or more acoustic signals. For example, virtual objects in a virtual environment may generate sounds arising from the object's position coordinates (e.g., a virtual character may speak or produce sound effects); or the virtual environment may be associated with musical cues or ambient sounds, which may or may not be associated with specific locations. The processor may determine an acoustic signal corresponding to the "listener" coordinates (for example, an acoustic signal that corresponds to a composite of sounds in a virtual environment and has been mixed and processed to stimulate an acoustic signal that can be heard by a listener at the listener coordinates), and present the acoustic signal to the user through one or more speakers.

[0037] Since a virtual environment exists only as a computational structure, users cannot directly perceive it using their normal senses. Instead, users can only perceive the virtual environment indirectly, for example, when it is presented to them through a display, speaker, haptic output device, etc. Similarly, while users cannot directly touch, manipulate, or otherwise interact with the virtual environment, they can provide input data to a processor that can update the virtual environment using device or sensor data via input devices or sensors. For example, a camera sensor may provide light data indicating that the user is attempting to move an object in the virtual environment, and the processor can use that data to cause the object to respond accordingly in the virtual environment.

[0038] A mixed reality system may present a user with a mixed reality ("MR") environment that combines aspects of the real environment and aspects of the virtual environment, for example, using a transparent display and / or one or more speakers (which may be incorporated, for example, into a wearable head device). In some embodiments, one or more speakers may be located outside the head-mounted wearable unit. As used herein, an MR environment is a simultaneous representation of the real environment and a corresponding virtual environment. In some examples, the corresponding real and virtual environments share a single coordinate space; in some examples, the real coordinate space and one or more corresponding virtual coordinate spaces are related to each other by a transformation matrix (or other suitable representation). Thus, in some embodiments, a single coordinate (in some examples, together with a transformation matrix) may define a first location in the real environment and a second corresponding location in the virtual environment; and vice versa.

[0039] In an MR environment, a virtual object (for example, in a virtual environment associated with the MR environment) may correspond to a real object (for example, in a real environment associated with the MR environment). For example, if the real environment within the MR environment has a real streetlamp (real object) at a certain location coordinate, the virtual environment within the MR environment may have a corresponding virtual streetlamp (virtual object) at the corresponding location coordinate. As used herein, a real object, together with its corresponding virtual object, constitutes a “mixed reality object.” It is not necessary for the virtual object to perfectly match or be located in the same position as the corresponding real object. In some examples, the virtual object may be a simplified version of the corresponding real object. For example, if the real environment includes a real streetlamp, the corresponding virtual object may be a cylinder with approximately the same height and radius as the real streetlamp (reflecting that the streetlamp may be approximately cylindrical in shape). Simplifying the virtual object in this manner may enable computational efficiency and simplify the calculations that would otherwise be performed on such a virtual object. Furthermore, in some examples of MR environments, not all real-world objects in the real world are necessarily associated with their corresponding virtual objects. Similarly, in some examples of MR environments, not all virtual objects in the virtual environment are necessarily associated with their corresponding real-world objects. In other words, some virtual objects can exist only in the virtual environment of the MR environment, without any corresponding real-world counterparts. In some examples, not all real-world objects are necessarily associated with their corresponding real-world objects.

[0040] In some examples, virtual objects may have characteristics that differ (sometimes significantly) from those of their corresponding real-world objects. For instance, while a real-world environment in an MR environment might feature a green, two-armed cactus (a thorny, inanimate object), the corresponding virtual object in the MR environment might have the characteristics of a green, two-armed virtual character with human facial features and an unfriendly expression. In this example, the virtual object resembles its corresponding real-world object in certain characteristics (color, number of arms) but differs in other characteristics (facial features, personality). Thus, virtual objects have the potential to represent real-world objects in creative, abstract, exaggerated, or fantastical ways; or to give behavior (e.g., human personality) to another inanimate real-world object. In some examples, virtual objects may be mere imaginary creations with no real-world counterparts (e.g., a virtual monster in a virtual environment located in a space that might correspond to a blank space in the real-world environment).

[0041] In contrast to VR systems, which present a virtual environment to the user by disrupting the real environment, mixed reality (MR) systems, which present a virtual environment, offer the advantage of allowing the real environment to remain perceptible while the virtual environment is presented. Therefore, users of mixed reality systems can experience and interact with the corresponding virtual environment using visual and auditory cues associated with the real environment. For example, as noted above, since users cannot directly perceive or interact with the virtual environment, VR system users struggle to perceive or interact with virtual objects displayed in the virtual environment, whereas MR system users may find it intuitive and natural to interact with virtual objects by seeing, hearing, and touching the corresponding real objects in their own real environment. This level of interactivity can enhance the user's immersion in, connection to, and engagement with the virtual environment. Similarly, by presenting the real and virtual environments simultaneously, mixed reality systems can reduce the negative feelings (e.g., cognitive dissonance) and negative physical sensations (e.g., motion sickness) associated with VR systems. Mixed reality systems offer many more possibilities for applications that can enhance or alter our experience of the real world.

[0042] Figure 12A illustrates an exemplary reality environment 1200 in which user 1210 uses a mixed reality system 1212. The mixed reality system 1212 may comprise a display (e.g., a transparent display) and one or more speakers, and one or more sensors (e.g., a camera), as described below, for example. The shown reality environment 1200 comprises a rectangular room 1204A in which user 1210 is standing, and reality objects 1222A (a lamp), 1224A (a table), 1226A (a sofa), and 1228A (a painting). Room 1204A further comprises a position coordinate 1206, which may be considered the origin of the reality environment 1200. As shown in Figure 12A, an environment / world coordinate system 1208 having its origin at point 1206 (world coordinates) (with x-axis 1208X, y-axis 1208Y, and z-axis 1208Z) can define the coordinate space for the real environment 1200. In some embodiments, the origin 1206 of the environment / world coordinate system 1208 may correspond to the location where the mixed reality system 1212 is powered on. In some embodiments, the origin 1206 of the environment / world coordinate system 1208 may be reset during operation. In some examples, the user 1210 may be considered a real object in the real environment 1200; similarly, body parts of the user 1210 (e.g., head, feet) may be considered real objects in the real environment 1200. In some examples, a user / listener / head coordinate system 1214, with point 1215 (e.g., user / listener / head coordinates) as its origin (with x-axis 1214X, y-axis 1214Y, and z-axis 1214Z), can define the coordinate space for the user / listener / head on which the mixed reality system 1212 is located. The origin 1215 of the user / listener / head coordinate system 1214 can be defined with respect to one or more components of the mixed reality system 1212. For example, the origin 1215 of the user / listener / head coordinate system 1214 can be defined with respect to the display of the mixed reality system 1212, for example, during the initial calibration of the mixed reality system 1212. Matrices (matrices may include translation matrices and quaternion matrices or other rotation matrices), or other suitable representations, can characterize transformations between the user / listener / head coordinate system 1214 space and the environment / world coordinate system 1208 space.In some embodiments, the left ear coordinates 1216 and the right ear coordinates 1217 may be defined relative to the origin 1215 of the user / listener / head coordinate system 1214. A matrix (which may include a translation matrix and a quaternion matrix or other rotation matrix), or other suitable representation, may characterize the transformation between the left ear coordinates 1216 and the right ear coordinates 1217 and the user / listener / head coordinate system 1214 space. The user / listener / head coordinate system 1214 may simplify the representation of location relative to the user's head, or the representation of location relative to a head-mounted device (e.g., relative to the environment / world coordinate system 1208). The transformation between the user coordinate system 1214 and the environment coordinate system 1208 may be determined and updated in real time using Simultaneous Localization and Mapping (SLAM), visual odometry, or other techniques.

[0043] Figure 12B illustrates an exemplary virtual environment 1230 corresponding to a real environment 1200. The shown virtual environment 1230 comprises a virtual rectangular room 1204B corresponding to a real rectangular room 1204A; a virtual object 1222B corresponding to a real object 1222A; a virtual object 1224B corresponding to a real object 1224A; and a virtual object 1226B corresponding to a real object 1226A. The metadata associated with virtual objects 1222B, 1224B, and 1226B may include information derived from the corresponding real objects 1222A, 1224A, and 1226A. The virtual environment 1230 further comprises a virtual monster 1232, which does not correspond to any real object in the real environment 1200. Real object 1228A in the real environment 1200 does not correspond to any virtual object in the virtual environment 1230. A persistent coordinate system 1233 (persistent coordinates) having its origin at point 1234 (with x-axis 1233X, y-axis 1233Y, and z-axis 1233Z) can define a coordinate space for virtual content. The origin 1234 of the persistent coordinate system 1233 may be defined with respect to / in relation to one or more real objects, such as real object 1226A. A matrix (which may include translation matrices and quaternion matrices or other rotation matrices), or other suitable representation, may characterize the transformation between the persistent coordinate system 1233 space and the environment / world coordinate system 1208 space. In some embodiments, each of the virtual objects 1222B, 1224B, 1226B, and 1232 may have its own persistent coordinate point with respect to the origin 1234 of the persistent coordinate system 1233. In some embodiments, multiple persistent coordinate systems may exist, and each of the virtual objects 1222B, 1224B, 1226B, and 1232 may have its own persistent coordinate points relative to one or more persistent coordinate systems.

[0044] With respect to Figures 12A and 12B, the environment / world coordinate system 1208 defines a shared coordinate space for both the real environment 1200 and the virtual environment 1230. In the examples shown, the coordinate space has its origin at point 1206. Furthermore, the coordinate space is defined by the same three orthogonal axes (1208X, 1208Y, 1208Z). Thus, a first location in the real environment 1200 and a second corresponding location in the virtual environment 1230 can be described in the same coordinate space. Since the same coordinate system can be used to identify both locations, this simplifies the identification and representation of corresponding locations in the real and virtual environments. However, in some examples, the corresponding real and virtual environments do not need to use a shared coordinate space. For example, in some examples (not shown), a matrix (which may include translation matrices and quaternion matrices or other rotation matrices), or other suitable representation, may characterize the transformation between the real environment coordinate space and the virtual environment coordinate space.

[0045] Figure 12C illustrates an exemplary MR environment 1250 that simultaneously presents to the user 1210 aspects of the real environment 1200 and aspects of the virtual environment 1230 via the mixed reality system 1212. In the example shown, the MR environment 1250 simultaneously presents to the user 1210 real objects 1222A, 1224A, 1226A, and 1228A from the real environment 1200 (e.g., via the transparent portion of the display of the mixed reality system 1212) and virtual objects 1222B, 1224B, 1226B, and 1232 from the virtual environment 1230 (e.g., via the active display portion of the display of the mixed reality system 1212). As described above, the origin 1206 serves as the origin for the coordinate space corresponding to the MR environment 1250, and the coordinate system 1208 defines the x, y, and z axes with respect to the coordinate space.

[0046] In the examples shown, a mixed reality object comprises corresponding pairs of real and virtual objects (i.e., 1222A / 1222B, 1224A / 1224B, 1226A / 1226B) that occupy corresponding locations in coordinate space 1208. In some examples, both real and virtual objects may be visible to user 1210 simultaneously. This may be desirable, for example, when presenting information designed so that the virtual object enhances the view of the corresponding real object (e.g., in a museum application where the virtual object presents a missing piece of a damaged ancient sculpture). In some examples, the virtual objects (1222B, 1224B, and / or 1226B) may be displayed in a way that obscures the corresponding real objects (1222A, 1224A, and / or 1226A) (e.g., via active pixelated occlusion with a pixelated occlusion shutter). This may be desirable, for example, when a virtual object functions as a visual replacement for a corresponding real-world object (such as in interactive narrative applications where an inanimate real-world object becomes a "living" character).

[0047] In some examples, real-world objects (e.g., 1222A, 1224A, 1226A) may be associated with virtual content or auxiliary data that does not necessarily constitute a virtual object. This virtual content or auxiliary data can facilitate the processing or manipulation of virtual objects in a mixed reality environment. For example, such virtual content may include a two-dimensional representation of the corresponding real-world object; a custom asset type associated with the corresponding real-world object; or statistical data associated with the corresponding real-world object. This information can enable or facilitate calculations about real-world objects without incurring unnecessary computational overhead.

[0048] In some examples, the presentations described above may also incorporate acoustic aspects. For example, in the MR environment 1250, the virtual monster 1232 may be associated with one or more acoustic signals, such as the effect of footsteps generated when the monster walks around the MR environment 1250. As will be further described below, the processor of the mixed reality system 1212 may compute an acoustic signal corresponding to a mixed and processed composite of all such sounds in the MR environment 1250 and present the acoustic signal to the user 1210 via one or more speakers included in the mixed reality system 1212 and / or one or more external speakers.

[0049] Exemplary Mixed Reality System

[0050] Referring to Figures 2A-2D, several schematic component options are illustrated. In the detailed explanation following the discussion in Figures 2A-2D, various systems, subsystems, and components are presented to address the objective of providing a high-quality and comfortably perceived display system for human VR and / or AR.

[0051] As shown in Figure 2A, a user (60) of the AR system is depicted wearing an exemplary head-mounted component (58) featuring a frame (64) structure coupled to a display system (62) positioned in front of the user's eyes. A speaker (66) may be coupled to the frame (64) in the depicted configuration and positioned adjacent to the user's ear canal. (In one embodiment, another speaker (not shown) may be positioned adjacent to the user's other ear canal to provide stereo sound control / shapeable sound control). The display (62) may be operably coupled to a local processing and data module (70) by, for example, a wired or wireless connection (68), and the local processing and data module (70) may be mounted in various configurations, such as being fixedly attached to a frame (64), being fixedly attached to a helmet or hat (80) as shown in the embodiment of Figure 2B, being incorporated as headphones, being detachably attached to the torso of a user (60) in a backpack configuration, or being detachably attached to the buttocks (84) of a user (60) in a belt-connected configuration as shown in the embodiment of Figure 2C.

[0052] The local processing and data module (70) may include not only a low-power processor or controller but also digital memory such as flash memory, both of which may be utilized to assist in the processing, caching, and storage of data captured from sensors that can be operably coupled to the frame (64), such as image capture devices (cameras, etc.), microphones, inertial measurement units, accelerometers, compasses, GPS units, radio devices, and / or gyroscopes; and / or data acquired and / or processed using the remote processing module (72) and / or remote data repository (74) (possibly for delivery to the display (62) after such processing or retrieval).

[0053] The local processing and data module (70) may be operably coupled to a remote processing module (72) and a remote data repository (74) via, for example, a wired or wireless link (76, 78), thereby operably coupling these remote modules (72, 74) with one another and making them available as resources to the local processing and data module (70).

[0054] In one embodiment, the remote processing module (72) may comprise one or more relatively powerful processors or controllers configured to analyze and process data and / or image information. In one embodiment, the remote data repository (74) may comprise a relatively large digital data storage facility, which may be available via the internet or other network configuration in a “cloud” resource configuration. In one embodiment, all data may be stored and all calculations performed in the local processing and data modules, thereby enabling fully autonomous use from any remote module.

[0055] Exemplary Mixed Reality System

[0056] An exemplary mixed reality system 1212 (which may correspond to the exemplary AR system depicted in Figures 2A–2D) may include a wearable head device (e.g., a wearable augmented reality head device, or a wearable mixed reality head device) which comprises: a display (which may include left and right transmissive displays, or near-ear displays, or associated components for coupling light from the displays to the user's eyes); left and right speakers (e.g., positioned adjacent to the user's left and right ears, respectively); an inertial measurement unit (IMU) (e.g., mounted on the temple arm of the head device); a quadrature coil electromagnetic receiver (e.g., mounted on the left temple piece); left and right cameras (e.g., depth (time-of-flight) cameras) directed away from the user; and left and right eye cameras directed towards the user (e.g., for detecting the user's eye movements). However, the mixed reality system 1212 may incorporate any suitable display technology and any suitable sensors (e.g., light, infrared, acoustic, LiDAR, EOG, GPS, magnetic). In addition, the mixed reality system 1212 may incorporate network features (e.g., Wi-Fi capabilities) and communicate with other devices and systems, including other mixed reality systems. The mixed reality system 1212 may further include a battery (which may be mounted as an auxiliary unit, such as a belt pack designed to be worn around the user's waist), a processor, and memory. The wearable head device of the mixed reality system 1212 may include a tracking component such as an IMU or other suitable sensor, which is configured to output a set of coordinates of the wearable head device relative to the user's environment. In some examples, the tracking component may provide input to a processor that performs simultaneous localization and mapping (SLAM) and / or visual odometry algorithms. In some examples, the mixed reality system 1212 may also include a handheld controller 1400 and / or an auxiliary unit 1420, which may be a wearable belt pack as described further below.

[0057] Figures 13A–13D illustrate components of an exemplary mixed reality system 1300 (which may correspond to mixed reality system 1212) that may be used to present an MRE (which may correspond to MRE1250) or other virtual environment to a user. Figure 13A illustrates a perspective view of a wearable head device 2102 included in the exemplary mixed reality system 1300. Figure 13B illustrates a top view of the wearable head device 2102 mounted on a user's head 2202. Figure 13C illustrates a front view of the wearable head device 2102. Figure 13D illustrates a side view of an exemplary eyepiece 2110 of the wearable head device 2102. As shown in Figures 13A–13C, the exemplary wearable head device 2102 includes an exemplary left eyepiece (e.g., a left transparent waveguide set eyepiece) 2108 and an exemplary right eyepiece (e.g., a right transparent waveguide set eyepiece) 2110. Each eyepiece 2108 and 2110 may include not only a transmissive element (through which the real environment may be visible) but also a display element for presenting a display that overlaps with the real environment (e.g., via image-modulated light). In some examples, such a display element may include a surface diffractive optical element for controlling the flow of image-modulated light. For example, the left eyepiece 2108 may include a left internally coupled diffraction grating set 2112, a left orthogonal pupil dilation (OPE) diffraction grating set 2120, and a left exit (output) pupil dilation (EPE) diffraction grating set 2122. Similarly, the right eyepiece 2110 may include a right internally coupled diffraction grating set 2118, a right OPE diffraction grating set 2114, and a right EPE diffraction grating set 2116. Image-modulated light can be transmitted to the user's eye via the internally coupled diffraction gratings 2112 and 2118, OPE 2114 and 2120, and EPE 2116 and 2122. Each internally coupled diffraction grating set 2112, 2118 may be configured to deflect light toward its corresponding OPE diffraction grating sets 2120, 2114. Each OPE diffraction grating set 2120, 2114 is designed to progressively deflect light downward toward its associated EPE 2122, 2116, thereby horizontally elongating the formed exit pupil.Each EPE 2122, 2116 is configured to redirect at least a portion of the light received from its corresponding OPE diffraction grating sets 2120, 2114 in an increasing outward direction toward the user's eyebox position (not shown) defined behind the eyepieces 2108, 2110, thereby elongating the exit pupil formed in the eyebox vertically. Alternatively, instead of the internally coupled diffraction grating sets 2112 and 2118, the OPE diffraction grating sets 2114 and 2120, and the EPE diffraction grating sets 2116 and 2122, the eyepieces 2108 and 2110 may include other arrangements of diffraction gratings and / or refractive and reflective features to control the coupling of image-modulated light to the user's eye.

[0058] In some examples, the wearable head device 2102 may include a left temple arm 2130 and a right temple arm 2132, the left temple arm 2130 may include a left speaker 2134, and the right temple arm 2132 may include a right speaker 2136. The orthogonal coil electromagnetic receiver 2138 may be located in the left temple piece or in another suitable location within the wearable head unit 2102. The inertial measurement unit (IMU) 2140 may be located in the right temple arm 2132 or in another suitable location within the wearable head device 2102. The wearable head device 2102 may also include a left depth (e.g., time-of-flight) camera 2142 and a right depth camera 2144. The depth cameras 2142 and 2144 may be appropriately oriented in different directions and together cover a wider field of view.

[0059] In the example shown in Figures 13A-13D, the left image modulation light source 2124 may be optically coupled to the left eyepiece 2108 through the left internal coupling diffraction grating set 2112, and the right image modulation light source 2126 may be optically coupled to the right eyepiece 2110 through the right internal coupling diffraction grating set 2118. The image modulation light sources 2124 and 2126 may include, for example, an optical fiber scanner; a projector including an electronic optical modulator such as a digital light processing (DLP) chip or a liquid crystal on silicon (LCoS) modulator; or a light-emitting display (such as a micro-light-emitting diode (μLED) or micro-organic light-emitting diode (μOLED) panel coupled to the internal coupling diffraction grating sets 2112 and 2118 using one or more lenses per side). The input coupling diffraction grating sets 2112 and 2118 can deflect the light from the image modulation light sources 2124 and 2126 to an angle greater than the critical angle of total internal reflection (TIR) ​​with respect to the eyepieces 2108 and 2110. The OPE diffraction grating sets 2114 and 2120 increasingly deflect the light propagating by TIR downward toward the EPE diffraction grating sets 2116 and 2122. The EPE diffraction grating sets 2116 and 2122 increasingly combine the light toward the user's face, including the pupils of the user's eyes.

[0060] In some examples, as shown in Figure 13D, each of the left eyepiece 2108 and the right eyepiece 2110 includes multiple waveguides 2402. For example, each eyepiece 2108, 2110 may include multiple individual waveguides, each waveguide dedicated to its respective color channel (e.g., red, blue, and green). In some examples, each eyepiece 2108, 2110 may include multiple sets of such waveguides, each set configured to impart different wavefront curvatures to the emitted light. The wavefront curvature is convex to the user's eye and can, for example, present a virtual object positioned at a distance in front of the user (e.g., a distance corresponding to the reciprocal of each wavefront curvature). In some examples, EPE diffraction grating sets 2116, 2122 include curved diffraction grooves that can produce convex wavefront curvature by changing the Poynting vector of the emitted light traversing each EPE.

[0061] In some examples, stereo-adjusted left-eye and right-eye images may be presented to the user through image light modulators 2124, 2126 and eyepieces 2108, 2110 to create the perception that the displayed content is three-dimensional. The perceived reality of the presentation of three-dimensional virtual objects can be enhanced by selecting waveguides (and therefore corresponding to wavefront curvature) such that the virtual objects are displayed at a distance close to the distance indicated by the stereoscopic left-eye and right-eye images. This technique may also reduce motion sickness experienced by some users (motion sickness can be caused by the difference between the depth perception cues provided by the stereoscopic left-eye and right-eye images and the autonomous accommodation of the human eye (e.g., object distance-dependent focus)).

[0062] Figure 13D illustrates a top-down side view of the right eyepiece 2110 of an exemplary wearable head device 2102. As shown in Figure 13D, the waveguides 2402 may include a first subset 2404 of three waveguides and a second subset 2406 of three waveguides. The two subsets of waveguides 2404 and 2406 are distinguished by different EPE diffraction gratings featuring different diffraction line curvatures, which can give different wavefront curvatures to the emitted light. Within each of the waveguide subsets 2404 and 2406, each waveguide may be used to couple a different spectral channel (e.g., one of the red, green, and blue spectral channels) to the user's right eye 2206. (Although not shown in Figure 13D, the structure of the left eyepiece 2108 is similar to that of the right eyepiece 2110.)

[0063] Figure 14A illustrates an exemplary handheld controller component 1400 of the mixed reality system 1300. In some examples, the handheld controller 1400 includes a grip portion 1446 and one or more buttons 1450 arranged along the top surface 1448. In some examples, the buttons 1450 may be configured for use as optical tracking targets with a camera or other optical sensor (the camera or other optical sensor may be mounted within the head unit of the mixed reality system 1300 (e.g., a wearable head device 2102)) (e.g., to track the 6-degree-of-freedom (6DOF) motion of the handheld controller 1400). In some examples, the handheld controller 1400 includes a tracking component (e.g., an IMU or other suitable sensor) for detecting position or orientation (e.g., position or orientation relative to the wearable head device 2102). In some examples, such a tracking component may be positioned within the handle of the handheld controller 1400 and / or mechanically coupled to the handheld controller. The handheld controller 1400 may be configured to provide one or more of the pressed button states; or one or more output signals corresponding to the position, orientation, and / or movement of the handheld controller 1400 (e.g., by an IMU). Such output signals may be used as inputs to the processor of the mixed reality system 1300. Such inputs may correspond to the position, orientation, and / or movement of the handheld controller (and, more broadly, the position, orientation, and / or movement of the user's hand holding the controller). Such inputs may also correspond to user push buttons 1450.

[0064] Figure 14B illustrates an exemplary auxiliary unit 1420 of the mixed reality system 1300. The auxiliary unit 1420 may include a battery to provide energy to operate the system 1300, and the auxiliary unit 1420 may include a processor to run a program to operate the system 1300. As shown, the exemplary auxiliary unit 1420 includes a clip 2128 for attaching the auxiliary unit 1420 to the user's belt. Other form factors, including form factors that do not involve mounting the unit to the user's belt, may be suitable for the auxiliary unit 1420 and will become apparent. In some examples, the auxiliary unit 1420 is coupled to a wearable head device 2102 via multiple conduit cables, which may include, for example, electrical wires and fiber optics. Wireless connectivity between the auxiliary unit 1420 and the wearable head device 2102 may also be used.

[0065] In some examples, the mixed reality system 1300 may include one or more microphones for detecting sound and providing corresponding signals to the mixed reality system. In some examples, the microphones may be attached to or integrated with a wearable head device 2102, and the microphones may be configured to detect the user's voice. In some examples, the microphones may be attached to or integrated with a handheld controller 1400 and / or an auxiliary unit 1420. Such microphones may be configured to detect ambient sounds, surrounding noise, the user's or a third party's voice, or other sounds.

[0066] Exemplary Mixed Reality Network Architecture

[0067] Referring here to Figure 3, the schematic diagram illustrates the coordination between a cloud computing asset (46) and a local processing asset, the local processing asset may reside, for example, in a head-mounted component (58) coupled to the user's head (120), and in a local processing and data module (70) coupled to the user's belt (308). Component 70 may also be referred to as a “beltpack” 70. In one embodiment, a cloud (46) asset, such as one or more server systems (110), may be directly (40, 42) operably coupled (115) to one or both of the local computing assets (such as processors and memory configurations) coupled to the user's head (120) and belt (308) as described above, via, for example, wired or wireless networking (wireless is preferred for mobility, and wired is preferred for certain high bandwidth or large data transfers). These user-local computing assets can also be coupled to each other operationally via wired and / or wireless connectivity configurations (44), such as wired coupling (68), as discussed below with reference to Figure 8. In one embodiment, in order to maintain a low-inertia, compact subsystem mounted on the user's head (120), the primary transfers between the user and the cloud (46) may be via a link between the belt-mounted subsystem and the cloud. The head-mounted (120) subsystem may be data-tethered to the belt-based (308) subsystem, primarily using wireless connectivity such as ultra-wideband ("UWB") connectivity, which is currently employed in personal computing peripheral connectivity applications.

[0068] Through efficient coordination of local and remote processing, and by a suitable display device for the user, such as the user interface or user display system (62) or its variations shown in Figure 2A, an aspect of the world appropriate to the user's current real or virtual location can be transferred or "passed" to the user and updated in an efficient manner. In other words, the world map can be continuously updated in a storage location, which may reside partly on the user's AR system and partly in cloud resources. The map (also referred to as the "passable world model") can be a large database containing raster images of the real world, 3D and 2D points, parameter information, and other information. As more and more AR users continuously acquire information about their real environment (e.g., through cameras, sensors, IMUs, etc.), the map can become increasingly accurate and perfect.

[0069] In configurations described on the premise that there exists a single world model that may reside on and be distributed from cloud computing resources, such a world may be "deliverable" to one or more users in a relatively low-bandwidth format, preferably by passing around real-time video data, etc. The augmented experience of a person standing near a statue (i.e., the experience shown in Figure 1) may be provided with information by a cloud-based world model, a subset of which may be passed on to them and their local display devices to complete the view. A person sitting in front of a remote display device, which may be as simple as a personal computer on a desk, may efficiently download the same section of information from the cloud and render it on their display. In fact, one person who is actually present near a statue in a park may take a walk through the park with a friend who is located remotely, and the friend may participate through virtual and augmented reality. The system may need to know where the paths are, where the trees are, and where the statues are. In some embodiments, such information is stored in the cloud, and the participating friend downloads aspects of the scenario from the cloud and begins walking as local augmented reality for the person actually in the park.

[0070] 3D points can be captured from the environment, and the pose of the camera capturing those images or points (i.e., vector and / or origin position information relative to the world) can be determined, thereby allowing these points or images to be "tagged" or associated with this pose information. Points captured by a second camera can be used to determine the pose of the second camera. That is, the second camera can be pointed and / or positioned based on a comparison with the tagged image from the first camera. This knowledge (since there are two positioned cameras in the surroundings) can be used to extract textures from the real world, create maps, and generate virtual copies.

[0071] Therefore, at a base level, in one embodiment, a wearable system may be utilized to capture both 3D points and 2D images that give rise to those points, and these points and images may be transmitted to cloud storage and processing resources. The points and images may be cached locally along with embedded pose information (i.e., cache tagged images); so the cloud may have, along with the 3D points, tagged (i.e., tagged with 3D pose) 2D images ready (i.e., in the available cache). If the user is observing something dynamic, additional motion-related information may be transmitted to the cloud (for example, if the user is looking at another person's face, a texture map of the face may be captured and pushed up at an optimized frequency, even if the surrounding world is otherwise fundamentally static). More information regarding the object recognition unit and the passable world model can be found in U.S. Patent Application No. 14 / 205,126, entitled "System and method for augmented and virtual reality," which is incorporated herein by reference in its entirety.

[0072] Mixed reality coordinate frame

[0073] Figure 4 illustrates an exemplary environment for a user wearing an HMD AR device with a “head” coordinate frame. The HMD can create a “world” coordinate frame, for example, by passable world creation and mapping as described above, and the user’s head position and orientation can be measured relative to the world coordinate frame. The environment may also include virtual content, which may have its own “local” coordinate frame. The placement of the virtual content can be handled by the HMD by applying a transformation from local to world coordinate frame.

[0074] By assigning virtual content its own unique coordinate frame, rather than being directly measured to a world coordinate frame that could be the user's head, virtual content can choose a more persistent frame position. For example, if a virtual ramp is placed on a table, there may be multiple data points on the table to provide placement inputs for the relative positioning of the virtual ramp, which is substantially unchanged over time. In contrast, if a world map is created as a function of a particular orientation and position, and the user changes its position or orientation, thus requiring a new world coordinate frame, the virtual ramp may continue to utilize the same local coordinate frame rather than adjusting to a new world framework that could induce jitter or position shifts in the ramp's appearance.

[0075] In some embodiments, a coordinate frame may be established by using sensors of a mixed reality system (e.g., mixed reality system 1212 or 1300). For example, a world coordinate frame may be created using a depth sensor, a time-of-flight camera, a LiDAR sensor, and / or an RGB camera to identify the relative positions of physical objects. A mixed reality system used in a room may identify the physical features of the room and the arrangement of those features. For example, a mixed reality system may determine the position of a desk relative to the wall, the floor, the cabinet, and the chair. As the user walks around the room, the world coordinate frame may be refined so that physical objects are viewed from different angles and / or different distances, allowing for a more accurate determination of their relative positions. A mixed reality system may also establish a local coordinate frame using more localized features compared to the world coordinate frame. For example, a local coordinate frame may be established for a desk by identifying the features of the desk and the relative arrangement of those features. The corners of the desk can be identified and positioned relative to each other, thereby allowing the virtual object to appear as if it were a real object placed on the desk. The virtual object can be positioned using a local coordinate frame (for example, the position of the virtual object is determined relative to the corners of the desk). Then, the local coordinate frame (for example, the desk's) can be converted to a world coordinate frame that allows the desk to be positioned relative to other physical objects in the room.

[0076] Figure 5 illustrates the further coordinate frame transformation between the head coordinate frame and the rendering camera for either display unit (i.e., left / right in a binocular system, or a single system in a monocular field of view). Since the display medium's position relative to the user's eyes may vary, further coordinate frame analysis may be required if virtual content is rendered relative to that position. The rendering camera's position relative to the head coordinate frame may be provided by calibration level intrinsics. If content is projected onto the display independently of the rendering camera, changes in eye position may warp the intended content position. A further discussion of rendering camera transformations is provided below.

[0077] Figure 6 shows a multi-user system where user 1 and user 2 are viewing the same virtual content. As depicted, a world coordinate frame may be created for each user's own device, the virtual content may have a local coordinate frame, and the local coordinate frame may change to either world coordinate frame (e.g., world 1 and world 2).

[0078] In some embodiments, as shown in Figure 7, User 1 and User 2 may share a world coordinate frame. This can prevent small variations in the quality of the world coordinate frame and prevent system noise from giving different views of common content.

[0079] For example, in Figure 8A, User 1 and User 2 are in the room, but User 2 is closer to the left wall, and as a result, when User 2 looks towards the content near that wall, there are fewer data points to collect and create a reliable long-term world coordinate frame. In contrast, when User 1 looks at virtual content, there are more objects in the line of sight to create a reliable world coordinate frame. User 2 can then utilize the world coordinate frame created by User 1. Analysis of map quality and suitability of world coordinate frames can be further enabled in conjunction with U.S. Patent Application No. 62 / 702829, “Methods and Apparatuses for Determining and / or Evaluating Localization Maps of Head-Worn Image Display Devices,” the contents of which are incorporated herein by reference in their entirety.

[0080] As shown in Figure 8B, if User 2 moves closer to the wall, User 1's world coordinate frame may no longer be visible to User 2. Since User 2's head pose may not be measurable, the virtual content may begin to float or shift due to User 2's movement. Alternatively, a new world 2 coordinate frame may be created, albeit with less reliability in the world, as depicted in Figure 8C.

[0081] In some embodiments, a mixed reality system (e.g., mixed reality system 1212 or 1300) may receive a world coordinate frame from a server. For example, a room may have been mapped by a previous mixed reality system, and the established world coordinate frame may be uploaded to the server. When a new user enters the same room, the mixed reality system may recognize that the room has already been mapped and receive an appropriate world coordinate frame from the server. The mixed reality system may identify the room using location tracking (e.g., GPS coordinates or Wi-Fi triangulation) and / or computer vision (e.g., recognizing features within the room and matching those features to already mapped room features). In some embodiments, the world coordinate frame received from the server may be more reliable than the world coordinate frame established by user 1 or user 2. For example, each mixed reality system mapping a room may upload additional information to the server, thereby increasing the reliability of the stored world coordinate frames stored in the server. When a mixed reality system or server recognizes that a room already has an established world coordinate frame, it can determine which of several world coordinate frames is more reliable, and the mixed reality system can then utilize the most reliable world coordinate frame.

[0082] Figure 9A illustrates a first rendering camera protocol for converting head coordinates to a display unit. As depicted, the pupil of one of the user's eyes moves from position A to position B. With respect to an optical eyepiece that displays a virtual object when optically powered, the virtual object, which should appear stationary, can be projected in 3D at one of two positions based on the pupil position (assuming the rendering camera is configured to use the pupil as the coordinate frame). In other words, using pupil coordinates converted to head coordinates can cause jitter in stationary virtual content as the user's eye moves. This is referred to as a view-dependent display or projection system.

[0083] In some embodiments, as depicted in Figure 9B, the camera rendering frame is positioned, for example, at the center of rotation of the eyeball, to include all pupil positions. The object projection CR area may coincide independently of pupil positions A and B. Head coordinates may change in the camera rendering frame, which is referred to as a view-independent display or projection system. In some embodiments, image warping is given to virtual content to account for changes in eye position, but jitter can be minimized as it still renders in the same position.

[0084] Synchronized virtual content In environments where virtual content may be shared, further embodiments are enabled by the coordinate frame protocol described in detail above.

[0085] Figure 10A illustrates a shared virtual object 1102 that is visible to both User 1 and User 2. User 1 and User 2 may each have view vectors 1 and 2, respectively, which are measured in an exemplary xyz world coordinate frame 1100 relative to the virtual object 1102. In some embodiments, the angular difference θ A However, it can be measured between two vectors. Vectors 1 and 2 can be measured from the user to the virtual content 1102 based on the user's angular position relative to the virtual content in the world coordinate frame, but θ A Since this can be a comparison of vectors rather than absolute measurements, it may not be constrained by a coordinate frame. In other words, if user 1 and user 2 are using different world coordinate frames, the conversion between the two frames is the angular difference θ. A It may only be necessary to determine θ, A The process itself does not depend on the use of a specific world coordinate frame.

[0086] In some embodiments, rather than displaying content to all viewers with content fixed to a local coordinate frame, the presentation may be created based on the angular relationship with a specified user to enhance shared interactions among all viewers.

[0087] Figure 10B illustrates a synchronous view between User 1 and User 2, where User 1 is designated as a "director" or host who controls the viewing perspective of virtual content 1102 and can control the viewing perspectives of all users viewing virtual content 1102. At time t when User 1 is the director of the view A the virtual content can be displayed to all users who are in the shared content viewing mode with User 1 at a viewing perspective equal to that of User 1. User 2, who is a viewer of the shared director view, may have virtual content 1102-A displayed to them, which A at time t A may be the virtual object 1102 rotated by an angular difference θ.

[0088] During the duration of User 1's position as director among the users, the virtual content 1102 may be displayed to multiple users at time t with an angular rotation θ that reflects the angular difference between each viewing user and the director user at a given time. n at time t n Figure 11A - B further illustrates this, where User 1, who is in the director mode, moves around virtual object 1102 at time t

[0089] and has an angular difference θ B with User 2 (as shown in Figure 11A). Figure 11B illustrates a new angular rotation and displays virtual content 1102 - B to User 2. The cumulative effect of the angular change of User 1 displayed to User 2 may make it appear as if virtual content 1102 rotates with respect to User 2 even though User 2 has not physically moved. B

[0090] ​In some embodiments, the director view may only involve angular changes, and the relative distance of the director to the virtual content does not affect how the virtual content is displayed to the viewing user. Figure 11C shows the angular difference θ between User 1 (Director) and User 2. B The diagram illustrates a scenario in which the user moves to a position having a virtual content 1102, but the distance to the virtual content 1102 is reduced compared to the distance to user 1 in Figures 11A and 11B. In some embodiments, this may only result in an angular rotation relative to the user, thereby displaying a rotated-only virtual content 1102-B. User 1 may see the virtual content as a scaled virtual content 1102-1, while user 2 may not see a similarly scaled version of 1102-B. In some embodiments, the display of the virtual content may be scaled relative to user 1's position relative to the local coordinate frame for the virtual content 1102, but such distance scaling may not be visible to the shared viewer because this embodiment of the shared viewpoint view takes angular differences into account.

[0091] Figures 11D-11E illustrate this relationship regarding how virtual content may be displayed to User 1 and User 2, with User 1 as the director. Figure 11D illustrates User 1's field of view 1104-1, through which virtual content 1102-1 may be displayed according to User 1's position relative to virtual content 1102 as a function of both angle and distance position. Figure 11E illustrates User 2's field of view 1104-2, through which virtual content 1102-B may be displayed according to User 2's angle difference relative to User 1, but not according to User 2's distance from virtual content 1102.

[0092] In some embodiments, virtual content displayed in synchronous viewing mode may be displayed to one or more viewing users exclusively from the hosting user's viewpoint. For example, the hosting user (which may be a director) may select virtual content and display it in synchronous viewing mode. One or more viewing users may then view the selected virtual content from the hosting user's viewpoint. In some embodiments, synchronous virtual content may be displayed independently of the viewing user's coordinate frame. For example, one or more viewing users may view the synchronous virtual content at the center of their field of view. Synchronized virtual content displayed independently of the viewing user's coordinate frame may display virtual content that is not fixed to the viewing user's head posture and / or location. For example, while a viewing user is wearing a mixed reality system (which may correspond to mixed reality system 1212 or 1300), they may move around in different directions and look in different directions, but the synchronous virtual content may continue to be displayed to the viewing user from the hosting user's viewpoint.

[0093] In some embodiments, synchronized virtual content may be displayed to one or more viewers using one or more viewer coordinate frames. For example, synchronized virtual content may be displayed to viewers using a world coordinate frame and a local coordinate frame. The center of the virtual content may be fixed to coordinates within the local coordinate frame, and the local coordinate frame may be converted to a world coordinate frame (or other coordinate frame). While the center of the virtual content may be fixed to a coordinate frame, the viewpoint of the synchronized virtual content may be synchronized to the viewpoint of the hosting user. For example, as the hosting user walks around the synchronized virtual content, the synchronized virtual content may rotate and be displayed to the viewer looking at the synchronized virtual content, thereby allowing the viewer to share a viewpoint with the hosting user. In some embodiments, as the hosting user moves closer to or further away from the synchronized virtual content, the synchronized virtual content displayed to the viewer may scale so that its size appears the same to both the viewer and the hosting user. In some embodiments, as the hosting user moves closer to or further away from the synchronized virtual content, the synchronized virtual content displayed to the viewer may not scale (for example, as the hosting user moves closer, the synchronized virtual content displayed to the viewer may not appear to be larger). If the viewer shifts their viewpoint and looks away from the synchronized virtual content, the synchronized virtual content may no longer be visible to the viewer, and the synchronized virtual content may remain centered in a fixed position within the coordinate frame. In some embodiments, the hosting user and the viewer user may share a coordinate frame (e.g., a world coordinate frame and / or a local coordinate frame) to maintain consistency in the placement of the synchronized virtual content.

[0094] In some embodiments, a hosting user may manipulate synchronized virtual content that can be displayed to a viewing user. For example, a hosting user may resize synchronized virtual content to a larger or smaller size. A hosting user may also rotate synchronized virtual content without physically moving or shifting the hosting user's viewpoint of the synchronized virtual content. A hosting user may also relocate synchronized virtual content to a different location, which may be established using a coordinate frame and / or transformation matrix. A viewing user may view synchronized virtual content while the hosting user manipulates the synchronized virtual content so that the viewing user maintains the same viewpoint of the synchronized virtual content as the hosting user. In some embodiments, a hosting user may be a user of a mixed reality system (e.g., mixed reality system 1212 or 1300). In other embodiments, a hosting user may manipulate or view synchronized virtual content using a computer or mobile device that may have a 2D screen.

[0095] Various exemplary embodiments of the present invention are described herein. These examples are mentioned in a non-limiting sense. They are provided to illustrate broader applicable aspects of the present invention. Various modifications may be made to the described invention and equivalents may be substituted without departing from the true intent and scope of the present invention. In addition, many modifications may be made to adapt a particular state, material, compound, process, process act(s) or step(s) of the present invention to the object(s), intent, or scope of the present invention. Furthermore, as will be recognized by those skilled in the art, each of the individual variations described and illustrated herein has distinct components and features, which may be readily separated from or combined with features of any of the many other embodiments without departing from the scope or intent of the present invention. All such modifications are intended within the scope of the claims associated with this disclosure.

[0096] The present invention includes methods that can be carried out using the device. The methods may include the act of providing such a suitable device. Such provision may be carried out by an end user. That is, the act of “providing” simply requires the end user to acquire, access, approach, position, configure, activate, or otherwise operate the essential device in the method. The methods described herein may be carried out in any logically possible order of the described events, as well as in the order in which the events are described.

[0097] Exemplary aspects of the present invention have been described above, along with details relating to material selection and manufacturing. Other details of the present invention may be understood in relation not only to the patents and publications mentioned above, but also to patents and publications generally known or recognized by those skilled in the art. The same may apply to the method-based aspects of the present invention in terms of additional actions that are generally or logically adopted.

[0098] In addition, although the present invention has been described with reference to several examples that optionally incorporate various features, the present invention should not be limited to those described or shown as being assumed with respect to each variation of the present invention. Various modifications may be made to the described invention without departing from the true intent and scope of the present invention, and equivalents may be substituted (whether described herein or not included for the sake of brevity). In addition, if a range of values ​​is provided, it should be understood that all intermediate values ​​between the upper and lower limits of that range, and any other described values ​​or intermediate values, are included in the present invention.

[0099] Similarly, it is assumed that any optional feature of the variations of the invention described herein may be stated and claimed independently or in combination with any one or more of the features described herein. References to singular items include the possibility that there may be plural versions of the same item. In particular, as used herein and in the claims relating thereto, the singular forms “a,” “an,” “said,” and “the” include plural references unless specifically stated otherwise. That is, the use of articles allows for “at least one” of the items, not only in the above description but also in the claims relating to this disclosure. It is noted that such claims may be drafted to exclude any optional elements. This statement is therefore intended to serve as a prior ground for the use of such exclusionary terms of the kind such as “solely,” “only,” or “negative” limitations relating to the description of claim elements.

[0100] Without the use of such exclusionary terms, the term “comprising” in the claims associated with this disclosure shall allow for the inclusion of any additional elements, regardless of whether a given number of elements are enumerated in such claims or whether the addition of features may be considered to alter the nature of the elements described in such claims. Unless specifically defined herein, all technical and scientific terms used herein should be given in a sense that is as broadly and generally understood as possible, while maintaining the validity of the claims.

[0101] The scope of the present invention should not be limited to the examples and / or the foregoing, but should be limited only by the scope of the words of the claims associated with this disclosure.

Claims

1. A method, wherein the said method is Determining the viewpoint of a virtual object, wherein the viewpoint is at least partially based on the axis of view of a first wearable head device, Determining whether the presentation mode includes the first mode or the second mode, In accordance with the determination that the presentation includes the first mode, the size of the virtual object is adjusted based on the distance between the first wearable head device and the virtual object, In accordance with the decision that the presentation includes the second mode, refrain from adjusting the size of the virtual object, In accordance with the view of the aforementioned viewpoint, the virtual object is presented via the display of the second wearable head device. Methods that include...

2. The method according to claim 1, wherein the presentation mode is determined based on input to the first wearable head device.

3. The method according to claim 1, wherein the view of the virtual object's viewpoint is determined via a server.

4. The method according to claim 1, wherein the viewpoint of the virtual object is determined via one or more sensors of the first wearable head device.

5. The method according to claim 1, wherein the viewpoint of the virtual object is determined based on input to the first wearable head device.

6. Determining the movement of the second wearable head device, Adjusting the size of the virtual object based on a second distance between the second wearable head device and the virtual object, in accordance with the determined movement and the second determination that the presentation includes the first mode, In accordance with the determined movement and the second determination that the presentation includes the second mode, refrain from adjusting the size of the virtual object. The method according to claim 1, further comprising:

7. To determine whether the aforementioned presentation mode includes a third mode, In accordance with the determination that the presentation mode includes the third mode, the size of the virtual object is adjusted in accordance with the input to the first wearable head device. The method according to claim 1, further comprising:

8. The method according to claim 1, wherein determining the viewpoint of the virtual object includes determining the position of the virtual object as presented through the display of the second wearable head device.

9. A system comprising one or more processors, wherein the one or more processors communicate with a first wearable head device, and further communicate with a second wearable head device, and the one or more processors are configured to perform a method, the method being Determining the viewpoint of a virtual object, wherein the viewpoint is at least partially based on the axis of view of a first wearable head device, Determining whether the presentation mode includes the first mode or the second mode, In accordance with the determination that the presentation includes the first mode, the size of the virtual object is adjusted based on the distance between the first wearable head device and the virtual object, In accordance with the decision that the presentation includes the second mode, refrain from adjusting the size of the virtual object, The virtual object is presented via the display of the second wearable head device in accordance with the view of the aforementioned viewpoint. A system that includes this.

10. The system according to claim 9, wherein the presentation mode is determined based on input to the first wearable head device.

11. The system according to claim 9, wherein the view of the virtual object's viewpoint is determined via a server.

12. The system according to claim 9, wherein the viewpoint of the virtual object is determined via one or more sensors of the first wearable head device.

13. The system according to claim 9, wherein the viewpoint of the virtual object is determined based on input to the first wearable head device.

14. The aforementioned method, Determining the movement of the second wearable head device, Adjusting the size of the virtual object based on a second distance between the second wearable head device and the virtual object, in accordance with the determined movement and the second determination that the presentation includes the first mode, In accordance with the determined movement and the second determination that the presentation includes the second mode, refrain from adjusting the size of the virtual object. The system according to claim 9, further comprising:

15. To determine whether the aforementioned presentation mode includes a third mode, In accordance with the determination that the presentation mode includes the third mode, the size of the virtual object is adjusted in accordance with the input to the first wearable head device. The system according to claim 9, further comprising:

16. The system according to claim 9, wherein determining the viewpoint view of the virtual object includes determining the position of the virtual object as presented through the display of the second wearable head device.

17. A non-transient computer-readable medium storing instructions, wherein, when an instruction is executed by one or more processors, the one or more processors cause the one or more processors to execute a method, the method is Determining the viewpoint of a virtual object, wherein the viewpoint is at least partially based on the axis of view of a first wearable head device, Determining whether the presentation mode includes the first mode or the second mode, In accordance with the determination that the presentation includes the first mode, the size of the virtual object is adjusted based on the distance between the first wearable head device and the virtual object, In accordance with the decision that the presentation includes the second mode, refrain from adjusting the size of the virtual object, In accordance with the view of the aforementioned viewpoint, the virtual object is presented via the display of the second wearable head device. Non-transient, computer-readable media, including those mentioned above.

18. The presentation mode is determined based on input to the first wearable head device, the non-transient computer-readable medium according to claim 17.

Images