3D object annotation

Abstract

Disclosed herein are systems and methods for presenting and annotating virtual content. According to an example method, a virtual object is presented to a first user at a first location via a transmissive display of a wearable device. A first input is received from the first user. In response to receiving the first input, a virtual annotation is presented via the transmissive display at a first displacement from the first location. First data is sent to a second user, the first data being associated with the virtual annotation and the first displacement. A second input is received from the second user. In response to receiving the second input, the virtual annotation is presented to the first user via the transmissive display at a second displacement from the first location. Second data is sent to a remote server, the second data being associated with the virtual object, the virtual annotation, the second displacement, and the first location.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application No. 62 / 977,073, filed February 14, 2020, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure relates generally to systems and methods for presenting and annotating virtual content, and more particularly to systems and methods for presenting and annotating virtual content in a mixed reality environment. Background Art

[0004] Virtual environments are ubiquitous in computing environments, with applications in video games (where a virtual environment can represent a game world); maps (where a virtual environment can represent a terrain to be navigated); simulations (where a virtual environment can simulate a real environment); digital storytelling (where virtual characters can interact with each other in a virtual environment); and many other applications. Modern computer users are generally comfortable perceiving and interacting with virtual environments. However, the user's experience of a virtual environment can be limited by the technology used to render the virtual environment. For example, conventional displays (e.g., 2D screens) and audio systems (e.g., fixed speakers) may not be able to render a virtual environment in a way that produces a convincing, realistic, and immersive experience.

[0005] Virtual reality (“VR”), augmented reality (“AR”), mixed reality (“MR”), and related technologies (collectively “XR”) share the ability to present sensory information corresponding to a virtual environment represented by data in a computer system to a user of an XR system. The present disclosure contemplates distinctions between VR, AR, and MR systems (although some systems may be classified as VR in one aspect (e.g., visually) and simultaneously as AR or MR in another aspect (e.g., audio)). As used herein, a VR system presents a virtual environment that replaces the user's real environment in at least one aspect; for example, a VR system may present a view of a virtual environment to a user while obscuring his or her view of the real environment, such as with a light-blocking head-mounted display. Similarly, a VR system may present audio corresponding to a virtual environment to a user while blocking (attenuating) audio from the real environment.

[0006] VR systems may experience various disadvantages due to replacing the user's real environment with a virtual one. One disadvantage is the feeling of motion sickness, which can occur when the user's field of view in the virtual environment no longer corresponds to the state of their inner ears, which detect a person's balance and orientation in the real (non-virtual) environment. Similarly, users may experience disorientation in VR environments where their own body and limbs (the view upon which the user feels "grounded" in the real environment) are not directly visible. Another disadvantage is the computational burden (e.g., storage, processing power) placed on VR systems that must render a full 3D virtual environment, particularly in real-time applications that attempt to immerse the user in the virtual environment. Similarly, such environments may need to meet very high standards of realism to be considered immersive, as users are often sensitive to even the slightest imperfections in the virtual environment—any imperfection can disrupt the user's sense of immersion in the virtual environment. Furthermore, another disadvantage of VR systems is that such applications of the systems fail to utilize the extensive sensory data from the real environment, such as the various visual and auditory experiences people have in the real world. A related disadvantage is that VR systems may have difficulty creating a shared environment in which multiple users can interact, because users who share physical space in the real environment may not be able to directly see or interact with each other in the virtual environment.

[0007] As used herein, an AR system presents a virtual environment that overlaps or overlays a real environment in at least one aspect. For example, an AR system may present to a user a view of a virtual environment that is overlaid on a view of the user's real environment, such as with a transmissive head-mounted display that presents the displayed image while allowing light to pass through the display to the user's eyes. Similarly, an AR system may present to the user audio corresponding to the virtual environment while mixing in audio from the real environment. Similarly, as used herein, an MR system presents a virtual environment that overlaps or overlays a real environment in at least one aspect, as an AR system does, and may additionally allow the virtual environment in the MR system to interact with the real environment in at least one aspect. For example, a virtual character in a virtual environment may toggle a light switch in the real environment, causing a corresponding light bulb in the real environment to turn on or off. As another example, the virtual character may react to audio signals in the real environment (such as with facial expressions). By maintaining a representation of the real environment, AR and MR systems can avoid some of the aforementioned shortcomings of VR systems; for example, the user experiences less motion sickness because visual cues from the real environment (including the user's own body) can remain visible, and the system does not need to present the user with a fully realized 3D environment in order to be immersed in it. Further, AR and MR systems can leverage real-world sensory input (e.g., scenery, objects, and views and sounds of other users) to create new applications that enhance that input.

[0008] XR systems may be uniquely positioned to enable greater collaboration between people. The ability to present virtual content in a persistent and three-dimensional manner may allow people to interact with virtual content more naturally. For example, arranging virtual objects in three-dimensional space may enable more natural recall of location than a two-dimensional screen can provide. Where a user of a two-dimensional screen might have to hunt for one of forty open tabs to reopen a desired application, a user of an XR system may be able to precisely locate a desired virtual object displayed on a table (such as picking up a real folder placed on the table). In addition, XR systems may allow users to see virtual avatars of other users to simulate the live presence of others. This may enable more natural collaboration than a phone call or even a video conference can provide. Therefore, it may be desirable to develop systems and methods for enabling deep user collaboration on XR systems.

[0009] XR systems can provide a uniquely enhanced sense of immersion and realism by combining virtual visual and audio cues with real sight and sound. Consequently, in some XR systems, it is desirable to present a virtual environment that augments, improves, or alters the corresponding real environment. The present disclosure relates to XR systems that enable consistent placement of virtual objects across multiple XR systems. Summary of the Invention

[0010] Examples of the present disclosure describe systems and methods for presenting and annotating virtual content. According to the example method, a virtual object is presented to a first user at a first location via a transmissive display of a wearable device. A first input is received from the first user. In response to receiving the first input, a virtual annotation is presented via the transmissive display at a first displacement from the first location. First data is sent to a second user, the first data being associated with the virtual annotation and the first displacement. A second input is received from the second user. In response to receiving the second input, the virtual annotation is presented to the first user via the transmissive display at a second displacement from the first location. Second data is sent to a remote server, the second data being associated with the virtual object, the virtual annotation, the second displacement, and the first location. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figures 1A to 1C An example mixed reality environment is shown in accordance with some embodiments.

[0012] Figures 2A to 2D Components of an example mixed reality environment are shown that may be used to generate and interact with a mixed reality environment in accordance with some embodiments.

[0013] Figure 3A An example mixed reality handheld controller is shown that can be used to provide input to a mixed reality environment in accordance with some embodiments.

[0014] Figure 3BAn example auxiliary unit is shown that may be used with an example mixed reality system in accordance with some embodiments.

[0015] Figure 4 An example functional block diagram for an example mixed reality system is shown in accordance with some embodiments.

[0016] Figures 5A-5C An example of a mixed reality collaboration session is shown in accordance with some embodiments.

[0017] Figure 6 An example of a session manager architecture is shown in accordance with some embodiments.

[0018] Figure 7 Shown are examples of session instances according to some embodiments.

[0019] Figure 8 An example of a mixed reality collaboration session is shown in accordance with some embodiments.

[0020] Figure 9 Shown is an example of an annotation menu in accordance with some embodiments.

[0021] Figure 10 An example of mixed reality annotation is shown in accordance with some embodiments.

[0022] Figure 11 An example of mixed reality annotation is shown in accordance with some embodiments. DETAILED DESCRIPTION

[0023] In the following description of the examples, reference is made to the accompanying drawings which form a part hereof, and in which is shown by way of illustration specific examples that may be practiced. It will be understood that other examples may be used and structural changes may be made without departing from the scope of the disclosed examples.

[0024] Mixed reality environment

[0025] Like all people, users of mixed reality systems exist within a real environment—that is, they can perceive a three-dimensional portion of the "real world" and all its contents. For example, users perceive the real environment using their normal human senses—sight, hearing, touch, taste, and smell—and interact with it by moving their bodies within it. Positioning within the real environment can be described as coordinates in a coordinate space; for example, coordinates can include latitude, longitude, and altitude relative to sea level; distance from a reference point in three orthogonal dimensions; or other suitable values. Similarly, vectors can describe quantities that have direction and magnitude in a coordinate space.

[0026] A computing device may maintain a representation of a virtual environment, for example, in a memory associated with the device. As used herein, a virtual environment is a computational representation of a three-dimensional space. The virtual environment may include representations of any objects, actions, signals, parameters, coordinates, vectors, or other characteristics associated with the space. In some examples, circuitry (e.g., a processor) of a computing device may maintain and update the state of the virtual environment; that is, the processor may determine, at a first time t0, the state of the virtual environment at a second time t1 based on data associated with the virtual environment and / or input provided by a user. For example, if an object in the virtual environment is located at a first coordinate at time t0 and has certain programmed physical parameters (e.g., mass, coefficient of friction); and input received from the user indicates that a force should be applied to the object in a direction vector; the processor may apply the laws of kinematics to determine the position of the object at time t1 using basic mechanics. The processor may use any suitable information known about the virtual environment and / or any suitable input to determine the state of the virtual environment at time t1. In maintaining and updating the state of the virtual environment, the processor may execute any suitable software, including software relating to creating and deleting virtual objects in the virtual environment; software (e.g., scripts) for defining the behavior of virtual objects or characters in the virtual environment; software for defining the behavior of signals (e.g., audio signals) in the virtual environment; software for creating and updating parameters associated with the virtual environment; software for generating audio signals in the virtual environment; software for processing input and output; software for implementing network operations; software for applying asset data (e.g., animation data that moves virtual objects over time); or many other possibilities.

[0027] An output device (such as a display or speakers) can present any or all aspects of the virtual environment to the user. For example, the virtual environment may include virtual objects (which may include representations of inanimate objects; people; animals; lights, etc.) that can be presented to the user. The processor can determine a view of the virtual environment (e.g., corresponding to a "camera" having a coordinate origin, viewing axes, and a viewing cone); and render a visual scene of the virtual environment corresponding to that view to the display. Any suitable rendering technique can be used for this purpose. In some examples, the visual scene may include only some virtual objects in the virtual environment and not certain other virtual objects. Similarly, the virtual environment may include audio aspects that can be presented to the user as one or more audio signals. For example, a virtual object in the virtual environment may generate sounds that originate from the object's location coordinates (e.g., a virtual character may speak or cause a sound effect); or the virtual environment may be associated with musical cues or ambient sounds that may or may not be associated with a specific location. The processor may determine an audio signal corresponding to the “listener” coordinates—e.g., an audio signal corresponding to a synthesis of sounds in the virtual environment, mixed and processed to simulate the audio signal heard by the listener at the listener coordinates—and present the audio signal to the user via one or more speakers.

[0028] Because the virtual environment exists only as a computational construct, the user cannot directly perceive the virtual environment using their ordinary senses. Instead, the user can only perceive the virtual environment indirectly, such as when it is presented to the user through a display, speakers, tactile output devices, etc. Similarly, the user cannot directly touch, manipulate, or otherwise interact with the virtual environment; however, input data can be provided to the processor via input devices or sensors, and the processor can use the device or sensor data to update the virtual environment. For example, a camera sensor may provide optical data indicating that the user is attempting to move an object in the virtual environment, and the processor can use this data to cause the object in the virtual environment to respond accordingly.

[0029] A mixed reality system can present a mixed reality environment ("MRE") to a user that combines aspects of a real environment and a virtual environment, for example using a transmissive display and / or one or more speakers (which can be incorporated into a wearable head device, for example). In some embodiments, the one or more speakers can be external to the head-mounted wearable unit. As used herein, an MRE is a simultaneous representation of a real environment and a corresponding virtual environment. In some examples, the corresponding real environment and virtual environment share a single coordinate space; in some examples, the real coordinate space and the corresponding virtual coordinate space are related to each other by a transformation matrix (or other suitable representation). Thus, a single coordinate (in some examples, together with a transformation matrix) can define a first location in the real environment, and a second, corresponding location in the virtual environment, or vice versa.

[0030] In an MRE, a virtual object (e.g., in a virtual environment associated with the MRE) can correspond to a real object (e.g., in the real environment associated with the MRE). For example, if the real environment of the MRE includes a real light pole (real object) at a location coordinate, the virtual environment of the MRE can include a virtual light pole (virtual object) at the corresponding location coordinate. As used herein, a real object and its corresponding virtual object, combined together, constitute a "mixed reality object." It is not necessary for a virtual object to perfectly match or align with the corresponding real object. In some examples, a virtual object can be a simplified version of the corresponding real object. For example, if the real environment includes a real light pole, the corresponding virtual object can include a cylinder having approximately the same height and radius as the real light pole (reflecting that the light pole may be approximately cylindrical in shape). Simplifying a virtual object in this manner can improve computational efficiency and simplify calculations to be performed on the virtual object. Further, in some examples of MREs, not all real objects in the real environment can be associated with corresponding virtual objects. Similarly, in some examples of MREs, not all virtual objects in the virtual environment can be associated with corresponding real objects. That is, some virtual objects may only exist in the virtual environment of the MRE without any real-world counterparts.

[0031] In some examples, virtual objects can have characteristics that differ from, and sometimes drastically differ from, those of their corresponding real-world counterparts. For example, while the real-world environment in an MRE might include a green, two-armed cactus—a spiny, inanimate object—the corresponding virtual object in the MRE might have the characteristics of a green, two-armed virtual character with human-like facial features and a rambunctious demeanor. In this example, the virtual object resembles its corresponding real-world counterpart in some characteristics (color, number of arms); but differs from the real object in other characteristics (facial features, personality). In this way, it is possible for virtual objects to represent real-world objects in creative, abstract, exaggerated, or imaginative ways, or to impart behaviors (e.g., human personalities) to otherwise inanimate real-world objects. In some examples, virtual objects can be purely imaginative creations with no real-world counterpart (e.g., a virtual monster in a virtual environment, perhaps in a location that corresponds to an empty space in the real environment).

[0032] Compared to VR systems that present a virtual environment to the user while blurring the real environment, mixed reality systems that present MREs offer the advantage of maintaining the perceptual presence of the real environment while presenting the virtual environment. Therefore, users of mixed reality systems are able to experience and interact with the corresponding virtual environment using visual and auditory cues associated with the real environment. For example, while users of VR systems may find it difficult to perceive or interact with virtual objects displayed in a virtual environment—because, as mentioned above, users cannot directly perceive or interact with the virtual environment—users of MR systems may find it intuitive and natural to interact with virtual objects by seeing, hearing, and touching the corresponding real objects in their own real environment. This level of interactivity can enhance the user's sense of immersion, connection, and engagement with the virtual environment. Similarly, by simultaneously presenting the real and virtual environments, mixed reality systems can reduce negative psychological sensations (e.g., cognitive dissonance) and negative physical sensations (e.g., motion sickness) associated with VR systems. Mixed reality systems further offer many possibilities for applications that can augment or alter our experience of the real world.

[0033] Figure 1A An example real environment 100 is shown in which a user 110 uses a mixed reality system 112. For example, as described below, the mixed reality system 112 may include a display (e.g., a transmissive display) and one or more speakers, as well as one or more sensors (e.g., a camera). The real environment 100 shown includes a rectangular room 104A in which the user 110 is standing; and real objects 122A (lamp), 124A (table), 126A (couch), and 128A (painting). The room 104A also includes position coordinates 106, which can be considered the origin of the real environment 100. Figure 1AAs shown, an environment / world coordinate system 108 (including an x-axis 108X, a y-axis 108Y, and a z-axis 108Z) with an origin at point 106 (world coordinates) can define the coordinate space of real environment 100. In some embodiments, origin 106 of environment / world coordinate system 108 can correspond to the location where mixed reality environment 112 is powered on. In some embodiments, origin 106 of environment / world coordinate system 108 can be reset during operation. In some examples, user 110 can be considered a real object in real environment 100; similarly, body parts of user 110 (e.g., hands, feet) can be considered real objects in real environment 100. In some examples, a user / audience / head coordinate system 114 (including an x-axis 114X, a y-axis 114Y, and a z-axis 114Z) with an origin at point 115 (e.g., user / audience / head coordinates) can define the coordinate space of the user / audience / head in which mixed reality system 112 resides. The origin 115 of the user / listener / head coordinate system 114 can be defined relative to one or more components of the mixed reality system 112. For example, the origin 115 of the user / listener / head coordinate system 114 can be defined relative to the display of the mixed reality system 112, such as during initial calibration of the mixed reality system 112. A matrix (which can include a translation matrix and a quaternion matrix or other rotation matrix) or other suitable representation can represent the transformation between the space of the user / listener / head coordinate system 114 and the space of the environment / world coordinate system 108. In some embodiments, left ear coordinates 116 and right ear coordinates 117 can be defined relative to the origin 115 of the user / listener / head coordinate system 114. A matrix (which can include a translation matrix and a quaternion matrix or other rotation matrix) or other suitable representation can represent the transformation between the left ear coordinates 116 and right ear coordinates 117 and the space of the user / listener / head coordinate system 114. The user / listener / head coordinate system 114 can simplify the representation of a position relative to a user's head or head-mounted device (e.g., relative to the environment / world coordinate system 108). The transformation between the user coordinate system 114 and the environment coordinate system 108 may be determined and updated in real time using simultaneous localization and mapping (SLAM), visual odometry, or other techniques.

[0034] Figure 1BAn example virtual environment 130 corresponding to real environment 100 is shown. The illustrated virtual environment 130 includes a virtual rectangular room 104B corresponding to real rectangular room 104A; a virtual object 122B corresponding to real object 122A; a virtual object 124B corresponding to real object 124A; and a virtual object 126B corresponding to real object 126A. Metadata associated with virtual objects 122B, 124B, and 126B may include information derived from the corresponding real objects 122A, 124A, and 126A. Virtual environment 130 additionally includes a virtual monster 132, which does not correspond to any real object in real environment 100. Real object 128A in real environment 100 does not correspond to any virtual object in virtual environment 130. A persistent coordinate system 133 (including an x-axis 133X, a y-axis 133Y, and a z-axis 133Z) with its origin at point 134 (a persistent coordinate) may define a coordinate space for virtual content. The origin 134 of the persistent coordinate system 133 can be defined relative to / with respect to one or more real objects (such as real object 126A). A matrix (which may include a translation matrix and a quaternion matrix or other rotation matrix) or other suitable representation can represent the transformation between the space of the persistent coordinate system 133 and the space of the environment / world coordinate system 108. In some embodiments, each of the virtual objects 122B, 124B, 126B, and 132 can have its own persistent coordinate point relative to the origin 134 of the persistent coordinate system 133. In some embodiments, there can be multiple persistent coordinate systems, and each of the virtual objects 122B, 124B, 126B, and 132 can have its own persistent coordinate point relative to one or more persistent coordinate systems.

[0035] Persistent coordinate data may be coordinate data that persists relative to the physical environment. MR systems (e.g., MR systems 112, 200) may use persistent coordinate data to place permanent virtual content that may not be tied to the motion of a display displaying the virtual object. For example, a two-dimensional screen may only display virtual objects relative to their positions on the screen. As the two-dimensional screen moves, the virtual content may move with the screen. In some embodiments, persistent virtual content may be displayed in the corner of a room. An MR user may look into the corner to see virtual content, look out of the corner (wherein the virtual content may no longer be visible because the virtual content may have moved from within the user's field of view to a position outside the user's field of view due to movement of the user's head), and then look back at the virtual content in the corner (similar to how real objects may appear).

[0036] In some embodiments, persistent coordinate data (e.g., a persistent coordinate system and / or a persistent coordinate frame) may include an origin and three axes. For example, a persistent coordinate system may be assigned to the center of a room by the MR system. In some embodiments, a user may move around the room, leave the room, re-enter the room, etc., and the persistent coordinate system may remain at the center of the room (e.g., because it persists relative to the physical environment). In some embodiments, a transformation of the persistent coordinate data may be used to display virtual objects, which may enable the display of persistent virtual content. In some embodiments, the MR system may generate persistent coordinate data using simultaneous positioning and mapping (e.g., the MR system may assign a persistent coordinate system to a point in space). In some embodiments, the MR system may map the environment by generating persistent coordinate data at regular intervals (e.g., the MR system may assign persistent coordinate systems in a grid, where a persistent coordinate system may be at least within five feet of another persistent coordinate system).

[0037] In some embodiments, persistent coordinate data can be generated by the MR system and sent to a remote server. In some embodiments, the remote server can be configured to receive the persistent coordinate data. In some embodiments, the remote server can be configured to synchronize persistent coordinate data from multiple observation instances. For example, multiple MR systems can map the same room with persistent coordinate data and send the data to the remote server. In some embodiments, the remote server can use the observation data to generate canonical persistent coordinate data, which can be based on one or more observations. In some embodiments, canonical persistent coordinate data may be more accurate and / or more reliable than a single observation of persistent coordinate data. In some embodiments, canonical persistent coordinate data can be sent to one or more MR systems. For example, an MR system can use image recognition and / or position data to identify that it is located in a room with corresponding canonical persistent coordinate data (for example, because other MR systems have previously mapped the room). In some embodiments, the MR system can receive canonical persistent coordinate data corresponding to its location from a remote server.

[0038] Relative to Figure 1A and Figure 1B, the environment / world coordinate system 108 defines a shared coordinate space for both the real environment 100 and the virtual environment 130. In the example shown, the coordinate space has an origin at point 106. Further, the coordinate space is defined by the same three orthogonal axes (108X, 108Y, 108Z). Therefore, a first position in the real environment 100 and a second corresponding position in the virtual environment 130 can be described relative to the same coordinate space. This simplifies identifying and displaying corresponding positions in the real environment and the virtual environment because the same coordinates can be used to identify the two positions. However, in some examples, the corresponding real environment and virtual environment do not need to use a shared coordinate space. For example, in some examples (not shown), a matrix (which may include a translation matrix and a quaternion matrix or other rotation matrix) or other suitable representation can characterize the transformation between the real environment coordinate space and the virtual environment coordinate space.

[0039] Figure 1C An example MRE 150 is shown that simultaneously presents aspects of a real environment 100 and a virtual environment 130 to a user via a mixed reality system 112. In the example shown, the MRE 150 simultaneously presents real objects 122A, 124A, 126A, and 128A from the real environment 100 (e.g., via a transmissive portion of the display of the mixed reality system 112); and virtual objects 122B, 124B, 126B, and 132 from the virtual environment 130 (e.g., via an actively displayed portion of the display of the mixed reality system 112) to the user 110. As described above, the origin 106 serves as the origin of a coordinate space corresponding to the MRE 150, and the coordinate system 108 defines the x-axis, y-axis, and z-axis of the coordinate space.

[0040] In the example shown, the mixed reality objects include corresponding real object and virtual object pairs (i.e., 122A / 122B, 124A / 124B, 126A / 126B) occupying corresponding positions in coordinate space 108. In some examples, both the real object and the virtual object can be visible to user 110 at the same time. This may be desirable in instances where, for example, the virtual object presents information designed to augment the view of the corresponding real object (such as in a museum application where the virtual object presents missing portions of an ancient damaged sculpture). In some examples, the virtual object (122B, 124B, and / or 126B) can be displayed (e.g., via active pixelated occlusion using a pixelated occlusion shutter) so as to occlude the corresponding real object (122A, 124A, and / or 126A). This may be desirable in instances where, for example, the virtual object acts as a visual replacement for the corresponding real object (such as in an interactive storytelling application where an inanimate real object becomes a "living" character).

[0041] In some examples, real objects (e.g., 122A, 124A, 126A) can be associated with virtual content or helper data (which may not necessarily constitute a virtual object). The virtual content or helper data can facilitate the processing or handling of virtual objects in a mixed reality environment. For example, such virtual content can include a two-dimensional representation of the corresponding real object; a custom asset type associated with the corresponding real object; or statistical data associated with the corresponding real object. This information can enable or facilitate calculations involving real objects without incurring unnecessary computational overhead.

[0042] In some examples, the presentation described above may also include an audio aspect. For example, in MRE 150, virtual monster 132 may be associated with one or more audio signals, such as footstep effects generated when the monster walks around MRE 150. As described further below, a processor of mixed reality system 112 may calculate an audio signal corresponding to the mixed and processed synthesis of all such sounds in MRE 150 and present the audio signal to user 110 via one or more speakers included in mixed reality system 112 and / or one or more external speakers.

[0043] Sample Mixed Reality System

[0044] An example mixed reality system 112 may include a wearable head device (e.g., a wearable augmented reality or mixed reality head device) that includes: a display (which may include left and right transmissive displays, which may be near-eye displays, and associated components for coupling light from the displays to the user's eyes); left and right speakers (e.g., positioned adjacent to the user's left and right ears, respectively); an inertial measurement unit (IMU) (e.g., mounted to the temples of the head device); an orthogonal coil electromagnetic receiver (e.g., mounted to the left temple piece); left and right cameras oriented away from the user (e.g., depth (time of flight) cameras); and left and right eye cameras oriented toward the user (e.g., for detecting eye movements of the user). However, the mixed reality system 112 may include any suitable display technology and any suitable sensors (e.g., optical, infrared, acoustic, LIDAR, EOG, GPS, magnetic). In addition, the mixed reality system 112 may include network features (e.g., Wi-Fi capabilities) for communicating with other devices and systems (including other mixed reality systems). The mixed reality system 112 may also include a battery (which may be mounted in an auxiliary unit, such as a belt pack designed to be worn around the user's waist), a processor, and memory. The wearable head device of the mixed reality system 112 may include a tracking component, such as an IMU or other suitable sensor, that is configured to output a set of coordinates of the wearable head device relative to the user's environment. In some examples, the tracking component may provide input to a processor that performs simultaneous localization and mapping (SLAM) and / or visual odometry. In some examples, the mixed reality system 112 may also include a handheld controller 300 and / or an auxiliary unit 320, which may be a wearable belt pack, as further described below.

[0045] Figures 2A-2D Components of an example mixed reality system 200 (which may correspond to mixed reality system 112 ) that may be used to present an MRE (which may correspond to MRE 150 ) or other virtual environment to a user are shown. Figure 2A A perspective view of a wearable head device 2102 included in an example mixed reality system 200 is shown. Figure 2B A top view of a wearable head device 2102 worn on a user's head 2202 is shown. Figure 2C A front view of the wearable head device 2102 is shown. Figure 2D An edge view of an example eyepiece 2110 of a wearable head device 2102 is shown. Figures 2A-2CAs shown, an example wearable head device 2102 includes an example left eyepiece (e.g., a left transparent waveguide set eyepiece) 2108 and an example right eyepiece (e.g., a right transparent waveguide set eyepiece) 2110. Each eyepiece 2108 and 2110 may include: a transmission element through which the real environment can be visible; and a display element for presenting a display overlaid with the real environment (e.g., via image-level modulated light). In some examples, such a display element may include a surface diffraction optical element for controlling the flow of image-level modulated light. For example, the left eyepiece 2108 may include a left incoupling grating set 2112, a left orthogonal pupil expansion (OPE) grating set 2120, and a left exit (output) pupil expansion (EPE) grating set 2122. Similarly, the right eyepiece 2110 may include a right incoupling grating set 2118, a right OPE grating set 2114, and a right EPE grating set 2116. The image-level modulated light can be delivered to the user's eyes via the incoupling gratings 2112 and 2118, the OPEs 2114 and 2120, and the EPEs 2116 and 2122. Each incoupling grating set 2112, 2118 can be configured to deflect light toward its corresponding OPE grating set 2120, 2114. Each OPE grating set 2120, 2114 can be designed to incrementally deflect light downward toward its associated EPE 2122, 2116, thereby horizontally extending the exit pupil formed. Each EPE 2122, 2116 can be configured to incrementally redirect at least a portion of the light received from its corresponding OPE grating set 2120, 2114 outward to a user's eyebox location (not shown) defined behind the eyepieces 2108, 2110, vertically extending the exit pupil formed at the eyebox. Alternatively, in place of the coupling-in grating sets 2112 and 2118, the OPE grating sets 2114 and 2120, and the EPE grating sets 2116 and 2122, the eyepieces 2108 and 2110 may include other arrangements of gratings and / or refractive and reflective features for controlling the coupling of image-level modulated light to the user's eye.

[0046] In some examples, the wearable head unit 2102 can include a left temple 2130 and a right temple 2132, wherein the left temple 2130 includes a left speaker 2134 and the right temple 2132 includes a right speaker 2136. An orthogonal coil electromagnetic receiver 2138 can be positioned in the left temple piece, or in another suitable location in the wearable head unit 2102. An inertial measurement unit (IMU) 2140 can be positioned in the right temple 2132, or in another suitable location in the wearable head unit 2102. The wearable head unit 2102 can also include a left depth (e.g., time of flight) camera 2142 and a right depth camera 2144. The depth cameras 2142, 2144 can be suitably oriented in different directions so as to cover a wider field of view together.

[0047] exist Figures 2A-2D , an image-level modulated left light source 2124 can be optically coupled into the left eyepiece 2108 via a left incoupling grating set 2112, and an image-level modulated right light source 2126 can be optically coupled into the right eyepiece 2110 via a right incoupling grating set 2118. The image-level modulated light sources 2124, 2126 can include, for example, a fiber scanner; a projector including an electronic light modulator such as a digital light processing (DLP) chip or a liquid crystal on silicon (LCoS) modulator; or an emissive display such as a micro light emitting diode (μLED) or micro organic light emitting diode (μOLED) panel, which is coupled into the incoupling grating sets 2112, 2118 using one or more lenses on each side. The incoupling grating sets 2112, 2118 can deflect light from the image-level modulated light sources 2124, 2126 to an angle greater than the critical angle for total internal reflection (TIR) ​​of the eyepieces 2108, 2110. The OPE grating sets 2114, 2120 incrementally deflect light propagating via TIR downwardly towards the EPE grating sets 2116, 2122. The EPE grating sets 2116, 2122 incrementally couple light toward the user's face, including the pupils of the user's eyes.

[0048] In some examples, such as Figure 2DAs shown, each of the left eyepiece 2108 and the right eyepiece 2110 includes a plurality of waveguides 2402. For example, each eyepiece 2108, 2110 can include a plurality of separate waveguides, each dedicated to a corresponding color channel (e.g., red, blue, and green). In some examples, each eyepiece 2108, 2110 can include multiple groups of such waveguides, wherein each group is configured to impart a different wavefront curvature to the emitted light. The wavefront curvature can be convex relative to the user's eye, for example to present a virtual object positioned a certain distance in front of the user (e.g., a distance corresponding to the inverse of the wavefront curvature). In some examples, the EPE grating sets 2116, 2122 can include curved grating recesses that achieve a convex wavefront curvature by altering the Poynting vector across each EPE exit light.

[0049] In some examples, to create the perception that the displayed content is three-dimensional, stereoscopically accommodated left and right eye images may be presented to the user via image-level light modulators 2124, 2126 and eyepieces 2108, 2110. The perceived realism of the presentation of three-dimensional virtual objects may be enhanced by selecting the waveguides (and therefore the corresponding wavefront curvature) so that the virtual objects are displayed at a distance that approximates the distance indicated by the stereoscopic left and right images. This technique may also reduce motion sickness experienced by some users, which may be caused by the difference between the depth perception cues provided by the stereoscopic left and right eye images and the automatic accommodation of the human eye (e.g., focus related to object distance).

[0050] Figure 2D An edge-facing view from the top of the right eyepiece 2110 of an example wearable head device 2102 is shown. Figure 2D As shown, the plurality of waveguides 2402 may include a first subset 2404 of three waveguides and a second subset 2406 of three waveguides. The two subsets 2404, 2406 of waveguides may be distinguished by different EPE gratings, which are characterized by different grating line curvatures to impart different wavefront curvatures to the outgoing light. Within each of the subsets 2404, 2406 of waveguides, each waveguide may be used to couple a different spectral channel (e.g., one of the red, green, and blue spectral channels) to the user's right eye 2206. (Although in Figure 2D (Not shown, but the structure of the left eyepiece 2108 is similar to that of the right eyepiece 2110.)

[0051] Figure 3AAn example handheld controller assembly 300 for mixed reality system 200 is shown. In some examples, handheld controller 300 includes a handle portion 346 and one or more buttons 350 disposed along a top surface 348. In some examples, buttons 350 can be configured to serve as optical tracking targets, for example, for tracking six degrees of freedom (6DOF) motion of handheld controller 300 in conjunction with a camera or other optical sensor (which can be mounted in a head unit of mixed reality system 200 (e.g., wearable head device 2102)). In some examples, handheld controller 300 includes a tracking component (e.g., an IMU or other suitable sensor) for detecting position or orientation (such as relative to wearable head device 2102). In some examples, the tracking component can be located in the handle of handheld controller 300 and / or can be mechanically coupled to the handheld controller. Handheld controller 300 can be configured to provide one or more output signals corresponding to one or more pressed states of the buttons; or the position, orientation, and / or motion of handheld controller 300 (e.g., via an IMU). This output signal can be used as an input to a processor of the mixed reality system 200. The input can correspond to the position, orientation, and / or movement of the handheld controller (e.g., by extension, the position, orientation, and / or movement of the user's hand holding the controller). The input can also correspond to the user button 350.

[0052] Figure 3B An example auxiliary unit 320 of the mixed reality system 200 is shown. The auxiliary unit 320 may include a battery that provides energy to operate the system 200 and may include a processor for executing programs of the operating system 200. As shown, the example auxiliary unit 320 includes a clip 2128, such as for attaching the auxiliary unit 320 to a user's belt. Other form factors are suitable for the auxiliary unit 320 and will be apparent, including form factors that do not involve mounting the unit to a user's belt. In some examples, the auxiliary unit 320 is coupled to the wearable head device 2102 via a multi-conductor cable, which may include, for example, electrical wires and optical fibers. A wireless connection between the auxiliary unit 320 and the wearable head device 2102 may also be used.

[0053] In some examples, the mixed reality system 200 can include one or more microphones that detect sound and provide corresponding signals to the mixed reality system. In some examples, the microphone can be attached to or integrated with the wearable head device 2102 and configured to detect the user's voice. In some examples, the microphone can be attached to or integrated with the handheld controller 300 and / or the auxiliary unit 320. The microphone can be configured to detect ambient sound, ambient noise, the user's or a third party's voice, or other sounds.

[0054] Figure 4 1 shows an example functional block diagram that may correspond to an example mixed reality system, such as the mixed reality system 200 described above (which may correspond to the mixed reality system 112 with respect to FIG. 1 ). Figure 4As shown, an example handheld controller 400B (which may correspond to the handheld controller 300 ("Totem")) includes a totem to wearable head device six degrees of freedom (6DOF) totem subsystem 404A, and an example wearable head device 400A (which may correspond to the wearable head device 2102) includes a totem to wearable head device 6DOF subsystem 404B. In the example, the 6DOF totem subsystem 404A and the 6DOF subsystem 404B cooperate to determine the six coordinates of the handheld controller 400B relative to the wearable head device 400A (e.g., offsets in three translational directions and rotations along three axes). The six degrees of freedom can be expressed relative to the coordinate system of the wearable head device 400A. The three translational offsets can be expressed as X, Y, and Z offsets in that coordinate system, a translation matrix, or some other representation. The rotational degrees of freedom can be expressed as a sequence of yaw, pitch, and roll rotations, a rotation matrix, a quaternion, or some other representation. In some examples, the wearable head device 400A; one or more depth cameras 444 (and / or one or more non-depth cameras) included in the wearable head device 400A; and / or one or more optical targets (e.g., the button 350 of the handheld controller 400B as described above, or a dedicated optical target included in the handheld controller 400B) can be used for 6DOF tracking. In some examples, the handheld controller 400B can include a camera, as described above; and the wearable head device 400A can include an optical target for optical tracking in conjunction with the camera. In some examples, the wearable head device 400A and the handheld controller 400B each include a set of three orthogonally oriented solenoids that are used to wirelessly send and receive three distinguishable signals. By measuring the relative amplitudes of the three distinguishable signals received in each of the coils used for receiving, the 6DOF of the wearable head device 400A relative to the handheld controller 400B can be determined. Additionally, the 6DOF totem subsystem 404A may include an inertial measurement unit (IMU) that may be used to provide improved accuracy and / or more timely information regarding rapid movements of the handheld controller 400B.

[0055] In some examples, it may be necessary to transform coordinates from a local coordinate space (e.g., a coordinate space that is fixed relative to the wearable head device 400A) to an inertial coordinate space (e.g., a coordinate space that is fixed relative to the real environment), for example, to compensate for motion of the wearable head device 400A relative to the coordinate system 108. For example, the transformation may be necessary for the display of the wearable head device 400A to present a virtual object at a desired position and orientation relative to the real environment (e.g., a virtual person sitting in a real chair, facing forward, regardless of the position and orientation of the wearable head device), rather than at a fixed position and orientation on the display (e.g., at the same position in the lower right corner of the display) to maintain the illusion that the virtual object exists in the real environment (and, for example, does not appear unnaturally positioned in the real environment when the wearable head device 400A moves and rotates). In some examples, the compensating transformation between coordinate spaces can be determined by processing images from the depth camera 444 using SLAM and / or visual odometry programs to determine the transformation of the wearable head device 400A relative to the coordinate system 108. Figure 4 In the example shown, the depth camera 444 is coupled to the SLAM / visual odometry block 406 and can provide images to the block 406. The SLAM / visual odometry block 406 embodiment may include a processor configured to process the image and determine the position and orientation of the user's head, which can then be used to identify a transformation between the head coordinate space and another coordinate space (e.g., an inertial coordinate space). Similarly, in some examples, an additional source of information about the user's head pose and position is obtained from the IMU 409. Information from the IMU 409 can be integrated with information from the SLAM / visual odometry block 406 to provide improved accuracy and / or more timely information for rapid adjustment of the user's head pose and position.

[0056] In some examples, depth camera 444 can supply 3D images to gesture tracker 411, which can be implemented in a processor of wearable head device 400A. Gesture tracker 411 can identify a user's gesture, for example, by matching the 3D images received from depth camera 444 with stored patterns representing gestures. Other suitable techniques for identifying a user's gesture will be apparent.

[0057] In some examples, one or more processors 416 can be configured to receive data from the wearable head device's 6DOF headband subsystem 404B, IMU 409, SLAM / visual odometry block 406, depth camera 444, and / or gesture tracker 411. Processor 416 can also send and receive control signals from 6DOF totem system 404A. Processor 416 can be wirelessly coupled to 6DOF totem system 404A, such as in an example where handheld controller 400B is not limited. Processor 416 can also communicate with additional components, such as audio-visual content storage 418, graphics processing unit (GPU) 420, and / or digital signal processor (DSP) audio spatializer. DSP audio spatializer 422 can be coupled to head-related transfer function (HRTF) storage 425. GPU 420 can include a left channel output coupled to an image-level modulated left light source 424 and a right channel output coupled to an image-level modulated right light source 426. GPU 420 can output stereoscopic image data to image-level modulated light sources 424, 426, for example as described above with respect to Figures 2A-2D As described. The DSP audio spatializer 422 can output audio to the left speaker 412 and / or the right speaker 414. The DSP audio spatializer 422 can receive an input from the processor 419 indicating a direction vector from the user to the virtual sound source (which can be moved by the user, for example, via the handheld controller 320). Based on the direction vector, the DSP audio spatializer 422 can determine a corresponding HRTF (for example, by accessing the HRTF, or by interpolating multiple HRTFs). The DSP audio spatializer can then apply the determined HRTF to an audio signal, such as an audio signal corresponding to a virtual sound generated by a virtual object. This can enhance the credibility and realism of the virtual sound by incorporating the user's relative position and orientation with respect to the virtual sound in the mixed reality environment, that is, by presenting a virtual sound that matches the user's expectations of the virtual sound, the virtual sound sounds like a real sound in a real environment.

[0058] In some examples, such as Figure 4 As shown, one or more of the processor 416, GPU 420, DSP audio spatializer 422, HRTF memory 425, and audio / video content memory 418 may be included in an auxiliary unit 400C (which may correspond to the auxiliary unit 320 described above). Auxiliary unit 400C may include a battery 427 for powering its components and / or providing power to the wearable head device 400A or the handheld controller 400B. Including such components in an auxiliary unit that can be mounted to the user's waist can limit the size and weight of the wearable head device 400A, which in turn can reduce fatigue on the user's head and neck.

[0059] Although Figure 4 Elements corresponding to various components of an example mixed reality system are presented, but various other suitable arrangements of these components will become apparent to those skilled in the art. Figure 4 Elements presented as being associated with auxiliary unit 400C may instead be associated with wearable head device 400A or handheld controller 400B. In addition, some mixed reality systems may completely forgo handheld controller 400B or auxiliary unit 400C. It will be understood that such changes and modifications are included within the scope of the disclosed examples.

[0060] Session Manager

[0061] MR systems can be uniquely positioned to enable interactive virtual collaboration between users. Because MR systems can present virtual content in three dimensions and within the user's physical environment, MR collaboration systems and methods can enable remote collaboration that is at least as effective as local collaboration. In some embodiments, MR collaboration can allow users to view and / or manipulate virtual content in three-dimensional space. For example, a first user can start an MR collaboration session and see two virtual 3D models, a text document, and a messaging interface. A second user can join the session locally (e.g., the second user may walk into the same room as the first user) and see the same two virtual 3D models, text document, and messaging interface at the same location as the first user. In some embodiments, a third user can join the session remotely (e.g., the third user may not be in the same room as the first and second users) and see the two virtual 3D models, text document, and messaging interface in the third user's environment. In some embodiments, virtual content can share a spatial relationship with each other for all session users (e.g., the virtual content can be arranged in the same manner). In some embodiments, MR collaboration can allow users in the same physical space to leverage a shared physical context to enjoy a more meaningful shared experience involving virtual content.

[0062] In some embodiments, displaying and / or synchronizing virtual content across multiple MR systems may present challenges. For example, it may be beneficial to develop systems and methods for ensuring that each MR system displays shared virtual content in a manner consistent with the other MR systems in the session. It may also be beneficial to develop systems and methods that can enable cross-application collaboration (e.g., virtual content generated by applications created by different developers can be used). In some embodiments, it may be beneficial to develop systems and methods that can allow users who are local to each other (e.g., users in the same room) to collaborate with each other and with remote users (e.g., in different rooms). In some embodiments, it may be beneficial to develop systems and methods that can enable collaborative sessions to continue over time so that session users can continue to collaborate at a later time. In some embodiments, it may be beneficial to develop systems and methods that can enable content persistence so that session users can continue to work on virtual content even when they are not collaborating on-site with other users.

[0063] In some embodiments, a session can be broadly defined as a group of users (with identifiers) that can collaborate and share a series of experiences over time and space. In some embodiments, a session can include a communication and collaboration experience that provides network connectivity, a common spatial reference, and a centralized user interface for chatting and sharing prisms with other MR users. Session participants can be remote or local in the same physical location. In some embodiments, a session manager can include a centralized backend service that manages some or all activities within a session. In some embodiments, the session manager can include one or more user-facing front-end controls and / or expressions that represent the session manager and / or are configured to receive user input (e.g., menus and / or session handles). In some embodiments, the session manager can include a background service and / or daemon that coordinates and manages various session events through various session states. The session manager can also facilitate the user experience by allowing users to discover and connect with other users. In some embodiments, the session manager can also manage various UI components, such as menus and / or session UI-related states.

[0064] In some embodiments, collaboration can be facilitated by configuring virtual content in a collaboration session to behave similarly to real objects in the collaboration session. For example, in a "real" collaboration session, users might sit around a table with documents and / or objects. Users can refer to "this" document and / or "that" document by pointing to a specific document. In some embodiments, users in a real collaboration session may refer to objects using relational terms (e.g., "the object on the right"). This behavior may come naturally to users due to years of conditioning and actually working with others. Therefore, it may be desirable to develop systems and methods for MR collaboration that enable natural interaction between users and the content they are collaborating on. In some embodiments, MR collaboration sessions can enable users to reference juxtaposed virtual content (e.g., virtual content that may appear in the same location in multiple users' real environments) as if the virtual content were real content in the users' physical environments. In some embodiments, MR collaboration sessions can be persistent. For example, all users can exit a session, and users can restart the same session weeks later. In some embodiments, users can see all virtual content in the same state as when the user exited the session (e.g., in the same relative position and / or with the same edits).

[0065] In some embodiments, a session may include a platform for presenting, synchronizing, managing, and / or storing virtual content used in a mixed reality collaborative session. For example, session users may have recurring weekly meetings where virtual content (e.g., Word documents, 3D models, presentation slides, conversation histories, etc.) is discussed and / or processed. In some embodiments, users can leverage the session platform to merge virtual content (which may be created by different developers) into a single virtual space that persists over time. For example, loading a single session instance may present the user with a 3D model (generated using a first application created by a first developer), a text document describing changes and / or goals to the 3D model (generated using a second application created by a second developer), and a history of conversations between session users associated with the session. This virtual content can persist across time and across session users, allowing the same user or a different session user to load a session and see the same session content as any other session user. In some embodiments, sessions can enable user presence flexibility (e.g., local users can share virtual content placement in their local space, but remote users can also see virtual content with the same spatial relationship in their remote space). In some embodiments, sessions can enable capability flexibility. For example, capabilities (e.g., corresponding to third-party applications) can be interacted / enabled / disabled without leaving the centralized session platform. In some embodiments, applications (e.g., third-party applications) can leverage the session platform to avoid building proprietary shared platforms that may be incompatible with other application software. In some embodiments, sessions can be time-flexible. For example, users may access a session at different times and may not need to be in real-time with other users. In some embodiments, changes made by a user can be synchronized so that the changes are reflected for other session users (whether they are currently in the session or enter the session later).

[0066] In some embodiments, a session may include virtual content shared with one or more users over time. A session may have one or more owners, and in some embodiments, the user who creates the session may be considered the session owner. A session may have one or more participants who can access the session. In some embodiments, the session owner may control which participants can join the session. In some embodiments, a session may have a session identifier. In some embodiments, each user (e.g., an owner or a participant) may have a user identifier. In some embodiments, a session may include one or more user avatars that may represent the location of the remote user relative to other objects in the session. In some embodiments, a session may include positioning data (e.g., positioning data corresponding to each user, positioning data corresponding to the location of the session that has been opened, etc.). The positioning data may include persistent coordinate data. In some embodiments, the positioning data may include one or more transformations (e.g., one or more transformation matrices) that may relate a location to persistent coordinate data.

[0067] In some embodiments, a session may include one or more capabilities. A session capability may include one or more features that a user may select and / or enable in a session. For example, virtual object sharing may be considered a session capability. In some embodiments, determining whether a user is local to other users may be considered a session capability. In some embodiments, projecting a user's avatar may be considered a session capability. In some embodiments, projecting a user's screen to other users may be considered a session capability. In some embodiments, a capability may have one or more capability instances (e.g., a capability may have multiple instances running simultaneously). For example, two virtual objects may be shared with users in a session, and each virtual object may be considered a separate capability instance.

[0068] In some embodiments, a session can be persistent. For example, a session can continue to exist even after all users have exited the session. In some embodiments, a session can continue to store session information, such as the session capabilities used (e.g., shared virtual objects, where virtual objects are located, etc.), user location, user identification, etc. Persistent sessions can facilitate long-term collaboration between users. For example, users can continue where they left off without having to rearrange their virtual workspace to their preferences. In some embodiments, session persistence can enable different users to enter the session at a later time and see the virtual content that was arranged when the previous user exited the session.

[0069] Figures 5A-5C An exemplary MR collaboration session is shown in accordance with some embodiments. Figure 5AAn exemplary mixed reality collaboration session is shown in which users 508a, 508b, and 508c may be together at a first location (eg, a first room). Figure 5B An exemplary mixed reality collaboration session is shown in which users 508d and 508e may be together at a second location (eg, a second room). Figure 5C An example mixed reality collaboration session showing a mobile session handle.

[0070] In some embodiments, users 508a, 508b, 508c, 508d, and 508e can all be part of the same mixed reality collaboration session 500. In some embodiments, the collaboration session can include a session handle 502a (which can be a virtual object). The session handle 502a can serve as a local anchor for the session. For example, virtual content positioned relative to the session handle 502a can be presented to all session users located in the same location (e.g., if users 508a, 508b, and 508c share common persistent coordinate data, users 508a, 508b, and 508c can be considered to be in the same location). This can make the virtual content appear to be located at a specific position and orientation in the real world, similar to real / physical objects. In some embodiments, the session handle 502a can be positioned relative to the persistent coordinate data (e.g., using a transformation). In some embodiments, users 508a, 508b, and 508c can use canonical persistent coordinate data, which can achieve consistent placement of the session handle 502a in each user's MR system. In some embodiments, users 508a, 508b, and 508c may all see session handle 502a at the same location (eg, the users may all see session handle 502a at the same location on the floor).

[0071] In some embodiments, persistent coordinate data can be used to determine whether users can be considered local to one another. For example, the MR system of user 508a can receive canonical persistent coordinate data based on the identified environment of user 508a (e.g., from one or more remote servers). The MR system of user 508a can use positioning data (e.g., GPS, WiFi, and / or cellular data) and / or image recognition data (e.g., identifying a known environment by comparing acquired images with images of known environments) to identify the environment of user 508a. In some embodiments, the MR system of user 508a can send the received persistent coordinate data to other MR systems in the session (e.g., the MR system of user 508b). In some embodiments, the other MR systems in the session can receive the canonical persistent coordinate data and compare the transmitted data received from the other MR systems with the canonical persistent coordinates already in use (and / or the canonical persistent coordinate data received from one or more remote servers). If it is determined (e.g., using a unique identifier) ​​that one or more instances of canonical persistent coordinate data are shared between the MR systems in the session, then the MR systems can be determined to be local to one another. In some embodiments, if the MR systems do not share instances of canonical persistent coordinate data, then the MR systems can be determined to be remote from one another. In some embodiments, a session handle (e.g., session handle 502a) can be displayed in association with one or more shared instances of persistent canonical persistent coordinate data, which can enable session handle 502a to be presented to users 508a, 508b, and 508c in the same location.

[0072] In some embodiments, the conversation 500 may include a shared virtual object 504a. The shared virtual object 504a may be considered a conversation capability instance. In some embodiments, users 508a, 508b, and 508c may all see the virtual object 504a in the same location (e.g., a user may see the virtual object 504a at the end of a real table). In some embodiments, the shared virtual object 504a may be positioned relative to the conversation handle 502a (e.g., using a transformation). In some embodiments, the shared virtual object 504a may be positioned relative to persistent coordinate data (e.g., canonical persistent coordinate data). In some embodiments, a user (e.g., user 508c) may manipulate the shared virtual object 504a. For example, user 508c may move the object 504a from the edge of a table to the center of the table. In some embodiments, users 508a and 508b may also see the object 504a move from the edge of a table to the center of the table. In some embodiments, if a user (eg, user 508b) points to a portion of object 504a (eg, a helmet), other users (eg, 508a and 508c) may also see user 508b pointing to the same portion of object 504a.

[0073] In some embodiments, the session handle 502a may also be moved. Figure 5C , user 508a can move session handle 502a to the left. In some embodiments, any virtual content displayed as part of the session can also move so as to maintain the same relative positioning with session handle 502a. For example, as session handle 502a moves to the left, object 504a can also move to the left by the same amount. In some embodiments, moving a session handle at one location (e.g., session handle 502a) may not move a session handle at a different location (e.g., session handle 502b). It may be beneficial to allow each group of local users to manage their own session handle placement. For example, because virtual content can be positioned relative to the session handle, each local group can determine the optimal positioning of their virtual content for their respective local physical environment.

[0074] Session 500 may involve users who may not share the same location. Figure 5B , users 508d and 508e may also be part of session 500. In some embodiments, users 508d and 508e may be considered remote from users 508a, 508b, and 508c (e.g., because there may not be common persistent coordinate data between users 508d / 508e and 508a / 508b / 508c). In some embodiments, users 508d and 508e may see a second session handle 502b. In some embodiments, each user (or user group) that does not have common persistent coordinate data with another user (or user group) may see their own session handle. Shared virtual content displayed to users 508d and 508e may be displayed relative to session handle 502b. For example, shared virtual object 504b may correspond to object 504a. In some embodiments, when object 504a is positioned relative to session handle 502a, object 504b may be positioned at the same location relative to session handle 502b. In some embodiments, if object 504a is moved relative to session handle 502a, object 504b may also be moved relative to session handle 502b (and vice versa). In some embodiments, if session handle 502a is moved, session handle 502b may not be moved. This may enable a local user to manage how session content is presented to a group of local users.

[0075] In some embodiments, session 500 can include user avatar 506e. In some embodiments, user avatar 506e can represent a user in session 500 who may be remote from other users in the session. For example, users 508a, 508b, and 508c can be considered local to each other (e.g., because they may share persistent coordinate data), and user 508e can be considered remote from users 508a, 508b, and 508c (e.g., because user 508e may not share persistent coordinate data with other users). In some embodiments, user 508e (in Figure 5B ) can also be part of session 500, and user avatar 506e can correspond to user 508e.

[0076] In some embodiments, user avatar 506e can enable user 508e to collaborate with users 508a, 508b, and 508c. In some embodiments, avatar 506e can reflect one or more movements of user 508e. For example, when user 508e approaches session handle 502b, user avatar 506e can approach session handle 502a, thereby maintaining the same relative positioning between user 508e and session handle 502b. In some embodiments, user 508e can point to object 504b, and avatar 506e can correspondingly point to object 504a at the same location. Similarly, avatar 506b can represent user 508b, while avatar 506a can represent user 508a. As user 508a approaches object 504a, avatar 506a can also correspondingly approach object 504b. In some embodiments, the remote user may not broadcast their avatar to other users. For example, user 508d may be remote from users 508a, 508b, and 508c, but user 508d may not project a corresponding avatar for session 502a.

[0077] In some embodiments, session persistence can allow a user to dynamically locate different session locations. For example, users 508a, 508b, and 508c can be in a first room, while users 508d and 508e can be in a second room, which can be down the hall from the first room. In some embodiments, user 508a can leave the first room, walk down the hall, and enter the second room, and virtual content can be displayed to user 508a relative to session handle 502b. In some embodiments, each MR system used by a user can periodically poll the user's location (e.g., using GPS data and / or image recognition). In some embodiments, the MR system can trigger a new location query (e.g., by using geofencing).

[0078] Figure 6An exemplary session manager architecture according to some embodiments is shown. In some embodiments, a session manager 604 can run on an MR system 602, which can include one or more computer systems and can correspond to MR systems 112, 200. In some embodiments, the session manager 604 can include processes, subprocesses, threads, and / or services. In some embodiments, the session manager 604 can include one or more data structures configured to store information. In some embodiments, the session manager 604 can include services (e.g., background operating system services). In some embodiments, the processes, subprocesses, threads, and / or services of the session manager 604 can be configured to run continuously (e.g., in the background) while the operating system of the host system is running. In some embodiments, the session manager 604 can include an instantiation of a parent background service, which can serve as a host process for one or more background processes and / or subprocesses. In some embodiments, the session manager 604 can include a subprocess of a parent process. In some embodiments, the session manager 604 can include a thread of a parent process.

[0079] The session manager 604 may include one or more session instances 606a and / or 606b. In some embodiments, a session instance may correspond to an MR collaboration session (e.g., session 500). In some embodiments, a session instance may manage information used in the MR collaboration session. In some embodiments, a session instance may include one or more data structures configured to store information. In some embodiments, a session instance may include one or more processes, subprocesses, threads, and / or services. In some embodiments, one or more session instances may be stored at one or more remote servers. In some embodiments, the stored session instance may be encrypted (locally at the MR device or at one or more remote servers) before being stored.

[0080] In some embodiments, a session instance can be configured to communicate with one or more capability instances. For example, session instance 606b can be configured to communicate with capability instances 608b and 608c. A capability instance can correspond to one or more session capabilities. For example, capability instance 608b can correspond to shared object 504a. In some embodiments, a capability instance can include one or more data structures configured to store information. In some embodiments, a capability instance can include one or more processes, subprocesses, threads, and / or services.

[0081] In some embodiments, a capability instance may be configured to communicate with one or more connection services, such as an application connection platform 610a and / or collaboration core 610b. In some embodiments, the application connection platform 610a and / or collaboration core 610b may include processes, sub-processes, threads, and / or services. In some embodiments, the application connection platform 610a and / or collaboration core 610b may include one or more data structures configured to store information. In some embodiments, the application connection platform 610a and / or collaboration core 610b may include services (e.g., background operating system services). In some embodiments, the processes, sub-processes, threads, and / or services of the application connection platform 610a and / or collaboration core 610b may be configured to run continuously (e.g., in the background) while the operating system of the host system is running. In some embodiments, the application connection platform 610a and / or collaboration core 610b may include an instantiation of a parent background service, which may serve as a host process for one or more background processes and / or sub-processes. In some embodiments, the application connection platform 610a and / or collaboration core 610b may include a sub-process of a parent process. In some embodiments, the application connection platform 610a and / or the collaboration core 610b may include threads of a parent process.

[0082] In some embodiments, the application connection platform 610a can provide a low-latency communication path between MR systems in a colocation session to achieve real-time virtual object colocation. In some embodiments, the application connection platform 610a may include one or more implementations of WebRTC. For example, in some embodiments, data can be sent via one or more Twilio tracks for low-latency communication. In some embodiments, the capability instance can utilize the application connection platform 610a to send and / or receive low-latency data (e.g., relational transformation data when a shared virtual object moves) from the MR system in a session. In some embodiments, the application connection platform 610a can be configured to communicate with other application connection platforms running on other MR systems.

[0083] In some embodiments, the collaboration core 610b can provide a data synchronization service for simultaneous editing. In some embodiments, the collaboration core 610b can be configured to receive edit data from one or more capability instances. In some embodiments, the collaboration core 610b can be configured to communicate with an external synchronization service (e.g., Firebase) to synchronize simultaneous edits to virtual content in a session.

[0084] In some embodiments, the application connection platform 610a and / or the collaboration core 610b can communicate with the session manager 604. In some embodiments, the session manager 604 can provide privileged information directly to the application connection platform 610a and / or the collaboration core 610b (e.g., user identification data). Because capability instances may be developed by unknown developers, which may pose a security risk to privileged data, it may be beneficial to mask privileged information from capability instances.

[0085] Although the application connection platform 610a and the collaboration core 610b are depicted as separate services, it is contemplated that the functionality provided by each may be provided as a single service or as two or more services.

[0086] In some embodiments, the session manager 604 can communicate with one or more remote servers and / or one or more MR systems to synchronize session instances. For example, a second MR system can initiate a session and invite MR system 602 to join the session. In some embodiments, the session manager 604 can create a new session instance corresponding to the newly joined session. In some embodiments, the new session instance can be a copy of the session instance on the second MR system. In some embodiments, the session instance can be received from one or more remote servers. In some embodiments, session instance data can be sent to one or more remote servers (e.g., if a capability instance has been updated, it may be desirable to send the update to other session users). In some embodiments, the session instance data can be sent to one or more remote servers at the end of a session (e.g., when the last user leaves the session) so that the session data can be saved and reaccessed later. In some embodiments, the session manager and / or session instances can communicate with one or more services (e.g., one or more services provided by the application connection platform 610a) to synchronize session instance data with other session instances (which may be stored on another MR system or remote server). In some embodiments, the session manager and / or session instances can communicate with one or more services to establish real-time and / or low-latency communication links with one or more remote endpoints (e.g., other MR systems in the session).

[0087] Figure 7An exemplary session instance architecture according to some embodiments is shown. In some embodiments, session instance 702 may correspond to session instance 604a and / or 604b. In some embodiments, session instance 702 may include one or more data structures that may be configured to store one or more additional data structures (e.g., capability module 704, participant module 708, location module 712, and / or presence module 716). Capability module 704 may manage data and / or data structures corresponding to one or more capability instances in a session. For example, instance 706a may correspond to a virtual object. In some embodiments, instance 706a may include transformation data that may relate the location of the virtual object to persistent coordinate data and / or one or more session handle locations. In some embodiments, instance 706a may include one or more references to collaborative core services. In some embodiments, if a user makes changes to instance 706a, the reference to the collaborative core service may enable instance 706a to be appropriately notified and / or updated. In some embodiments, instance 706a may include application connection platform data (e.g., where data should be sent, what pipeline should be used, etc.). In some embodiments, capability module 704 may be configured to communicate with one or more capability instances (eg, capability instance 608a).

[0088] In some embodiments, the session instance 702 may include a participant module 708. The participant module 708 may manage data and / or data structures corresponding to one or more users in the session. For example, user 710a may include an identifier of the MR system used by the user. In some embodiments, user 710a may include avatar data (e.g., appearance, size, color, etc.). In some embodiments, user 710a may include location data. In some embodiments, the location data may include GPS data, WiFi data, cellular data, persistent coordinate data, etc.

[0089] In some embodiments, session instance 702 may include a location module 712. Location module 712 may manage data and / or data structures corresponding to one or more locations in a session. For example, location 714a may include persistent coordinate data, transformation data, data corresponding to a ground plane, and the like. In some embodiments, location 714a may correspond to a user location. In some embodiments, location 714a may correspond to a session handle location.

[0090] In some embodiments, the session instance 702 may include a presence module 716. The presence module 716 may manage data and / or data structures corresponding to the local and / or remote status of one or more users. For example, an instance 718a may indicate that a first user is remote from a second user, while a third user is local to the second user. In some embodiments, the instance 718a may include data used for communication between users (e.g., using the application connection platform 610a).

[0091] 3D object annotation

[0092] MR collaboration is particularly useful for creating 3D virtual content. Leveraging virtual object persistence, MR systems can enable users to view virtual content as if it were real. For example, a virtual object can be displayed as if placed on a real table. In some embodiments, users can walk around the table and observe the virtual object from different angles, as if it were actually placed on the table. This ability to naturally view and / or interact with virtual content can offer advantages over other methods. For example, viewing a 3D model on a 2D screen can require multiple workarounds. Users may have to use a computer mouse to drag the 3D model to display different perspectives. However, due to the nature of displaying 3D content on a 2D screen, this experience can be frustrating, as the 3D content may change in unexpected ways. In some embodiments, MR systems can also enable multiple users to collaborate on 3D content. For example, two users working on the same 3D content can use the MR system to view the 3D content projected in 3D space. In some embodiments, the 3D content can be synchronized and / or positioned identically for both users of the MR system. Users can then collaborate by referencing various aspects of the 3D content, moving around to view different angles, and so on. In some embodiments, annotations on the virtual content can be made available to collaborating users in real time. For example, a first user can add virtual tags and / or comments to virtual content, and a second user can see the virtual tags and / or comments as the first user creates them. Therefore, it may be beneficial to develop systems and methods that enable real-time collaboration on 3D objects.

[0093] Figure 8An exemplary mixed reality collaboration session according to some embodiments is shown. In some embodiments, users 802 and 804 can use one or more MR systems (e.g., MR system 806, which can correspond to MR systems 112, 200) to collaborate on 3D virtual content. In some embodiments, users 802 and 804 can utilize a session to display virtual content and / or collaborate on virtual content. For example, a virtual model 808 can be presented to users 802 and 804. In some embodiments, the virtual model 808 can be presented to users 802 and 804 at the same location (e.g., position and / or orientation). In some embodiments, the virtual model 808 can be presented to users 802 and 804 using the same session handle. In some embodiments, the virtual model 808 can be managed by a capability instance. In some embodiments, the characteristics of the virtual model 808 can be stored in the capability instance and / or managed by the capability instance. In some embodiments, the capability instance can be stored in the session instance and / or managed by the session instance.

[0094] In some embodiments, user 802 can annotate virtual model 808 (e.g., by creating a virtual tag and / or adding a virtual comment). For example, user 802 can create a virtual tag 812 that can indicate where a cafe can be placed in virtual model 808. In some embodiments, user 804 can see virtual tag 812. In some embodiments, user 804 can see virtual tag 812 at the same time user 802 created it. In some embodiments, user 804 can see virtual tag 812 at the same location as user 802 saw virtual tag 812. In some embodiments, user 802 can create one or more virtual comments. In some embodiments, a virtual comment can include a location indicator 814 and / or a comment bubble 816. In some embodiments, comment bubble 816 can include a visual indicator corresponding to location indicator 814 (e.g., comment bubble 816 and location indicator 814 can share a number). In some embodiments, location indicator 814 can be presented to user 804 in the same location as it was presented to user 802 (e.g., relative to the real world and / or relative to other virtual content). In some embodiments, comment bubbles 816 can be presented in different locations for different users. For example, comment bubble 816 can be presented to user 802 facing user 802, and comment bubble 816 can be presented to user 804 facing user 804. In some embodiments, comment bubble 816 can be configured to continue facing the user as the user looks at different locations. In some embodiments, comment bubble 816 can be presented in the same location for multiple users of a session (e.g., all local users).

[0095] In some embodiments, data corresponding to the virtual annotation can be sent from a capability instance (e.g., capability instance 608c) to a session instance (e.g., session instance 606b). In some embodiments, data corresponding to the virtual annotation can be sent from a capability instance to a collaboration core 610b. In some embodiments, the collaboration core 610b can send the data corresponding to the virtual annotation to one or more remote servers (e.g., one or more remote servers configured to handle data synchronization and / or synchronization conflicts). In some embodiments, the one or more remote servers can send the data corresponding to the virtual annotation to other session users. In some embodiments, the data corresponding to the virtual annotation can be stored in the session instance. In some embodiments, the session instance can be closed and reopened, and one or more capability instances (e.g., virtual model 808 and / or virtual tag 812) can be loaded and / or displayed to the user.

[0096] In some embodiments, user 802 may be remote from user 804. For example, user 802 may be in a first room, and user 804 may be in a second room that is different from the first room. In some embodiments, users 802 and 804 may collaborate on virtual model 808 using a conversation instance. In some embodiments, user 802 may see virtual model 808 in the first room, and user 804 may see virtual model 808 in the second room. In some embodiments, virtual model 808 may be presented relative to a first conversation handle for user 802, and virtual model 808 may be presented relative to a second conversation handle for user 804. In some embodiments, virtual annotations made by one user (e.g., user 802) may be visible to all conversation users (e.g., user 804).

[0097] Figure 9An exemplary annotation menu according to some embodiments is shown. In some embodiments, user 902 can use MR system 904 to view virtual object 906. In some embodiments, MR system 904 can display annotation menu 908, which allows user 902 to annotate virtual object 906. In some embodiments, MR system 904 can dynamically display annotation menu 908 to face user 902, regardless of which direction user 902 is facing. For example, as user 902 moves around object 906, annotation menu 908 can rotate so that the entire menu is visible to user 902. Because selecting options on annotation menu 908 can be difficult for user 902 if annotation menu 908 is displayed at an angle to user 902 (e.g., because the effective visible area of ​​buttons on annotation menu 908 may be too small for user 902 to identify and / or select), it may be desirable to dynamically orient annotation menu 908. In some embodiments, the desired orientation of menu 908 can be determined by determining the position (e.g., location and / or orientation) of user 902's head. In some embodiments, the orientation of the menu 908 may be determined such that the normal vector of the menu 908 points toward the user 902. In some embodiments, the MR system 904 may display the annotation menu 908 proximate to a corresponding virtual object (eg, virtual object 906).

[0098] In some embodiments, the MR system 904 can display the annotation menu 908 so that the virtual object 906 does not obstruct the annotation menu 908. For example, if the annotation menu 908 is displayed in front of the object 906 and the user 902 moves to the opposite side of the object 906, the object 906 may completely or partially obstruct the annotation menu 908. It may be desirable to dynamically reposition the annotation menu 908 so that the corresponding virtual object (e.g., a virtual object annotated with the menu 908) does not obstruct the annotation menu 908 (e.g., because if the menu 908 is obstructed by the corresponding virtual object, the user 902 may not be able to interact with the menu 908). In some embodiments, occlusion can be determined by determining whether the virtual object 906 (and / or a prism associated with the virtual object 906) intersects a direct path between the user 902 and the annotation menu 908.

[0099] The annotation menu 908 may include several features for annotating virtual objects. For example, the annotation menu 908 may include a virtual drawing button. In some embodiments, the virtual drawing button may toggle drawing mode, which allows users to create virtual markers on and / or around virtual objects. In some embodiments, virtual markers created in drawing mode may be visible to other session users. In some embodiments, the annotation menu 908 may include a virtual delete button. In some embodiments, the virtual delete button may delete the selected virtual object and / or virtual marker. In some embodiments, the annotation menu 908 may include a virtual comment button. In some embodiments, the virtual comment button may allow users to place a location indicator for a virtual comment and / or add a virtual comment. In some embodiments, the annotation menu 908 may include a virtual color selector button. In some embodiments, selecting the virtual color selector button may allow users to select a color for the virtual marker. In some embodiments, the annotation menu 908 may include a virtual size toggle button. In some embodiments, selecting the virtual size toggle button may toggle whether the virtual object is displayed at its true size. For example, a 3D model may include parameters such as dimensions. In some embodiments, the MR system may display the 3D model at a modified size to make it easier to view. For example, a 3D model of a building may initially be presented to the user much smaller than its actual size so that the user can easily see the entire 3D model. In some embodiments, the annotation menu 908 may include a virtual annotation visibility button. In some embodiments, selecting the virtual annotation visibility button may toggle the visibility of annotations (e.g., virtual tags and / or virtual comments) corresponding to the virtual object (e.g., the display may display or not display them). In some embodiments, the annotation menu 908 may include a virtual clear button. In some embodiments, selecting the virtual clear button may remove all virtual annotations corresponding to the virtual object.

[0100] In some embodiments, user 902 can move virtual object 906. For example, user 902 can use handheld controller 910 (which can correspond to handheld controller 300) and virtual selection beam 912 to select virtual object 906 and drag virtual object 906 to a new location. In some embodiments, annotation menu 908 can move with virtual object 906 and maintain its relative position to virtual object 906.

[0101] Figure 10An example of mixed reality annotation in accordance with some embodiments is shown. In some embodiments, virtual object 1002 can be a prism and can include one or more virtual objects (e.g., virtual object 1004) therein. In some embodiments, the prism can include a bounding volume and / or parameters of the included virtual objects. In some embodiments, virtual object 1012 can include a prism that can include an annotation menu (which can correspond to annotation menu 908). In some embodiments, virtual object 1012 may not intersect prism 1002 (e.g., because object 1004 may block the view of annotation menu 1012, which may prevent the user from interacting with annotation menu 1012).

[0102] In some embodiments, the user can select a position indicator 1008, which can be displayed on and / or near virtual objects 1002 and / or 1004. In some embodiments, selecting position indicator 1008 can cause virtual object 1010 to be displayed to the user. Virtual object 1010 can include a prism that can include a virtual comment bubble. In some embodiments, virtual object 1010 can be displayed near position indicator 1008. In some embodiments, virtual object 1010 may not intersect virtual object 1002 (e.g., because it may obscure the view of the virtual comment bubble). In some embodiments, virtual objects 1012 and / or 1010 can continue to face the user as the user moves through the environment. In some embodiments, if the view of virtual objects 1012 and / or 1010 becomes obscured (e.g., if the user moves such that virtual object 1002 blocks the user's line of sight to virtual object 1010), virtual objects 1012 and / or 1010 can reposition themselves.

[0103] Figure 11An example of mixed reality annotation in accordance with some embodiments is shown. In some embodiments, virtual object 1108 (which may include a prism) may be very large (e.g., because the included virtual object 1102 is very large). In some embodiments, virtual object 1104 may be displayed within another virtual object (e.g., virtual object 1108). For example, virtual object 1108 may include the bounding prism of virtual object 1102, but due to the size of virtual object 1102, there may be significant space where virtual object 1102 does not block the view of other virtual objects. In some embodiments, virtual object 1104 may include a virtual comment bubble in which a user can enter text. In some embodiments, a virtual keyboard 1106 may be displayed when a user edits a virtual comment bubble. In some embodiments, virtual keyboard 1106 may be displayed proximate to the corresponding virtual comment bubble. For example, a user may edit comment bubble 1104, and virtual keyboard 1106 may be displayed proximate to bubble 1104 (instead of, for example, comment bubble 1106). It may be helpful to visually indicate which virtual comment bubble a user is editing (e.g., by displaying a keyboard proximate to the corresponding comment bubble).

[0104] Example systems, methods, and computer-readable media are disclosed. According to some examples, a system includes: a wearable device including a transmissive display; one or more processors configured to execute a method, the method comprising: presenting a virtual object to a first user at a first location via the transmissive display of the wearable device; receiving a first input from the first user; in response to receiving the first input, presenting a virtual annotation at a first displacement from the first location via the transmissive display; sending first data to a second user, the first data associated with the virtual annotation and the first displacement; receiving a second input from the second user; in response to receiving the second input, presenting the virtual annotation to the first user via the transmissive display at a second displacement from the first location; and sending second data to a remote server, the second data associated with the virtual object, the virtual annotation, the second displacement, and the first location. In some examples, the virtual object is presented based on a first application configured to run on the wearable device, and the virtual annotation is presented based on data from a plugin library, the plugin library being configured to be accessed by multiple applications configured to run on the wearable device. In some examples, the second data is sent at a first time, and the method further includes: exiting the session instance, wherein the session instance is configured to store the second data; receiving a third input from the first user; in response to receiving the third input, requesting the second data; presenting the virtual object at the first location to the first user at a second time later than the first time; and presenting the virtual annotation at a second displacement from the first location to the first user at the second time. In some examples, the annotation includes a virtual marker. In some examples, the virtual object is presented at a first size, the virtual object including target size data, and the method further includes: receiving a third input from the first user; in response to receiving the third input, presenting the virtual object at a second size to the first user, wherein the second size is associated with the target size data; and sending third data to the second user, the third data associated with the second size. In some examples, the method further includes: receiving a third input from the first user; in response to receiving the third input, presenting a virtual location indicator to the first user, wherein the virtual location indicator is associated with the virtual comment; and sending the third data to the second user, the third data associated with the virtual location indicator. In some examples, the method further includes: presenting the virtual annotation menu to the first user via the transmissive display; and repositioning the virtual annotation menu so that the virtual annotation menu is not obscured by the virtual object.

[0105] According to some examples, a method includes: presenting a virtual object to a first user at a first location via a transmissive display of a wearable device; receiving a first input from the first user; in response to receiving the first input, presenting a virtual annotation at a first displacement from the first location via the transmissive display; sending first data to a second user, the first data associated with the virtual annotation and the first displacement; receiving a second input from the second user; in response to receiving the second input, presenting the virtual annotation to the first user via the transmissive display at a second displacement from the first location; and sending second data to a remote server, the second data associated with the virtual object, the virtual annotation, the second displacement, and the first location. In some examples, the virtual object is presented based on a first application configured to run on the wearable device, and the virtual annotation is presented based on data from a plug-in library, the plug-in library being configured to be accessed by multiple applications configured to run on the wearable device. In some examples, the second data is sent at a first time, and the method further includes: exiting the session instance, wherein the session instance is configured to store the second data; receiving a third input from the first user; in response to receiving the third input, requesting the second data; presenting the virtual object at the first location to the first user at a second time later than the first time; and presenting the virtual annotation at a second displacement from the first location to the first user at the second time. In some examples, the annotation includes a virtual marker. In some examples, the virtual object is presented at a first size, the virtual object including target size data, and the method further includes: receiving a third input from the first user; in response to receiving the third input, presenting the virtual object at a second size to the first user, wherein the second size is associated with the target size data; and sending third data to the second user, the third data associated with the second size. In some examples, the method further includes: receiving a third input from the first user; in response to receiving the third input, presenting a virtual location indicator to the first user, wherein the virtual location indicator is associated with the virtual comment; and sending the third data to the second user, the third data associated with the virtual location indicator. In some examples, the method further includes: presenting the virtual annotation menu to the first user via the transmissive display; and repositioning the virtual annotation menu so that the virtual annotation menu is not obscured by the virtual object.

[0106] According to some examples, a non-transitory computer-readable medium stores instructions that, when executed by one or more processors, cause the one or more processors to perform a method, the method comprising: presenting a virtual object to a first user at a first location via a transmissive display of a wearable device; receiving a first input from the first user; in response to receiving the first input, presenting a virtual annotation via the transmissive display at a first displacement from the first location; sending first data to a second user, the first data associated with the virtual annotation and the first displacement; receiving a second input from the second user; in response to receiving the second input, presenting the virtual annotation to the first user via the transmissive display at a second displacement from the first location; and sending second data to a remote server, the second data associated with the virtual object, the virtual annotation, the second displacement, and the first location. In some examples, the virtual object is presented based on a first application configured to run on the wearable device, and the virtual annotation is presented based on data from a plugin library, the plugin library being configured to be accessed by multiple applications configured to run on the wearable device. In some examples, the second data is sent at a first time, and the method further includes: exiting the session instance, wherein the session instance is configured to store the second data; receiving a third input from the first user; in response to receiving the third input, requesting the second data; presenting the virtual object at the first location to the first user at a second time later than the first time; and presenting the virtual annotation at a second displacement from the first location to the first user at the second time. In some examples, the annotation includes a virtual marker. In some examples, the virtual object is presented at a first size, the virtual object including target size data, and the method further includes: receiving a third input from the first user; in response to receiving the third input, presenting the virtual object at a second size to the first user, wherein the second size is associated with the target size data; and sending third data to the second user, the third data associated with the second size. In some examples, the method further includes: receiving a third input from the first user; in response to receiving the third input, presenting a virtual location indicator to the first user, wherein the virtual location indicator is associated with the virtual comment; and sending the third data to the second user, the third data associated with the virtual location indicator. In some examples, the method further includes: presenting the virtual annotation menu to the first user via the transmissive display; and repositioning the virtual annotation menu so that the virtual annotation menu is not obscured by the virtual object.

[0107] Although the disclosed examples have been fully described with reference to the accompanying drawings, it should be noted that various changes and modifications will become apparent to those skilled in the art. For example, elements of one or more embodiments may be combined, deleted, modified, or supplemented to form further embodiments. It will be understood that such changes and modifications are within the scope of the disclosed examples as defined by the appended claims.

Claims

1. A system for presenting and annotating virtual content, comprising: A wearable device comprising a transmissive display; as well as One or more processors configured to perform a method comprising: presenting a virtual object to a first user at a first location via the transmissive display of the wearable device; receiving a first input from the first user; in response to receiving the first input, presenting a virtual annotation via the transmissive display at a first displacement from the virtual object; sending first data to a second user, the first data being associated with the virtual annotation and the first displacement; receiving a second input from the second user; In response to receiving the second input, presenting the virtual annotation to the first user via the transmissive display at a second displacement from the first position, the second displacement being different from the first displacement; and sending second data to a remote server, the second data being associated with the virtual object, the virtual annotation, the second displacement, and the first position; wherein receiving the first input from the first user comprises receiving the first input via a first annotation menu that is dynamically presented to the first user in a fixed orientation relative to an orientation of the first user, and Receiving the second input from the second user includes receiving the second input via a second annotation menu that is dynamically presented to the second user in a fixed orientation relative to an orientation of the second user.

2. The system according to claim 1, wherein: Rendering the virtual object comprises rendering the virtual object based on a first application configured to run on the wearable device, and wherein rendering the virtual annotation comprises rendering the virtual annotation based on data from a plug-in library configured to be accessed by multiple applications configured to run on the wearable device.

3. The system according to claim 1, wherein: Sending the second data includes sending the second data at a first time, and wherein the method further includes: exiting the session instance, wherein the session instance is configured to store the second data; receiving a third input from the first user; In response to receiving the third input, requesting the second data; presenting the virtual object at the first location to the first user at a second time that is later than the first time; and The virtual annotation is presented to the first user at the second displacement from the first location at the second time.

4. The system according to claim 1, wherein: The annotation includes a virtual marker.

5. The system according to claim 1, wherein: Rendering the virtual object includes rendering the virtual object at a first size, wherein the virtual object includes target size data, and wherein the method further comprises: receiving a third input from the first user; In response to receiving the third input, presenting the virtual object to the first user at a second size, wherein the second size is associated with the target size data; and Third data is sent to the second user, where the third data is associated with the second size.

6. The system of claim 1 , wherein the method further comprises: receiving a third input from the first user; In response to receiving the third input, presenting a virtual location indicator to the first user, wherein the virtual location indicator is associated with a virtual review; as well as Third data is sent to the second user, the third data being associated with the virtual location indicator.

7. The system according to claim 1, wherein: The method further comprises: receiving an instruction to present the first annotation menu to the first user via the transmissive display; determining whether a view of the first annotation menu is obscured by the virtual object; and In response to determining that a view of the first annotation menu is obscured by the virtual object, the first annotation menu is repositioned so that the first annotation menu is not obscured by the virtual object.

8. The system according to claim 1, wherein: Rendering the virtual annotation includes rendering the virtual annotation at a first orientation at the first displacement relative to the virtual object; as well as The second wearable device is configured to: presenting the virtual object to the second user at the first location, and The virtual annotation is presented at the first displacement relative to the first position and in a second orientation, the second orientation being different from the first orientation.

9. A method for presenting and annotating virtual content, comprising: presenting a virtual object to a first user at a first location via a transmissive display of the wearable device; receiving a first input from the first user; in response to receiving the first input, presenting a virtual annotation via the transmissive display at a first displacement from the first location; sending first data to a second user, the first data being associated with the virtual annotation and the first displacement; receiving a second input from the second user; in response to receiving the second input, presenting the virtual annotation to the first user via the transmissive display at a second displacement from the virtual object, the second displacement being different than the first displacement; as well as sending second data to a remote server, the second data being associated with the virtual object, the virtual annotation, the second displacement, and the first position; wherein receiving the first input from the first user comprises receiving the first input via a first annotation menu that is dynamically presented to the first user in a fixed orientation relative to an orientation of the first user, and Receiving the second input from the second user includes receiving the second input via a second annotation menu that is dynamically presented to the second user in a fixed orientation relative to an orientation of the second user.

10. The method according to claim 9, wherein: The virtual object is rendered based on a first application configured to run on the wearable device, and wherein the virtual annotation is rendered based on data from a plugin library configured to be accessed by a plurality of applications configured to run on the wearable device.

11. The method according to claim 9, wherein Sending the second data at a first time, the method further includes: exiting the session instance, wherein the session instance is configured to store the second data; receiving a third input from the first user; In response to receiving the third input, requesting the second data; presenting the virtual object at the first location to the first user at a second time that is later than the first time; and The virtual annotation is presented to the first user at the second displacement from the first location at the second time.

12. The method according to claim 9, wherein The annotation includes a virtual marker.

13. The method according to claim 9, wherein presenting the virtual object at a first size, wherein the virtual object includes target size data, and wherein the method further comprises: receiving a third input from the first user; In response to receiving the third input, presenting the virtual object to the first user at a second size, wherein the second size is associated with the target size data; and Third data is sent to the second user, where the third data is associated with the second size.

14. The method according to claim 9, further comprising: receiving a third input from the first user; In response to receiving the third input, presenting a virtual location indicator to the first user, wherein the virtual location indicator is associated with a virtual review; as well as Third data is sent to the second user, the third data being associated with the virtual location indicator.

15. The method according to claim 9, further comprising: receiving an instruction to present the first annotation menu to the first user via the transmissive display; determining whether a view of the first annotation menu is obscured by the virtual object; as well as In response to determining that a view of the first annotation menu is obscured by the virtual object, the first annotation menu is repositioned so that the first annotation menu is not obscured by the virtual object.

16. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method comprising: presenting a virtual object to a first user at a first location via a transmissive display of the wearable device; receiving a first input from the first user; in response to receiving the first input, presenting a virtual annotation via the transmissive display at a first displacement from the first location; sending first data to a second user, the first data being associated with the virtual annotation and the first displacement; receiving a second input from the second user; in response to receiving the second input, presenting the virtual annotation to the first user via the transmissive display at a second displacement from the virtual object, the second displacement being different than the first displacement; as well as sending second data to a remote server, the second data being associated with the virtual object, the virtual annotation, the second displacement, and the first position; wherein receiving the first input from the first user comprises receiving the first input via a first annotation menu that is dynamically presented to the first user in a fixed orientation relative to an orientation of the first user, and Receiving the second input from the second user includes receiving the second input via a second annotation menu that is dynamically presented to the second user in a fixed orientation relative to an orientation of the second user.

17. The non-transitory computer readable medium of claim 16, wherein: Rendering the virtual object comprises rendering the virtual object based on a first application configured to run on the wearable device, and wherein rendering the virtual annotation comprises rendering the virtual annotation based on data from a plug-in library configured to be accessed by multiple applications configured to run on the wearable device.

18. The non-transitory computer readable medium of claim 16, wherein: Sending the second data includes sending the second data at a first time, and the method further includes: exiting the session instance, wherein the session instance is configured to store the second data; receiving a third input from the first user; In response to receiving the third input, requesting the second data; presenting the virtual object at the first location to the first user at a second time that is later than the first time; and The virtual annotation is presented to the first user at the second displacement from the first location at the second time.

19. The non-transitory computer readable medium of claim 16, wherein: Rendering the virtual object includes rendering the virtual object at a first size, wherein the virtual object includes target size data, and wherein the method further comprises: receiving a third input from the first user; In response to receiving the third input, presenting the virtual object to the first user at a second size, wherein the second size is associated with the target size data; and Third data is sent to the second user, where the third data is associated with the second size.

20. The non-transitory computer readable medium of claim 16, the method further comprising: receiving a third input from the first user; In response to receiving the third input, presenting a virtual location indicator to the first user, wherein the virtual location indicator is associated with a virtual review; as well as Third data is sent to the second user, the third data being associated with the virtual location indicator.

Images