Spatial Command and Guidance in Mixed Reality

Abstract

To provide a system and method for presenting audio and visual signals corresponding to user-generated content within a mixed reality environment.SOLUTION: The method according to the present invention includes: receiving a first input from a first user at a first time; determining a location of the first user at the first time in a coordinate space using a sensor of a first wearable head device; and generating a persistent virtual content associated with the location of the first user at the first time. The method includes: determining a location of a second user at a second time in the coordinate space at the second time; displaying the persistent virtual content to the second user via the display; determining a location of the second user associated with the persistent virtual content; and determining a new location associated with the persistent virtual content and based on the location of the first user and the location of the second user.SELECTED DRAWING: None

Description

[Technical Field]

[0001] (Related Applications) This patent application claims priority to U.S. Provisional Patent Application No. 62 / 884,153, filed August 7, 2019, and U.S. Provisional Patent Application No. 62 / 894,448, filed August 30, 2019, which are incorporated by reference in their entireties.

[0002] (Field) The present disclosure relates generally to systems and methods for presenting audio and visual signals, and more particularly to systems and methods for presenting audio and visual signals corresponding to user-generated content within a mixed reality environment. [Background technology]

[0003] (background) Virtual environments are ubiquitous in computing environments, finding use in video games (where a virtual environment may represent a game world), maps (where a virtual environment may represent a terrain to be navigated), simulations (where a virtual environment may simulate a real environment), digital storytelling (where virtual characters may interact with one another within a virtual environment), and many other applications. Modern computer users are generally comfortable perceiving and interacting with virtual environments. However, a user's experience with a virtual environment may be limited by the technology for presenting the virtual environment. For example, traditional displays (e.g., 2D display screens) and audio systems (e.g., fixed speakers) may be unable to realize a virtual environment in a way that creates a compelling, realistic, and immersive experience.

[0004] Virtual reality (“VR”), augmented reality (“AR”), mixed reality (“MR”), and related technologies (collectively, “XR”) share the ability to present to a user of an XR system sensory information corresponding to a virtual environment represented by data in a computer system. This disclosure considers uniqueness among VR, AR, and MR systems (although some systems may be categorized as VR in one aspect (e.g., visual aspect) and simultaneously categorized as AR or MR in another aspect (e.g., audio aspect)). As used herein, a VR system presents a virtual environment that replaces the user's real environment in at least one aspect. For example, a VR system may present a user with a view of the virtual environment while simultaneously obscuring that view of the real environment, such as with an optically blocking head-mounted display. Similarly, a VR system may present a user with audio corresponding to the virtual environment while simultaneously blocking (attenuating) the audio from the real environment.

[0005] VR systems may suffer from various drawbacks resulting from replacing a user's real environment with a virtual environment. One drawback is motion sickness, which can occur when a user's field of view within the virtual environment no longer corresponds to the state of their inner ear, which detects their balance and orientation in the real (but not the virtual) environment. Similarly, a user may experience disorientation within a VR environment if their body and limbs (the view upon which the user relies to feel "grounded" in the real environment) are not directly visible. Another drawback is the computational burden (e.g., memory, processing power) imposed on a VR system that must present a fully 3D virtual environment, especially in real-time applications that seek to immerse a user in the virtual environment. Similarly, such an environment may need to reach a very high level of realism to be considered immersive, as users tend to be sensitive to even slight imperfections in the virtual environment, any of which can destroy the user's sense of immersion in the virtual environment. Furthermore, another disadvantage of VR systems is that such applications of the systems cannot take advantage of the wide range of sensory data in the real environment, such as the various sights and sounds experienced in the real world. A related disadvantage is that VR systems may struggle to create shared environments in which multiple users can interact, because users who share physical space in the real environment may not be able to see or interact with each other directly in the virtual environment.

[0006] As used herein, an AR system presents a virtual environment that overlaps or overlays the real environment in at least one aspect. For example, an AR system may present a user with a view of the virtual environment overlaid on the user's view of the real environment, such as using a see-through head-mounted display that presents a displayed image while allowing light to pass through the display into the user's eyes. Similarly, an AR system may present a user with audio corresponding to the virtual environment while simultaneously mixing in audio from the real environment. Similarly, as used herein, an MR system, like an AR system, may present a virtual environment that overlaps or overlays the real environment in at least one aspect, and may additionally allow the virtual environment in the MR system to interact with the real environment in at least one aspect. For example, a virtual character in the virtual environment may flip a light switch in the real environment, causing a corresponding light bulb in the real environment to turn on or off. As another example, the virtual character may react to audio signals in the real environment (such as with facial expressions). By maintaining the presentation of the real environment, AR and MR systems may avoid some of the aforementioned disadvantages of VR systems. For example, motion sickness in a user is reduced because visual cues from the real environment (including the user's own body) can remain visible and such systems do not need to present the user with a fully realized 3D environment to be immersive. Furthermore, AR and MR systems can create new applications that utilize real-world sensory input (e.g., views and sounds of scenery, objects, and other users) to augment that input.

[0007] XR systems may provide users with various ways to interact with the virtual environment. For example, XR systems may include various sensors (e.g., cameras, microphones, etc.) to detect a user's position and orientation, facial expressions, speech, and other characteristics and present this information as input to the virtual environment. Interaction with an XR system can increase user engagement compared to user engagement in other virtual environments (e.g., through a 2D screen). Social features can further increase interaction, such as when XR system users are able to interact with each other or with content. Increased engagement can promote better learning or more enjoyable entertainment as users experience a closer visual and physical connection with the virtual content.

[0008] XR systems can provide a unique high level of immersion and realism by combining virtual visual and audio cues with real sights and sounds. Thus, in some XR systems, it is desirable to present a virtual environment that enhances, improves, or modifies a corresponding real environment. This disclosure relates to XR systems that incorporate interactive virtual environments for training, education, or entertainment purposes. Summary of the Invention [Means for solving the problem]

[0009] (Brief summary)

[0006] An embodiment of the present disclosure describes a system and method for creating spatial instructions and guides within a mixed reality environment. In one exemplary method, a first input is received from a first user at a first time. The first user's position at the first time in a coordinate space is determined using a sensor of a first wearable head device. Persistent virtual content corresponding to the first input is generated. The persistent virtual content is associated with the first user's position at the first time. A location of the first user is determined. The first user's location is associated with the persistent virtual content. At a second time, a second user's position at the second time in the coordinate space is determined. The persistent virtual content is displayed to a second user via a display. Presenting the persistent virtual content to the second user includes presenting the persistent virtual content at a location in the coordinate space corresponding to the first user's position at the first time. The second user's location is determined. The second user's location is associated with the persistent virtual content. A new location is determined, the new location being based on the location of the first user and the location of the second user. The new location is associated with persistent virtual content. The present invention provides, for example, the following. (Item 1) 1. A method comprising: receiving a first input from a first user at a first time; determining a location associated with the first user in coordinate space at the first time using a sensor of a first wearable head device; generating persistent virtual content corresponding to the first input, the persistent virtual content being associated with a location associated with the first user at the first time; At a second time, determining a location of a second user at the second time within the coordinate space; determining whether to present the persistent virtual content to the second user based on a location associated with the first user at the first time and a location of the second user at the second time; presenting the persistent virtual content to the second user via a display in accordance with a decision to present the persistent virtual content to the second user; not presenting the persistent virtual content to the second user in accordance with a decision not to present the persistent virtual content to the second user. Including, presenting the persistent virtual content to the second user includes presenting the persistent virtual content at a location in the coordinate space that corresponds to a location associated with the first user at the first time. (Item 2) Item 10. The method of item 1, wherein the first input includes at least one of a head movement of the first user, a hand movement of the first user, and a handheld controller input. (Item 3) Item 10. The method of item 1, wherein the sensor includes at least one of a position sensor and a camera. (Item 4) Item 10. The method of item 1, wherein the virtual content comprises virtual markings. (Item 5) Item 10. The method of claim 1, wherein determining whether to present the persistent virtual content to the second user includes using a sensor of a second wearable head device to determine whether the second user's field of view includes a location associated with the first user at the first time. (Item 6) 2. The method of claim 1, further comprising receiving a filter selection from the second user, wherein determining whether to present the persistent virtual content to the second user comprises determining whether the persistent virtual content is associated with the filter selection. (Item 7) determining a new location, the new location being based on a location associated with the first user and a location of the second user; Associating the persistent virtual content with the new location; Item 1, the method of claim 1 further comprising: (Item 8) At a third time, determining a location of a third user at the third time; presenting the persistent virtual content to the third user pursuant to determining that the third user's location is at the new location; and 8. The method of claim 7, further comprising: (Item 9) 8. The method of claim 7, wherein the new location is further based on a location accuracy associated with the first user and a location accuracy of the second user. (Item 10) In response to a decision not to present the persistent virtual content to the second user, the method further comprises: determining whether the second user's field of view includes a location associated with the first user at the first time; presenting second content associated with the persistent virtual content in accordance with determining that the second user's field of view includes a location associated with the first user at the first time; and Item 1. The method according to item 1, comprising: (Item 11) While the persistent virtual content is presented to the second user, At a third time, determining a location of a third user at the third time within the coordinate space; pursuant to determining that the third user location is at the second location; receiving input from the third user; presenting the persistent virtual content to the third user from a start of the persistent virtual content in response to receiving a first input; presenting the persistent virtual content to the third user in parallel with the second user in response to receiving a second input; and Item 1, the method of claim 1 further comprising: (Item 12) Item 10. The method of item 1, wherein the location associated with the first user is the first user's physical location within the coordinate space. (Item 13) Item 10. The method of item 1, wherein the location associated with the first user is a virtual location of the first user in the coordinate space, the virtual location being different from the physical location of the first wearable head device. (Item 14) 1. A system comprising: A sensor, One or more processors and wherein the one or more processors are configured to perform a method, the method comprising: receiving a first input from a first user at a first time; determining a location associated with the first user at the first time in coordinate space using the sensor; generating persistent virtual content corresponding to the first input, the persistent virtual content being associated with a location associated with the first user at the first time; At a second time, determining a location of a second user at the second time within the coordinate space; determining whether to present the persistent virtual content to the second user based on a location associated with the first user at the first time and a location of the second user at the second time; presenting the persistent virtual content to the second user via a display in accordance with a decision to present the persistent virtual content to the second user; not presenting the persistent virtual content to the second user in accordance with a decision not to present the persistent virtual content to the second user. Including, Presenting the persistent virtual content to the second user includes presenting the persistent virtual content at a location within the coordinate space that corresponds to a location associated with the first user at the first time. (Item 15) The method further comprises: determining a new location, the new location being based on a location associated with the first user and a location of the second user; Associating the persistent virtual content with the new location; Item 15. The system according to item 14, comprising: (Item 16) Item 15. The system of item 14, wherein the location associated with the first user is the first user's physical location within the coordinate space. (Item 17) Item 15. The system of item 14, wherein the location associated with the first user is a virtual location of the first user in the coordinate space, the virtual location being different from the physical location of the first wearable head device. (Item 18) A computer-readable medium having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to perform a method, the method comprising: receiving a first input from a first user at a first time; determining a location associated with the first user in coordinate space at the first time using a sensor of a first wearable head device; generating persistent virtual content corresponding to the first input, the persistent virtual content being associated with a location associated with the first user at the first time; At a second time, determining a location of a second user at the second time within the coordinate space; determining whether to present the persistent virtual content to the second user based on a location associated with the first user at the first time and a location of the second user at the second time; presenting the persistent virtual content to the second user via a display in accordance with a decision to present the persistent virtual content to the second user; not presenting the persistent virtual content to the second user in accordance with a decision not to present the persistent virtual content to the second user. Including, presenting the persistent virtual content to the second user includes presenting the persistent virtual content at a location in the coordinate space corresponding to a location associated with the first user at the first time. (Item 19) The method further comprises: determining a new location, the new location being based on a location associated with the first user and a location of the second user; Associating the persistent virtual content with the new location; Item 19. The computer-readable medium of item 18, comprising: (Item 20) Item 19. The computer-readable medium of item 18, wherein the location associated with the first user is the first user's physical location within the coordinate space. [Brief explanation of the drawings]

[0010] [Figure 1A] 1A-1C illustrate an example mixed reality environment. [Figure 1B] 1A-1C illustrate an example mixed reality environment. [Figure 1C] 1A-1C illustrate an example mixed reality environment.

[0011] [Figure 2A] 2A-2D illustrate components of an example mixed reality system that can be used to generate and interact with a mixed reality environment. [Figure 2B] 2A-2D illustrate components of an example mixed reality system that can be used to generate and interact with a mixed reality environment. [Figure 2C] 2A-2D illustrate components of an example mixed reality system that can be used to generate and interact with a mixed reality environment. [Figure 2D] 2A-2D illustrate components of an example mixed reality system that can be used to generate and interact with a mixed reality environment.

[0012] [Figure 3A] FIG. 3A illustrates an example mixed reality handheld controller that can be used to provide input to a mixed reality environment.

[0013] [Figure 3B] FIG. 3B illustrates an example auxiliary unit that may be used in conjunction with an example mixed reality system.

[0014] [Figure 4] FIG. 4 illustrates an example functional block diagram for an example mixed reality system.

[0015] [Figure 5] FIG. 5 illustrates an example flowchart of a process for creating and displaying spatial content within a mixed reality system.

[0016] [Figure 6] FIG. 6 illustrates an example of a user interacting with prompts to create or display spatial content within a mixed reality system.

[0017] [Figure 7]FIG. 7 illustrates an example of filtering spatial content in a mixed reality system.

[0018] [Figure 8] FIG. 8 illustrates an example of a user creating spatial content within a mixed reality system.

[0019] [Figure 9] FIG. 9 illustrates an example of creating spatial content in a mixed reality system.

[0020] [Figure 10] FIG. 10 illustrates an example of creating spatial content in a mixed reality system.

[0021] [Figure 11] FIG. 11 illustrates an example of displaying information about spatial content within a mixed reality system.

[0022] [Figure 12] FIG. 12 illustrates an example of a user selecting spatial content within a mixed reality system.

[0023] [Figure 13] FIG. 13 illustrates an example of a user viewing spatial content in a mixed reality system.

[0024] [Figure 14] FIG. 14 illustrates an example of sharing spatial content in a mixed reality system.

[0025] [Figure 15] FIG. 15 illustrates an example of sharing spatial content in a mixed reality system.

[0026] [Figure 16] FIG. 16 illustrates an example of updating a location associated with spatial content in a mixed reality system. DETAILED DESCRIPTION OF THE INVENTION

[0027] (Detailed explanation) In the following description of the embodiments, reference is made to the accompanying drawings which form a part hereof, and in which is shown, by way of illustration, specific embodiments which may be practiced. It is to be understood that other embodiments may be used and structural changes may be made without departing from the scope of the disclosed embodiments.

[0028] Mixed Reality Environment

[0029] Like all people, users of mixed reality systems exist in a real environment, i.e., three-dimensional portions of the "real world" and all of its content are perceptible to the user. For example, users perceive the real environment using normal human senses, i.e., sight, hearing, touch, taste, and smell, and interact with the real environment by moving their body within the real environment. Locations within the real environment can be described as coordinates within a coordinate space. For example, coordinates can include latitude, longitude, and altitude relative to sea level, distance in three orthogonal dimensions from a reference point, or other suitable values. Similarly, a vector can describe a quantity, having a direction and magnitude within the coordinate space.

[0030] A computing device may maintain a representation of a virtual environment, for example, in a memory associated with the device. As used herein, a virtual environment is a computed representation of a three-dimensional space. The virtual environment may include representations of any objects, actions, signals, parameters, coordinates, vectors, or other properties associated with that space. In some examples, circuitry (e.g., a processor) of a computing device may maintain and update the state of the virtual environment. That is, the processor may determine the state of the virtual environment at a second time t1 based on data associated with the virtual environment and / or input provided by a user at a first time t0. For example, if an object in the virtual environment is located at a first coordinate and has certain programmed physical parameters (e.g., mass, coefficient of friction) at time t0, and input received from the user indicates that a force should be applied to the object in a certain directional vector, the processor may apply the laws of kinematics and use basic mechanics to determine the location of the object at time t1. The processor may determine the state of the virtual environment at time t1 using any suitable information known about the virtual environment and / or any suitable input. In maintaining and updating the state of the virtual environment, the processor may execute any suitable software, including software related to creating and deleting virtual objects within the virtual environment, software (e.g., scripts) for defining the behavior of virtual objects or characters within the virtual environment, software for defining the behavior of signals (e.g., audio signals) within the virtual environment, software for creating and updating parameters associated with the virtual environment, software for generating audio signals within the virtual environment, software for handling input and output, software for implementing network operations, software for applying asset data (e.g., animation data for moving a virtual object over time), or many other possibilities.

[0031] An output device, such as a display or speakers, can present any or all aspects of the virtual environment to the user. For example, the virtual environment may include virtual objects (which may include representations of inanimate objects, people, animals, lights, etc.) that can be presented to the user. A processor can determine a view of the virtual environment (e.g., corresponding to a "camera," with its origin coordinates, viewing axis, and frustum) and render on the display a viewable scene of the virtual environment corresponding to that view. Any suitable rendering technique may be used for this purpose. In some examples, the viewable scene may include only some virtual objects in the virtual environment and exclude certain other virtual objects. Similarly, the virtual environment may include audio aspects that can be presented to the user as one or more audio signals. For example, a virtual object in the virtual environment may generate a sound originating from the object's location coordinates (e.g., a virtual character may speak or produce a sound effect), or the virtual environment may be associated with a musical cue or ambient sound that may or may not be associated with a particular location. The processor can determine audio signals corresponding to the "listener" coordinates, e.g., audio signals corresponding to the synthesis of sounds in the virtual environment and mixed and processed to simulate the audio signals that would be heard by a listener at the listener coordinates, and present the audio signals to the user via one or more speakers.

[0032] Because the virtual environment exists only as a computational construct, the user cannot directly perceive the virtual environment using their normal senses. Instead, the user can only indirectly perceive the virtual environment, as presented to the user, for example, by a display, speakers, tactile output device, etc. Similarly, the user cannot directly touch, manipulate, or otherwise interact with the virtual environment, but can provide input data via input devices or sensors to a processor, which can use the device or sensor data to update the virtual environment. For example, a camera sensor can provide optical data indicating that the user is attempting to move an object in the virtual environment, and the processor can use that data to cause the object to respond appropriately within the virtual environment.

[0033] A mixed reality system can present a user with a mixed reality environment (“MRE”) that combines aspects of a real environment and a virtual environment, for example, using a see-through display and / or one or more speakers (which may, for example, be incorporated into a wearable head device). In some embodiments, the one or more speakers may be external to the head-mounted wearable unit. As used herein, an MRE is a simultaneous representation of a real environment and a corresponding virtual environment. In some examples, the corresponding real and virtual environments share a single coordinate space. In some examples, the real coordinate space and the corresponding virtual coordinate space are related to each other by a transformation matrix (or other suitable representation). Thus, a single coordinate (in some examples, together with the transformation matrix) may define a first location in the real environment and a second corresponding location in the virtual environment, and vice versa.

[0034] In an MRE, a virtual object (e.g., in a virtual environment associated with the MRE) may correspond to a real object (e.g., in a real environment associated with the MRE). For example, if the real environment of the MRE includes a real lamppost (real object) at certain location coordinates, the virtual environment of the MRE may include a virtual lamppost (virtual object) at the corresponding location coordinates. As used herein, a real object combined with its corresponding virtual object constitutes a “mixed reality object.” It is not necessary for a virtual object to perfectly match or match the corresponding real object. In some embodiments, a virtual object can be a simplified version of the corresponding real object. For example, if the real environment includes a real lamppost, the corresponding virtual object may include a cylinder of approximately the same height and radius as the real lamppost (reflecting that a lamppost may be approximately cylindrical in shape). Simplifying virtual objects in this way can enable computational efficiency and simplify calculations to be performed on such virtual objects. Furthermore, in some embodiments of an MRE, not all real objects in the real environment may be associated with a corresponding virtual object. Similarly, in some embodiments of an MRE, not all virtual objects in the virtual environment may be associated with corresponding real objects, i.e., some virtual objects may exist solely within the virtual environment of the MRE without any real-world counterpart.

[0035] In some embodiments, virtual objects may have characteristics that differ, sometimes significantly, from those of their corresponding real objects. For example, a real environment in an MRE may include a green, two-pronged cactus, i.e., a thorny, inanimate object, while the corresponding virtual object in the MRE may have the characteristics of a green, two-armed virtual character with human facial features and a surly attitude. In this embodiment, the virtual object resembles its corresponding real object in some characteristics (color, number of arms) but differs from the real object in other characteristics (facial features, personality). In this manner, virtual objects have the potential to represent real objects in a creative, abstract, exaggerated, or fictional manner, or to impart behavior (e.g., human personality) to otherwise inanimate real objects. In some embodiments, a virtual object may be a purely fictional creation with no real-world counterpart (e.g., a virtual monster in a virtual environment, perhaps in a location that corresponds to a void in the real environment).

[0036] Compared to VR systems, which present a virtual environment to a user while obscuring the real environment, mixed reality systems that present an MRE offer the advantage that the real environment remains perceptible while the virtual environment is presented. Thus, a user of a mixed reality system can experience and interact with the corresponding virtual environment using visual and audio cues associated with the real environment. As an example, a user of a VR system may struggle to perceive or interact with virtual objects displayed in the virtual environment because, as noted above, the user cannot directly perceive or interact with the virtual environment. However, a user of an MR system may find it intuitive and natural to interact with virtual objects by seeing, hearing, and touching the corresponding real objects in their own real environment. This level of interaction may enhance the user's sense of immersion, connection, and engagement with the virtual environment. Similarly, by simultaneously presenting a real environment and a virtual environment, a mixed reality system may reduce negative psychological sensations (e.g., cognitive dissonance) and negative physical sensations (e.g., motion sickness) associated with VR systems. Mixed reality systems also offer many possibilities for applications that can augment or modify our experience of the real world.

[0037] 1A illustrates an exemplary real environment 100 in which a user 110 uses a mixed reality system 112. The mixed reality system 112 may include a display (e.g., a see-through display) and one or more speakers, as well as one or more sensors (e.g., cameras), for example, as described below. The illustrated real environment 100 includes a rectangular room 104A in which the user 110 is standing and real objects 122A (lamp), 124A (table), 126A (sofa), and 128A (painting). The room 104A further includes a location coordinate 106, which may be considered the origin of the real environment 100. As shown in FIG. 1A, an environment / world coordinate system 108 (comprising an x-axis 108X, a y-axis 108Y, and a z-axis 108Z), with its origin at point 106 (world coordinates), may define a coordinate space for the real environment 100. In some embodiments, origin 106 of environment / world coordinate system 108 may correspond to where mixed reality system 112 is powered on. In some embodiments, origin 106 of environment / world coordinate system 108 may be reset during operation. In some examples, user 110 may be considered a real object in real environment 100. Similarly, body parts (e.g., hands, feet) of user 110 may be considered real objects in real environment 100. In some examples, user / listener / head coordinate system 114 (comprising x-axis 114X, y-axis 114Y, and z-axis 114Z), with its origin at point 115 (e.g., user / listener / head coordinate), may define a coordinate space for user / listener / head on which mixed reality system 112 is located. Origin 115 of user / listener / head coordinate system 114 may be defined relative to one or more components of mixed reality system 112. For example, the origin 115 of the user / listener / head coordinate system 114 may be defined relative to the display of the mixed reality system 112, such as during an initial calibration of the mixed reality system 112. A matrix (which may include a translation matrix and a quaternion matrix or other rotation matrix) or other suitable representation can characterize the transformation between the user / listener / head coordinate system 114 space and the environment / world coordinate system 108 space.In some embodiments, left ear coordinates 116 and right ear coordinates 117 may be defined relative to the origin 115 of the user / listener / head coordinate system 114. A matrix (which may include a translation matrix and a quaternion matrix or other rotation matrix) or other suitable representation can characterize the transformation between the left ear coordinates 116 and right ear coordinates 117 and the user / listener / head coordinate system 114 space. The user / listener / head coordinate system 114 can simplify the representation of location relative to the user's head or head-mounted device, for example, relative to the environment / world coordinate system 108. Using simultaneous localization and mapping (SLAM), visual odometry, or other techniques, the transformation between the user coordinate system 114 and the environment coordinate system 108 can be determined and updated in real time.

[0038] 1B illustrates an exemplary virtual environment 130 that corresponds to real environment 100. The illustrated virtual environment 130 includes a virtual rectangular room 104B that corresponds to real rectangular room 104A, a virtual object 122B that corresponds to real object 122A, a virtual object 124B that corresponds to real object 124A, and a virtual object 126B that corresponds to real object 126A. Metadata associated with virtual objects 122B, 124B, and 126B may include information derived from the corresponding real objects 122A, 124A, and 126A. Virtual environment 130 additionally includes a virtual monster 132, which does not correspond to any real object in real environment 100. Real object 128A in real environment 100 does not correspond to any virtual object in virtual environment 130. A persistent coordinate system 133 (with x-axis 133X, y-axis 133Y, and z-axis 133Z), with its origin at point 134 (persistent coordinate), may define a coordinate space for the virtual content. Origin 134 of persistent coordinate system 133 may be defined relative to / with respect to one or more real objects, such as real object 126A. Matrices (which may include translation matrices and quaternion or other rotation matrices) or other suitable representations can characterize the transformation between persistent coordinate system 133 space and environment / world coordinate system 108 space. In some embodiments, virtual objects 122B, 124B, 126B, and 132 may each have its own persistent coordinate point relative to origin 134 of persistent coordinate system 133. In some embodiments, there may be multiple persistent coordinate systems, and virtual objects 122B, 124B, 126B, and 132 may each have its own persistent coordinate point relative to one or more persistent coordinate systems.

[0039] 1A and 1B, environment / world coordinate system 108 defines a shared coordinate space for both real environment 100 and virtual environment 130. In the illustrated embodiment, the coordinate space has its origin at point 106. Furthermore, the coordinate space is defined by the same three orthogonal axes (108X, 108Y, 108Z). Thus, a first location in real environment 100 and a second corresponding location in virtual environment 130 can be described with respect to the same coordinate space. This simplifies identifying and displaying corresponding locations in the real and virtual environments because the same coordinates can be used to identify both locations. However, in some embodiments, corresponding real and virtual environments need not use a shared coordinate space. For example, in some embodiments (not shown), a matrix (which may include a translation matrix and a quaternion matrix or other rotation matrix) or other suitable representation can characterize the transformation between the real environment coordinate space and the virtual environment coordinate space.

[0040] 1C illustrates an exemplary MRE 150 that simultaneously presents aspects of real environment 100 and virtual environment 130 to user 110 via mixed reality system 112. In the example shown, MRE 150 simultaneously presents to user 110 real objects 122A, 124A, 126A, and 128A from real environment 100 (e.g., through a transparent portion of the display of mixed reality system 112) and virtual objects 122B, 124B, 126B, and 132 from virtual environment 130 (e.g., through an active display portion of the display of mixed reality system 112). As described above, origin 106 serves as the origin for a coordinate space corresponding to MRE 150, and coordinate system 108 defines the x-, y-, and z-axes for the coordinate space.

[0041] In the illustrated example, the mixed reality objects comprise corresponding pairs of real and virtual objects (i.e., 122A / 122B, 124A / 124B, 126A / 126B) that occupy corresponding locations in coordinate space 108. In some examples, both real and virtual objects may be visible to user 110 simultaneously. This may be desirable in instances where, for example, a virtual object presents information designed to augment the view of the corresponding real object (such as in a museum application where a virtual object presents a missing portion of an ancient, damaged statue). In some examples, the virtual objects (122B, 124B, and / or 126B) may be displayed so as to occlude the corresponding real objects (122A, 124A, and / or 126A) (e.g., via active pixelated occlusion using a pixelated occlusion shutter). This may be desirable, for example, in instances where a virtual object acts as a visual replacement for a corresponding real object (such as in interactive storytelling applications where inanimate real objects become "living" characters).

[0042] In some examples, real objects (e.g., 122A, 124A, 126A) may be associated with virtual content or helper data that does not necessarily constitute a virtual object. The virtual content or helper data can facilitate processing or handling of the virtual object within a mixed reality environment. For example, such virtual content may include a two-dimensional representation of the corresponding real object, a custom asset type associated with the corresponding real object, or statistical data associated with the corresponding real object. This information can enable or facilitate calculations involving the real object without incurring unnecessary computational overhead.

[0043] In some embodiments, the presentation described above may also incorporate audio aspects. For example, in MRE 150, virtual monster 132 may be associated with one or more audio signals, such as footstep effects, that are generated as the monster walks around MRE 150. As described further below, a processor in mixed reality system 112 may calculate an audio signal corresponding to a mixed and processed combination of all such sounds within MRE 150 and present the audio signal to user 110 via one or more speakers included within mixed reality system 112 and / or one or more external speakers.

[0044] Exemplary Mixed Reality System

[0045] An exemplary mixed reality system 112 can include a wearable head device (e.g., a wearable augmented reality or mixed reality head device) that includes a display (which may include left and right see-through displays, which may be eyepiece displays, and associated components for coupling light from the displays to the user's eyes), left and right speakers (e.g., positioned adjacent the user's left and right ears, respectively), an inertial measurement unit (IMU) (e.g., mounted on temple arms of the head device), a quadrature coil electromagnetic receiver (e.g., mounted on the left temple component), left and right cameras (e.g., depth (time-of-flight) cameras) oriented away from the user, and left and right eye cameras oriented toward the user (e.g., to detect the user's eye movements). However, the mixed reality system 112 can incorporate any suitable display technology and any suitable sensors (e.g., optical, infrared, acoustic, LIDAR, EOG, GPS, magnetic). Additionally, mixed reality system 112 may incorporate networking features (e.g., Wi-Fi capabilities) to communicate with other devices and systems, including other mixed reality systems. Mixed reality system 112 may further include a battery (which may be mounted in an auxiliary unit, such as a belt pack designed to be worn around the user's waist), a processor, and memory. The wearable head device of mixed reality system 112 may include a tracking component, such as an IMU or other suitable sensor, configured to output a set of coordinates of the wearable head device relative to the user's environment. In some examples, the tracking component may provide input to a processor and implement simultaneous localization and mapping (SLAM) and / or visual odometry algorithms. In some examples, mixed reality system 112 may also include a handheld controller 300 and / or an auxiliary unit 320, which may be a wearable belt pack, as described further below.

[0046] 2A-2D illustrate components of an exemplary mixed reality system 200 (which may correspond to mixed reality system 112) that may be used to present an MRE (which may correspond to MRE 150) or other virtual environment to a user. FIG. 2A illustrates a perspective view of a wearable head device 2102 included within the exemplary mixed reality system 200. FIG. 2B illustrates a top view of the wearable head device 2102 worn on a user's head 2202. FIG. 2C illustrates a front view of the wearable head device 2102. FIG. 2D illustrates an edge view of an exemplary eyepiece 2110 of the wearable head device 2102. As shown in FIGS. 2A-2C, the exemplary wearable head device 2102 includes an exemplary left eyepiece (e.g., a left transparent waveguide set eyepiece) 2108 and an exemplary right eyepiece (e.g., a right transparent waveguide set eyepiece) 2110. Each eyepiece 2108 and 2110 can include a transmissive element through which the real environment is visible and a display element for presenting a display (e.g., via image-modulated light) that is overlaid on the real environment. In some embodiments, such display elements can include surface diffractive optical elements for controlling the flow of the image-modulated light. For example, the left eyepiece 2108 can include a left internal coupling grating set 2112, a left orthogonal pupil-extension (OPE) grating set 2120, and a left exit (output) pupil-extension (EPE) grating set 2122. Similarly, the right eyepiece 2110 can include a right internal coupling grating set 2118, a right OPE grating set 2114, and a right EPE grating set 2116. The image-modulated light can be transferred to the user's eye via the internal coupling gratings 2112 and 2118, the OPEs 2114 and 2120, and the EPEs 2116 and 2122. Each internal coupling grating set 2112, 2118 can be configured to deflect light toward its corresponding OPE grating set 2120, 2114. Each OPE grating set 2120, 2114 can be designed to progressively deflect light downward toward its associated EPE 2122, 2116, thereby extending the exit pupil formed horizontally.Each EPE 2122, 2116 can be configured to progressively redirect at least a portion of the light received from its corresponding OPE grating set 2120, 2114 outward toward a user eyebox location (not shown), defined behind the eyepieces 2108, 2110, such that the exit pupil formed in the eyebox extends vertically. Alternatively, instead of the internal coupling grating sets 2112 and 2118, the OPE grating sets 2114 and 2120, and the EPE grating sets 2116 and 2122, the eyepieces 2108 and 2110 can include gratings and / or other arrangements of refractive and reflective features to control the coupling of image-modulated light into the user's eye.

[0047] In some examples, the wearable head device 2102 can include a left temple arm 2130 and a right temple arm 2132, where the left temple arm 2130 includes a left speaker 2134 and the right temple arm 2132 includes a right speaker 2136. A quadrature coil electromagnetic receiver 2138 can be located in the left temple assembly or another suitable location within the wearable head unit 2102. An inertial measurement unit (IMU) 2140 can be located in the right temple arm 2132 or another suitable location within the wearable head device 2102. The wearable head device 2102 can also include a left depth (e.g., time-of-flight) camera 2142 and a right depth camera 2144. The depth cameras 2142, 2144 can preferably be oriented in different directions so that both cover a wider field of view.

[0048] 2A-2D , a left source of image-wise modulated light 2124 can be optically coupled into the left eyepiece 2108 through a left internal coupling grating set 2112, and a right source of image-wise modulated light 2126 can be optically coupled into the right eyepiece 2110 through a right internal coupling grating set 2118. The source of image-wise modulated light 2124, 2126 can include, for example, a fiber optic scanner, a projector including an electronic light modulator such as a digital light processing (DLP) chip or a liquid crystal on silicon (LCoS) modulator, or an emissive display such as a micro light emitting diode (μLED) or micro organic light emitting diode (μOLED) panel coupled into the internal coupling grating sets 2112, 2118 using one or more lenses per side. The input coupling grating sets 2112, 2118 can deflect light from the source of image-wise modulated light 2124, 2126 to an angle above the critical angle for total internal reflection (TIR) ​​for the eyepieces 2108, 2110. The OPE grating sets 2114, 2120 progressively deflect the propagating light downward by TIR towards the EPE grating sets 2116, 2122. The EPE grating sets 2116, 2122 progressively couple the light towards the user's face, including the pupils of the user's eyes.

[0049] In some embodiments, as shown in FIG. 2D , the left eyepiece 2108 and the right eyepiece 2110 each include multiple waveguides 2402. For example, each eyepiece 2108, 2110 can include multiple individual waveguides, each dedicated to a separate color channel (e.g., red, blue, and green). In some embodiments, each eyepiece 2108, 2110 can include multiple sets of such waveguides, each configured to impart a different wavefront curvature to the emitted light. The wavefront curvature may be convex with respect to the user's eye, for example, to present a virtual object positioned at a distance in front of the user (e.g., a distance corresponding to the inverse of the wavefront curvature). In some embodiments, the EPE grating sets 2116, 2122 can include curved grating grooves to impart a convex wavefront curvature by modifying the Poynting vector of light exiting across each EPE.

[0050] In some examples, stereoscopically adjusted left and right eye images can be presented to the user through light modulators 2124, 2126 and eyepieces 2108, 2110 for each image to create the perception that the displayed content is three-dimensional. The perceived realism of the presentation of three-dimensional virtual objects can be enhanced by selecting the waveguides (and thus the corresponding wavefront curvatures) so that the virtual objects are displayed at distances that approximate the distances indicated by the stereoscopic left and right images. This technique can also reduce motion sickness experienced by some users, which can be caused by differences between the depth perception cues provided by the stereoscopic left and right eye images and the automatic accommodation (e.g., object distance-dependent focus) of the human eye.

[0051] FIG. 2D illustrates an edge view from above of the right eyepiece 2110 of the exemplary wearable head device 2102. As shown in FIG. 2D , the plurality of waveguides 2402 can include a first subset 2404 of three waveguides and a second subset 2406 of three waveguides. The two subsets 2404, 2406 of waveguides can be distinguished by different EPE gratings featuring different grating line curvatures to impart different wavefront curvatures to the exiting light. Within each of the subsets 2404, 2406 of waveguides, each waveguide can be used to couple a different spectral channel (e.g., one of the red, green, and blue spectral channels) to the user's right eye 2206. (Although not shown in FIG. 2D , the structure of the left eyepiece 2108 is similar to that of the right eyepiece 2110.)

[0052] 3A illustrates example handheld controller components 300 of mixed reality system 200. In some examples, handheld controller 300 includes a grip portion 346 and one or more buttons 350 disposed along a top surface 348. In some examples, button 350 may be configured for use as an optical tracking target to track six degrees of freedom (6DOF) movement of handheld controller 300, for example, in conjunction with a camera or other optical sensor (which may be mounted in a head unit (e.g., wearable head device 2102) of mixed reality system 200). In some examples, handheld controller 300 includes a tracking component (e.g., an IMU or other suitable sensor) for detecting a position or orientation, such as a position or orientation relative to wearable head device 2102. In some examples, such a tracking component may be positioned in a handle of handheld controller 300 and / or may be mechanically coupled to the handheld controller. The handheld controller 300 can be configured to provide one or more output signals corresponding to one or more of a button press state, or the position, orientation, and / or movement (e.g., via an IMU) of the handheld controller 300. Such output signals may be used as inputs to a processor of the mixed reality system 200. Such inputs may correspond to the position, orientation, and / or movement of the handheld controller (or, for that matter, the position, orientation, and / or movement of a user's hand holding the controller). Such inputs may also correspond to a user pressing a button 350.

[0053] 3B illustrates an example auxiliary unit 320 of the mixed reality system 200. The auxiliary unit 320 can include a battery for providing energy to operate the system 200 and can include a processor for executing programs to operate the system 200. As shown, the example auxiliary unit 320 includes a clip 2128 for attaching the auxiliary unit 320 to a user's belt, etc. It will also be apparent that other form factors are suitable for the auxiliary unit 320, including form factors that do not involve mounting the unit on a user's belt. In some embodiments, the auxiliary unit 320 is coupled to the wearable head device 2102 through a multi-tube cable, which may include, for example, electrical wires and optical fibers. A wireless connection between the auxiliary unit 320 and the wearable head device 2102 can also be used.

[0054] In some examples, mixed reality system 200 can include one or more microphones to detect sound and provide a corresponding signal to the mixed reality system. In some examples, the microphones may be attached to or integrated with wearable head device 2102 and configured to detect the user's voice. In some examples, microphones may be attached to or integrated with handheld controller 300 and / or auxiliary unit 320. Such microphones may be configured to detect environmental sounds, ambient noise, the user's or a third party's voice, or other sounds.

[0055] 4 shows an example functional block diagram that may correspond to an example mixed reality system, such as mixed reality system 200 described above (which may correspond to mixed reality system 112 with respect to FIG. 1). As shown in FIG. 4, example handheld controller 400B (which may correspond to handheld controller 300 (“totem”)) includes a totem / wearable head device six degrees of freedom (6DOF) totem subsystem 404A, and example wearable head device 400A (which may correspond to wearable head device 2102) includes a totem / wearable head device 6DOF subsystem 404B. In an example, 6DOF totem subsystem 404A and 6DOF subsystem 404B cooperate to determine six coordinates of handheld controller 400B relative to wearable head device 400A (e.g., offsets in three translational directions and rotations along three axes). The six degrees of freedom may be expressed relative to the coordinate system of wearable head device 400A. The three translational offsets may be represented as X, Y, and Z offsets within such a coordinate system, a translation matrix, or some other representation. The rotational degrees of freedom may be represented as a sequence of yaw, pitch, and roll rotations, as a rotation matrix, as a quaternion, or some other representation. In some examples, the wearable head device 400A, one or more depth cameras 444 (and / or one or more non-depth cameras) included within the wearable head device 400A, and / or one or more optical targets (e.g., buttons 350 of handheld controller 400B as described above or dedicated optical targets included within handheld controller 400B) can be used for 6DOF tracking. In some examples, the handheld controller 400B can include a camera as described above, and the wearable head device 400A can include an optical target for optical tracking in conjunction with the camera. In some embodiments, the wearable head device 400A and the handheld controller 400B each include a set of three orthogonally oriented solenoids, which are used to wirelessly transmit and receive three distinguishable signals.By measuring the relative magnitudes of the three distinguishable signals received in each of the coils used to receive, the 6DOF of the wearable head device 400A relative to the handheld controller 400B can be determined. Additionally, the 6DOF totem subsystem 404A can include an inertial measurement unit (IMU), which is useful for providing improved accuracy and / or more timely information regarding high speed movements of the handheld controller 400B.

[0056] In some examples, it may be necessary to transform coordinates from a local coordinate space (e.g., a coordinate space that is fixed relative to the wearable head device 400A) to an inertial coordinate space (e.g., a coordinate space that is fixed relative to the real environment), e.g., to compensate for movement of the wearable head device 400A relative to coordinate system 108. For example, such a transformation may be necessary so that the display of the wearable head device 400A presents virtual objects in an expected position and orientation relative to the real environment (e.g., a virtual person sitting in a real chair facing forward, regardless of the position and orientation of the wearable head device), rather than in a fixed position and orientation on the display (e.g., the same position in the bottom right corner of the display), preserving the illusion that the virtual objects exist in the real environment (and do not appear unnaturally positioned in the real environment, e.g., as the wearable head device 400A shifts and rotates). In some examples, a compensatory transformation between coordinate spaces can be determined by processing images from depth camera 444 using SLAM and / or visual odometry procedures to determine the transformation of wearable head device 400A relative to coordinate system 108. In the example shown in FIG. 4 , depth camera 444 is coupled to SLAM / visual odometry block 406 and can provide images to block 406. The SLAM / visual odometry block 406 implementation can include a processor configured to process the images and then determine the position and orientation of the user's head, which can be used to identify a transformation between the head coordinate space and another coordinate space (e.g., an inertial coordinate space). Similarly, in some examples, an additional source of information about the user's head pose and location is obtained from IMU 409. Information from IMU 409 can be integrated with information from SLAM / visual odometry block 406 to provide improved accuracy and / or more timely information for rapid adjustments of the user's head pose and position.

[0057] In some examples, depth camera 444 can provide 3D images to hand gesture tracker 411, which can be implemented within a processor of wearable head device 400A. Hand gesture tracker 411 can identify the user's hand gestures, for example, by matching the 3D images received from depth camera 444 to stored patterns representing hand gestures. Other suitable techniques for identifying the user's hand gestures will also be apparent.

[0058] In some embodiments, one or more processors 416 may be configured to receive data from the wearable head device's 6DOF headgear subsystem 404B, IMU 409, SLAM / visual odometry block 406, depth camera 444, and / or hand gesture tracker 411. The processor 416 may also send and receive control signals to and from the 6DOF totem system 404A. The processor 416 may be wirelessly coupled to the 6DOF totem system 404A, such as in embodiments in which the handheld controller 400B is untethered. The processor 416 may further communicate with additional components, such as an audio / visual content memory 418, a graphical processing unit (GPU) 420, and / or a digital signal processor (DSP) audio spatializer 422. The DSP audio spatializer 422 may be coupled to a head-related transfer function (HRTF) memory 425. The GPU 420 may include a left channel output coupled to a left source of imagewise modulated light 424 and a right channel output coupled to a right source of imagewise modulated light 426. The GPU 420 may output stereoscopic image data to the sources of imagewise modulated light 424, 426, for example, as described above with respect to FIGS. 2A-2D . The DSP audio spatializer 422 may output audio to the left speaker 412 and / or the right speaker 414. The DSP audio spatializer 422 may receive an input from the processor 419 indicating a direction vector from the user to a virtual sound source (which may be moved by the user, e.g., via the handheld controller 320). Based on the direction vector, the DSP audio spatializer 422 may determine a corresponding HRTF (e.g., by accessing an HRTF or by interpolating multiple HRTFs). The DSP audio spatializer 422 may then apply the determined HRTF to an audio signal, such as an audio signal corresponding to a virtual sound generated by a virtual object.This can improve the believability and realism of virtual sounds by incorporating the user's relative position and orientation to the virtual sounds in the mixed reality environment, i.e., by presenting virtual sounds that match the user's expectations of what they would hear if the virtual sounds were real sounds in a real environment.

[0059] 4 , one or more of the processor 416, GPU 420, DSP audio spatializer 422, HRTF memory 425, and audio / visual content memory 418 may be included in auxiliary unit 400C (which may correspond to auxiliary unit 320 described above). Auxiliary unit 400C may include battery 427 to power its components and / or provide power to wearable head device 400A or handheld controller 400B. Including such components in an auxiliary unit, which may be mounted on the user's waist, can limit the size and weight of wearable head device 400A, which in turn can reduce fatigue in the user's head and neck.

[0060] While Figure 4 presents elements corresponding to various components of an exemplary mixed reality system, various other suitable arrangements of these components will be apparent to those skilled in the art. For example, elements shown in Figure 4 as associated with auxiliary unit 400C may instead be associated with wearable head device 400A or handheld controller 400B. Furthermore, some mixed reality systems may dispense with handheld controller 400B or auxiliary unit 400C entirely. Such variations and modifications should be understood as being within the scope of the disclosed embodiments.

[0061] Spatial Content

[0062] The present disclosure relates to creating and viewing spatial content within an MRE, which may include spatial instructions and guides ("SIGs"). According to some embodiments, a user (or more than one user) can "record" spatial content within an MRE, which can be persistently tied to the physical environment in which the events were recorded. For example, a user in a room may walk around the room, talking about objects in the room, pointing to them, and drawing virtual pictures in the MRE. A user (either the same user or a different user) may then play back the MRE "recording." In some embodiments, the user may be present in the same room in which the MRE recording was created (during MRE recording playback). In some embodiments, the user may then view a virtual representation of the user moving around the room, describing objects, pointing to them, and drawing virtual pictures in the MRE. In some embodiments, the virtual representation may behave in a similar manner to the user who created the MRE recording (e.g., the virtual representation may move around the physical environment in a similar manner to how the user who created the MRE recording moves around the physical environment). In some embodiments, spatial content can be created without recording the user's movements. For example, spatial content can be at least partially simulated by a computer without requiring a user to physically perform the actions to be recorded. In some embodiments, spatial content may have no association with a physical environment. For example, spatial content intended for entertainment purposes can be configured to be played in any physical environment. In some embodiments, spatial content can be configured to be adaptable to various playback environments (e.g., spatial entertainment content can be configured to scale to the size of a user's living room). As used herein, a "SIG event" may refer to virtual content that is created in a spatial location and persists (permanently, semi-permanently, or for a limited amount of time) relative to that spatial location.A SIG event may include text, images, audio, video, interactive content, web links, or any other suitable information. A SIG recording may comprise one or more recorded SIG events.

[0063] Recording and playing back SIG events within an MRE can be beneficial, for example, in educational or entertainment applications. For example, an instructor can record a SIG event about safety procedures (e.g., the proper way to operate machinery or where personal protective equipment is stored) in a physical location where a user may be working (e.g., in a warehouse). A user (e.g., a new employee) can then be present in the relevant physical location (e.g., the warehouse where they will be working) and view the SIG event within the MRE (e.g., the new employee can follow the virtual representation of the instructor within the physical warehouse as the virtual representation teaches the proper safety procedures). In some embodiments, SIG events within an MRE can be more engaging to users than other virtual environments (e.g., watching a video on a 2D screen). In some embodiments, a user can view or interact with the virtual environment simultaneously with the associated physical environment (e.g., the user can view the SIG event while the user is present in the physical environment where the SIG event was recorded), leading to a greater sense of engagement. Other methods of viewing a virtual environment (e.g., watching a video on a 2D screen) generally do not allow a user to simultaneously interact with the physical environment depicted in the video (e.g., because the user is watching the video from a location different from where the video was recorded). Therefore, it may be desirable to develop systems and methods for increasing a user's engagement with a virtual environment by allowing the virtual environment to composite with the physical environment within an MRE.

[0064] FIG. 5 depicts a flowchart of an exemplary use of the SIG system. In step 502, a SIG recording session is initialized (e.g., a user can interact with a user interface to start recording or set parameters for the recording session). In step 504, one or more SIG events are recorded. The SIG recording can incorporate data from one or more sensors on the wearable head device, which can include, for example, the user's hand and head movements, audio signals, eye movements, location information, vital signs, or spatial location. In step 506, a SIG playback session is initialized (e.g., a user can select a SIG recording for playback). In step 508, one or more SIG events are displayed within the SIG playback session (e.g., a SIG recording selected by the user can be played back to the user). In some embodiments, multiple users can participate in a single SIG playback session. Steps 502, 504, 506, and 508 of process 500 are described in further detail below.

[0065] Initializing SIG Records

[0066] 6 depicts an embodiment in which a SIG recording session is initiated. In the depicted embodiment, a user 602 may use a mixed reality system 603 (which may correspond to mixed reality systems 112 and 200), and the user 602 may be located in a physical environment 604 (which may correspond to real environment 100). The mixed reality system 603 may be implemented via one or more mixed reality devices, such as a wearable head device (e.g., wearable head device 2102 described above), a handheld controller (e.g., handheld controller 300 described above), and / or an auxiliary unit (e.g., auxiliary unit 320 described above). In some embodiments, the user 602 may initiate a SIG recording session using a mobile device (e.g., a mobile phone). In some embodiments, the mixed reality system 603 may display virtual objects to the user 602 (which may correspond to virtual environment 130) while simultaneously allowing the user 602 to perceive the physical environment 604. In the depicted embodiment, the mixed reality system 603 can display a virtual menu 606 to the user. In some embodiments, the virtual menu can be at least partially transparent so that the user can see through the virtual menu to the physical environment behind the virtual menu. In some embodiments, the virtual menu (generally, a virtual object) may not be transparent, such that the virtual menu (generally, a virtual object) appears to be part of the physical environment. In some embodiments, the virtual menu can be generally fixed in position relative to the physical environment (e.g., the virtual menu 606 is fixed above a table and is only perceptible when the user views that area in the physical environment). In some embodiments, the mixed reality system 603 can place virtual objects within the physical environment using a coordinate system (which may correspond to coordinate system 108) and an origin (which may correspond to origin 106). In some embodiments, the coordinate system can define a shared coordinate space for both the physical environment and the virtual environment, such that the same coordinates can be used to identify locations in both environments.In some embodiments, the virtual menu can be generally fixed in a fixed position relative to the user's field of view. For example, the virtual menu can be generally fixed to the center of the user's field of view so that the virtual menu is always centered in the user's field of view regardless of where the user is looking. In some embodiments, the virtual menu can exhibit inertia once it is generally fixed in location. For example, if the virtual menu is generally fixed to the center of the user's field of view, it can follow changes in the user's field of view so that it is not always centered in the field of view. In some embodiments, the virtual menu can then return to the center of the field of view once the field of view remains unchanged for a period of time.

[0067] In some embodiments, the virtual menu 606 may display one or more options to the user (e.g., record a SIG event, toggle a SIG layer, or start a SIG event). In some embodiments, the mixed reality system 603 may track the user's eye movements, and the user may select between one or more displayed options by looking at a particular option. In some embodiments, the mixed reality system 603 may track the movements of a handheld controller (e.g., handheld controller component 300), which may be part of the mixed reality system, and the user may confirm a particular selection by pressing a button on the handheld controller (e.g., handheld controller component 300). However, other suitable selection methods may also be used (e.g., using a button on the handheld controller to select and confirm an option). In some embodiments, the virtual menu 606 may display an option for recording a SIG event. The user may select an option for recording a SIG event, and the mixed reality system may, in response, display a prompt to the user (e.g., press a button to start recording). In some embodiments, the user can use voice input to provide input to the virtual menu.

[0068] In some embodiments, a remote user who is not physically present in the mixed reality environment records or uploads SIG events to a location in the mixed reality environment. That is, the SIG events can be uploaded to a virtual location associated with the remote user, regardless of the remote user's physical location. For example, the location associated with the remote user can be a virtual location on a display of a remote user device (e.g., a mobile device, a computer, a second wearable head device) that presents the mixed reality environment. The presentation of the mixed reality environment may be captured (e.g., recorded, streamed) using a first wearable head device in the mixed reality environment. The remote user may see the mixed reality environment (e.g., captured by the first wearable head device) through the display of the remote user device, and the virtual location on the display may correspond to the physical location of the first wearable head device that shares the view of the mixed reality environment. In addition to seeing the mixed reality environment, additional information about the mixed reality environment may be displayed (e.g., a map of the environment, a menu of options, a grid showing the location of the first wearable head device). From the remote user's device, the remote user may record or upload a SIG event to a physical location in the mixed reality environment by selecting the corresponding virtual location. The SIG event can then be accessed within the mixed reality environment by other users of the mixed reality environment, regardless of whether they are physically present in the mixed reality environment or access the mixed reality environment remotely. A SIG event can comprise any suitable type of data, such as a text file, an audio file, a video file, a URL, or social media content. A SIG event can also comprise executable computer code, such as a computer application or script.

[0069] As an example, a first user may remotely communicate with a second user who accesses a wearable device such that the first user can see a view provided by the second user. For example, the first user may remotely access the second user's view of a mixed reality environment. The second user's wearable device may show the first user their collection of musical instruments. The first user may upload or record a SIG event (e.g., in this example, content associated with the musical instrument) at the location of the instrument on the display of the device such that the SIG event becomes associated with the physical location of the instrument in the mixed reality environment. The second user may access the SIG event at the physical location. At a later time, a third user at the location of the instrument may access the SIG event recorded or uploaded by the first user. The third user may be present at the physical location of the instrument, or the third user may be a remote user virtually present at the location of the instrument.

[0070] In some embodiments, a user at a virtual location (e.g., a remote user not physically present in the mixed reality environment) records or uploads a SIG event without a presentation of the mixed reality environment from a wearable head device. For example, the location associated with the user is a virtual location on a map of the mixed reality environment, and based on the map of the mixed reality environment, the user records or uploads a SIG event to the location in the mixed reality environment. As another example, the user may define coordinates within the mixed reality environment for recording or uploading a SIG event.

[0071] FIG. 7 depicts an embodiment in which a user may select to record a SIG event on a particular layer. In some embodiments, the mixed reality system presents a virtual menu 702 to the user. In some embodiments, the virtual menu 702 may be presented after the user selects an option to record a SIG event. The virtual menu 702 may display one or more options 704 corresponding to one or more layers for recording a SIG event. A layer may serve as an organizational tool for arranging multiple SIG events. For example, a layer may act as a filter that selectively shows or hides groups of information (e.g., SIG events or SIG recordings). In some embodiments, different layers may correspond to different educational-oriented material. For example, a "Discovery" layer may include SIG events about features within a dorm building. SIG events in the "Discovery" layer may instruct the user on where the laundry or computer lab is located, proper kitchen etiquette, or how to connect to the campus wireless network. In another example, a "Campus Tour" layer may include SIG events about buildings within the campus. One SIG event may be dedicated to the admissions office, while another SIG event may be dedicated to classroom navigation within an engineering building. In some embodiments, a user can select from a predefined set of tiers (e.g., one or more tiers) to record. In some embodiments, a user can add or remove tiers from the list of tiers presented. In some embodiments, SIG events can be automatically assigned to tiers based on contextual information.

[0072] While tiers have been described, it is also contemplated that SIG events (and other spatial content) may be filtered by other suitable means. For example, SIG events may be filtered by associated keywords. In some embodiments, SIG events may be filtered with more granularity. For example, a specific element of a SIG event may be associated with a filter (e.g., a tier or one or more keywords). In some embodiments, toggling a filter may toggle one or more associated elements of a SIG event, and in some embodiments, toggling a filter off will not affect unassociated elements of a SIG event. For example, a virtual bar graph associated with a first tier may be displayed, and a chart value descriptor may be associated with a second tier. The chart value descriptor may or may not be overlaid over the virtual bar graph, depending on whether the associated tier is toggled on or off. Filters may also be used by a user to select the content the user desires to view. For example, a user may toggle between different language filters or genre filters. In some embodiments, filters can be used to restrict a user's access to space content (e.g., content not suitable for children). For example, an administrator may allow only specific employees to access content associated with a specific filter. In some embodiments, filters can be associated with social media content (e.g., groups, friends, etc.).

[0073] SIG Event Records

[0074] In some embodiments, once a user selects one or more layers to record, the user can begin recording the SIG event, as shown in FIG. 8 . FIG. 8 depicts user 802 using mixed reality system 803 while located in physical environment 804 (which may correspond to real environment 100). In some embodiments, once the user begins recording the SIG event, a virtual object associated with the SIG event can be generated (which may correspond to virtual environment 130). In some embodiments, virtual markers 806 can be placed where the user is located in the physical environment (e.g., using coordinate system 108). The virtual object (e.g., virtual markers 806) can generally be fixed in position relative to the physical environment. For example, virtual markers 806 can be placed next to a set of shelves 807 in physical environment 804. A user using the mixed reality system can then see virtual markers 806 near shelves 807 when the user views an area with shelves 807. In some embodiments, virtual objects may persist in location relative to the physical environment using a persistent coordinate system (which may correspond to coordinate system 108), as described herein with respect to Figures 1A-1C. In some embodiments, the coordinate system used for the physical environment may be the same as the coordinate system used for the virtual environment, such that coordinates may be described for the same location in both the physical and virtual environments.

[0075] In some embodiments, a user can create virtual markings 808 while recording a SIG event. In some embodiments, the creation of virtual markings 808 (e.g., the user's movements while creating virtual markings 808) may be recorded. FIG. 9 depicts an example of a user creating virtual markings 902. In some embodiments, the user can activate a marker function of the SIG system through any suitable means (e.g., pressing a button on a handheld controller 904, selecting a function through a virtual user interface, or simply drawing with the user's hand without using a handheld controller). In some embodiments, once the marker function is activated, the user can create virtual markings 902 using handheld controller 904 (which may correspond to handheld controller 300). In some embodiments, the mixed reality system can track the movement of the user's hand through the movement of handheld controller 904 through the systems and methods described herein (e.g., with respect to FIG. 3A ). In some embodiments, a user can press and hold a button on the handheld controller 904 to “write” a virtual marker 906. In some embodiments, the virtual marker 906 can be displayed during the recording session and overlaid on the handheld controller 904 when the handheld controller comes into the user's field of view. In some embodiments, a virtual controller and / or hand can be displayed during the recording session (e.g., when a virtual marker is not engaged) and overlaid on the handheld controller when the handheld controller comes into the user's field of view. In some embodiments, when the marker function is activated, the user can use their hand to create virtual markings 902. In some embodiments, the mixed reality system can track the movement of the user's hand through various sensors (e.g., a depth sensor, an RGB camera, other optical sensors, or motion sensors).In some embodiments, a user can activate the marker function by pointing with the user's hand. A virtual marker can be displayed and optionally overlaid on the user's hand when the hand is within the user's field of view, indicating that the marker function is active.

[0076] In some embodiments, the virtual markings 902 will follow the movement of the handheld controller 904 as if the user were “writing” with the handheld controller 904. In some embodiments, the virtual markings may visually persist after a user creates the virtual markings. In some embodiments, the virtual markings may visually persist within a SIG event only for a threshold amount of time after the user creates the markings. The virtual markings and their associated visual persistence may be associated with a particular SIG playback session such that if the SIG playback session is resumed or initiated for a different user, the virtual markings may reappear and persist again for the threshold amount of time. In some embodiments, virtual markings previously created for more than the threshold amount of time may fade from the user's view. It is also contemplated that the virtual markings may fade based on a threshold length of the virtual markings (e.g., a virtual marking may have a threshold length of 2 feet). In some embodiments, the virtual markings may be generally fixed relative to their location in the physical environment.

[0077] Referring back to FIG. 8 , in some embodiments, a virtual trajectory 810 around the physical environment can be recorded in the SIG recording. In some embodiments, the mixed reality system can track movement and determine the user's movement through the physical environment through a wearable head device, a handheld controller, or other suitable means (e.g., a combination of both). Other sensors can be used as well (e.g., sensors in the physical environment). In some embodiments, the user's head movement and direction can be recorded in the SIG recording. In some embodiments, the mixed reality system can include a wearable head device capable of tracking the user's head movement and the direction the user is facing using the systems and methods described herein with respect to FIGS. 2A-2D . Other sensors can be used as well (e.g., sensors in the physical environment). In some embodiments, audio signals (e.g., the user's speech) can be recorded in the SIG recording. For example, the mixed reality system can include a microphone, which may be built into the mixed reality system's wearable head device. Other microphones can be used as well (e.g., microphones fixed in the room). In some embodiments, virtual audio markers 812 can be placed within the SIG recording. The virtual audio markers 812 can correspond to audio recordings and can generally be fixed in position relative to the physical environment. In some embodiments, video recordings can be included within the SIG recording. For example, one or more cameras located on the mixed reality system 112 or 200 (or external to the mixed reality system) can record video during the SIG session. In some embodiments, the recorded video can be from the perspective of the user wearing the mixed reality system (i.e., the camera records what the user sees). In some embodiments, the mixed reality system can record augmented video, which records both the physical environment and the overlaid virtual environment.In some embodiments, augmented video can record what a user experiences in a mixed reality environment. In some embodiments, video recorded in a SIG recording session can have an associated virtual video marker, which can be placed in the SIG recording. In some embodiments, a user can activate video recording by approaching an associated virtual marker. In some embodiments, the recorded video can be of the user (e.g., the user's face, including facial expressions, the user's body, including gestures, etc.).

[0078] In some embodiments, a user can end a SIG recording session (e.g., by pressing a button on a handheld controller). In some embodiments, ending a SIG recording session can prompt the mixed reality system to display a virtual menu. In some embodiments, the virtual menu can indicate the length of the recorded SIG event. In some embodiments, the virtual menu can present options to publish the SIG recording or delete the SIG recording. In some embodiments, the virtual menu can present an option to preview the recorded SIG event. In some embodiments, the preview option can allow a user to view the recorded SIG event in the same way that other users (e.g., users who did not create the SIG event) may experience the SIG recording. A SIG recording comprising one or more SIG events can include one or more virtual objects, recordings, points, and / or markers. For example, a single SIG event can include one or more virtual markings, audio recordings, and / or video recordings.

[0079] In some embodiments, a virtual marker 1006 can be created after a user publishes a recorded SIG event, as shown in FIG. 10 . In some embodiments, the virtual marker 1006 can be generally fixed in a fixed position relative to the physical environment 804. In some embodiments, the virtual marker 1006 can be located where the user started recording the corresponding SIG event. In some embodiments, the virtual marker 1006 can display the length of the recorded SIG event and corresponding tier information. In some embodiments, the virtual marker 1006 can be color-coded according to the tier associated with the recorded SIG event. In some embodiments, the virtual marker can reflect an associated context or filter. For example, a virtual marker corresponding to a SIG event associated with a video tier can be a remote controller. In another example, a virtual marker corresponding to a SIG event associated with a wizard tier can be a wand.

[0080] SIG playback initialization

[0081] In some embodiments, a user may receive a notification (e.g., on a mobile device) after a recorded SIG event is published. In some embodiments, the notification may include information about the user who published the recorded SIG event. In some embodiments, the notification may include information about the layer where the SIG event was recorded. In some embodiments, the notification may include information about where the SIG event occurred (e.g., GPS coordinates, direction). In some embodiments, the notification may include information about when the SIG event was recorded. In some embodiments, the mixed reality system may receive the notification and display a virtual walking route to the newly recorded SIG event. In some embodiments, only users within a specific area may receive notifications of the recorded SIG event (e.g., users within a threshold radius of the location of the recorded SIG event).

[0082] FIG. 11 depicts an example application launching on a mobile device 1102. In some embodiments, selecting a notification about a published SIG event may open an application (e.g., the application depicted in FIG. 11). In some embodiments, the application may display a map with a marker 1104. The marker 1104 may correspond to a recorded SIG event. In some embodiments, the marker 1104 may be placed at a location where a recorded SIG event may be played back (e.g., where a SIG event recording was initiated). In some embodiments, the application may display a marker associated with one or more layers. In some embodiments, the application may present an option 1106 to toggle one or more layers on or off.

[0083] In some embodiments, a mobile device can be coupled to a mixed reality system. In some embodiments, the mobile device can present a scannable code (e.g., a barcode or QR code) that the mixed reality system can scan using one or more cameras and pair with the mobile device. However, other coupling mechanisms, such as wired methods (e.g., via a cable connecting the mobile device to the mixed reality system) or wireless methods (e.g., via Bluetooth or near-field communication), are also envisioned. In some embodiments, the mobile device (or other suitable computing device) can edit metadata associated with a SIG event. In some embodiments, the mobile device (or other suitable computing device) can associate a SIG event with another SIG event. For example, SIG events can be linked to play sequentially after the previous SIG event completes. SIG events can also be linked in other structures, such as a tree structure (e.g., a single SIG start point can end with a different SIG event end).

[0084] In some embodiments, a user can proceed to a physical location corresponding to a marker 1104 displayed on a mobile device 1102. FIG. 12 depicts an embodiment in which a user wearing a mixed reality system is present in physical location 1204 (which may correspond to physical location 804). The mixed reality system can display one or more virtual markers 1202. In some embodiments, as the user physically approaches the location of a virtual marker, the virtual marker can display information corresponding to the virtual marker (e.g., the record length of the corresponding SIG record or layer information for the corresponding SIG record). In some embodiments, the virtual marker can display information corresponding to the virtual marker according to the user's eye movements (e.g., the virtual marker may display additional information only when the user is looking at the virtual marker).

[0085] In some embodiments, the virtual marker may correspond to a physical location where the user created the SIG recording. In some embodiments, the virtual marker can be placed at a location other than the recording location (e.g., a virtual marker for simulated spatial content that is not associated with a physical location can be placed anywhere). In some embodiments, the virtual marker can be placed within a virtual environment. For example, the virtual environment can include a miniature amusement park, which can be sized to fit within a user's living room. A virtual marker corresponding to spatial content can be placed within the virtual environment (e.g., within the amusement park). A user can activate a virtual marker by approaching the virtual marker in the virtual environment.

[0086] SIG playback

[0087] FIG. 13 depicts an embodiment in which a user has initiated playback of a SIG recording. In some embodiments, the user can begin playback of the SIG recording by approaching a virtual marker and then selecting a playback option via suitable means (e.g., pressing a button on a handheld controller, using voice input). In some embodiments, once the user begins the SIG playback session, all virtual markers that do not correspond to the selected SIG event can disappear. In some embodiments, the mixed reality system can display a virtual representation 1304. The virtual representation can be a representation of the user who recorded the SIG event and can be based on the user's tracked movements during the SIG recording session. In some embodiments, a virtual hand 1306 can also be displayed as part of the virtual representation. In some embodiments, the virtual hand can be based on a handheld controller operated by the user who recorded the SIG event, and the virtual hand can be based on tracked movements of the handheld controller. In some embodiments, the virtual hand can be based on one or more hands of the user who recorded the SIG event, and the virtual hand can be based on tracked movements of the one or more hands of the user who recorded the SIG event. In some embodiments, a SIG playback session can function as a video recording that can be overlaid over the physical environment in which the video recording occurred.

[0088] In some embodiments, the mixed reality system can display a virtual representation moving around physical location 1310 (which may correspond to physical location 1204). In some embodiments, the virtual representation can mimic the movements of the user who recorded the SIG event. For example, the mixed reality headset can track the position and orientation of the recording user's head, hands, and / or other body parts. As the recording user moves around the physical location and looks around, a virtual representation can be created that moves and looks around in a manner generally similar to the recording user. In some embodiments, the mixed reality system can display a virtual representation 1304 in which the recording user creates virtual markings 1302 in a manner generally similar to how the recording user created virtual markings during the recording session. In some embodiments, the mixed reality system can display a virtual trajectory 1308. In some embodiments, the virtual trajectory indicates the previous location of the virtual representation. In some embodiments, the virtual trajectory can facilitate following the virtual representation around the physical location during the SIG playback session. In some embodiments, the SIG playback session can be paused if the distance between the user and the virtual content exceeds a customizable threshold. The threshold can be either static (eg, a set distance in feet) or dynamic (eg, varying with the size of the virtual object).

[0089] In some embodiments, a virtual marker may display one or more completion indications when a user plays a SIG recording associated with the virtual marker. In some embodiments, a virtual marker may indicate whether a user has only partially completed a SIG playback session. In some embodiments, a virtual marker may indicate whether a user did not start a SIG playback session.

[0090] FIG. 14 depicts an embodiment in which multiple users may participate in a single SIG playback session. In the depicted embodiment, user 1402 approaches virtual marker 1406. In the depicted embodiment, user 1404 has already started a SIG playback session corresponding to virtual marker 1406. In the depicted embodiment, both users 1402 and 1404 may use a mixed reality system. In some embodiments, as user 1402 approaches virtual marker 1406, when another user has already started a playback session corresponding to that virtual marker, virtual marker 1406 may display a virtual menu 1408 to user 1402. In some embodiments, virtual menu 1408 may include options to join an existing session or to start a new session. In some embodiments, virtual menu 1408 may display the remaining or elapsed time of the existing playback session. In some embodiments, selecting the option to start a new session may start a separate SIG playback session that is not synchronized with another user. In some embodiments, selecting the option to start a new session will start a SIG playback session from the beginning of the recorded SIG event. A virtual marker can be associated with one or more display parameters, such as parameters, that determine aspects of the presentation of the virtual marker. As one example, the display parameters can specify that the virtual marker be presented at a certain orientation relative to the user's visual axis.

[0091] FIG. 15 depicts an embodiment in which a user selects to join an existing SIG playback session. In some embodiments, the mixed reality system can synchronize the SIG playback session with another SIG playback session. For example, if user 1502 selects to join an existing SIG playback session already started by user 1504, user 1502's mixed reality system can start the SIG playback session such that user 1502's SIG playback session may start from the current time of user 1504's SIG playback session instead of the start of the SIG recording. In some embodiments, users 1502 and 1504 can view virtual representation 1506, which moves and creates virtual markings 1508, in a synchronized manner. In some embodiments, the synchronized playback session can take into account the individual users' line of sight of the synchronized SIG playback. For example, even if the virtual representations displayed to each user are time-synchronized with each other, a user standing to the left of the virtual representation may see the left side of the virtual representation, and a user standing to the right of the virtual representation may see the right side of the virtual representation.

[0092] In some embodiments, synchronized SIG playback sessions can facilitate group activity. For example, a group of newly hired employees can go through a training session together if each employee participates in the same SIG playback session. In some embodiments, it may be beneficial for users to experience the same SIG playback session simultaneously and facilitate interaction between users (e.g., users can discuss the training material with each other as they watch it).

[0093] SIG Network

[0094] In some embodiments, a network can include one or more mobile devices. In some embodiments, the network can organize the one or more mobile devices into one or more groups. In some embodiments, a user can choose to publish a SIG record to one or more groups so that mobile devices in the one or more groups receive notifications, but mobile devices outside the one or more groups do not. In some embodiments, only mobile devices in the one or more groups to which the SIG record is published can view the SIG record. In some embodiments, groups can be based on users rather than devices (e.g., authorized users can view SIG records on any of the user's devices). In some embodiments, social media features can be integrated. For example, SIG records can be published directly to a social media website. In another example, social media groups (e.g., friends) can be inherited from a social media website, for example, using an API. The SIG record can then be published to a specific social media group.

[0095] In some embodiments, spatial content can be shared across different platforms. For example, spatial content created on a mixed reality system can be shared with a non-mixed reality computer system. In some embodiments, spatial content can generate a virtual environment based on a physical environment and add virtual content to the virtual environment. Benefits of doing so include expanding the audience for spatial content to users who may not have access to a mixed reality system. Users may view the spatial content, for example, on a 2D screen.

[0096] In some embodiments, the spatial content can be stored on a mixed reality system. In some embodiments, the mixed reality system can communicate with a server and forward the SIG records to the server. In some embodiments, the SIG records can be stored on a mobile device as an intermediary between the mixed reality system and the server (e.g., the mixed reality system forwards the SIG records to the mobile device, which then forwards the SIG records to the server). In some embodiments, the mixed reality system communicates with the mobile device during the SIG recording session such that the SIG records are initially stored on the mobile device. In some embodiments, the mixed reality system communicates with the server during the SIG recording session such that the SIG records are initially stored on the server. In some embodiments, the server can provide the SIG records to the mobile device for use in the SIG playback session. In some embodiments, the server can provide the SIG records to the mixed reality system for use in the SIG playback session. It is also contemplated that other network topologies (e.g., peer-to-peer networking or ad-hoc networks of mixed reality devices) can be used as well.

[0097] Determining a location for SIG recording

[0098] 16 illustrates an example process 1600 for determining a location for a SIG recording. In step 1602, an initial location for the SIG recording can be estimated. The initial estimate can be obtained while a user is recording a SIG event. For example, a user using the mixed reality system 603 can record a SIG event, and during recording, the mixed reality system 603 can estimate an initial location for the SIG event. The mixed reality system 603 can estimate the initial location using any suitable method. In some embodiments, such as when the mixed reality system 603 is equipped with a GPS sensor, the initial location can be identified based on the GPS coordinates of the mixed reality system 603.

[0099] In some embodiments, a remote user who is not physically present in the mixed reality environment records or uploads SIG events to a location in the mixed reality environment, i.e., the SIG events can be uploaded to a virtual location associated with the remote user, regardless of the remote user's physical location, as described herein.

[0100] In some embodiments, such as when the mixed reality system 603 lacks GPS functionality, other methods of locating the SIG event can be used. For example, in some embodiments, the mixed reality system 603 can use known locations for wireless access points to identify a location, e.g., using received signal strength indications, signatures, angle of arrival, and / or time-of-flight techniques (e.g., via triangulation). In some embodiments, the location of the SIG event relative to the mixed reality system 603 is known or may be determined. The location for the wireless access point can be determined at the time of installation of the wireless access point, e.g., using a device that may detect the wireless access point while determining the device's location using GPS. In some embodiments, the device may be a device, such as a mobile phone, that communicates with the mixed reality system 603. The location can then be associated with the wireless access point. In some embodiments, the mixed reality system 603 can use GPS signals from nearby the device directly to estimate an initial location for the SIG event. In some embodiments, the mixed reality system 603 can identify a location using known locations for cellular base stations. In some embodiments, the mixed reality system 603 can identify a location using a known location relative to a 5G base station. For example, the 5G base station can have a known location, and the location of the mixed reality system 603 can be determined relative to the 5G base station (e.g., by determining a vector from the 5G base station to the mixed reality system 603 through beamforming).

[0101] In some embodiments, location can be estimated visually, for example, using object recognition of "landmark" objects with known locations. For example, while recording a SIG event, the mixed reality system 603 can recognize one or more rooms with associated location information (which may be predetermined). The mixed reality system 603 can use the associated location information of one or more recognized objects (e.g., a table, a chair, a corner, etc.) to determine an initial location estimate for the SIG recording. In some embodiments, the object can be tagged (e.g., using a QR code or barcode) such that location information about the object is embedded in the tag. The tag, and therefore the embedded location information about the object, can be detected using the camera of the mixed reality system 603.

[0102] Other methods for estimating an initial location for a SIG record can also be used. In some embodiments, a different device in communication with the mixed reality system 603 can be used to estimate an initial location for a SIG record. For example, a mobile device in communication with the mixed reality system 603 can estimate its location using wireless access points, GPS signals, cellular base stations, 5G base stations, and / or recognized objects. The location of the mobile device can then be associated with the SIG record. If the mobile device is located near the mixed reality system 603 (e.g., a user of the mixed reality system 603 carries the mobile device), the location of the mobile device can be directly associated with the SIG record. If the mobile device is located away from the mixed reality system 603, its location can be used alone or in conjunction with other information (e.g., wireless access points, GPS, cellular base stations, 5G base stations, and / or recognized objects) to estimate an initial location for the SIG record. In some embodiments, the estimated location for the SIG record can include location data (e.g., longitude and latitude coordinates) and an accuracy estimate. The accuracy estimate may include, for example, a radius that is estimated to contain a location with a certain confidence. In some embodiments, the position of the mixed reality system 603 relative to the mobile device is known or may be determined.

[0103] In step 1604 of the example process shown in FIG. 16 , a SIG recording can be accessed. For example, a user can play the SIG recording using a mobile device or a mixed reality system. In some embodiments, the user can be prompted to access the SIG recording, or the SIG recording may automatically start playing. For example, a user may walk near a location that has an associated SIG recording. The user may be notified (e.g., through the user's mobile device or mixed reality system) that a SIG recording is present at or near the user's location, and the user may be prompted to view the SIG recording, or the SIG recording may automatically start playing. In some embodiments, a SIG record with an associated location can be loaded (e.g., pushed) onto the user's device (e.g., from a remote server) when the SIG recording is present at or near the user's associated location (e.g., within a certain radius of the associated location).

[0104] In some embodiments, the associated location of the SIG record can be the estimated initial location determined in step 1602. In step 1606, the mobile device can estimate its own location (e.g., during a playback session) using any suitable method, including, but not limited to, triangulation using wireless access points, GPS signals, cellular base stations, 5G base stations, and / or recognized objects. Similarly, a mixed reality system 603 can also be used to play back the SIG recording, and the mixed reality system 603 can estimate its location using any suitable means. The new location estimate and a new accuracy estimate associated with the new location estimate can be stored on the mobile device, the mixed reality system, and / or a server communicatively coupled to the mobile device or the mixed reality system.

[0105] In step 1608, the location of the SIG record may be updated. In some embodiments, the new location estimate determined in step 1606 may be more accurate than the initial location estimate determined in step 1602. For example, the initial location determined in step 1602 may be estimated using triangulation from wireless access points, which may produce a location estimate with poor accuracy (e.g., a large radius that is confident in the location). The mobile device may produce a more accurate location estimate in step 1606 by using a more accurate location method (e.g., using a GPS signal). In some embodiments, the location associated with the SIG record may be updated to the more accurate location. For example, in step 1608, a determination may be made regarding the accuracy of two or more location estimates (e.g., identifying the location estimate with the smallest confidence radius). The location estimate with the highest accuracy may be associated with the SIG record and may be used to share the SIG record. For example, the location estimate with the highest accuracy may be used to notify a user when the user walks near the location of the estimate with the highest accuracy. The location estimate with the highest accuracy can also be used to mark the SIG record on a map so that the user can navigate to the SIG record.

[0106] In some embodiments, one or more location estimates determined in steps 1602 or 1606 may be combined to update the location of the SIG record. For example, one or more location estimates may be averaged to form a combined location estimate, and the combined location estimate may be used to share the SIG record. In some embodiments, one or more location estimates may be weighted to form a combined location estimate. In some embodiments, a location estimate with a higher estimated accuracy may be weighted more heavily than a location estimate with a lower estimated accuracy. In some embodiments, a location estimate that is more recent in time may be weighted more heavily than a location estimate that is older in time.

[0107] In some embodiments, steps 1604, 1606, and 1608 can be performed each time a SIG record is accessed. For example, each time a mobile device or mixed reality system plays a SIG record, a background process can estimate a location for the mobile device or mixed reality system during the playback session. The location estimate can then be stored on the mobile device, the mixed reality system, and / or a server communicatively coupled to the mobile device or mixed reality system. The location associated with the SIG record can be updated based on new location estimates received (e.g., by selecting the location estimate with the highest accuracy or by calculating a new weighted average based on the new location estimates).

[0108] With regard to the systems and methods described above, elements of the systems and methods can be implemented by one or more computer processors (e.g., CPUs or DSPs) as needed. The present disclosure is not limited to any particular configuration of computer hardware, including computer processors, used to implement these elements. In some cases, multiple computer systems can be employed to implement the systems and methods described above. For example, a first computer processor (e.g., a processor of a wearable device coupled to one or more microphones) can be utilized to receive input microphone signals and perform initial processing of those signals (e.g., signal conditioning and / or segmentation as described above). A second (and perhaps more computationally powerful) processor can then be utilized to perform more computationally intensive processing, such as determining probability values ​​associated with speech segments in those signals. Another computing device, such as a cloud server, can host a speech processing engine to which the input signals are ultimately provided. Other suitable configurations will be apparent and are within the scope of the present disclosure.

[0109] Although the disclosed embodiments have been fully described with reference to the accompanying drawings, it should be noted that various changes and modifications will be apparent to those skilled in the art. For example, elements of one or more implementations may be combined, deleted, modified, or supplemented to form further implementations. Such changes and modifications are to be understood as being included within the scope of the disclosed embodiments as defined by the appended claims.

Claims

1. A method comprising: receiving an input from a first user, the input indicating a movement of the first user; determining, via a sensor, a location associated with the first user; generating content corresponding to the input, the content being associated with the movement of the first user and further associated with the location associated with the first user; Determining a first location of a second user; determining whether to present the content to the second user based on the location associated with the first user and further based on the first location of the second user; In accordance with a decision to present the content to the second user, presenting the content to the second user via a display, wherein presenting the content to the second user includes presenting a video of the movements of the first user; determining whether a second location of the second user is within a threshold distance from the first location of the second user; Pausing presentation of the content to the second user in accordance with determining that the second location of the second user is not within the threshold distance; and not pausing the presentation of the content to the second user pursuant to determining that the second location of the second user is within the threshold distance. and in accordance with a decision not to present the content to the second user, not presenting the content to the second user; determining a new location, the new location being based on the location associated with the first user and further based on the first location of the second user; Associating the new location with the content; and A method comprising:

2. The method of claim 1, wherein the content includes a video of the movement of the first user.

3. The method of claim 1, wherein the input from the first user is received during a recording session.

4. The sensor is a sensor of a first wearable head device associated with the first user; The method of claim 1 , wherein the display is a display of a second wearable head device associated with the second user.

5. The method of claim 1, wherein the input includes at least one of head movement of the first user, hand movement of the first user, and handheld controller input.

6. The method of claim 1, wherein the sensor includes at least one of a position sensor and a camera.

7. The method of claim 1, wherein the location associated with the first user is associated with a virtual marking.

8. The method of claim 1, wherein determining whether to present the content to the second user includes determining, via a second sensor, whether the field of view of the second user includes the location associated with the first user.

9. In accordance with a decision not to present the content to the second user, determining whether the second user's field of view includes the location associated with the first user; and presenting second content via the display in accordance with determining that the second user's field of view includes the location associated with the first user. The method of claim 1 , further comprising: determining a location of a third user while the content is presented to the second user; and presenting the content to the third user via a second display in parallel with presenting the content to the second user pursuant to determining that the location of the third user is within a second threshold distance from the first location of the second user; and The method of claim 1 further comprising:

11. The method of claim 1, wherein the locations associated with the first user include one or more of the first user's physical location and the first user's virtual location. determining a location of a third user; associating the content with the location of the third user; and The method of claim 1 further comprising:

13. A system comprising: A sensor, The display and one or more processors and Equipped with The one or more processors are configured to perform a method, the method comprising: receiving an input from a first user, the input indicating a movement of the first user; determining, via the sensor, a location associated with the first user; generating content corresponding to the input, the content being associated with the movement of the first user and further associated with the location associated with the first user; Determining a first location of a second user; determining whether to present the content to the second user based on the location associated with the first user and further based on the first location of the second user; In accordance with a decision to present the content to the second user, presenting the content to the second user via the display, wherein presenting the content to the second user includes presenting a video of the movements of the first user; determining whether a second location of the second user is within a threshold distance from the first location of the second user; Pausing presentation of the content to the second user in accordance with determining that the second location of the second user is not within the threshold distance; and not pausing the presentation of the content to the second user pursuant to determining that the second location of the second user is within the threshold distance. and in accordance with a decision not to present the content to the second user, not presenting the content to the second user; determining a new location, the new location being based on the location associated with the first user and further based on the first location of the second user; Associating the new location with the content; and Including, the system.

14. The sensor is a sensor of a first wearable head device associated with the first user; The system of claim 13 , wherein the display is a display of a second wearable head device associated with the second user.

15. The system of claim 13, wherein the input includes at least one of head movement of the first user, hand movement of the first user, and handheld controller input.

16. The system described in claim 13, wherein the sensor includes at least one of a position sensor and a camera.

17. Further comprising a second sensor, 14. The system of claim 13, wherein determining whether to present the content to the second user includes determining, via the second sensor, whether a field of view of the second user includes the location associated with the first user.

18. The system of claim 13, wherein the locations associated with the first user include one or more of the first user's physical location and the first user's virtual location.

19. The method comprising: Determining a location of a third user; associating the content with the location of the third user; and The system of claim 13 further comprising:

20. A non-transitory computer-readable medium, the non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method, the method comprising: receiving an input from a first user, the input indicating a movement of the first user; determining, via a sensor, a location associated with the first user; generating content corresponding to the input, the content being associated with the movement of the first user and further associated with the location associated with the first user; Determining a first location of a second user; determining whether to present the content to the second user based on the location associated with the first user and further based on the first location of the second user; In accordance with a decision to present the content to the second user, presenting the content to the second user via a display, wherein presenting the content to the second user includes presenting a video of the movements of the first user; determining whether a second location of the second user is within a threshold distance from the first location of the second user; Pausing presentation of the content to the second user in accordance with determining that the second location of the second user is not within the threshold distance; and not pausing the presentation of the content to the second user pursuant to determining that the second location of the second user is within the threshold distance. and in accordance with a decision not to present the content to the second user, not presenting the content to the second user; determining a new location, the new location being based on the location associated with the first user and further based on the first location of the second user; Associating the new location with the content; and 1. A non-transitory computer-readable medium comprising:

Images