Non-uniform volumetric rendering

Abstract

Examples of the present disclosure describe systems and methods for recording augmented reality and mixed reality experiences. In an example method, an image of a real environment is received via a camera of a wearable head device. A pose of the wearable head device is estimated, and a first image of a virtual environment is generated based on the pose. A second image of the virtual environment is generated based on the pose, where the second image of the virtual environment includes a larger field of view than a field of view of the first image of the virtual environment. A combined image is generated based on the second image of the virtual environment and the image of the real environment.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of U.S. Provisional Application No. 62 / 926,306, filed October 25, 2019, the entire disclosure of which is incorporated herein by reference for all purposes. Technical Field

[0003] This disclosure generally relates to systems and methods for rendering and displaying visual information, and more particularly to systems and methods for rendering and displaying visual information in mixed reality environments. Background Technology

[0004] Virtual environments are ubiquitous in computing environments, finding applications in video games (where a virtual environment can represent a game world); maps (where a virtual environment can represent terrain to navigate); simulations (where a virtual environment can simulate a real environment); digital storytelling (where virtual characters can interact with each other within a virtual environment); and many other applications. Modern computer users are generally comfortable perceiving and interacting with virtual environments. However, the user experience with virtual environments can be limited by the technologies used to render them. For example, traditional displays (e.g., 2D screens) and audio systems (e.g., fixed speakers) may not be able to render virtual environments in a way that creates a compelling, realistic, and immersive experience.

[0005] Virtual reality (“VR”), augmented reality (“AR”), mixed reality (“MR”), and related technologies (collectively, “XR”) share the ability to present to a user of an XR system sensory information corresponding to a virtual environment represented by data from a computer system. This disclosure considers the distinctions between VR, AR, and MR systems (although some systems may be classified as VR in one aspect (e.g., visual aspect) and simultaneously as AR or MR in another aspect (e.g., audio aspect)). As used herein, a VR system presents a virtual environment that replaces the user’s real environment in at least one aspect; for example, a VR system may present the user with a view of the virtual environment while blurring his or her view of the real environment, for example, by using light occlusion on a head-mounted display. Similarly, a VR system may present the user with audio corresponding to the virtual environment while blocking (attenuating) audio from the real environment.

[0006] VR systems can encounter various drawbacks due to replacing the user's real-world environment with a virtual one. One drawback is motion sickness, which can occur when the user's field of view in the virtual environment no longer corresponds to the state of their inner ear, which detects balance and orientation in the real environment (not the virtual one). Similarly, users may experience disorientation in VR environments where their own bodies and limbs (the view users rely on to feel "grounded" in the real environment) cannot be directly seen. Another drawback is the computational burden (e.g., storage, processing power) of VR systems that must render a complete 3D virtual environment, especially in real-time applications seeking to immerse users. Likewise, such environments may need to meet very high standards of realism to be considered immersive, as users are often sensitive to even minor imperfections in virtual environments—any imperfection can disrupt the user's sense of immersion. Furthermore, another drawback of VR systems is that such applications cannot utilize the extensive sensory data of the real environment, such as the various visual and auditory experiences people have in the real world. A related drawback is that VR systems may struggle to create shared environments where multiple users can interact, as users sharing a physical space in the real environment may not be able to directly see or interact with each other in the virtual environment.

[0007] As used herein, AR systems present a virtual environment that overlaps or overlays the real environment in at least one aspect. For example, an AR system can present a view of a virtual environment overlaid on a view of the user's real environment, such as using a transmissive head-mounted display that presents the displayed image while allowing light to pass through the display into the user's eyes. Similarly, an AR system can present the user with audio corresponding to the virtual environment while mixing audio from the real environment. Likewise, as used herein, MR systems present a virtual environment that overlaps or overlays the real environment in at least one aspect, as do AR systems, and can additionally allow the virtual environment in an MR system to interact with the real environment in at least one aspect. For example, a virtual character in a virtual environment might toggle a light switch in the real environment, causing the corresponding light bulb in the real environment to turn on or off. As another example, a virtual character can react to audio signals in the real environment (e.g., with facial expressions). By maintaining the presentation of the real environment, AR and MR systems can avoid some of the aforementioned drawbacks of VR systems; for example, motion sickness is reduced because visual cues from the real environment (including the user's own body) remain visible, and such systems do not require presenting a fully realized 3D environment to the user for immersion. In addition, AR and MR systems can utilize real-world sensory input (such as landscapes, objects, and other user views and sounds) to create new applications that enhance that input.

[0008] It may be desirable to capture and record MR experiences so that they can be shared with other users. Compared to simple video recording systems, MR systems (especially wearable head-mounted devices) can be better positioned to record environments (real and virtual) in the same way that the user experiences the environment. Providing a first-hand view of the user experience in both real and virtual environments can make the recording feel more immersive, as if the viewer were with the original user while the recording was being done. Recording MR experiences can include combining recordings of real and virtual environments. The recorded MR content can be used for both casual content sharing and professional content creation. For example, it may be desirable to share casual MR experiences (e.g., concerts) with other users who can view the recorded MR content on a 2D screen or MR system. Professional content creators may also want to record MR content (e.g., advertisements) for display on other MR systems or 2D screens. However, recording MR experiences may not be as straightforward as recording simple 2D video, at least because the camera used to record the real environment may be offset from the user's eye position to avoid obstructing their view. However, virtual content rendered for the user can be rendered from the user's eye's perspective. When combining two recording streams, this offset can create a difference in perspective between what the camera captures and the virtual content rendered for the user. Furthermore, the camera may record at a different resolution and / or aspect ratio than the displayed virtual content. Therefore, combining recordings of the real environment with recordings of the virtual environment can lead to poor integration between the real and virtual content (e.g., the real and virtual content may be presented from different angles). Thus, it may be desirable to create an accurate MR recording system that simulates the first-person experience and displays the recorded MR content in an integrated and accurate manner. Summary of the Invention

[0009] This disclosure describes examples of systems and methods for recording augmented reality and mixed reality experiences. In the example method, an image of a real-world environment is received via a camera on a wearable head-mounted device. The pose of the wearable head-mounted device is estimated, and a first image of a virtual environment is generated based on the pose. A second image of the virtual environment is generated based on the pose, wherein the second image of the virtual environment includes a larger field of view than the first image of the virtual environment. A combined image is generated based on the second image of the virtual environment and the image of the real-world environment. Attached Figure Description

[0010] Figures 1A to 1C An example mixed reality environment according to one or more embodiments of the present disclosure is shown.

[0011] Figures 2A to 2D Components of an example mixed reality system according to one or more embodiments of the present disclosure are shown. The mixed reality system can be used to generate and interact with a mixed reality environment.

[0012] Figure 3A An example mixed reality handheld controller, which can be used to provide input to a mixed reality environment according to one or more embodiments of the present disclosure, is shown.

[0013] Figure 3B An example auxiliary unit that can be used with an example mixed reality system according to one or more embodiments of the present disclosure is shown.

[0014] Figure 4 An example functional block diagram of an example mixed reality system according to one or more embodiments of the present disclosure is shown.

[0015] Figures 5A to 5B Examples of mixed reality recordings according to one or more embodiments of this disclosure are shown.

[0016] Figure 6 Examples of mixed reality recordings according to one or more embodiments of this disclosure are shown.

[0017] Figure 7 Examples of mixed reality recordings according to one or more embodiments of this disclosure are shown.

[0018] Figure 8 Examples of mixed reality recordings according to one or more embodiments of this disclosure are shown.

[0019] Figure 9 Examples of mixed reality recordings according to one or more embodiments of this disclosure are shown. Detailed Implementation

[0020] In the following description of the examples, reference is made to the accompanying drawings, which form part of the drawings and illustrate, by way of illustration, examples that can be practiced. It should be understood that other examples may be used and structural changes may be made without departing from the scope of the disclosed examples.

[0021] Mixed Reality Environment

[0022] Like everyone else, users of mixed reality systems exist within a real-world environment—that is, the three-dimensional portion of the “real world” and all its contents that are perceptible to the user. For example, users use ordinary human senses to perceive the real-world environment—sight, sound, touch, taste, smell—and interact with it by moving their own bodies within it. A location within the real-world environment can be described as coordinates in a coordinate space; for example, coordinates may include latitude, longitude, and altitude relative to sea level; distance from a reference point in three orthogonal dimensions; or other suitable values. Similarly, a vector can describe a quantity having direction and magnitude in coordinate space.

[0023] Computing devices may maintain a representation of a virtual environment, for example, in memory associated with the device. As used herein, a virtual environment is a computational representation of a three-dimensional space. A virtual environment may include a representation of any object, action, signal, parameter, coordinate, vector, or other characteristic associated with that space. In some examples, the circuitry of the computing device (e.g., a processor) may maintain and update the state of the virtual environment; that is, the processor may determine the state of the virtual environment at a second time t1 at a first time t0 based on data associated with the virtual environment and / or input provided by the user. For example, if an object in the virtual environment is located at a first coordinate at time t0 and has some programmed physical parameter (e.g., mass, coefficient of friction); and input received from the user indicates that a force should be applied to the object in a direction vector; the processor may apply laws of kinematics to determine the position of the object at time t1 using fundamental mechanics. The processor may use any suitable information known about the virtual environment and / or any suitable input to determine the state of the virtual environment at time t1. While maintaining and updating the state of the virtual environment, the processor may execute any suitable software, including software related to the creation and deletion of virtual objects in the virtual environment; software for defining the behavior of virtual objects or characters in the virtual environment (e.g., scripts); software for defining the behavior of signals (e.g., audio signals) in the virtual environment; software for creating and updating parameters associated with the virtual environment; software for generating audio signals in the virtual environment; software for processing inputs and outputs; software for implementing network operations; software for applying asset data (e.g., animation data of virtual objects moving over time); or many other possibilities.

[0024] Output devices (such as displays or speakers) can present any or all aspects of a virtual environment to a user. For example, a virtual environment may include virtual objects that can be presented to the user (which may include representations of inanimate objects; people; animals; light, etc.). A processor can determine a view of the virtual environment (e.g., corresponding to a “camera” with a coordinate origin, view axes, and view frustum); and render a visual scene of the virtual environment corresponding to that view to the display. Any suitable rendering technique can be used for this purpose. In some examples, the visual scene may include only some virtual objects in the virtual environment and exclude certain other virtual objects. Similarly, a virtual environment may include audio aspects that can be presented to the user as one or more audio signals. For example, virtual objects in the virtual environment may generate sounds originating from the object's location coordinates (e.g., a virtual character can speak or cause sound effects); or the virtual environment may be associated with musical cues or ambient sounds that may or may not be associated with a specific location. The processor can determine the audio signal corresponding to the “listener” coordinates—for example, a composite audio signal corresponding to sounds in a virtual environment, and mix and process it to simulate the audio signal that will be heard by the listener at the listener coordinates—and present the audio signal to the user via one or more speakers.

[0025] Because the virtual environment exists solely as a computational construct, users may not directly perceive it using their ordinary senses. Instead, users can only perceive it indirectly, such as through displays, speakers, haptic output devices, etc. Similarly, users may not directly touch, manipulate, or otherwise interact with the virtual environment; however, they can provide input data to a processor that can update the virtual environment using this data via input devices or sensors. For example, a camera sensor can provide optical data indicating that a user is attempting to move an object in the virtual environment, and the processor can use this data to cause the object to respond accordingly within the virtual environment.

[0026] Mixed reality systems can present a mixed reality environment (“MRE”) to a user, combining aspects of a real and a virtual environment, for example, using a transmissive display and / or one or more speakers (which may be incorporated, for example, into a wearable head-mounted device). In some embodiments, one or more speakers may be external to the head-mounted wearable unit. As used herein, an MRE is a simultaneous representation of a real environment and a corresponding virtual environment. In some examples, the corresponding real and virtual environments share a single coordinate space; in some examples, the real coordinate space and the corresponding virtual coordinate space are correlated with each other through a transformation matrix (or other suitable representation). Thus, a single coordinate (in some examples, together with the transformation matrix) can define a first position in the real environment and a second corresponding position in the virtual environment; and vice versa.

[0027] In MRE, virtual objects (e.g., in a virtual environment associated with the MRE) can correspond to real objects (e.g., in a real environment associated with the MRE). For example, if the real environment of the MRE includes a real lamppost (real object) at location coordinates, then the virtual environment of the MRE can include a virtual lamppost (virtual object) at the corresponding location coordinates. As used herein, real objects combined with their corresponding virtual objects constitute "mixed reality objects." Virtual objects do not need to perfectly match or align with their corresponding real objects. In some examples, virtual objects can be simplified versions of their corresponding real objects. For example, if the real environment includes a real lamppost, the corresponding virtual object can include a cylinder with roughly the same height and radius as the real lamppost (reflecting that the lamppost can be roughly cylindrical in shape). Simplifying virtual objects in this way allows for computational efficiency and simplifies the computations performed on such virtual objects. Furthermore, in some examples of the MRE, not all real objects in the real environment can be associated with corresponding virtual objects. Similarly, in some examples of the MRE, not all virtual objects in the virtual environment can be associated with corresponding real objects. That is, some virtual objects can exist only in the virtual environment of MRE without any real-world counterparts.

[0028] In some examples, virtual objects can possess characteristics that differ (sometimes drastically) from their corresponding real-world counterparts. For instance, while a real-world environment in an MRE might include a green, two-armed cactus—a thorny, inanimate object—the corresponding virtual object in the MRE could possess characteristics of a green, two-armed virtual character with human facial features and aggressive behavior. In this example, the virtual object resembles its real-world counterpart in some characteristics (color, number of arms); but differs from the real object in other characteristics (facial features, personality). In this way, virtual objects have the potential to represent real-world objects in a creative, abstract, exaggerated, or imaginative way; or to impart behavior (e.g., human personality) to other inanimate real-world objects. In some examples, virtual objects can be purely imaginary creations without any real-world counterpart (e.g., a virtual monster in a virtual environment, perhaps in a location corresponding to an empty space in a real environment).

[0029] Compared to VR systems, which present a virtual environment to the user while blurring the real environment, mixed reality (MRE) systems offer the advantage of maintaining the perceptibility of the real environment while the virtual environment is presented. Therefore, users of MRE systems can experience and interact with the corresponding virtual environment using visual and audio cues associated with the real environment. For example, while a VR system user might strive to perceive or interact with virtual objects displayed in the virtual environment—because, as mentioned above, the user cannot directly perceive or interact with the virtual environment—an MR system user can find interacting with virtual objects intuitive and natural by seeing, hearing, and touching corresponding real objects in their own real environment. This level of interactivity can enhance the user's sense of immersion, connection, and engagement with the virtual environment. Similarly, by simultaneously presenting the real and virtual environments, MRE systems can reduce the negative psychological sensations (e.g., cognitive dissonance) and negative physical sensations (e.g., motion sickness) associated with VR systems. Furthermore, MRE systems offer numerous possibilities for applications that can enhance or alter our experience of the real world.

[0030] Figure 1A An exemplary real-world environment 100 is shown where user 110 uses a mixed reality system 112. The mixed reality system 112 may include a display (e.g., a transmissive display) and one or more speakers, as well as one or more sensors (e.g., cameras), such as those described below. The real-world environment 100 shown includes a rectangular room 104A where user 110 stands; and real-world objects 122A (lamp), 124A (table), 126A (sofa), and 128A (painting). Room 104A also includes positional coordinates 106, which can be considered the origin of the real-world environment 100. Figure 1AAs shown, an environment / world coordinate system 108 (including x-axis 108X, y-axis 108Y, and z-axis 108Z) with its origin at point 106 (world coordinates) can define the coordinate space for the real environment 100. In some embodiments, the origin 106 of the environment / world coordinate system 108 can correspond to the location where the mixed reality environment 112 is powered on. In some embodiments, the origin 106 of the environment / world coordinate system 108 can be reset during operation. In some examples, the user 110 can be considered a real object in the real environment 100; similarly, body parts of the user 110 (e.g., hands, feet) can be considered real objects in the real environment 100. In some examples, a user / audience / head coordinate system 114 (including x-axis 114X, y-axis 114Y, and z-axis 114Z) with its origin at point 115 (e.g., user / audience / head coordinates) can define the coordinate space for the user / audience / head where the mixed reality system 112 is located. The origin 115 of the user / audience / head coordinate system 114 can be defined relative to one or more components of the mixed reality system 112. For example, the origin 115 of the user / audience / head coordinate system 114 can be defined relative to the display of the mixed reality system 112, such as during the initial calibration of the mixed reality system 112. Matrices (which may include translation matrices and quaternion matrices or other rotation matrices) or other suitable representations can characterize the transformation between the user / audience / head coordinate system 114 space and the environment / world coordinate system 108 space. In some embodiments, left ear coordinates 116 and right ear coordinates 117 can be defined relative to the origin 115 of the user / audience / head coordinate system 114. Matrices (which may include translation matrices and quaternion matrices or other rotation matrices) or other suitable representations can characterize the transformation between the left ear coordinates 116 and right ear coordinates 117 and the user / audience / head coordinate system 114 space. The user / audience / head coordinate system 114 can simplify the representation of position relative to the user's head or head-mounted device (e.g., relative to the environment / world coordinate system 108). Using simultaneous localization and mapping (SLAM), visual ranging, or other techniques, the transformation between the user coordinate system 114 and the environment coordinate system 108 can be determined and updated in real time.

[0031] Figure 1BAn exemplary virtual environment 130 corresponding to the real environment 100 is shown. The virtual environment 130 shown includes a virtual rectangular room 104B corresponding to the real rectangular room 104A; a virtual object 122B corresponding to the real object 122A; a virtual object 124B corresponding to the real object 124A; and a virtual object 126B corresponding to the real object 126A. Metadata associated with the virtual objects 122B, 124B, and 126B may include information derived from the corresponding real objects 122A, 124A, and 126A. The virtual environment 130 additionally includes a virtual monster 132, which does not correspond to any real object in the real environment 100. The real object 128A in the real environment 100 does not correspond to any virtual object in the virtual environment 130. A persistent coordinate system 133 (including x-axis 133X, y-axis 133Y, and z-axis 133Z) with its origin at point 134 (persistent coordinates) can define the coordinate space used for the virtual content. The origin 134 of the persistent coordinate system 133 can be defined relative to / about one or more real objects (such as real object 126A). Matrices (which may include translation matrices and quaternion matrices or other rotation matrices) or other suitable representations can characterize the transformation between the space of the persistent coordinate system 133 and the environment / world coordinate system 108. In some embodiments, each of the virtual objects 122B, 124B, 126B, and 132 can have its own persistent coordinate point relative to the origin 134 of the persistent coordinate system 133. In some embodiments, multiple persistent coordinate systems may exist, and each of the virtual objects 122B, 124B, 126B, and 132 can have its own persistent coordinate point relative to one or more persistent coordinate systems.

[0032] Compared to Figure 1A and Figure 1B The environment / world coordinate system 108 defines a shared coordinate space for both the real environment 100 and the virtual environment 130. In the example shown, the coordinate space has its origin at point 106. Furthermore, the coordinate space is defined by the same three orthogonal axes (108X, 108Y, 108Z). Therefore, a first position in the real environment 100 and a second corresponding position in the virtual environment 130 can be described relative to the same coordinate system. This simplifies the identification and display of corresponding positions in the real and virtual environments, as the same coordinates can be used to identify both positions. However, in some examples, the corresponding real and virtual environments do not need to use a shared coordinate space. For example, in some examples (not shown), matrices (which may include translation matrices and quaternion matrices or other rotation matrices) or other suitable representations can characterize the transformation between the real environment coordinate space and the virtual environment coordinate space.

[0033] Figure 1CAn exemplary MRE 150 is shown that simultaneously presents aspects of a real environment 100 and a virtual environment 130 to a user 110 via a mixed reality system 112. In the example shown, the MRE 150 simultaneously presents to the user 110 real objects 122A, 124A, 126A, and 128A from the real environment 100 (e.g., via the transmissive portion of the display of the mixed reality system 112); and virtual objects 122B, 124B, 126B, and 132 from the virtual environment 130 (e.g., via the active display portion of the display of the mixed reality system 112). As described above, the origin 106 serves as the origin for the coordinate space corresponding to the MRE 150, and the coordinate system 108 defines the x-axis, y-axis, and z-axis for the coordinate space.

[0034] In the illustrated example, the mixed reality objects include corresponding real and virtual objects occupying corresponding positions in coordinate space 108 (i.e., 122A / 122B, 124A / 124B, 126A / 126B). In some examples, both the real and virtual objects can be simultaneously visible to the user 110. This is desirable in instances where the virtual object presents information designed to enhance the view of the corresponding real object (such as in a museum application where the virtual object presents a missing piece of an ancient damaged sculpture). In some examples, the virtual object (122B, 124B, and / or 126B) can be displayed (e.g., via active pixelation occlusion using a pixelation occlusion shutter) to occlude the corresponding real object (122A, 124A, and / or 126A). This is desirable in instances where the virtual object acts as a visual replacement for the corresponding real object (such as in an interactive story application where an inanimate real object becomes a "living" character).

[0035] In some examples, real-world objects (e.g., 122A, 124A, 126A) may be associated with virtual content or helper data that may not necessarily constitute a virtual object. Virtual content or helper data can facilitate the processing or disposal of virtual objects in a mixed reality environment. For example, such virtual content may include a two-dimensional representation of: the corresponding real-world object; a custom asset type associated with the corresponding real-world object; or statistical data associated with the corresponding real-world object. This information enables or facilitates computations involving real-world objects without incurring unnecessary computational overhead.

[0036] In some examples, the presentation described above may also include audio aspects. For example, in MRE 150, the virtual monster 132 may be associated with one or more audio signals, such as footsteps generated as the monster walks around MRE 150. As further described below, the processor of Mixed Reality System 112 may calculate the audio signal corresponding to the mixture and processing of all such sounds in MRE 150, and present the audio signal to user 110 via one or more speakers included in Mixed Reality System 112 and / or one or more external speakers.

[0037] Example Mixed Reality System

[0038] Example mixed reality system 112 may include a wearable head-mounted device (e.g., a wearable augmented reality or mixed reality head-mounted device) comprising: a display (which may include left and right transmissive displays, which may be near-eye displays, and associated components for coupling light from the displays to the user's eyes); left and right speakers (e.g., positioned adjacent to the user's left and right ears, respectively); an inertial measurement unit (IMU) (e.g., mounted to a side arm of the head-mounted device); an orthogonal coil electromagnetic receiver (e.g., mounted to the left side arm); left and right cameras (e.g., depth (time-of-flight) cameras) oriented away from the user; and left and right eye cameras oriented towards the user (e.g., for detecting the user's eye movements). However, mixed reality system 112 may incorporate any suitable display technology and any suitable sensors (e.g., optical, infrared, acoustic, LiDAR, EOG, GPS, magnetic). Furthermore, mixed reality system 112 may include networking features (e.g., Wi-Fi capability) for communicating with other devices and systems, including other mixed reality systems. The mixed reality system 112 may also include a battery (which may be housed in an auxiliary unit, such as a belt bag designed to be worn around the user's waist), a processor, and memory. The wearable head-mounted device of the mixed reality system 112 may include tracking components, such as an IMU or other suitable sensors, configured to output a set of coordinates of the wearable head-mounted device relative to the user's environment. In some examples, the tracking components may provide input to a processor performing simultaneous localization and mapping (SLAM) and / or visual ranging algorithms. In some examples, the mixed reality system 112 may also include a handheld controller 300 and / or an auxiliary unit 320, which may be a wearable belt bag, as further described below.

[0039] Figures 2A to 2D Components of an exemplary mixed reality system 200 (which may correspond to mixed reality system 112) that can be used to present an MRE (which may correspond to MRE 150) or other virtual environment to a user are shown. Figure 2A A perspective view of a wearable head-mounted device 2102 included in an example mixed reality system 200 is shown. Figure 2B A top view of a wearable head device 2102 worn on a user's head 2202 is shown. Figure 2C A front view of the wearable head device 2102 is shown. Figure 2D A side view of an example eyepiece 2110 of a wearable head device 2102 is shown. Figures 2A to 2C As shown, the example wearable head device 2102 includes an exemplary left eyepiece (e.g., a left transparent waveguide set eyepiece) 2108 and an exemplary right eyepiece (e.g., a right transparent waveguide set eyepiece) 2110. Each eyepiece 2108 and 2110 may include: a transmission element through which the real environment can be visible; and a display element for presenting a display superimposed on the real environment (e.g., via image-by-image modulated light). In some examples, such a display element may include surface diffraction optics for controlling the flow of image-by-image modulated light. For example, the left eyepiece 2108 may include a left-coupled-in grating set 2112, a left orthogonal pupil expansion (OPE) grating set 2120, and a left-exit (output) pupil expansion (EPE) grating set 2122. Similarly, the right eyepiece 2110 may include a right-coupled-in grating set 2118, a right OPE grating set 2114, and a right EPE grating set 2116. Image-by-image modulated light can be delivered to the user's eye via coupling gratings 2112 and 2118, OPEs 2114 and 2120, and EPEs 2116 and 2122. Each coupling grating set 2112, 2118 can be configured to deflect light toward its corresponding OPE grating set 2120, 2114. Each OPE grating set 2120, 2114 can be designed to deflect light downwards incrementally toward its associated EPEs 2122, 2116, thereby horizontally extending the forming exit pupil. Each EPE 2122, 2116 can be configured to incrementally redirect at least a portion of the light received from its corresponding OPE grating set 2120, 2114 outwards to a user eyebox location (not shown) defined behind eyepieces 2108, 2110, vertically extending the exit pupil formed at the eyebox. Alternatively, instead of coupling grating sets 2112 and 2118, OPE grating sets 2114 and 2120, and EPE grating sets 2116 and 2122, eyepieces 2108 and 2110 may include gratings and / or other arrangements for controlling the refractive and reflective characteristics of the image-by-image modulated light coupled to the user's eye.

[0040] In some examples, the wearable head device 2102 may include a left support arm 2130 and a right support arm 2132, wherein the left support arm 2130 includes a left speaker 2134 and the right support arm 2132 includes a right speaker 2136. An orthogonal coil electromagnetic receiver 2138 may be positioned in the left support arm or at another suitable location within the wearable head unit 2102. An inertial measurement unit (IMU) 2140 may be positioned in the right support arm 2132 or at another suitable location within the wearable head device 2102. The wearable head device 2102 may also include a left depth (e.g., time-of-flight) camera 2142 and a right depth camera 2144. The depth cameras 2142 and 2144 may be suitably oriented in different directions to cover a wider field of view together.

[0041] exist Figures 2A to 2D In the example shown, the left source 2124 of the image-modulated light can be optically coupled to the left eyepiece 2108 via the left coupling grating set 2112, and the right source 2126 of the image-modulated light can be optically coupled to the right eyepiece 2110 via the right coupling grating set 2118. The image-modulated light sources 2124, 2126 may include, for example, fiber optic scanners; projectors including electro-optic modulators such as digital light processing (DLP) chips or liquid crystal on silicon (LCoS) modulators; or emitting displays such as micro-LED or micro-organic light-emitting diode (μOLED) panels, which are coupled to the coupling grating sets 2112, 2118 using one or more lenses on each side. The input coupling grating sets 2112, 2118 can deflect light from the image-modulated light sources 2124, 2126 to an angle greater than the critical angle for total internal reflection (TIR) ​​for the eyepieces 2108, 2110. OPE grating sets 2114 and 2120 incrementally deflect light propagating through TIR downwards toward EPE grating sets 2116 and 2122. EPE grating sets 2116 and 2122 incrementally couple light toward the user's face, including the pupils of the user's eyes.

[0042] In some examples, such as Figure 2DAs shown, each of the left eyepiece 2108 and the right eyepiece 2110 includes multiple waveguides 2402. For example, each eyepiece 2108, 2110 may include multiple individual waveguides, each dedicated to a corresponding color channel (e.g., red, blue, and green). In some examples, each eyepiece 2108, 2110 may include multiple sets of such waveguides, wherein each set is configured to impart a different wavefront curvature to the emitted light. The wavefront curvature may be convex relative to the user's eye, for example, to present a virtual object positioned at a distance in front of the user (e.g., by a distance corresponding to the inverse of the wavefront curvature). In some examples, the EPE grating sets 2116, 2122 may include curved grating recesses to achieve a convex wavefront curvature by varying the Poynting vector of the emitted light across each EPE.

[0043] In some examples, to create a perception that the displayed content is three-dimensional, stereoscopically accommodated left and right eye images can be presented to the user via image-by-image light modulators 2124, 2126 and eyepieces 2108, 2110. The perceived realism of the presentation of the three-dimensional virtual object can be enhanced by selecting waveguides (and therefore corresponding wavefront curvatures) so that the virtual object is displayed at a distance close to that indicated by the stereoscopic left and right images. This technique can also reduce motion sickness experienced by some users, which can be caused by the discrepancy between the depth perception cues provided by the stereoscopic left and right eye images and the autoacoustic function of the human eye (e.g., object distance-related focus).

[0044] Figure 2D An edge-facing view is shown from the top of the right eyepiece 2110 of the example wearable head device 2102. Figure 2D As shown, multiple waveguides 2402 may include a first subset of three waveguides 2404 and a second subset of three waveguides 2406. The two subsets of waveguides 2404 and 2406 can be distinguished by different EPE gratings, each with a different line curvature to impart different wavefront curvatures to the emitted light. Within each subset of waveguides 2404 and 2406, each waveguide can be used to couple a different spectral channel (e.g., one of the red, green, and blue spectral channels) to the user's right eye 2206. Although not shown in the diagram... Figure 2D However, the structure of the left eyepiece 2108 is similar to that of the right eyepiece 2110.

[0045] Figure 3AAn exemplary handheld controller component 300 of a mixed reality system 200 is shown. In some examples, the handheld controller 300 includes a grip portion 346 and one or more buttons 350 disposed along a top surface 348. In some examples, the buttons 350 may be configured to serve as optical tracking targets, for example, for combining a camera or other optical sensor (which may be mounted in the head unit of the mixed reality system 200 (e.g., wearable head device 2102)) to track the six degrees of freedom (6DOF) motion of the handheld controller 300. In some examples, the handheld controller 300 includes a tracking component (e.g., an IMU or other suitable sensor) for detecting position or orientation (such as position or orientation relative to the wearable head device 2102). In some examples, such a tracking component may be positioned in the handle of the handheld controller 300 and / or may be mechanically coupled to the handheld controller. The handheld controller 300 may be configured to provide one or more output signals corresponding to one or more of the button press states; or the position, orientation, and / or movement of the handheld controller 300 (e.g., via an IMU). Such an output signal can be used as an input to the processor of the mixed reality system 200. Such input can correspond to the position, orientation, and / or movement of the handheld controller (e.g., by extension, to the position, orientation, and / or movement of the user's hand holding the controller). Such input can also correspond to the user pressing a button 350.

[0046] Figure 3B An exemplary auxiliary unit 320 of a mixed reality system 200 is shown. The auxiliary unit 320 may include a battery that provides power to the operating system 200 and may include a processor for executing programs on the operating system 200. As shown, the example auxiliary unit 320 includes a chip 2128, such as for attaching the auxiliary unit 320 to a user's belt. Other form factors suited to the auxiliary unit 320 and will be apparent, including form factors not involving mounting the unit to the user's belt. In some examples, the auxiliary unit 320 is coupled to a wearable head device 2102 via a multi-conduit cable, which may include, for example, wires and optical fibers. A wireless connection between the auxiliary unit 320 and the wearable head device 2102 may also be used.

[0047] In some examples, the mixed reality system 200 may include one or more microphones that detect sound and provide corresponding signals to the mixed reality system. In some examples, the microphone may be attached to or integrated with a wearable head device 2102 and configured to detect the user's voice. In some examples, the microphone may be attached to or integrated with a handheld controller 300 and / or auxiliary unit 320. Such a microphone may be configured to detect ambient sound, ambient noise, the user's or a third party's voice, or other sounds.

[0048] Figure 4 An exemplary functional block diagram is shown that can correspond to an exemplary mixed reality system, such as the mixed reality system 200 described above (which can correspond to the mixed reality system 112 relative to FIG1). Figure 4 As shown, the example handheld controller 400B (which may correspond to the handheld controller 300 (“Totem”)) includes a totem-to-wear head-mounted device six-DOF (6DOF) totem subsystem 404A, and the example wearable head-mounted device 400A (which may correspond to the wearable head-mounted device 2102) includes a totem-to-wearable head-mounted device 6DOF subsystem 404B. In the example, the 6DOF totem subsystem 404A and the 6DOF subsystem 404B cooperate to determine the six coordinates of the handheld controller 400B relative to the wearable head-mounted device 400A (e.g., offsets in three translational directions and rotations along three axes). The six degrees of freedom can be represented relative to the coordinate system of the wearable head-mounted device 400A. The three translational offsets can be represented as X, Y, and Z offsets in such a coordinate system, as translation matrices, or as some other representation. The rotational degrees of freedom can be represented as a sequence of yaw, pitch, and roll rotations, as rotation matrices, as quaternions, or as some other representation. In some examples, a wearable head device 400A; one or more depth cameras 444 (and / or one or more non-depth cameras) included in the wearable head device 400A; and / or one or more optical targets (e.g., a button 350 of the handheld controller 400B as described above, or a dedicated optical target included in the handheld controller 400B) can be used for 6DOF tracking. In some examples, the handheld controller 400B may include a camera as described above; and the wearable head device 400A may include optical targets for combined camera optical tracking. In some examples, the wearable head device 400A and the handheld controller 400B each include a set of three orthogonally oriented solenoids for wirelessly transmitting and receiving three distinguishable signals. The 6DOF of the wearable head device 400A relative to the handheld controller 400B can be determined by measuring the relative amplitudes of the three distinguishable signals received in each of the coils used for receiving. In addition, the 6DOF totem subsystem 404A may include an inertial measurement unit (IMU) that can be used to provide improved accuracy and / or more timely information about the rapid movement of the handheld controller 400B.

[0049] In some examples, it may become necessary to transform coordinates from a local coordinate space (e.g., a coordinate space fixed relative to the wearable head-mounted device 400A) to an inertial coordinate space (e.g., a coordinate space fixed relative to the real environment), for example, to compensate for movement of the wearable head-mounted device 400A relative to coordinate system 108. For example, such a transformation may be necessary for the display of the wearable head-mounted device 400A to present virtual objects at a desired position and orientation relative to the real environment (e.g., a virtual person sitting in a real chair, facing forward, regardless of the position and orientation of the wearable head-mounted device), rather than at a fixed position and orientation on the display (e.g., at the same position in the lower right corner of the display), to maintain the illusion that the virtual objects exist in the real environment (and, for example, not appear unnaturally positioned in the real environment when the wearable head-mounted device 400A moves and rotates). In some examples, the compensation transformation between coordinate spaces can be determined by processing images from depth camera 444 using SLAM and / or visual ranging procedures to determine the transformation of the wearable head-mounted device 400A relative to coordinate system 108. Figure 4 In the example shown, a depth camera 444 is coupled to a SLAM / visual ranging block 406 and can provide images to the block 406. SLAM / visual ranging block 406 implementations may include a processor configured to process the image and determine the position and orientation of the user's head, which can then be used to identify a transformation between the head coordinate space and another coordinate space (e.g., inertial coordinate space). Similarly, in some examples, an additional source of information about the user's head pose and position is obtained from an IMU 409. Information from the IMU 409 can be integrated with information from the SLAM / visual ranging block 406 to provide improved accuracy and / or more timely information regarding the user's head pose and position.

[0050] In some examples, depth camera 444 can feed 3D images to gesture tracker 411, which can be implemented in the processor of wearable head device 400A. Gesture tracker 411 can identify a user's gestures, for example, by matching the 3D image received from depth camera 444 with a stored pattern representing the gesture. Other suitable techniques for identifying a user's gestures will be apparent.

[0051] In some examples, one or more processors 416 may be configured to receive data from a 6DOF helmet subsystem 404B, IMU 409, SLAM / visual ranging block 406, depth camera 444, and / or gesture tracker 411 of a wearable head-mounted device. Processor 416 may also send and receive control signals from a 6DOF totem system 404A. Processor 416 may be wirelessly coupled to the 6DOF totem system 404A, as in an example where the handheld controller 400B is unrestricted. Processor 416 may also communicate with additional components such as audio-visual content memory 418, graphics processing unit (GPU) 420, and / or digital signal processor (DSP) audio spatializer 422. DSP audio spatializer 422 may be coupled to head-related transfer function (HRTF) memory 425. GPU 420 may include a left channel output coupled to a left source 424 of per-image modulated light and a right channel output coupled to a right source 426 of per-image modulated light. GPU 420 can output stereoscopic image data to image-by-image modulated light sources 424 and 426, as shown above relative to... Figure 2A-2D As described, the DSP audio sound field locator 422 can output audio to the left speaker 412 and / or the right speaker 414. The DSP audio sound field locator 422 can receive input from the processor 419 indicating a direction vector from the user to a virtual sound source (which can be moved by the user, e.g., via a handheld controller 320). Based on the direction vector, the DSP audio sound field locator 422 can determine the corresponding HRTF (e.g., by accessing the HRTF or by interpolating multiple HRTFs). The DSP audio sound field locator 422 can then apply the determined HRTF to an audio signal, such as an audio signal corresponding to a virtual sound generated by a virtual object. This can enhance the credibility and realism of the virtual sound by interpolating the user's relative position and orientation to the virtual sound in the mixed reality environment—that is, by presenting a virtual sound that matches the user's expectation of what the virtual sound will sound like (if it is a real sound in a real environment).

[0052] In some examples, such as Figure 4 As shown, one or more of the processor 416, GPU 420, DSP audio sound field locator 422, HRTF memory 425, and audio / visual content memory 418 may be included in the auxiliary unit 400C (which may correspond to the auxiliary unit 320 described above). The auxiliary unit 400C may include a battery 427 that powers its components and / or powers the wearable head device 400A or the handheld controller 400B. Including such components in an auxiliary unit that can be mounted to the user's waist can limit the size and weight of the wearable head device 400A, which in turn can reduce fatigue in the user's head and neck.

[0053] Although Figure 4 Elements corresponding to various components of the example mixed reality system are presented, but various other suitable arrangements of these components will become apparent to those skilled in the art. For example, Figure 4 The elements presented as associated with the auxiliary unit 400C may alternatively be associated with the wearable head device 400A or the handheld controller 400B. Furthermore, some mixed reality systems may completely omit the handheld controller 400B or the auxiliary unit 400C. Such changes and modifications will be understood to be included within the scope of the disclosed examples.

[0054] Mixed Reality Video Capture

[0055] Capturing and recording AR / MR experiences can be more complex than capturing 2D video and may present new challenges. For example, a 2D video recording system may only capture one stream of information (e.g., what the optical / image sensors on the video recording system can capture). AR / MR experiences, on the other hand, may have two or more streams of information. For example, one stream could be 2D video, while another could be rendered virtual content. Sometimes three or more streams can be used (e.g., when capturing stereoscopic video from two 2D videos along with accompanying virtual content). Combining multiple streams into an accurate AR / MR recording may involve taking into account the perspective differences between the streams. Processing the combination process in a computationally efficient manner may be further beneficial, especially for portable AR / MR systems that may have limited battery power and / or processing capabilities.

[0056] Users may have various reasons to capture and record AR / MR content. In some embodiments, users may wish to record personal AR / MR content and share it on social media (e.g., AR / MR recordings of children's games). In some embodiments, users may wish to record and share commercial AR / MR content (e.g., advertisements). Recorded AR / MR content can be shared to 2D screens (e.g., by combining a virtual content stream with a real content stream) and other MR systems (e.g., by combining two virtual content streams with two real content streams). While the original user of the AR / MR system may experience the real environment through transmissive optics (e.g., through partially transmissive lenses), viewers of the recorded AR / MR content may not be in the same real environment. Therefore, to present the complete AR / MR recording to the viewer, video capture of the real environment may be included so that virtual content can be overlaid on the video capture of the real environment.

[0057] Capturing MRE in video recordings can present additional challenges beyond simply displaying it. For example, mixed reality video capture can include two recording streams: a real-world environment recording stream and a virtual-world environment recording stream. The real-world environment recording stream can be captured using one or more optical sensors (e.g., cameras). Mounting one or more optical sensors on an AR / MR system (e.g., MR systems 112, 200) can be advantageous. In embodiments where the AR / MR system is a wearable head-mounted device, one or more optical sensors can be mounted on the wearable head-mounted device, close to the user's eyes, and optionally facing in the same direction as the user's eyes. In the described mounting system, one or more optical sensors can be positioned to capture video recordings of the real-world environment in a manner very close to the user's experience of the real environment.

[0058] Virtual environment recording streams can be captured by recording the virtual content displayed to the user. In some embodiments, AR / MR systems (e.g., MR systems 112, 200) can simultaneously display two virtual content streams to the user (e.g., one stream per eye) to provide a stereoscopic view. Each virtual render can be rendered from a slightly different perspective (corresponding to the difference in perspective between the user's left and right eyes), and this difference in perspective can make the virtual content appear stereoscopic. The virtual environment recording stream can include one or two virtual renders displayed to the user's eyes. In some embodiments, the virtual renders can be overlaid with the real environment recording stream to simulate the MRE experienced by the user when using the AR / MR system. It may be advantageous to use virtual renders rendered for the user's eyes, where the user's eyes are physically closest to one or more optical sensors used for the real environment recording stream. The physical proximity of the user's eyes (and the virtual renders rendered for that corresponding perspective) to the optical sensors can make the combination process more accurate because the shift in perspective is minimal.

[0059] One problem that can arise from combining real-environment and virtual-environment recording streams is that the two streams may not have the same resolution and / or aspect ratio. For example, a real-environment recording stream might capture video at a resolution of 1920x1080 pixels, but a virtual-environment recording stream might only render virtual content at a resolution of 1280x720 pixels. Another potential issue is that the real-environment and virtual-environment recording streams may have different fields of view. For example, a camera might use a larger field of view to capture the real-environment recording stream than it does to render the virtual-environment recording stream. Therefore, more information (e.g., a larger field of view) can be captured in the real-environment recording stream than in the virtual-environment recording stream. In some embodiments, real-environment content located outside the field of view of the virtual content may not have a corresponding rendered virtual content in the virtual rendering. When real-environment and virtual-environment recording streams are merged, an artificial “boundary” may appear in the combined video, outside which virtual content is not displayed, but real content is still recorded and displayed. This boundary can be discordant for viewers because it disrupts the integration of virtual and real content. It can further present inaccuracies to the viewer. For example, a real object recorded by a camera may have corresponding virtual content (e.g., information overlay). However, if the real object is outside the virtual rendering's field of view but still within the camera's field of view, the corresponding virtual content may not be displayed to the viewer until the real object moves within the artificial boundary. This behavior can confuse viewers of AR / MR recordings because the virtual content corresponding to the real object may or may not be visible depending on the real object's position relative to the boundary.

[0060] Figure 5AExample embodiments are shown that can mitigate and / or eliminate the boundary between virtual and real content. The example embodiments shown can be implemented using one or more components of a mixed reality system, such as one or more of the wearable head-mounted device 2102, handheld controller 300, and auxiliary unit 320 of the example mixed reality system 200 described above. In the depicted embodiment, an RGB camera may be located near the user's left eye. The RGB camera can record a view 502 that may overlap and / or include a virtually rendered view 504 (and / or view 506) presented to the user's left eye. View 502 may be larger than view 504, so combining (e.g., overlaying or superimposing) view 504 on view 502 may result in an artificial boundary 503. In this embodiment, AR / MR recordings may display real content throughout view 502, but virtual content may only be displayed within boundary 503. Boundary 503 can be jarring for viewers because virtual content may exist outside boundary 503 (at least within view 502), but it won't be displayed until it moves within boundary 503. One solution to mitigate and / or eliminate boundary 503 is to crop view 502 to the size and position of boundary 503. A cropped view can produce virtual content of the same size as the real content, creating an integrated and immersive viewing experience. However, cropping the view reduces the viewer's visible area by "discarding" information captured by the RGB camera.

[0061] Figure 5BAn example embodiment for merging perspectives from views 502 and 504 is shown. The example embodiment shown can be implemented using one or more components of a mixed reality system, such as one or more of the wearable head-mounted device 2102, handheld controller 300, and auxiliary unit 320 of the example mixed reality system 200 described above. Because the RGB camera may have a focal point 508 offset from focal point 510 (which may correspond to the center of the user's left eye), view 502 may present a different perspective than view 504, resulting in virtual content potentially not being "synchronized" (e.g., aligned and / or aligned) with real content. This can be problematic because the virtual content may not correspond correctly to the real content (e.g., if the face of a virtual object is visible from the user's eye perspective but should not be visible from the RGB camera's perspective), potentially breaking the immersive experience of AR / MR recording. One solution could be to use planar projection to "map" view 504 to view 502. In planar projection, one or more points P1 in view 504 can be mapped onto a line between focal point 508 and one or more corresponding points P2 in view 502 (and vice versa). This can be accomplished using computer vision algorithms that determine that observations at one or more points P1 are observations with the same features as observations at one or more points P2. Combining these mappings can mitigate viewpoint differences caused by different focal points 508, 510. In some embodiments, the combined AR / MR recordings can display virtual content in a manner substantially similar to that shown to a user of the AR / MR system. However, as mentioned above, in some embodiments, useful information captured by the RGB camera can be discarded to avoid artificial boundaries between real and virtual content.

[0062] Figure 6An example embodiment for generating AR / MR recordings is shown. The example embodiment shown can be implemented using one or more components of a mixed reality system, such as one or more of the wearable head-mounted device 2102, handheld controller 300, and auxiliary unit 320 of the example mixed reality system 200 described above. The AR / MR system (e.g., MR systems 112, 200) can render virtual content from one or more perspectives (e.g., the virtual content can be rendered once from the user's left eye perspective and once from the user's right eye perspective). The AR / MR system can also render the virtual content a third time from the perspective of an RGB camera, which may be located near the user's eye (e.g., the user's left eye) and facing approximately the same direction as the user's forward gaze. In some embodiments, this third-pass rendering of the virtual content can be rendered at the same resolution, aspect ratio, and / or field of view as the RGB camera. This can allow the generation of AR / MR recordings using synchronized virtual and real content without discarding any information from the RGB camera or the virtual rendering. However, in some embodiments, third-pass rendering of the virtual content can be computationally expensive. Third-pass rendering can include a full geometry transfer, which may include full draw calls and / or rasterization. Further details can be found in U.S. Patent Application No. 15 / 924,144, the contents of which are incorporated herein by reference in their entirety. Performing third-channel rendering on a portable AR / MR system with limited computing resources may not be feasible (e.g., because the AR / MR system has already rendered twice for the user's two eyes). Due to the increased computational load, it would consume additional power, negatively impacting the battery life of AR / MR systems that rely on portable power supplies.

[0063] Figure 7An example embodiment for generating AR / MR recordings is shown. The example embodiment shown can be implemented using one or more components of a mixed reality system, such as one or more of the wearable headpiece 2102, handheld controller 300, and auxiliary unit 320 of the example mixed reality system 200 described above. The AR / MR system (e.g., MR systems 112, 200) can render virtual content from one or more perspectives (e.g., the virtual content can be rendered once from the user's left eye perspective and once from the user's right eye perspective). In some embodiments, the two virtual renderings (e.g., once for each eye) may be non-uniform. For example, the virtual rendering for the user's left eye may be rendered to view 702, which may differ from the virtual rendering for the user's right eye, which may be rendered to view 706. View 706 may be approximately the same size as the user's right eye's field of view (e.g., therefore, virtual content that will not be displayed to the user is not rendered, thus minimizing computational load). View 702 may be larger than view 706 and may be rendered to a larger field of view than the user's left eye can see. View 702 may include view 704, which may be the field of view available to the user's left eye (e.g., all virtual content displayed to the user may be included in view 702 except for some virtual content that may not be displayed to the user). View 702 may optionally have the same or substantially the same field of view as view 708, which may be the field of view available to an RGB camera (e.g., an RGB camera located near the user's left eye and facing the same direction as the user's forward gaze).

[0064] During AR / MR recording, views 702 and 706 can be rendered (e.g., via MR systems 112, 200). View 702 can be cropped into view 704, which may have the same field of view as view 706, but may be rendered from a slightly different perspective (e.g., from the left eye's perspective instead of the right eye's). Views 704 and 706 can display virtual content to the user, and can create stereoscopic images to simulate 3D virtual content. An RGB camera can simultaneously record video of the real environment from a perspective similar to the user's (e.g., the RGB camera may be positioned near one of the user's eyes). This real-environment recording stream can then be combined with view 702 (which may be a virtual environment recording stream) to create an MR recording. In some embodiments, the virtual environment recording stream from view 702 may have approximately the same resolution, aspect ratio, and / or field of view as view 708. Therefore, AR / MR recordings that combine the real-environment recording stream with the virtual environment recording stream can discard little or no information captured by the RGB camera and / or virtual rendering. Rendering non-uniform stereoscopic virtual content can also be beneficial because extending one of the two rendering passes is computationally more efficient than using the full third pass for rendering. Extending an existing rendering pass may only include additional rasterization for the extra pixels, but may not include the full geometry pass (unlike third-pass rendering in some embodiments).

[0065] Figure 8An example process for generating AR / MR recordings is illustrated, which can be augmented video capture. The illustrated example process can be implemented using one or more components of a mixed reality system, such as one or more of the wearable head-mounted device 2102, handheld controller 300, and auxiliary unit 320 of the example mixed reality system 200 described above. In step 802, a request to begin augmented video capture can be received. The request can be user-initiated (e.g., the user selects to start recording, or the user speaks aloud to start recording, which the AR / MR system can process as a user request, or the user performs a gesture, which the AR / MR system can process as a user request) or automatically processed (e.g., a computer vision algorithm detects a machine learning algorithm to determine a scene the user might be interested in, and then the MR system automatically begins recording accordingly). In step 804, the user's pose and the pose of the RGB camera can be determined / estimated. The user's pose can include information about the user's position and / or orientation in the three-dimensional environment. The user's pose may be the same as or similar to the pose of the AR / MR system (e.g., if the AR / MR system is a wearable head-mounted device fixed relative to the user's head). The pose can be estimated using SLAM, visual-inertial ranging, and / or other suitable methods. The pose of the RGB camera can be independently determined / estimated and / or derived from the pose of the AR / MR system (which can approximate the user's pose). For example, the RGB camera can be mounted at a fixed position on the AR / MR system, a position known to the AR / MR system. The AR / MR system can determine / estimate the pose of the RGB camera by combining its own estimated pose with the position of the RGB camera relative to the AR / MR system. In step 806, virtual content can be rendered based on a pose estimate for the user's first eye (e.g., the user's right eye) for the user and / or the AR / MR system. The virtual content can be rendered by simulating a first virtual camera, which can be positioned at the center of the user's eye (e.g., the user's first eye).

[0066] In step 808, virtual content can be rendered for the user's second eye (e.g., the user's left eye) based on pose estimation for the user and / or the AR / MR system and the RGB camera. The virtual content rendered for the user's second eye can simulate a second virtual camera, which can be located at the center of the user's eye (e.g., the user's second eye and / or left eye). Compared to the virtual content rendered for the user's first eye, the virtual content rendered for the user's second eye can include variations other than changes in viewpoint and / or virtual camera position. In some embodiments, the frame buffer size (e.g., image plane size) of the second virtual camera (e.g., a virtual camera to be composited with the RGB camera, or also a virtual camera for the user's left eye) can be increased compared to the frame buffer size of the virtual camera for the user's first eye (e.g., the first virtual camera). The aspect ratio of the frame buffer for the second eye can match the aspect ratio of the frame buffer for the first eye (although other aspect ratios can be used). In some embodiments, the projection matrix of the virtual camera for the second eye can be changed such that a larger field of view is rendered compared to the virtual camera for the first eye. It may be desirable to change the projection matrix of the virtual camera to accommodate a larger framebuffer size. Failure to increase the field of view and / or change the projection matrix of the virtual camera to accommodate a larger framebuffer size may result in a mismatch between the virtual content and the virtual content presented to different eyes, potentially reducing and / or eliminating stereoscopic effects. In some embodiments, an increased field of view and / or a changed projection matrix can produce a virtual render that has the same field of view as an RGB camera located near the user's second eye. In some embodiments, the view matrix may remain unchanged for a virtual camera to be composited with an RGB camera (e.g., the virtual camera can still render from the same location with the same vector to the target). In some embodiments, a minimum clipping distance (e.g., the minimum distance from the user's eye to the virtual content below which the virtual content may not be rendered) can prevent virtual objects from being rendered in areas that may be outside the field of view of the RGB camera, but within the field of view of the virtual camera located at the center of the user's second eye.

[0067] In step 810, the virtual content rendered for the second eye in step 808 can be cropped to be displayed to the user's second eye. In some embodiments, the center of the expanded field of view can also be the center of the cropped field of view presented to the user's second eye. The cropped field of view can be the same size as the field of view of the virtual content rendered for the user's first eye. The virtual content rendered for the user's first eye and the cropped virtual content rendered for the user's second eye can be combined to create a stereoscopic effect for the virtual content displayed to the user, which can simulate the three-dimensionality of the virtual content. In step 812, the virtual content can be displayed to the user. In some embodiments, steps 806, 808, and / or 810 can occur simultaneously or substantially simultaneously. For example, an AR / MR system can render virtual content for the user's first and second eyes, crop the virtual content rendered for the user's second eye, and display stereoscopic virtual content to the user with little or no noticeable delay (e.g., the stereoscopic virtual content can track real content as the user looks around the real environment).

[0068] In step 814, RGB video capture can begin (e.g., using an RGB camera mounted on an AR / MR system). In step 816, an expanded (e.g., uncropped) virtual content field of view can be composited with the captured RGB video. For example, the expanded virtual content field of view can be projected (e.g., using planar projection) onto the captured RGB video. In some embodiments, the captured RGB video can be projected (e.g., using planar projection) onto the virtual content field of view. In some embodiments, the virtual content field of view can be the same size as the field of view of the RGB camera. This may result in very little loss of RGB camera information and / or virtual information. In some embodiments, some information may be lost due to the projection process. In some embodiments, the virtual content field of view can be larger than the field of view of the RGB camera, so the loss due to projection may not require discarding information captured by the RGB camera. In some embodiments, the virtual content field of view can be smaller than the field of view of the RGB camera, so the loss due to projection may require discarding information captured by the RGB camera. In step 818, the enhanced video capture can be composited and stored. In some embodiments, the enhanced video capture can display more information than presented to the user recording the AR / MR system. For example, AR / MR recordings can have an expanded field of view compared to the user's field of view when using an AR / MR system.

[0069] Figure 9An example system for compositing and storing augmented video captures is shown, which may correspond to steps 816 and 818. The example system shown can be implemented using one or more components of a mixed reality system, such as one or more of the wearable head-mounted device 2102, handheld controller 300, and auxiliary unit 320 of the example mixed reality system 200 described above. Camera image frames from camera 902 can be sent to compositor 906 along with pose data for the user's head / eyes and / or pose data for the camera from IMU 904. The pose data can then be sent from compositor 906 to AR / MR application and / or graphics processing unit (“GPU”) 908, which can pull information (e.g., models) from a 3D database. AR / MR application 908 can send an expanded virtual render to compositor 906. Compositor 906 can then composite the expanded virtual render with the camera image frames from camera 902 (e.g., using planar projection) and send the composited frame to media encoder 912. The media encoder 912 can then send the encoded image frames to the recording database 914 for storage.

[0070] While an embodiment with a single RGB camera has been described above, it is also envisioned that the systems and methods described herein can be applied to any number of RGB cameras. For example, two RGB cameras could be used, with one mounted near each of the user's eyes. The virtual rendering for each eye could then be either expanded (although only a limited field of view might be presented to the user during use) or projected into the field of view of the RGB cameras (the RGB camera images could also be projected onto the virtual images). This could allow for stereoscopic augmented video capture and could be particularly well-suited for playback on another MR system, providing a more immersive experience than playback on a 2D screen. It is also envisioned that the systems and methods could be used to shift the RGB camera's viewpoint to be as close as possible to the user's eye. For example, mirrors and / or other optical elements could be used to reflect light so that the RGB cameras view content from the same (or substantially the same) viewpoint as the user's eyes. In addition to synthesized video, augmented video capture could also include synthesized audio. For example, virtual audio signals could be recorded and synthesized with recorded real audio signals (which could be captured by one or more microphones on the MR system). The synthesized audio could then be further synchronized with the synthesized video.

[0071] Although the disclosed examples have been fully described with reference to the accompanying drawings, it should be noted that various changes and modifications will become apparent to those skilled in the art. For example, elements of one or more embodiments may be combined, deleted, modified, or supplemented to form further embodiments. Such changes and modifications will be understood to be included within the scope of the disclosed examples as defined by the appended claims.

Claims

1. A method for rendering and displaying visual information, comprising: Images of the real environment are received via a camera on a wearable head-mounted device; The pose of the wearable head device is estimated based on the output of the camera by performing one or more of simultaneous localization and mapping (SLAM) and visual inertial ranging. Based on the posture, a first image of the virtual environment is generated; Based on the pose, a second image of the virtual environment is generated, wherein the second image of the virtual environment includes a larger field of view than the first image of the virtual environment; and A combined image is generated based on the second image from the virtual environment and the image from the real environment, wherein, Generating the combined image includes: projecting the second image of the virtual environment onto the image of the real environment by performing a planar projection, wherein the planar projection includes mapping points in the second image of the virtual environment to corresponding points in the image of the real environment, and The second image of the virtual environment has the same field of view as the image of the real environment, in order to mitigate the loss of information present in the image of the real environment and / or the second image of the virtual environment during the generation of the combined image.

2. The method according to claim 1, further comprising: The second image of the virtual environment is cropped, wherein the field of view of the cropped second image of the virtual environment is the same as the field of view of the first image of the virtual environment, and the cropped second image of the virtual environment and the first image of the virtual environment are combined to generate a stereoscopic image.

3. The method according to claim 1, further comprising: A stereoscopic image is presented via the display of the wearable head device based on the first image of the virtual environment and the cropped second image of the virtual environment.

4. The method according to claim 3, wherein, The cropped second image of the virtual environment includes the center of the cropped second image of the virtual environment, wherein the center of the cropped second image of the virtual environment includes the center of the second image, and wherein the center of the cropped second image of the virtual environment is the same as the center of the second image.

5. The method according to claim 1, wherein, The second image of the virtual environment includes an aspect ratio, and the image of the real environment includes the same aspect ratio.

6. The method according to claim 1, wherein, The second image of the virtual environment includes an aspect ratio, and wherein the first image of the virtual environment includes the same aspect ratio.

7. The method according to claim 1, wherein, The first image of the virtual environment includes a first frame buffer size, wherein the second image of the virtual environment includes a second frame buffer size, and wherein the second frame buffer size is greater than the first frame buffer size.

8. The method according to claim 1, wherein, The first image of the virtual environment includes a frame buffer size, and the cropped second image of the virtual environment includes the same frame buffer size.

9. The method according to claim 1, wherein, The first image of the virtual environment includes a first resolution, wherein the second image of the virtual environment includes a second resolution, and wherein the second resolution is greater than the first resolution.

10. The method according to claim 1, wherein, The camera is configured to be placed to the left of the user's left eye or to the right of the user's right eye.

11. The method according to claim 1, wherein, The camera is configured to be placed between the user's left eye and the user's right eye.

12. The method according to claim 1, further comprising: The combined image is presented to a second device, different from the wearable head-mounted device, via a display.

13. The method according to claim 1, wherein, The first image of the virtual environment is a simulated virtual camera positioned at the center of the user's first eye on the wearable head device.

14. The method according to claim 1, wherein, The second image in the virtual environment simulates a second virtual camera positioned at the center of the user's second eye.

15. A system for rendering and displaying visual information, comprising: Camera for wearable head-mounted devices; One or more processors are configured to perform the method according to any one of claims 1-14.

16. A non-transient computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the method according to any one of claims 1-14.

Images