Method and system for generating and displaying 3D videos in virtual, augmented, or mixed reality environment

Abstract

To provide a method and system for generating and displaying 3D videos in a virtual, augmented, or mixed reality environment.SOLUTION: Disclosed is an approach for displaying 3D videos in a virtual reality (VR) and / or augmented reality (AR) system 100. The 3D videos include 3D animated objects that escape from a screen of a display system 104, interact with objects within a VR and / or AR environment, and are interactive with a user 50, based on user input corresponding to decisions elected by the user at certain portions of the 3D video, such that a different storyline and possibly a different conclusion may result for the 3D video. The 3D video is a 3D icon displayed within a portal of a final 3D render world.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates generally to systems and methods configured to facilitate interactive virtual, augmented, or mixed reality environments for one or more users. [Background technology]

[0002] Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality" (VR), "augmented reality" (AR), or "mixed reality" (MR) experiences, in which digitally reproduced images or parts thereof are presented to a user in a manner that appears or may be perceived as real. VR scenarios typically involve the presentation of digital or virtual image information without transparency to other actual real-world visual inputs, while AR / MR scenarios typically involve the presentation of digital or virtual image information as an extension to the visualization of the real world around the user, such that the digital or virtual image (e.g., virtual content) may appear to be part of the real world. Mixed reality may be similar to augmented reality scenarios, since mixed reality scenarios have a blend of real and digital worlds. AR may hereafter refer to both AR and MR.

[0003] In a VR / AR environment, there is all kinds of virtual content that needs to be displayed. Virtual content such as 3D videos may need to be displayed within the VR / AR environment, where a user may be able to move freely around the VR / AR environment and view the 3D videos being displayed. Current techniques for rendering 3D videos on a display screen may not look realistic when displayed within the VR / AR environment, especially when these 3D videos may be interactive with the user.

[0004] Therefore, there is a need for an approach for generating and displaying 3D video in a more realistic manner within a VR / AR environment. Summary of the Invention [Means for solving the problem]

[0005] According to some embodiments, one embodiment is directed to a method for displaying 3D video extending beyond a surface of a display screen in a virtual and / or augmented reality environment, the method including identifying the 3D video, generating one or more 3D models corresponding to the 3D video, and rendering the one or more 3D models at an appropriate trigger time along with rendering the 3D video.

[0006] In one or more embodiments, the 3D video is a stereoscopic 3D video, and the one or more 3D models are generated with animation. The method may further include displaying an animation of the one or more 3D models at least partially outside a display screen of the 3D video, the animation of the one or more 3D models appearing to move out of the plane of the 3D video and into the user's 3D environment. The animation of the one or more 3D models may appear to move out of the plane of the 3D video and into the user's 3D environment. The method may still further include rendering the one or more 3D models on one or more video panes, and displaying the one or more video panes simultaneously for one or more viewing orientations.

[0007] In one or more embodiments, the stereoscopic 3D video is displayed with one or more 3D models, and the depth information comprises a distance from a user of a virtual and / or augmented reality device to the 3D video, and the separate one or more 3D models are displayed within the virtual and / or augmented reality environment. The 3D video may have a video file format comprising control data that instructs a video player to render the one or more 3D models at appropriate trigger times along with the rendering of the 3D video. The rendering of the one or more 3D models is based at least in part on the voxel-based video stream.

[0008] Some embodiments are directed to a display system for displaying 3D video extending beyond a surface of a display screen, the system including an augmented reality head mounted display system and one or more modules for processing data, the one or more modules being stored in one or more memories, the one or more modules may be configured to perform a step of identifying the 3D video. The one or more modules may also be configured to perform a step of generating one or more 3D models corresponding to the 3D video. The one or more modules may also be configured to perform a step of rendering the one or more 3D models at an appropriate trigger time along with the rendering of the 3D video.

[0009] In one or more embodiments of the display system, the 3D video is a stereoscopic 3D video, and the one or more 3D models are generated with animation. The method may further include displaying an animation of the one or more 3D models at least partially outside a display screen of the 3D video, the animation of the one or more 3D models appearing to move out of the plane of the 3D video and into the user's 3D environment. The animation of the one or more 3D models may appear to move out of the plane of the 3D video and into the user's 3D environment. The method may still further include rendering the one or more 3D models on one or more video panes, and displaying the one or more video panes simultaneously for one or more viewing orientations.

[0010] Some embodiments are directed to a video file format that may include one or more animation streams, data storage, and at least one of a video stream or an audio stream.

[0011] One embodiment is directed to a method that includes identifying a portal within a 3D rendered world, the portal being an opening in a plane of the 3D rendered world. Additionally, a first 2D stereoscopic image and a second 2D stereoscopic image may be rendered into the portal of the 3D rendered world.

[0012] In one or more embodiments, the rendered 3D rendering world may be rendered to a user through an augmented reality device. The first 2D stereo image is for the right eye and the second 2D stereo image is for the left eye. In one embodiment, the first 2D stereo image and the second 2D stereo image are sourced from a conventional 3D video (e.g., a 3D movie). In a different embodiment, the first 2D stereo image and the second 2D stereo image are sourced from two virtual rendering cameras located in different parts of the 3D rendering world.

[0013] In one or more embodiments, the first 2D stereo image and the second 2D stereo image are sourced from two cameras capturing real-world content. In a different embodiment, the first 2D stereo image and the second 2D stereo image are sourced from different 3D rendered worlds. In some embodiments, the portal can be displayed as a portal icon or as a virtual television screen.

[0014] Another embodiment is directed to a computer-implemented method for displaying 3D video within a virtual and / or augmented reality environment, the method including identifying the 3D video. The method also includes identifying a volumetric space for displaying the 3D video within a portion of the virtual and / or augmented reality environment. Then, rendering the 3D video within the volumetric space by a 3D media player.

[0015] In one or more embodiments, the 3D video is a stereoscopic 3D video. The volume space is a portal view of a 3D object, and the 3D object is a virtual television having a flat-view display. The 3D video is rendered within the flat-view display of the virtual television. First depth information from the 3D video is added to second depth information from a first location of a portion of the virtual and / or augmented environment volume space to a second location of the user viewing the 3D video.

[0016] Another embodiment is directed to 3D video that may be rendered to display within a portion of a volumetric space within a virtual and / or augmented reality environment. The 3D video may include 3D objects that escape the screen. The 3D video may interact with the virtual and / or augmented reality environment. The 3D video may interact with the user such that different storyline endings may result for the 3D video based on user input that may affect the storyline.

[0017] Another embodiment is directed to a method that includes placing two first stage cameras in a 3D environment. In addition, the method may include capturing one or more pairs of 2D images from the two first stage cameras. Further, the method may include placing the one or more pairs of 2D images into locations within a final stage scene (sometimes alternatively referred to as a final 3D rendering world). And, the method may include rendering the final stage scene from the two final stage cameras.

[0018] In one or more embodiments, the method may also include displaying the end-stage scene via a virtual and / or augmented reality device. The display location may be a portal. A first of the two first-stage cameras captures a 2D image from a left-eye perspective, and a second of the two first-stage cameras captures a 2D image from a right-eye perspective. One or more other pairs of two 2D images may be captured from two end-stage cameras, corresponding to the user's eyes.

[0019] In one or more embodiments, the two first stage cameras are at different locations in the 3D environment than the two final stage rendering cameras. The two first stage cameras may be virtual cameras and the 3D environment may be a 3D virtual world. The 3D environment may be a digital representation of the real world.

[0020] Another embodiment is directed to a method including placing a first 3D content from a first source within a second 3D content from a second source. In one or more embodiments, the method may also include displaying the second 3D content via a virtual and / or augmented reality device. The first 3D content may be placed within a portal within the second 3D content. A first camera of the first source captures a 2D image from a left eye perspective and a second camera of the first source captures a 2D image from a right eye perspective.

[0021] In one or more embodiments, another pair of two 2D images is captured from two other cameras from a second source, the pair of two 2D images corresponding to the eyes of a user viewing the scene from the second source. The two cameras from the first source are at a different location in the 3D environment than the two other cameras from the second source. The first 3D content may be captured from two virtual cameras, and the first source may be a 3D virtual world. In some embodiments, the first source may be the real world.

[0022] Another embodiment is directed to a method including placing a first set of stereoscopic images inside a second set of stereoscopic images. In one or more embodiments, the method may also include displaying the second set of stereoscopic images via a virtual and / or augmented reality device. The first set of stereoscopic images is placed within a portal within the second set of stereoscopic images. The first set of stereoscopic images is captured by a first camera that may capture 2D images from a left eye perspective and a second camera that may capture 2D images from a right eye perspective.

[0023] In one or more embodiments, the second set of stereo images is captured by two other cameras, which may correspond to the eyes of a user viewing the scene showing the first set of stereo images within the portal. The first set of stereo images may be captured from the two cameras from a different location than the two other cameras capturing the second set of stereo images. The first set of stereo images may be captured from two virtual cameras in a 3D virtual world. The first set of stereo images may be captured from two cameras in the real world.

[0024] Some embodiments are directed to a method including identifying two input images, where one input image may correspond to a left eye perspective and a second input image may correspond to a right eye perspective. In addition, the method may also include placing the two input images in a defined location within the final 3D rendered world. In one or more embodiments, the method may also include displaying the final 3D rendered world via a virtual and / or augmented reality device. The defined location may be a portal within the final 3D rendered world. The two input images may be captured by a first camera that captures a 2D image from a left eye perspective and a second camera that captures a 2D image from a right eye perspective.

[0025] In one or more embodiments, the final 3D rendered world may be captured by two other cameras, which capture the final 3D rendered world and correspond to the eyes of a user viewing a scene showing the two input images in a portal. The two input images may be captured from the two cameras from different locations than the two other cameras capturing the final 3D rendered world. The two input images may be captured from two virtual cameras in a 3D virtual world. The two input images may be captured from two cameras in the real world.

[0026] Some embodiments are directed to a method that includes identifying a 3D data input. Additionally, the method may also include placing the 3D data input in a location within the virtual final 3D rendered world. In one or more embodiments, the method may also include displaying the virtual final 3D rendered world via a virtual and / or augmented reality device. The location may be a portal within the virtual final 3D rendered world. The 3D data input may be captured by a first camera that captures a 2D image from a left eye perspective and a second camera that may capture a 2D image from a right eye perspective.

[0027] In one or more embodiments, the virtual final 3D rendering world may be captured by two other cameras, which capture the virtual final 3D rendering world and correspond to the two eyes of a user viewing a scene showing the 3D data input in the portal. The 3D data input may be captured from the two cameras from a different location than the two other cameras capturing the virtual final 3D rendering world. The 3D data input may be captured from two virtual cameras in the 3D virtual world. The 3D data input may be captured from two cameras in the real world.

[0028] Some embodiments are directed to a video file format, which may include one or more animation streams, a data store, and at least one of a video stream or an audio stream. In one or more embodiments, the data store may include one or more control data, one or more 3D models, one or more textures, and one or more materials. The control data may be fully loaded into memory when an application that reads the video file format is first accessed. The control stream may be preloaded into memory from one or more control data from the data store when the control data is fully loaded into memory. The control stream may be configured to have a corresponding time visible on the timeline ahead of the video stream or audio stream.

[0029] In one or more embodiments, the control stream may comprise commands and / or simple logic gates to determine the combination of models and animations to play at the appropriate time in the video timeline. The appropriate time in the video timeline corresponds to a corresponding time in the video and audio streams. The control stream comprises keyframes. The animation stream, video stream, and audio stream may be configured to look ahead and behind a keyframe. The one or more animation streams may correspond to animation instructions associated with one or more 3D models. Each animation stream of the one or more animation streams may correspond to at least one of a 3D model, a texture, or a material of the data store.

[0030] Some embodiments are directed to a method including receiving a video file of a video, the video file may include a data store comprising one or more animation streams and control data, and at least one of a video stream or an audio stream. The method may also include dynamically generating the control stream from the control data and a timeline controller. In addition, the method may also include loading a model of a 3D object received from the data store. The method may also include attaching the 3D object to an animation stream of the one or more animation streams.

[0031] In one or more embodiments, the one or more animation streams correspond to individual 3D models stored in the data store. The one or more animation streams may control the movement, orientation, and / or position of the 3D object relative to the video. The model of the 3D object may be loaded, at least in part, based on a control stream that looks ahead in time in the video stream and anticipates when the 3D object needs to be displayed.

[0032] In one or more embodiments, the method may also include determining a lead time for loading the model based on at least one of the size of the model, the network bandwidth, or the processing power of the user's virtual and / or augmented reality system. In addition, the method may also include displaying the video via the virtual and / or augmented reality device. The data store may be fully loaded into memory when the video file is received. The control stream may be fully loaded into memory when the control stream is generated. The video may be a stereoscopic 3D video.

[0033] Some embodiments are directed to a computer system implementing a mixed reality video player, including a computer processor for executing a set of program code instructions and a memory for holding the program code instructions, the program code instructions may include program code for performing a step of receiving a video file of a video, the video file may include a data store comprising one or more animation streams, control data, and at least one of a video stream or an audio stream. The program code instructions may also include program code for performing a step of dynamically generating a control stream from the control data and a timeline controller. The program code instructions may further include program code for performing a step of loading a model of a 3D object received from the data store. The program code instructions may also include program code for performing a step of attaching a 3D object to an animation stream of the one or more animation streams.

[0034] Another embodiment is directed to a method including receiving a video file of a video, the video file may include one or more animation streams, a data store comprising control data, and at least one of a video stream or an audio stream. The method may also include dynamically generating the control stream from the control data and a timeline controller. In addition, the method may also include requesting user interaction to answer a question displayed to the user at a point in the video. Further, the method may also include receiving an answer to the question via the user interaction. The method may also include notifying the timeline controller of the answer. Still further, the method may also include skipping to an appropriate portion of the video corresponding to the answer. The method may also include displaying the video from the appropriate portion.

[0035] In one or more embodiments, the method may also include displaying the video via a virtual and / or augmented reality device. The data store may be fully loaded into memory when the video file is received. The control stream may be fully loaded into memory when the control stream is generated. The video may be a stereoscopic 3D video. The control stream may control when questions are displayed to the user at specific points in the video based at least in part on data from the data store.

[0036] In one or more embodiments, the user interaction includes at least one of head pose, eye tracking, gaze, user hand gestures, totem gestures, or an object recognizer. A timeline controller controls the position of the video stream. The method may also include storing a state of the video with run-time interactivity data based at least in part on the answers received. The questions displayed to the user correspond to questions that change how the storyline of the video may end.

[0037] Some embodiments are directed to a computer system implementing a mixed reality video player, including a computer processor for executing a set of program code instructions and a memory for holding the program code instructions, the program code instructions including program code for performing steps of receiving a video file of a video, the video file including one or more animation streams, a data store comprising control data, and at least one of a video stream or an audio stream. The program code instructions may also include program code for performing steps of dynamically generating a control stream from the control data and a timeline controller, answering a question displayed to a user at a point in the video, requesting user interaction, receiving an answer to the question via user interaction, notifying the timeline controller of the answer, skipping to an appropriate portion of the video corresponding to the answer, and displaying the video from the appropriate portion.

[0038] In one or more embodiments, the program code instructions may also include program code for performing steps of displaying the video via a virtual and / or augmented reality device. The data store may be fully loaded into memory when the video file is received. The control stream may be fully loaded into memory when the control stream is generated. The video may be a stereoscopic 3D video. The control stream may control when a question is displayed to the user at a particular point in the video based at least in part on data from the data store.

[0039] In one or more embodiments, the user interaction includes at least one of head pose, eye tracking, gaze, user hand gestures, totem gestures, or an object recognizer. A timeline controller controls the position of the video stream. The method may also include storing a state of the video with run-time interactivity data based at least in part on the answers received. The questions displayed to the user correspond to questions that change how the storyline of the video may end.

[0040] Another embodiment is directed to a method for displaying environmentally aware 3D video in a virtual and / or augmented reality environment, the method including identifying the 3D video. The method also includes receiving environmental information of a user's environment from one or more sensors, the environmental information identifying objects in the environment. The method may further include rendering a portion of the 3D video onto the one or more objects identified from the environment.

[0041] In one or more embodiments, the 3D video is a stereoscopic 3D video. The environment may be the user's physical environment. The one or more sensors comprise one or more cameras for capturing scene information of the physical environment. The method further includes interpreting the scene information captured from the one or more sensors and mapping one or more elements of the environment by detecting and registering one or more elements from the environment. The 3D video has a video file format comprising control data, the control data instructing a video player to display a portion of the 3D video on one or more objects identified from the environment.

[0042] Some embodiments are directed to a virtual and / or augmented reality display system including a camera for capturing a virtual and / or augmented reality environment and a module for processing data, the module being stored in a memory that, when executed, performs a process for identifying a 3D video, a process for receiving environmental information of a user environment from one or more sensors, the environmental information identifying objects within the environment, and a process for rendering a portion of the 3D video onto one or more objects identified from the environment.

[0043] In one or more embodiments, the 3D video is a stereoscopic 3D video. The environment may be the user's physical environment. The one or more sensors comprise one or more cameras for capturing scene information of the physical environment. The method further includes interpreting the scene information captured from the one or more sensors and mapping one or more elements of the environment by detecting and registering one or more elements from the environment. The 3D video has a video file format comprising control data, the control data instructing a video player to display a portion of the 3D video on one or more objects identified from the environment.

[0044] Some embodiments are directed to a method for displaying interactive 3D video in a virtual and / or augmented reality environment, the method including identifying the 3D video as a stream of non-executable data periodically interrupted by a decision node. Further, the method includes receiving one or more interactions from a user as input corresponding to a decision taken by the user. The method may also include rendering the 3D video based at least on the decision taken by the user.

[0045] In one or more embodiments, the 3D video is a stereoscopic 3D video. The 3D video may have a video file format with control data that instructs a video player to render a portion of the 3D video based at least in part on a decision selected by a user. The one or more interactions received from the user include at least one of a head pose, eye tracking, a gesture, a totem gesture, or an object recognizer. An interaction from the one or more interactions received from the user causes the 3D video to jump to a different portion of the 3D video.

[0046] Some embodiments are directed to a virtual and / or augmented reality display system including a camera for capturing a virtual and / or augmented reality environment and a module for processing data, the module being stored in a memory and, when executed, performing a process of identifying 3D video as a stream of non-executable data that is periodically interrupted by a decision node, a process of receiving one or more interactions from a user as input corresponding to a decision selected by the user, and a process of rendering the 3D video based at least on the decision selected by the user.

[0047] In one or more embodiments, the 3D video is a stereoscopic 3D video. The 3D video may have a video file format with control data that instructs a video player to render a portion of the 3D video based at least in part on a decision selected by a user. The one or more interactions received from the user include at least one of a head pose, eye tracking, a gesture, a totem gesture, or an object recognizer. An interaction from the one or more interactions received from the user causes the 3D video to jump to a different portion of the 3D video.

[0048] Some embodiments are directed to a method for displaying a 3D object, the method may include storing graphical data representing the 3D object in a database. The method may further include displaying a 2D opaque pane with a viewing aperture for viewing the 3D object behind the 2D opaque pane, rendering a first portion of the 3D object viewable through the viewing aperture from a first perspective, receiving a request to display the 3D object viewable through the viewing aperture from a second viewing perspective, rendering a second portion of the 3D object viewable through the viewing aperture from the second perspective, and displaying the second portion of the 3D object from the second perspective.

[0049] In one or more embodiments, the graphical data representing the 3D object is a complete representation of the entire 3D object. The rendering of a first portion of the 3D object may be based at least in part on a viewing angle for a view of the 3D object from a first viewpoint. The rendering of a second portion of the 3D object may be based at least in part on a viewing angle for a view of the 3D object from a second viewpoint. The opaque pane may further comprise a plurality of viewing openings for viewing a plurality of 3D objects located behind the 2D opaque pane.

[0050] In one or more embodiments, each 3D object of the plurality of 3D objects is located behind a respective viewing aperture. The 3D object may be a 3D icon. The 2D opaque pane with the viewing aperture may be an icon grid for displaying the plurality of 3D icons. The 3D object may move through the viewing aperture as the user gazes at the 3D object. The 3D object may be displayed as a 2D image when the user is not focused on the 3D object.

[0051] Additional and other objects, features, and advantages of the present disclosure are set forth in the detailed description, the drawings, and the claims. The present specification also provides, for example, the following items: (Item 1) 1. A method for displaying 3D video extending beyond a surface of a display screen, the method comprising: Identifying the 3D video; generating one or more 3D models corresponding to the 3D video; and rendering said one or more 3D models at appropriate trigger times along with the rendering of said 3D video. A method comprising: (Item 2) Item 2. The method of item 1, wherein the 3D video is a stereoscopic 3D video. (Item 3) 3. The method of claim 2, wherein the one or more 3D models are generated with animation. (Item 4) 4. The method of claim 3, further comprising displaying an animation of the one or more 3D models outside of a display screen of the 3D video. (Item 5) 5. The method of claim 4, wherein the animation of the one or more 3D models appears to move out of the plane of the 3D video and into the user's 3D environment. (Item 6) 5. The method of claim 4, wherein the animation of the one or more 3D models appears to come out of the non-planar plane of the 3D video and into the user's 3D environment. (Item 7) rendering the one or more 3D models on one or more video panes; and displaying the one or more video panes simultaneously for one or more viewing orientations; and The method of claim 1, further comprising: (Item 8) 3. The method of claim 2, wherein the stereoscopic 3D video is displayed in conjunction with the one or more 3D models. (Item 9) 2. The method of claim 1, wherein depth information is taken into account in the rendering of the 3D video and the one or more 3D models, the depth information comprising a distance from a user of a virtual and / or augmented reality device to the 3D video, and wherein the individual one or more 3D models are displayed within a virtual and / or augmented reality environment. (Item 10) 2. The method of claim 1, wherein the 3D video comprises control data that instructs a video player to render the one or more 3D models at the appropriate trigger times along with rendering the 3D video. (Item 11) 2. The method of claim 1, wherein the rendering of the one or more 3D models is based at least in part on a voxel-based video stream. (Item 12) 12. A display system for displaying 3D video extending beyond the surface of a display screen comprising means for implementing any of the methods described in items 1-11. (Item 13) 1. A video file format, comprising: One or more animation streams; Data storage and At least one of the video stream or audio stream A video file format comprising: (Item 14) 1. A method comprising: receiving a video file of a video, the video file comprising: One or more animation streams; a data store comprising control data; At least one of the video stream or audio stream and dynamically generating a control stream from the control data and a timeline controller; loading a model of a 3D object received from said data store; attaching the 3D object to an animation stream of the one or more animation streams; A method comprising: (Item 15) 1. A computer system implementing a mixed reality video player, comprising: a computer processor for executing a set of program code instructions; a memory for holding said program code instructions, said program code instructions comprising: receiving a video file of a video, the video file comprising: One or more animation streams; a data store comprising control data; At least one of the video stream or audio stream and dynamically generating a control stream from the control data and a timeline controller; loading a model of a 3D object received from said data store; attaching the 3D object to an animation stream of the one or more animation streams; a memory having program code for implementing the A computer system comprising: (Item 16) 1. A method comprising: receiving a video file of a video, the video file comprising: One or more animation streams; a data store comprising control data; At least one of the video stream or audio stream and dynamically generating a control stream from the control data and a timeline controller; requesting user interaction to answer a question displayed to the user at a point in the video; receiving an answer to the question via the user interaction; and notifying a timeline controller of the answer; skipping to an appropriate portion of the video corresponding to the answer; and displaying the video from the appropriate portion; and A method comprising: (Item 17) 1. A computer system implementing a mixed reality video player, comprising: a computer processor for executing a set of program code instructions; a memory for holding said program code instructions, said program code instructions comprising: receiving a video file of a video, the video file comprising: One or more animation streams; a data store comprising control data; at least one of a video stream or an audio stream; and dynamically generating a control stream from the control data and a timeline controller; requesting user interaction to answer a question displayed to the user at a point in the video; receiving an answer to the question via the user interaction; and notifying a timeline controller of the answer; skipping to an appropriate portion of the video corresponding to the answer; and displaying the video from the appropriate portion; and a memory having program code for implementing the A computer system comprising: (Item 18) 1. A method for displaying environmentally aware 3D video within a virtual and / or augmented reality environment, the method comprising: Identifying the 3D video; receiving environmental information of a user's environment from one or more sensors, the environmental information identifying objects within the environment; Rendering a portion of the 3D video onto one or more objects identified from the environment; and A method comprising: (Item 19) 1. A virtual and / or augmented reality display system, comprising: a camera for capturing a virtual and / or augmented reality environment; A module for processing data, said module being stored in a memory, said module being capable of, when executed, A process for identifying 3D videos; a process for receiving environmental information of a user's environment from one or more sensors, the environmental information identifying objects within the environment; rendering a portion of the 3D video onto one or more objects identified from the environment; Module and A virtual and / or augmented reality display system comprising: (Item 20) 1. A method for displaying interactive 3D video in a virtual and / or augmented reality environment, the method comprising: identifying the 3D video as a stream of non-actionable data that is periodically interrupted by a decision node; receiving one or more interactions from a user as input corresponding to a decision selected by the user; rendering the 3D video based at least on the decision selected by the user; and A method comprising: (Item 21) 1. A virtual and / or augmented reality display system, comprising: a camera for capturing a virtual and / or augmented reality environment; A module for processing data, said module being stored in a memory, said module being capable of, when executed, a process for identifying 3D video as a stream of non-actionable data that is periodically interrupted by a decision node; a process for receiving one or more interactions from a user as input corresponding to a decision to be selected by said user; rendering said 3D video based at least on said decision selected by said user; Module and A virtual and / or augmented reality display system comprising: [Brief description of the drawings]

[0052] The drawings illustrate the design and utility of preferred embodiments of the present disclosure, where like elements are referenced by common reference numerals. In order to better understand how the foregoing and other advantages and objects of the present disclosure are obtained, a more particular description of the present disclosure briefly described above will be given by reference to specific embodiments thereof, which are illustrated in the accompanying drawings. With the understanding that these drawings depict only exemplary embodiments of the present disclosure and are therefore not to be considered as limiting its scope, the present disclosure will be described and explained with additional particularity and detail through the use of the accompanying drawings.

[0053] The drawings use like reference numbers to identify like elements. Letters following a reference number, such as "120a," indicate that the text specifically refers to the element with that particular reference number. A reference number in text without a letter following it, such as "120," refers to any or all of the elements in the drawings with that reference number (e.g., "120" in the text refers to reference numbers "120a" and / or "120b" in the drawings).

[0054] [Figure 1]FIG. 1 illustrates an exemplary user physical environment and an augmented reality system for displaying 3D video within an AR environment, according to some embodiments.

[0055] [Diagram 2] FIG. 2 illustrates a flowchart for an approach for displaying 3D video in an augmented reality system, according to some embodiments.

[0056] [Figure 3A] 3A-3B show examples of 3D images and / or 3D animations escaping from a screen, according to some embodiments. [Figure 3B] 3A-3B show examples of 3D images and / or 3D animations escaping from a screen, according to some embodiments.

[0057] [Figure 4] FIG. 4 shows a flowchart for an approach for displaying 3D video that extends beyond the surface of a display screen in a virtual and / or augmented reality environment, according to some embodiments.

[0058] [Diagram 5] FIG. 5 illustrates a flowchart for an approach for displaying environmentally aware 3D video within a virtual and / or augmented reality environment, according to some embodiments.

[0059] [Figure 6] FIG. 6 shows a flowchart for displaying interactive 3D video within a virtual and / or augmented reality environment, according to some embodiments.

[0060] [Figure 7] FIG. 7 illustrates a video format for displaying 3D video according to some embodiments.

[0061] [Figure 8]FIG. 8 shows a component diagram of a video player according to some embodiments.

[0062] [Figure 9] FIG. 9 illustrates a flow of a video player handling video formats according to some embodiments.

[0063] [Figure 10] FIG. 10 is a block diagram of an illustrative computing system suitable for implementing one or more of the embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0064] The present disclosure is directed to generating and displaying 3D videos within a VR, MR, and / or AR system. The 3D video may include 3D objects that appear to escape the screen. The 3D video may interact with objects within a virtual reality (VR), mixed reality (MR), augmented reality (AR), and / or real environment. The 3D video may interact with the user such that different storylines, and possibly different endings, may result for the 3D video based on user input corresponding to decisions chosen by the user at certain portions of the 3D video. The 3D video may be a 3D icon displayed within a portal of the final 3D rendered world.

[0065] This disclosure provides a description of an illustrative augmented reality (AR) system with which some embodiments of the disclosure may be practiced, followed by one or more embodiments of processes and mechanisms for generating and displaying 3D video within the illustrative AR system. Illustrative Augmented Reality Systems and Environments

[0066] The following description relates to an exemplary AR system with which the present disclosure may be practiced, however, it should be understood that the present disclosure is also suitable for use in other types of AR, MR, and virtual reality (VR) systems, and thus the present disclosure is not limited solely to the exemplary system disclosed herein.

[0067] 1, one embodiment of an AR system 100 constructed in accordance with the present disclosure is illustrated. The AR system 100 may be operated in conjunction with an AR projection subsystem 110, which may provide 3D video within the field of view of an end user 50 (hereafter referred to as "user 50"). The present approach employs one or more eyepieces, which may include at least partially transparent surfaces through which a surrounding environment, such as the user's physical environment / scenery 105, is viewed and may be augmented with images of virtual content (e.g., 3D video, virtual objects, etc.).

[0068] The AR system 100 includes a frame structure 102 worn by a user 50 and a display system 104 carried by the frame structure 102 such that the display system 104 is positioned in front of the eyes of the user 50.

[0069] The display system 104 is configured to present to the eyes of the user 50 light-based radiation patterns that may be comfortably perceived as an augmentation to physical reality, along with both two-dimensional and three-dimensional content. The display system 104 presents a sequence of frames at a high frequency that provides the perception of a single coherent scene that blends real and virtual content. To this end, the display system 104 includes a projection subsystem 110 and an eyepiece in a pair of glasses, which may be, for example, a partially transparent surface onto or through which the projection subsystem 110 projects an image. The eyepiece is positioned within the field of view of the user 50 between the eyes of the user 50 and the surrounding environment. In some embodiments, the eyepiece may be a fully transparent, partially transparent, or semi-transparent surface such that the user 50 may see at least a portion of the user's real world / surroundings while wearing the display system 104.

[0070] In some embodiments, the projection subsystem 110 may take the form of a scanning-based projection device and an eyepiece into which light from the projection subsystem 110 is injected to produce, for example, an image at a single optical viewing distance closer than infinity (e.g., arm's length), images at multiple optical viewing distances or focal planes, and / or image layers stacked at multiple viewing distances or focal planes to represent a stereoscopic 3D object. The layers in the light field may be stacked close enough together to appear continuous to the human visual system (e.g., one layer is within the noise cone of an adjacent layer). The layers in the light field may be stacked at predefined depth intervals and utilized one at a time or in combination to create depth planes at discrete viewing distances. Additionally or alternatively, photographic elements may be blended across two or more layers to increase the perceived continuity of the transition between layers in the light field even when the layers are more sparsely stacked (e.g., one layer is outside the noise cone of an adjacent layer). The display system 104 may be monocular or binocular. The scanning assembly includes one or more light sources that generate one or more light beams (e.g., emit light in multiple colors in a defined pattern). The light sources may take any of a wide variety of forms, such as a set of RGB sources (e.g., laser diodes capable of outputting red, green, and blue light) that are operable to generate coherent collimated light in red, green, and blue, respectively, according to a defined pixel pattern defined in an individual frame of pixel information or data. Laser light provides high color saturation and is highly energy efficient. In some embodiments, light emitting diodes (LEDs) may be used and the light may not be collimated. The optical coupling subsystem includes an optical waveguide input device, such as, for example, one or more reflective surfaces, diffraction gratings, mirrors, dichroic mirrors, or prisms, to optically couple the light into the end or edge or side of the eyepiece. The optical coupling subsystem may further include a collimation element that collimates the light from the optical fiber.

[0071] Optionally, the optical coupling subsystem includes an optical modulation device configured to converge light from the collimation element toward a focal point in the center of the optical waveguide input device, thereby allowing the size of the optical waveguide input device to be minimized. Thus, the display system 104 generates a series of composite image frames of pixel information that presents an undistorted image of one or more virtual objects to the user. Further details describing the display subsystem are provided in U.S. Non-Provisional Patent Application No. 14 / 212,961, entitled "Display System and Method," and in U.S. Pat. No. 6,313,363, entitled "Planar No. 9,671,566, entitled "Waveguide Apparatus With Diffraction Element(s) and Subsystem Employing Same," which is expressly incorporated herein by reference.

[0072] In some embodiments, the projection subsystem 110 may take the form of a liquid crystal on silicon (LCoS or LCOS). LCOS may be a miniaturized reflective active matrix liquid crystal display or "microdisplay" that uses a liquid crystal layer on top of a silicon backplane. LCOS may also be referred to as a spatial light modulator. LCOS may be used for wavelength selective switching, structured illumination, near-eye displays, and / or optical pulse shaping. More information regarding LCOS is disclosed in U.S. patent application Ser. No. 15 / 807,486, entitled "Augmented and virtual reality eyewear, systems, and methods for delivering polarized light and determining glucose levels," previously incorporated herein by reference.

[0073] The AR system 100 further includes one or more sensors mounted to the frame structure 102 to detect the position and movement of the user's 50 head and / or the user's 50 eye position and interocular distance. Such sensors may include image capture devices, microphones, inertial measurement units (IMUs), accelerometers, compasses, GPS units, wireless devices, and / or gyroscopes. For example, in one embodiment, the AR system 100 includes a head-mounted transducer subsystem that includes one or more inertial transducers and captures inertial measurements indicative of the moving head of the user 50. Such devices may be used to sense, measure, or collect information about the head movement of the user 50. For example, these devices may be used to detect the measured movement, velocity, acceleration, and / or position of the user's 50 head. Once the location and / or position of the user's head is known, the mixed reality engine 166 may be able to determine the distance between the user and real and / or virtual objects in the user's environment. The distance may be determined using a variety of different methods. In some embodiments, one or more depth sensors, or other environmental sensing sensors (e.g., outward-facing cameras), may be used to create a digital replica of the real world, which in some embodiments may be referred to as a world mesh. A wide variety of data may be associated with the world mesh, thus creating an environmentally aware system, which may in effect be referred to as a passable world. For example, data associated with the world mesh may be names or types of objects or surfaces (e.g., horizontal surfaces, tables, chairs, television screens, etc.). The user's position may be compared to the passable world, thus allowing distances between real and virtual objects to be calculated. In some embodiments, the passable world may correspond to all or a portion of a 3D rendering world.Further details describing methods and systems for creating a 3D mesh (e.g., a world mesh) of a real-world environment are provided in U.S. Non-Provisional Patent Application No. 15 / 274,823, entitled "Methods and Combining Structural Features in 3D Reconstruction," which is expressly incorporated herein by reference in its entirety. In some embodiments, a user simply requires a digital model of the real world or at least a subset of the real world that includes the user's environment. The digital model of the real world may be created using a depth sensor, a passable world system (e.g., as described above), and / or a stored map.

[0074] The AR system 100 may further include a user orientation detection module that detects the instantaneous position of the head of the user 50 and may predict the position of the head of the user 50 based on the position data received from the sensors. The user orientation detection module also tracks the eyes of the user 50, and in particular the direction and / or distance at which the user 50 focuses, based on the tracking data received from the sensors.

[0075] The AR system 100 also includes a mixed reality media player 164. The mixed reality media player 164 may select and retrieve data having the mixed reality video format 162 from the 3D video database 160. The mixed reality media player 164 may interpret the mixed reality video format 162 and provide the interpretation to the mixed reality engine 166 to provide a final composition of the video as displayed to the user 50 based at least in part on the user's head pose. The mixed reality engine 166 may render videos and models that may be used in the final composition of the scene based on the user's head pose. A more detailed disclosure of the mixed reality video format 162 and the mixed reality media player 164 will be discussed below.

[0076] The AR system 100 further includes a control subsystem, which may take any of a wide variety of forms. The control subsystem includes several controllers, such as one or more microcontrollers, microprocessors or central processing units (CPUs), digital signal processors, graphics processing units (GPUs), other integrated circuit controllers such as application specific integrated circuits (ASICs), programmable gate arrays (PGAs), such as field programmable gate arrays (FPGAs), and / or programmable logic controllers (PLUs). The control subsystem may include a digital signal processor (DSP), a central processing unit (CPU), a graphics processing unit (GPU) 152, and one or more frame buffers 154. The CPU controls the overall operation of the system, while the GPU 152 renders frames (i.e., converts a three-dimensional scene into a two-dimensional image) and stores these frames in the frame buffer 154. Although not shown, one or more additional integrated circuits may control the reading of frames into and / or from the frame buffer 154 and the operation of a scanning device of the display system 104. Reading into and / or from the frame buffer 154 may employ dynamic addressing, e.g., frames are over-rendered. The AR system 100 further includes a read-only memory (ROM) and a random access memory (RAM). The AR system 100 further includes a 3D database 160, from which the GPU 152 can access image data of one or more scenes for rendering frames and synthesized sound data associated with virtual sound sources contained within the 3D scenes.

[0077] Various processing components of the AR system 100 may be physically contained within a distributed subsystem. For example, the AR system 100 may include a local processing and data module that is operatively coupled to a portion of the display system 104, such as by wired leads or wireless connectivity. The local processing and data module may be mounted in a variety of configurations, such as fixedly attached to the frame structure 102, fixedly attached to a helmet or hat, built into headphones, removably attached to the torso of the user 50, or removably attached to the waist of the user 50 in a belt-type configuration.

[0078] The AR system 100 further includes a remote processing module and a remote data repository operably coupled to the local processing and data module, such as by wired conductors or wireless connectivity, such that these remote modules are operably coupled to each other and available as resources to the local processing and data module. The local processing and data module may include a power-efficient processor or controller and digital memory, such as flash memory, both of which may be utilized to aid in processing, caching, and storing data captured from the sensors and / or obtained and / or processed using the remote processing module and / or remote data repository (possibly for passage to the display system 104 after such processing or retrieval).

[0079] The remote processing module may include one or more relatively powerful processors or controllers configured to analyze and process the data and / or image information. The remote data repository may include a relatively large digital data storage facility, which may be available through the Internet or other networking configurations in a "cloud" resource configuration.

[0080] In one embodiment, all data is stored and all calculations are performed in the local processing and data module, allowing for fully autonomous use from any remote module. The coupling between the various components described above may include one or more wired interfaces or ports to provide wire or optical communication, or one or more wireless interfaces or ports, such as via radio frequency (RF), microwave, and infrared (IR), to provide wireless communication. In some implementations, all communication may be wired, while in other implementations, all communication may be wireless, except for optical fiber.

[0081] For example, a user 50 using the display system 104 of the AR system 100 may be looking towards the user's physical environment / landscape 105. The user's physical environment / landscape 105 may include a virtual television 120 displayed on a vertical wall 125. The vertical wall 125 may be any vertical wall in a room where the user 50 may be located. The vertical wall may also be a wall on the side of a building (inside or outside), or any natural vertical wall found anywhere. In some embodiments, the virtual television 120 may either be anchored and / or fixed to a blank vertical wall 125 or displayed across a photo frame (not shown) hung on a vertical wall in the user's physical environment / landscape. The virtual television 120 may be a virtual object generated by the AR system 100 that is anchored or fixed on the vertical wall 125 or, in some embodiments, floating in space. The virtual television 120 may be a virtual object on or into which the AR system 100 may display the 3D video 115. The virtual television 120 may be a portal within the user's physical environment / landscape 105. Portals are discussed below.

[0082] The 3D video 115 may include 3D animated objects that may be rendered and displayed along with the 3D video to provide a realistic view of the 3D effect to a user 50 using the display system 104. The 3D animated objects may appear to actually spill / leave the virtual television screen. If the user moves to a side view viewing viewpoint, the user may actually see the 3D objects and / or 3D animated objects exiting the virtual television display screen and into the user's physical environment / scenery 105.

[0083] Additionally, in another embodiment, the 3D video may be aware of the user's physical environment / landscape 105 based on environmental sensing sensors configured on the AR system 100. Because the 3D video may be aware of the user's physical environment, the 3D video may utilize its environmental awareness information while presenting the 3D video by rendering portions of the 3D video on actual objects in the user's physical environment. For example, sensors in the AR system 100 may detect two chairs 135a and 135b located in the user's physical environment / landscape. A 3D video about a news broadcast may place a reporter 130 sitting in chairs 135a and 135b in the user's physical environment / landscape, and the reporter 130 reports the news as if the reporter 130 were in the user's physical environment. A more detailed discussion of the environmental awareness features of the present disclosure is below.

[0084] In another embodiment, the 3D video may interact with a user of the VR and / or AR system. As an example, at certain portions or points in the 3D video, the user may be provided with decisions to make that may affect the storyline of the video. Based on the user's interaction to select a decision, the 3D video may be altered and a different ending or outcome of the 3D video may be reached. A more detailed discussion of interactive 3D video may be found below. (Displays 3D video within a portal in an AR scene)

[0085] A portal may be an opening and / or location in a plane of a 2D and / or 3D volumetric space where an object or other virtual content may be rendered and / or displayed inside, behind, in front of, and / or at the opening of the plane. An object may be seen or viewed by a user 50 using the display system 104 through an opening in the plane and / or location in the 3D volumetric space. The plane may be opaque such that the user may not be able to see through the plane. However, the user 50 may view through an opaque plane through an opening (e.g., a portal) in the plane. For example, when a user views an object from a first perspective having a direct frontal view position, the object may appear to be a flat 2D frontal view of the object with the portal constituting a boundary around the object. The portal may appear to be any shape, such as a circle, rectangle, square, polygon, etc., from this first perspective.

[0086] Continuing with the example, when a user views an object from a second viewpoint having a side view position, a portion of the object may be visible to the user and other portions of the object may be occluded or not visible based on the position of the object rendered and / or displayed relative to the front of the plane depending on the side view angle of the second viewpoint, such that a larger portion of the object may be visible if the object is positioned towards the front of the plane as opposed to a smaller portion of the object may be visible if the object is positioned towards the rear or behind the plane. A portion of the object may be cut off by the plane if the object is positioned towards the rear or behind the plane. The viewed object may be a 2D image, 2D video, 3D model (e.g., a computer-generated 3D icon), and / or 3D video. In some embodiments, the plane may be completely opaque such that the user may not be able to see anything located on the other side of the plane. In other embodiments, the plane may be semi-transparent such that the user may be able to view some objects or images through the plane, although not a clear view through the plane, when viewing through a portal.

[0087] In some embodiments, the portal effect may be created by placing the 3D virtual content directly in the final rendered world in a similar manner to any other 3D virtual content to be displayed to the user. For example, the 3D virtual content will share a coordinate system with any other 3D virtual content in the final rendered world that is being displayed to the user. The coordinate system may correspond to the real world coordinate system, so that the 3D virtual content is fixed relative to the real world. To create the portal opening effect, a mask such as an occlusion mask may be used. In some embodiments, the occlusion mask may be placed relative to a real world mesh to appear as a virtual opening in a real world wall. In this example, when the user walks around the wall, the user may enter the portal space such that the 3D virtual content previously viewed through the portal is now in the user's immediate environment, or alternatively, the user may be considered to be present in the portal world. In some embodiments, a portal may have a corresponding boundary such that if a user moves beyond a certain area, the portal "closes" (e.g., the 3D content associated with the portal is removed from the final rendered world), thus preventing the user from actually entering the portal or directly interacting with the portal 3D virtual content.

[0088] 2 shows a flow chart for an approach for displaying 3D video within an augmented reality system. At 210, 3D video may be identified for display within a user's physical environment / landscape. The 3D video may be stereoscopic 3D video.

[0089] At 220, a volumetric space for displaying 3D video within the user's physical environment / landscape may be identified. The volumetric space may be a portal for displaying 3D objects (e.g., 3D video). The portal may be a virtual television having a flat-view display (e.g., a virtual display screen).

[0090] At 230, 3D video may be rendered within a volumetric space (e.g., virtual television 120). The virtual television may include a plane with a portal within which the 3D video may be rendered and ultimately displayed. For example, the virtual television may include a boundary separating the portal (e.g., the virtual television screen) and the television frame itself. In some embodiments, the boundary may not be present, and in such embodiments, the plane of the virtual television may be the entire virtual television. The 3D video may be rendered and displayed in only a portion of the user's physical environment / landscape, since the 3D video may only be rendered and / or displayed within the fixed boundaries of the volumetric space (e.g., virtual television 120).

[0091] The 3D video may be a conventionally generated 3D video using stereoscopic images. Stereoscopic is a process where two pictures of the same object taken at slightly different angles are viewed together to create an impression of depth and solidity (e.g., a 3D effect). Here, as an example, the 3D video is rendered and / or displayed to a user in a virtual television (e.g., a portal) within the user's physical environment / landscape. The rendering of the 3D video may include first depth information obtained from the 3D video and second depth information corresponding to the depth information from a location of the virtual television screen to a location of the user, such that the final rendered 3D video displayed to the user 50 may include depth information that may help alleviate accommodation-vergence problems for the user when viewing the 3D video using the display system 104. By collecting depth information from the stereoscopic images and adding it to the depth buffer of the screen, the quality of the collected depth information may be greatly improved, based at least in part on the scene and an algorithm that may determine depth at run time.

[0092] In some embodiments, 3D objects (e.g., stereoscopic 3D objects or stereoscopic 3D videos) may be rendered and / or displayed through a portal. Traditionally, 2D images are rendered and displayed through a portal. However, in an augmented reality environment, 3D objects and 3D videos may be rendered and displayed through a portal (e.g., a virtual television) within the AR environment.

[0093] In one embodiment, a portal may be identified within a 3D rendering world (or alternatively, a 3D rendering scene, rendering scene, or 3D scene). A 3D rendering world may be a digital representation of 2D and 3D digital content that is placed relative to a digital representation of the real world (e.g., the user's physical environment / scenery 105) where the 2D and 3D digital content may be rendered as virtual content and displayed to the user. Two or more virtual rendering cameras may be placed within the 3D rendering world so that the virtual content is rendered from the correct perspective relative to the user and the real world. One virtual rendering camera may be for the right eye and the other virtual rendering camera may be for the left eye. A portal may be located within a 3D rendering world that may require additional virtual rendering cameras to be placed at different locations in the 3D rendering scene to capture what the user would see from the portal perspective. A portal may be a part within a plane that allows the user to see through or into the plane. A portal may be a virtual television screen, which may provide a plane within the 3D rendering world, and the virtual television screen may be an opening within the plane of the virtual television.

[0094] In addition, the first 2D stereoscopic image and the second 2D stereoscopic image may be rendered within a portal such that a user using the display system 104 may perceive both the first 2D stereoscopic image and the second 2D stereoscopic image as a 3D image or a 3D model. The first 2D stereoscopic image may be an image captured using a right-eye viewpoint, and the second 2D stereoscopic image may be an image captured using a left-eye viewpoint. The 3D image may be a 3D video, and the first 2D stereoscopic image may be a first 2D stereoscopic video captured from a right-eye viewpoint, and the second 2D stereoscopic image may be a second 2D stereoscopic video captured from a left-eye viewpoint. As discussed above, the 3D video may be a traditionally generated 3D video (e.g., for use on a real screen in the real world) that uses stereoscopic images. Here, the 3D video may be rendered and displayed within a portal of a 3D rendering world.

[0095] Furthermore, the first 2D stereoscopic image and the second 2D stereoscopic image may be sourced from two different virtual cameras located in different locations of the 3D rendered world. In some embodiments, the first 2D stereoscopic image and the second 2D stereoscopic image may be sourced from two different real-world cameras capturing real-world content. For example, the user 50 may be watching a 3D video on a virtual television screen using a display system 104 in the user's physical environment / scenery 105 receiving video data from two cameras (e.g., security cameras in the user's actual home) capturing real-world images that may be rendered as a 3D video. In addition, the first and second 2D stereoscopic images may also be sourced from different 3D rendered worlds, such as portal icons.

[0096] In one embodiment, an icon may be displayed within the portal. The icon may be a computer-generated 3D object. The icon may be composed of several parts. In some embodiments, the portal icon may have a background contained within the portal, an opening, and foreground elements that can move in and out of the opening, even including complete removal of the foreground elements from the icon and the opening. The icon may represent applications that the user may launch from the application launcher menu. The application launcher menu may comprise one or more 3D icons, each presented within their respective portal. This may allow 3D content associated with the application to be easily contained within the icon, but still be visible by the user. When the user is looking at an icon located inside the portal, the icon may begin to activate and animate. When the user is not looking at the icon, the icon may appear to be a flat 2D image displayed within the portal. However, as the user's gaze begins to focus on an icon displayed idle within the portal (e.g., a flat 2D icon), the icon may begin to transform (e.g., into a 3D icon) within the portal. The (e.g., 3D) icon may even begin to move toward the user, exiting the portal and appearing to float outside of the portal so that the user may view the icon as a 3D icon object. The user may view the 3D icon from multiple angles. This may be advantageous since a common problem with icons is that they are relatively small in size and are typically 2D images with limited space to illustrate the application they represent. However, this problem may be solved by displaying a 3D icon, which may have a larger surface simply due to the additional surface area available for a 3D icon versus a 2D icon. In some embodiments, the 3D icon may display a smaller version of the actual application that may be launched from the icon so that the user may be able to see and recognize the icon.

[0097] In another embodiment, two first stage cameras may be placed in the 3D environment. One or more pairs of 2D images may be captured from the two first stage cameras. The one or more pairs of 2D images may be placed in locations within the final 3D rendered world. The final 3D rendered world may be rendered to be displayed to a user using a virtual and / or augmented reality device (e.g., display system 104), which may include two virtual cameras that capture 2D images of the world scene, each of the two cameras corresponding to an eye of the user. In other words, the two first stage cameras provide content for a 3D stereoscopic video. The 3D stereoscopic video is placed in locations within the final 3D rendered scene. The final 3D rendered world / scene may be captured from a separate pair of two virtual cameras from the perspective of a user using display system 104, with each camera of the pair corresponding to an eye of the user using display system 104, such that the final 3D stereoscopic video displayed to the user via display system 104 is a final 3D video with at least one other 3D video displayed in a portal within the final 3D video. As discussed above, depth information may be included during the rendering of the final 3D video to accommodate the vergence-divergence movements experienced by the user's unique visual system (e.g., the user's eyes).

[0098] In one or more embodiments, the rendering location may be a portal, such as a virtual television, or a plane of the user's environment for displaying 3D video. A first of the two first stage cameras may capture one or more 2D images from a left eye perspective, and a second of the two first stage cameras may capture one or more 2D images from a right eye perspective. The two first stage cameras may be virtual cameras (e.g., capturing images of a digital / virtual world scene) and / or real world cameras (e.g., capturing images of a real world scene).

[0099] In some embodiments, two second stage cameras may capture 2D images and generate a second 3D stereoscopic video. The second 3D stereoscopic video may be placed in another location / portal in the final 3D rendering world to be rendered such that when the final 3D rendering scene is displayed to the user, the user may see two 3D stereoscopic videos in the user's virtual and / or augmented reality environment, and the two 3D stereoscopic videos may each be displayed in two different portals. There may be multiple stage cameras providing 3D stereoscopic video sources to the final 3D rendering world, each of which may correspond to an additional stage, but there may be only one final 3D rendering world that is rendered to be displayed to the user using the display system 104. The number of stages may correspond to the number of input sources of 3D video to be rendered in the final 3D rendering world, and the number of stages may determine the number of 3D videos displayed to the user from multiple locations (e.g., portals) in the user's 3D environment. Alternatively, any number of inputs may be input to the final 3D rendering world for two-stage rendering, with more than two sets of inputs. In some embodiments, 3D videos may be nested, so that, for example, one 3D video plays inside a different 3D video, which may then be included in the final rendered scene for display to the user.

[0100] In one or more embodiments, the 2D images captured from one or more stage cameras (not including the backstage camera) may originate from a virtual world, a real world, or a combination of both the virtual and real worlds. The location of the one or more stage cameras may be different from the location of the backstage camera.

[0101] In another embodiment, a first 3D content from a first source may be placed within a second 3D content from a second source. The first 3D content may be a 3D stereoscopic video, and the 3D stereoscopic video may be an input data source to a second 3D video that includes the first 3D stereoscopic video. The second 3D content may be a 3D stereoscopic video, and a source of video input for rendering and displaying the second 3D content may include the first 3D content.

[0102] In another embodiment, a first set of stereoscopic images may be placed inside a second set of stereoscopic images to provide a final 3D video that shows the first set of stereoscopic images as a 3D video.

[0103] In another embodiment, a pair of input images are identified, corresponding to an image captured for a left eye and an image captured for a right eye. The pair of input images may be specified to be rendered into specified locations within a scene to be rendered, and once the scene is rendered, the pair of input images may be displayed as 3D video within the scene.

[0104] In another embodiment, the 3D data input may be identified such that the 3D data input may be placed into a location within a virtual and / or augmented reality final 3D rendered world, which may correspond to the scene as viewed by a user using a virtual and / or augmented reality device.

[0105] In alternative embodiments, portal-like effects may be generated through the use of one or more stencil rendering stages. In some embodiments, instead of having two input streams, one for each eye, and instead of two first stage cameras as described above, the data input may be a 3D graphics data file. The data file may contain one or more data files associated with a single application or a single piece of content, and may contain various flags, such as stencil flags. When the final 3D rendering world is rendering, the content for a portal in the final 3D rendering world may be rendered in four stages: 1) draw the stencil, 2) draw the skybox for the stencil, 3) draw the portal content, and 4) erase the stencil. These four steps may be repeated for each portal in the final 3D rendering world. (3D video extends beyond the display surface)

[0106] 1, a user 50 viewing a 3D video 115 using a display system 104 may notice that objects in the 3D video 115 (e.g., blocks as depicted in the video 115) may appear to spill over or extend beyond the surface of a display screen in the user's physical environment / scenery 105. For example, traditional 3D video may provide the illusion or appearance of a 3D effect, and in some cases may influence / trick the user into believing that objects from within the 3D video are actually moving toward the user and are actually moving out of the screen. However, traditional 3D video achieves the 3D effect using stereoscopic processes, as discussed above.

[0107] A conventional 3D video placed in a VR / AR environment can only be viewed from one viewpoint. Conventional 3D video appears visibly flat as soon as the viewer deviates from the initial position of the camera. One solution to this problem may include a video file format and player that includes the mesh (e.g., 3D model and / or 3D animation), rig control (e.g., animation stream), and its corresponding control data (e.g., when playing the portion of the animation stream in a video file format that corresponds to the video and audio of the conventional 3D video).

[0108] 3A-3B show examples of 3D images and / or 3D animations escaping from a screen, according to some embodiments. FIG. 3A shows the intended 3D effect 305 and the actual 3D effect 310 of a conventional stereoscopic 3D video. The display screen / plane 320 may be one or more of a television, a computer monitor, a theater display screen, or any planar or non-planar surface for displaying 3D video, or any combination thereof. For example, the non-planar surface may be a curved display screen, which in some embodiments may appear to at least partially surround the viewer. A 3D object 330 is shown based on the intended 3D effect and the actual 3D effect. For example, current 3D technology employing stereoscopic images may desire to generate the intended 3D effect of an object 330a. However, given the limitations of the traditional stereoscopic 3D process, the actual 3D effect that the current traditional system generates is shown as a 3D object 330b. The intended effect of object 330a may be to show a 3D animated object displayed outside of plane 320 that may animate / move around the user's environment such that if the user moves to a second position having a different viewing angle of plane 320, the user may see a complete (or relevant portion) 3D representation of object 330a displayed and located outside of plane 320.

[0109] Figure 3B shows an example of how a user may perceive the intended 3D effect. Objects 330a and 330b are objects as viewed in Figure 3A, where object 330a may appear to have spilled / stepped out of plane 320 (e.g., a television screen display). Object 340 may be considered an object that has completely detached from plane 320 such that object 340 achieves one of the intended 3D effects 305 from Figure 3A.

[0110] For example, the 3D video may include a 3D video of an individual scuba diving in tropical waters with multiple fish swimming around the individual. From a traditional 3D video perspective, a user may watch the 3D video and feel a 3D effect based at least in part on a stereoscopic process. However, according to some embodiments of the present disclosure, at some portion of the 3D video, a 3D model of one of the fish may be generated for display against the 3D video. At some appropriate trigger time in the 3D video, the 3D model of the fish may be displayed as swimming inside the 3D video, and then the 3D model of the fish may start to leave the surface of the display screen and swim into the user's physical environment / landscape. As an example, a 3D animation of a fish may swim around an actual virtual television displaying the 3D video. In this example, if the user moves to a different position, such as a 90 degree angle parallel to the surface of the display screen, the user should still see the 3D animated fish swimming around the virtual television. If the user returns to a viewing angle where the user can see both the 3D animated fish swimming outside the display screen of the virtual television and the display screen, the user may view 3D video playing with the 3D model moving outside the display screen. In some embodiments, the user may view 3D video from more than one display. For example, a fish may swim out of display 1, around the user, and into display 2. In some embodiments, the user may have one large screen (e.g., dome-shaped, partial dome-shaped, ring-shaped, etc.) surrounding the user, and the 3D content may exit the screen at a first location and re-enter the screen at a second location. Regardless of where the 3D content exits or enters, the user may view the 3D object from the correct perspective in a realistic manner (similar to a real 3D object).

[0111] 1, the 3D video 115 may include 3D animations of individuals or objects that appear to spill / flow out of the display screen of the virtual television 120. The virtual television 120 and 3D video 115 may show objects (e.g., monsters) spilling / flowing out of the display screen of the virtual television when viewed from a second angle within the user's physical environment / landscape 105.

[0112] 4 shows a flow chart for an approach for displaying 3D video extending beyond the surface of a display screen in a virtual and / or augmented reality environment, according to some embodiments. At 410, 3D video may be identified. The 3D video may be traditional stereoscopic 3D video. Legacy implementations of 3D video may provide a large amount of 3D video content to be provided as a 3D video input data source to be used to implement embodiments of the present disclosure.

[0113] At 420, a 3D model of the 3D object may be generated to correspond to the 3D video. The 3D object may be a static or animated 3D object, including the movement of the 3D object from a fixed location or the movement of the entire 3D object within a 3D environment. The 3D model may correspond to the 3D video such that if the 3D video scene is a certain blue color and the 3D model of the 3D object is the same or a substantially similar blue color, the 3D model may not be visible to the user. Thus, the 3D model may be slightly adjusted in color, texture, contrast, or other characteristics for the user to detect the 3D model as it is displayed with the 3D video. The 3D model of the 3D object may be generated and stored in a data store. Animation data may be associated with the 3D model to control and direct the movement, orientation, and / or position of the 3D model relative to the 3D video. The animation data may be streamed as a stream separate and distinct from the traditional 3D video and audio streams in the media file. The animation data will be disclosed in more detail below.

[0114] At 430, the 3D model may be rendered at the appropriate trigger time along with the rendering of the 3D video. The 3D animation data may be received and rendered by an engine that renders the 3D animation data along with the 3D video and the audio portion of the 3D video. The 3D animation and 3D video may appear to the user as a single 3D video, and the 3D animated objects may appear to move out of the plane of the 3D video and spill out of / into the user's 3D environment.

[0115] In one or more embodiments, one or more 3D models may be rendered on one or more video panes, and one or more video panes may be rendered simultaneously for one or more viewing orientations. The one or more 3D models may appear to physically spill / exit the display screen of the virtual television. In some embodiments, the video panes may function as a background and / or backdrop, displaying the 3D models corresponding to the viewing angle from a user 50 using the display system 104.

[0116] Depth information may be taken into account in the rendering of the 3D video and one or more 3D models to solve accommodation-vergence-divergence mismatch problems that may generally be associated with legacy VR systems. Distance from the user using the display system 104 and the 3D model may be taken into account in how the 3D video and images or videos of the 3D model may be displayed to the user. For example, multiple depth information may be determined for the 3D video being displayed to the user. The first depth information may include depth information of the original 3D video, such as a determined distance of a camera capturing the 3D video to an object in the 3D video. The second depth information may include a distance from the user viewing the 3D video to a portal / location of the 3D video that is placed within the user's 3D environment. The third depth information may include a distance from the 3D model / 3D animated object to the user viewing the 3D video, where the 3D animated object spills / goes out of the video (e.g., out of the plane of the video display screen). In some embodiments, the 3D animated object may be moving towards the user, thereby decreasing the distance between the user and the 3D animated object. If depth information is not included in the calculation resulting in the display of the final scene, including the 3D video and the 3D animated object, to the user, the user's brain may be confused as to how to orient the user's eyes and address the accommodation-vergence-divergence problem common in most 3D video systems. In some embodiments, the distance from a user of a virtual and / or augmented reality device to the 3D video and the respective one or more 3D models displayed within the virtual and / or augmented reality environment may be calculated in any suitable manner. In some embodiments where more than one 3D model is displayed, the location of the 3D model used in the distance calculation may be the closest point to the user of any of the displayed 3D models. In some embodiments, the location of the 3D model used in the distance calculation may be the average of the origins of the 3D models displayed to the user.

[0117] In one or more embodiments, the 3D video may include a video format that may include control data. The control data may instruct the video player to display the 3D model outside of the display screen so that the 3D model appears to the user to be spilling out of the display screen. The control data may help coordinate and integrate the 3D model and the 3D video so that the 3D model and the 3D video appear to the user as if they were integrated as a single 3D video.

[0118] In some embodiments, one or more 3D models may be rendered based at least in part on a voxel-based video stream. Voxels represent values ​​on a regular grid in three-dimensional space. Like pixels in a bitmap, voxels themselves typically do not have their location (their coordinates) explicitly encoded along with their value. Instead, the rendering system infers the location of a voxel based on its location relative to other voxels (e.g., its location in the data structures that make up a single volumetric image). In contrast to pixels and voxels, points and polygons are often explicitly represented by the coordinates of their vertices. A direct consequence of this difference is that polygons can efficiently represent simple 3D structures with lots of empty or uniformly filled space, while voxels excel at representing regularly sampled spaces that are non-uniformly filled. Voxel-based video may require very high bandwidth / performance, which may translate into higher production computational costs and potentially larger amounts of data storage to implement volumetric video.

[0119] In another embodiment, one or more 3D models and / or 3D videos may be based at least in part on volumetric video. Volumetric video is a format of video that features moving images of real people that actually exist in three dimensions, like a hologram, allowing people to be viewed from any angle at any moment. The trick is that this medium requires a fundamentally different video technology capable of capturing 3D images of actors at high frame rates. Volumetric video may require very high bandwidth / performance, which may translate into higher production costs, not from a monetary standpoint, but from the standpoint of the computational process itself, by requiring greater computing power and potentially larger amounts of data to implement volumetric video. (Environmental Awareness Video)

[0120] Traditional video has always presented its content without awareness of the viewer's environment. The immersive experience can be greatly enhanced if the video content can be customized to the user's environment. In a virtual and / or augmented reality system, a user using a virtual and / or augmented reality system device (e.g., display system 104 from FIG. 1) may provide additional user environment information to the video so that the video can utilize the additional user environment information when displaying the video's content.

[0121] For example, an object recognizer (e.g., a sensor) may recognize and categorize items in a room of the user's physical environment / scenery, e.g., chair 1, table 2, etc. Head pose and eye tracking may be used to provide hints to the mixed reality video player about where the user is looking. Having additional user environment information may enable 3D video to break out of its limited 2D display space.

[0122] With reference to FIG. 1, a 3D video of a news broadcast may place reporter 130, seated in chairs 135a and 135b, in the user's physical environment / landscape (e.g., the user's living room, dining room, car, etc.). Instead of displaying the 3D video within a traditional video screen, an environmentally aware 3D video may instead render reporter 130 to appear on or against an object that may be found within the user's physical environment. The object may be a rendered version of a physical object in the user's physical environment. The object may be a rendered version of an object in a virtual world. The object may be a rendered virtual object that is placed in the user's physical environment for the purpose of displaying the object outside of a traditional video screen. The 3D video may be 3D stereoscopic video, voxel-based video, and / or volumetric video.

[0123] 5 illustrates a flowchart for an approach for displaying environmentally aware 3D video within a virtual and / or augmented reality environment, according to some embodiments. At 510, a 3D video may be identified. The 3D video may be at least one of a traditional stereoscopic 3D video, a voxel video, or a volumetric video.

[0124] At 520, environmental information may be received regarding objects detected / identified within the user's environment. The user's environment may be a physical and / or virtual environment. The user's physical environment may include the user's current physical environment as viewed from the user's perspective via a VR and / or AR device (e.g., AR system 100). The user's virtual environment may comprise virtual content displayed to the user as viewed from the user's perspective via a VR and / or AR display device (e.g., display system 104). The environmental information may include information about objects captured by sensors of the VR and / or AR system (e.g., cameras) and interpreted by the VR and / or AR system to identify the objects captured by the sensors. In some embodiments, the environmental information may be sourced from and / or stored in a passable world.

[0125] Environmental information captured by the sensors (e.g., information of objects in a scene) may be mapped to pre-identified objects stored in an external system (e.g., a cloud system), and detailed information of the identified objects (e.g., point clouds) may be available to further define the objects. Elements of the user's environment (e.g., physical environment and / or virtual environment) may be mapped to the pre-identified objects to provide more detailed information to the environmental information.

[0126] At 530, some and / or all of the 3D video content may be rendered on an object identified from the user's environment. The rendered video may be displayed to a user using a VR and / or AR display device (e.g., display system 104), where the user may view a portion of the 3D video at a first location (e.g., a portal location) in the user's environment, and the user may view a portion of the 3D video at a second location in the user's environment. For example, a reporter sitting in a chair in a newsroom in a traditional video may be displayed as sitting in a chair in the user's living room, where the reporter provides the user with the evening news, for example, in the user's physical environment / landscape. Complementary information may be displayed on an associated 2D virtual television screen / portal. The 3D video content may include a video file format with control data. The control data may instruct a video player to display a portion of the 3D video (e.g., the reporter) on an object identified from the user's environment. (User interactive movie)

[0127] The problem with traditional movies (e.g., a movie is a type of video) is that they are already edited by the director / producer of the movie. The user or viewer of the movie generally has no control over what is being shown or the decisions that the characters in the movie may make. This is because the plot of the movie is already set by the author / director / producer of the movie. However, in a VR and / or AR environment, the video may be interactive. The user may be able to affect the plot and / or outcome of the video based at least in part on user decisions provided through the VR and / or AR system while the video is being displayed to the user, e.g., via the display system 104.

[0128] User decisions provided through the VR and / or AR system may be in the form of the user selecting or providing an answer to a question provided by the VR and / or AR system that asks, for example, how the user would decide on a certain option during the video. The action may be one performed by an actor or actress or may be outside the actor's control, such as a weather event, natural disaster, timing, etc. Based on the decision made by the user, the storyline of the video may change, such that further events in the video may be affected and various endings may be reached for the video.

[0129] FIG. 6 illustrates a flow chart for displaying interactive 3D video within a virtual and / or augmented reality environment, according to some embodiments. At 610, a 3D video may be identified. The 3D video may be a stream of non-executable data that is periodically interrupted by a decision node. Typically, the video is formatted as an executable file that may be played by a video player. The non-executable data may include video data, audio data, and / or animation data. The non-executable data may be interrupted by a decision node, which may correspond to different versions of a storyline based on decisions made at the decision node, for example, by interaction from a user providing input corresponding to the individual decisions.

[0130] At 620, interactions from a user may be received as inputs corresponding to decisions to be made by the user. Currently, DVD and Blu-ray videos have a simple navigation system via a remote controller. In a VR and / or AR system, the remote controller may extend this behavior and incorporate various other user input sources, such as head pose, eye tracking, gestures, totem gestures, and / or object recognizers. A simple interaction received from the user (e.g., a decision made at a particular decision node) may cause the video to "jump" to a different part of the video. The options and decisions made may be stored for the duration of the video to maintain the state of the video. Interactions may be triggered, for example, by a user clicking an option, a hand gesture, a gaze combined with a selection from a user input device, etc.

[0131] For example, during a debate broadcast, depending on which presenter a user is looking at, a 3D animation of that presenter may be played in the video or in a chair in the user's environment. As another example, different video endings may be reached based on the approval and / or participation of the viewer (e.g., user 50 from FIG. 1). As yet another example, military officers may be discussing war strategies with a map displayed in front of the users on a horizontal table. The users may participate in the war strategy discussion by providing input regarding the strategy the officers should implement. In response to providing a decision of which strategy to implement, the video may jump / navigate to the appropriate stream of data and display the video based on the decision.

[0132] At 630, the 3D video may be rendered based on the determination and rendered in portions of the video that correspond to the appropriate streams of data (e.g., video, audio, and / or animation streams) for display to the user.

[0133] In one or more embodiments, the 3D video may be a stereoscopic 3D video. The 3D video may have a video file format comprising the control data. The control data may instruct a video player to render a portion of the 3D video based on at least a decision taken by a user. The portion of the 3D video may be associated with a particular storyline based on one or more decisions taken by the user. The interaction received from the user may include at least one of head pose, eye tracking, gaze, user hand gestures, totem gestures, or an object recognizer. An interaction from one of the interactions received from the user may cause the 3D video to jump to a different portion of the 3D video. (Mixed Reality Video Format)

[0134] Traditional video formats have a common high-level format that is for streaming. Some video formats may include only independent frame based, where all frames of data are unrelated to previous frames and are essentially independent from each other. Another form of video format may be key delta frame based. A common streaming compression technique involves sending in a key frame that is independent from all frames, and only subsequent delta frames contain the difference from that key frame until another key frame is encountered. In theory, it is possible to have only a single key frame with all subsequent delta frames. However, since seeking a frame (e.g., fast forwarding / rewinding) requires reconstruction from the closest key frame, it may be beneficial to have key frames at intervals throughout the video with delta frames in between, as opposed to having only a single key frame and only subsequent delta frames.

[0135] Traditional video is pre-rendered with video and audio clips along with general control of the video such as chapter changes, closed captions for multiple languages, sound quality during playback, etc. An embodiment of the present disclosure may include both pre-rendered and runtime generated 3D frames, which may be reused in some parts of the video. The runtime generated 3D frames may incorporate some assets within the video format. For example, some assets may include 3D models, meshes, animations, textures, shaders, and lights. Furthermore, more sophisticated and complex control mechanisms may control the integration of the runtime generated 3D frames with the pre-rendered frames of the video. A more capable and flexible format will be disclosed herein to encapsulate such assets, which may also be easier to stream.

[0136] FIG. 7 illustrates a video format for displaying 3D video, according to some embodiments. The video format 700 may be a virtual and / or augmented / mixed reality video format. The video format 700 may include a data store 710 and a data stream 750. The data store 710 may be a collection of data that may be loaded or unloaded depending on the section of the video being played. Unlike streaming data, data from the data store 710 may be loaded into memory. In some embodiments, for example, all models 718 and their respective textures 724 may be loaded into memory and remain in memory until the section of the video no longer requires them. In some embodiments, all data from the data store 710 may be loaded into memory. Data from a particular section of the video may be reused in several sections of the video. The data store 710 may include information about the video 712 (e.g., menus), subtitles 714, locales 716, models 718 (e.g., meshes of 3D models), materials 720, textures 724 (e.g., images used by the materials), and control data 726.

[0137] Control data 726 may control the flow and rendering of the video / audio / animation streams via a control stream (disclosed below). The control data may look ahead of the video to determine what is predicted for the video / audio / animation streams so that data-intensive objects (e.g., 3D models) can be preloaded in advance so that the 3D models can be ready for use when the time arrives and the animation stream needs to control the 3D models. The control data may take into account the user's bandwidth and user processing power and determine, for example, the amount of lead time that the control data may need to load a 3D model before the 3D model is needed by the animation stream. Traditionally, video and audio streams do not need to look forward or look ahead very much because their data is simply streamed and played back continuously by the video player. Now, the video, audio, and animation streams may look forward or backward to the extent of the next or previous key frame, respectively. However, the control data may need to look even further ahead because the control data may be involved in controlling what happens next in the rendering and display of the video content. In some embodiments, the entire contents of the control data for a video may be loaded upon the start of the video.

[0138] The control data 726 may determine the models 718 that may be used and animations to link / associate with the models 718 so that the models 718 (e.g., 3D models) may move. For example, the control data 726 may select, from a data store, a 3D model of a fish from the models 718, associate animations with the fish, and cause the 3D model of the fish to swim out of the display screen and into the 3D environment, swim in circles around the display screen displaying the pre-rendered 3D video, and swim back into the display screen so as to be integrated with a similar fish swimming in the pre-rendered 3D video displayed on the display screen within the 3D environment.

[0139] Subtitles 714 may be subtitles corresponding to an audio stream in the video. Subtitles 714 may include subtitles in several languages. Locales 716 may be localized versions of several languages ​​for the in-video audio. Locales 716 may be referenced through locale tags so that videos may be created for several languages. Models 718 may be 3D models of objects in the video that may be rendered and displayed at least partially outside of a display screen. The 3D models may be meshes of the 3D objects. The meshes of the 3D objects may be represented as a network of lines that connect a set of vertices to form a 3D model of the 3D object. Materials 720 may be various materials that may be used to cover the 3D model / mesh. Textures 724 may be various textures that may be used to render models 718. In some embodiments, additional or different 3D model data may be included, so long as the data represents a renderable 3D model.

[0140] The data stream 750 section of the mixed reality video format may be a sliding window of data including key frames and delta frames of video, audio, and animation streams. The data stream 750 may include a video stream 760, an audio stream 770, and an animation stream 780. The video stream 760 may include video data for the video. The audio stream 770 may include audio data corresponding to the video stream 760.

[0141] The animation stream 780 may include model animations that may be applied to the compatible models 718. The animation stream 780 may control how the 3D model / mesh moves and behaves relative to the video and audio streams. For example, the 3D model / mesh may be a 3D model of a fish. The animation stream may include instructions on how the fish moves and where the fish is displayed relative to the display screen. For example, at some point during the video, the animation stream may instruct the 3D model of the fish to be displayed as swimming out of the video screen and into the user's environment. The fish may swim around the video screen, and the fish may swim back into the video screen, at which time the animation stream for the fish may end. Each 3D model may have its own animation stream. Some 3D models may be associated / linked to more than one animation stream. The multiple animation streams depicted in FIG. 7 indicate that in this embodiment, there is more than one animation stream, and thus one or more models may be associated with multiple animation streams. In other embodiments, a single animation stream may be used to manage the display of one or more 3D models in the video.

[0142] A control stream (not shown in FIG. 7) may be responsible for indicating models / meshes / textures, etc. to load from a data store and link the models to an animation stream associated with the models at the appropriate time in the video's timeline. In some embodiments, the control stream may be dynamically generated from the control data and a timeline controller module (discussed below) as the video player reads / receives data from the video format. The control stream may contain only key frames. The control stream may contain commands (which may be called control commands) and / or simple logic gates to determine the combination of one or more models and / or animations to play at the appropriate time in the video's timeline.

[0143] In some embodiments, the control stream is data that is not streamed from a video file, but is preloaded into memory from the control data 726 in the data store 710. Like subtitles, the control stream data does not need to be streamed from a file because both the subtitle and control data file sizes are generally not very large, and therefore both can be loaded into memory without putting much strain on the amount of memory used. Preloading the control stream into memory provides significant control over the buffering required to load other data from the data store 710. For example, the control stream that is preloaded into memory may be configured to look ahead in time in the video (e.g., ahead of the video or audio stream) to determine future models and animations to be displayed. When the control stream determines that a future model is needed (e.g., relative to the current time according to a time controller for the video currently being displayed to the user), the appropriate model and its associated animation (if applicable) may be preloaded into buffered memory such that the 3D model may be displayed in sync with the video, the video's audio, and animation streams when the model is executed by the control stream. In some embodiments, the control stream is streamed instead of being preloaded into memory.

[0144] In some embodiments, the 3D model may only be displayed without any animation. The control stream may determine, based at least in part on the size of the model, the user's network bandwidth, and the processing power of the user's VR and / or AR system, how much time in advance the model may need to be loaded into memory so that when the time arrives to display the model and its associated animation (if applicable), video playback will not be delayed due to the model not being ready or not displaying in time. In some embodiments, the control stream may determine that a model that is loaded into memory may not be needed for a certain amount of time and therefore may be unloaded from memory until the model is needed again at a future time.

[0145] In one embodiment, the video format may include an animation stream, a data store, and at least one of a video stream or an audio stream. One or more animation streams may be applied to individual compatible models. In some embodiments, the data store may be a collection of data that is loaded and unloaded depending on the section of the video being played.

[0146] In one or more embodiments, the data store may be fully loaded into memory when an application that reads the video file format is first accessed. The control stream may be preloaded into memory from one or more control data 726 from the data store 710 when the control data is fully loaded into memory. The control stream may be configured to read the control data prior to the corresponding video and / or audio data in the video file. In some embodiments, the control stream is read by one or more processors of the MR system asynchronously with the video and / or audio streams, but may be executed synchronously. The control stream may comprise commands (e.g., control commands) and / or simple logic gates to determine the combination of models and animations to play at the appropriate time in the video timeline. The appropriate time in the video timeline corresponds to the corresponding time in the video and audio streams.

[0147] In one or more embodiments, the control stream comprises only key frames. The animation stream, video stream, and audio stream may be configured to be visible before and after a key frame. The one or more animation streams may correspond to animation instructions associated with one or more 3D models. Each animation stream of the one or more animation streams corresponds to at least one of the 3D models, textures, or materials of the data store.

[0148] The richness of the features of the mixed reality video format may be handled by several different control commands supported by the mixed reality video player. In some embodiments, the control data may include a set of control commands to allow the user to control objects and functions within the video and outside of the video. In some embodiments, the control commands may be extensible. In some embodiments, the control commands may be updated. In some embodiments, the control commands may be dynamic. Examples of types of control commands may be 1) changing the speed of the movie (which may be based on a user history of, for example, fast-forwarding or skipping certain parts of the movie, for example, fast-forwarding through the credits and reaching the post-credits scene), 2) providing a user prompt, or automatically dimming the lights at the beginning of the movie and returning to the original lights at the end of the movie, 3) automatically calling a pizza restaurant at the beginning of the movie, or 4) playing the movie differently depending on the user's line of sight. In some embodiments, the control stream may be a full scripting language. In some embodiments, the mixed reality movie of the present disclosure is a movie or video with a programming language. In some embodiments, the terms "control data" and "control stream" are used synonymously. (Mixed reality video player)

[0149] The video player may interpret a 3D video format (e.g., video format 700). The video player may be a virtual and / or augmented / mixed reality video player. The video player may have access to the animation streams and data in the data store 710 of the video format 700, so the video player may read and / or execute the video format. A typical video player may read the data, separate the audio data and / or streams from the video data and / or streams, decode the data, and display the video. However, a mixed reality video player may perform several more actions, such as loading a 3D model into memory, attaching animation streams to the 3D model, and removing the model and corresponding animation from memory when they are no longer needed or have not been needed for a threshold time period. Furthermore, additional interactive controls may allow the user to make decisions to modify the video or to cause parts of the video to interact with the environment.

[0150] For example, the video may adapt to the environment by placing a portion of the video content to be displayed on a certain object in the world (e.g., placing reporter 130 on the user's physical chair in the user's environment), e.g., using additional interactive controls provided by the mixed reality video player. As another example, a user may be able to provide input regarding decisions made at certain points in the video that may alter and adjust the video content that may be displayed to the user, and possibly the way the video may end.

[0151] A video director typically creates a traditional video. With a mixed reality video player, a user may create a video. For example, when playing the video, the video may ask the user if the user would like to attack an enemy. If the user chooses to attack, the user may be defeated and the video may reach a particular ending. However, if the user chooses not to attack, another portion of the video may be played and a different ending of the video may be reached.

[0152] 8 shows a component diagram of a video player according to some embodiments. The video player 805 may be a virtual and / or augmented / mixed reality player. The video player 805 may include a time controller 810, a data stream buffer and splitter 815, a data storage buffer and controller 820, a video / audio decoder 825, an animation decoder 830, a control stream 835, and a scene setting module 840.

[0153] The time controller 810 may control the position of a video (e.g., movie) stream. For regular video, the time controller may simply increment the position of a video slider relative to a timeline. In response to commands received from the command executor 850 (disclosed below), the timeline of the video may jump to various sections of the video.

[0154] The data stream buffer and splitter 815 may be responsible for buffering in data and splitting the incoming file data stream 750 into individual streams such as video stream, audio stream, and animation stream. The data store buffer and controller 825 may be responsible for buffering in data from the data store 710 and determining what data to preload. Unlike the data stream buffer and splitter 815, the data store buffer and controller 825 needs to establish what data to start streaming based on the control stream 835 and the time controller 810.

[0155] Incoming data may be streamed into a Video / Audio Decoder 825, and output from the Video / Audio Decoder 825 may be written to one or more buffer frames. Audio may be routed to the audio handling portion of the engine. An Animation Decoder 830 may decode the animation stream into a format that can be applied directly to the model.

[0156] The control stream 835 may be dynamically generated from the control data 726 and the time controller 810 from Figure 7, as disclosed above. The control stream 835 may indicate commands to execute within the current frame (e.g., the frame being displayed to the user via the MR system).

[0157] Scene Settings 840 may be a collection of modules that may provide additional interactive control to the user, allowing the video to interact with the environment and / or allowing the user to interact with the video. Scene Settings 840 may include input controllers 845, command executors 850, runtime interactivity progress / save data 860, and environmentally aware positioning system 865.

[0158] The input controllers 845 may be responsible for delivering input from user actions, such as gestures, controller inputs, dialog boxes, the environment (world object recognizers, e.g., a chair), and the like.

[0159] Runtime interactive progress / save data 860 may store the current nature / state of the video. Similar to a video game, runtime interactive progress / save data 860 may be data that will be written to a save file for the video game as the user progresses through the video game. Here, as the user progresses through the video, runtime interactive progress / save data 860 may store the video flow that has been rendered and displayed thus far. In particular, if the user makes a decision that has sent a flow of video to a particular section of the video file, runtime interactive progress / save data 860 may include the previous key frames that have been displayed in a particular order to maintain the state of the video for the user by capturing the video history.

[0160] The environmentally aware positioning system 865 may be involved in dynamically positioning videos, models, etc. based on the user watching the video. For example, depending on the user's home setting, the 3D model and video frames may be suitably positioned. The suitable position may be determined in various ways. In some embodiments, the 3D model may have corresponding placement data. This placement data may specify the type of object (e.g., chair, floor, etc.) on or near which the object may be placed, or may specify a set of characteristics required to display the content (e.g., instead of specifying a chair, a horizontal surface 1-3 feet above the floor may instead be specified). The environmentally aware positioning system 865 may communicate with a mixed reality engine (e.g., to access a passable world) to suitably position the 3D model.

[0161] Commands from the control stream 835 may be received and interpreted by the command executor 850, and appropriate logical actions may be performed. This may be where the initial integration of the model and animation streams takes place. The logical actions may also be determined at least in part from user input, and the results may be stored in the runtime interactivity data 860. One or more command examples may be added to the command executor 850 that extend and enrich the capabilities of the video player. Some examples include (1) presenting a model on the screen and associating it with an animation stream, (2) relocating the model to a location in the user's environment (if there is no chair, a default location may be used), and (3) presenting an interaction dialog for the user to select a decision.

[0162] The output of the mixed reality video player 805 may be sent to a mixed reality engine 890. The mixed reality engine 890 may be similar to a game engine. The mixed reality engine 890 may be an engine capable of rendering videos and models, which may be used for the final composition of the scene based on the user's head pose. Interaction and access to other VR and / or AR technologies and / or capabilities may be interfaced through this mixed reality engine 890.

[0163] FIG. 9 illustrates the flow of a video player that processes video formats, according to some embodiments. The video player 900 may be a virtual and / or augmented / mixed reality video player. The flow of the video player 900 is depicted relative to a timeline, where time flows starting from the top of FIG. 9, progressing to the bottom of FIG. 9, and ending at the video end 970. The timeline controller 810 may be the time controller 810 from FIG. 8. The timeline controller 810 may control the position of the video stream, which may indicate the current frame being displayed to the user. The video player 900 may read / receive data from a mixed reality video file in a mixed reality video file format. The video feature of the mixed reality video format may be a stereoscopic 3D video. The mixed reality video file format may include an animation stream 915, a data store with control data, a video stream 905, and an audio stream 910. The animation stream 915, the video stream 905, and the audio stream 910 may correspond to the animation stream 780, the video stream 760, and the audio stream 770, respectively, from FIG.

[0164] The video stream 905 may include one or more streams of video data for the video. The audio stream 910 may include one or more streams of audio data for the video. The one or more audio streams may correspond to different languages ​​that the video may use depending on a user selected option. The one or more video streams may correspond to multiple views that are captured, for example. In current video compression approaches, the video and audio data may be data intensive such that the video and audio data may be streamed to a video player. As discussed above, a common streaming compression technique involves sending in a key frame 925 that may be independent of every frame, and a subsequent delta frame 930 that may carry only the differences from the key frame 925 until another key frame 933 is encountered.

[0165] As disclosed above, the animation stream 915 may include model animations that may be applied to compatible 3D models. The animation stream 915 may control how the 3D models / meshes move and behave (e.g., movement, orientation, position, etc.) relative to the video and audio streams (e.g., video). Animation data may be streamed to the video player 900 in one or more animation streams. The one or more animation streams may correspond to one or more models / meshes / textures that may be displayed at specific times during the video, and the one or more models may be displayed outside of the display screen of the 3D video.

[0166] The control stream 920 may be dynamically generated from control data received from a data store and interpreted by the time controller 810. As disclosed above, the control stream may be responsible for loading from the data store 945 and indicating models / meshes / textures etc. to link to associated animation streams (e.g., at 940) at the appropriate times in the video timeline to correspond with frames from the video stream 905 and audio stream 910. The control stream 920 may include key frames (e.g., key frame 935a and key frame 935b). The control stream 920 may include commands and simple logic gates to determine the combination of models and animations to play at the appropriate times in the video timeline.

[0167] In some embodiments, the control stream 920 is not streamed from a video file, but instead is data that is preloaded into memory. Because the control stream 920 is preloaded into memory, the video player 900 may look ahead in time in the video timeline and / or video stream to determine when a 3D model from a data store, animated or not, may need to be preloaded into memory in anticipation of the 3D model being displayed. In some embodiments, the lead time to load the model may be determined based on the size of the 3D model's data, the network bandwidth, and / or the processing power of the user's virtual and / or augmented reality system (e.g., the display system 104). In some embodiments, the control stream is streamed instead of being preloaded into memory. In some embodiments, the control stream and control data may be used synonymously. In some embodiments, this may be because the control stream is the control data that is being executed. In some embodiments, this may be because they are one and the same. In some embodiments, this may be because the control data is being streamed.

[0168] In some embodiments, the video player 900 may receive a video file comprising a video stream, an audio stream, an animation stream, and / or control data. The control stream may be generated from the control data and a time controller of the video player. At 940, the 3D model may be preloaded into a memory (not shown) based at least in part on a control stream prediction of when the 3D model may need to be displayed in the video. The video player may determine (e.g., calculate) a lead time for loading the 3D model to ensure that the 3D model is ready to be displayed to the user, with or without animation. If animation is included with the 3D model, the control stream may attach the 3D object to the animation stream by associating a link between the loaded 3D model and the animation stream from the animation stream 915. The video may be displayed to the user using a VR and / or AR device (e.g., the display system 104). The user may see the 3D video displayed on the virtual television such that in some parts of the video, the 3D object may appear to move out of front of the virtual television, move around in the user's environment, and optionally re-enter the virtual television.

[0169] In another embodiment, the control stream 920 may instruct the video player 900 to display a question to the user, for example, at 950, whether to attack at dawn. If the user selects "yes" and attacks at dawn, the control stream may tag the timeline with "success" at 965 and communicate a mission "success" tag to the timeline controller 810. Alternatively, if the user selects "no" and does not attack at dawn, the control stream may tag the timeline with "failure" at 960 and communicate a mission "fail" tag to the timeline controller 810.

[0170] After receiving the timeline tag for a mission, the timeline controller 810 may skip / jump the timeline to the appropriate portion of the video and continue the video, whether it is to a portion of the video that continues based on a successful mission or a portion of the video that continues based on a failed mission. The interaction received from the user may include at least one of head pose, eye tracking, gaze, user hand gesture, totem gesture or input, or object recognizer. At 955, the state of the video may be stored with the run-time interactivity data to capture answers provided by the user. This embodiment discloses a method in which a video player 900 that receives a video file (e.g., a mixed reality video format file) may display a user interactive 3D video using a control stream 920 and capture user interactions that indicate the portion of the video to be displayed next, thus allowing the user to indicate and control how the storyline of the video may end. In some embodiments, the run-time interactivity data 955 may at least partially interpret the control command. For example, 955 may interpret the user selection when the control command comprises a user option. Runtime interactivity data may, in some embodiments, be passed to command executor 850 where user selections may affect the state of the video. In some embodiments, 955 may interpret control commands that do not comprise user choices. For example, the control commands may function to automatically censor video content based on the user's age (which may be stored in the user's profile on the mixed reality system).

[0171] A mixed reality video editor (not shown) for creating and editing a mixed reality video file format may include tools for editing video streams, audio streams, animation streams, and data stores of the mixed reality video format. The tools may, for example, allow a user to (1) create or import models into the data store, (2) define animation streams from stored animation streams and / or create new animation streams for controlling models, (3) associate animation streams with compatible models, (4) define when models may be preloaded, for example, based on the configured user bandwidth and the processing power of the user's VR and / or AR system, (5) define portions of the video and audio streams that may be displayed on objects in the user's environment, and / or (6) define how and / or what questions are asked to a user watching the video and how each answer from the user may skip to a portion of the video, etc.

[0172] The mixed reality video file format may include at least one animation stream and additional assets within a traditional video file format and data storage for implementing one or more embodiments of the present disclosure. A mixed reality video player may be used to interpret and process the mixed reality video within the mixed reality video file format.

[0173] A video player that interprets and executes the control stream may enable the VR and / or AR system to display 3D video in new and novel ways. For example, displaying a 3D object as if it is outside the display screen may be accomplished by preloading the 3D model into memory based on a prediction of when the 3D model needs to be displayed and attaching the 3D model to an animation stream for controlling the movement of the 3D model. This may improve the ability of the virtual and / or augmented reality system to display the 3D model in combination with the 3D video and provide a more realistic view of the 3D video, where the 3D objects in the 3D video may appear to actually spill or spill out of a traditional display screen that displays traditional 3D video. In some embodiments, the addition of the animation stream 915, control data, and control stream 920, when interpreted by the video player 900, enables the VR and / or AR system to display 3D video with 3D objects that may actually spill or spill out of the display screen in an efficient manner that reduces computer processing power and reduces the amount of memory that may be required to generate the 3D video compared to other techniques such as voxel and / or volumetric based video streams. Additional Embodiments

[0174] Additional embodiments of the present disclosure are described below. These additional embodiments may incorporate elements from the embodiments disclosed above. Embodiment 1. 1 is an embodiment, identifying a portal within the final 3D rendered world, a portal being an opening within a surface of the final 3D rendered world; rendering the first 2D stereoscopic image and the second 2D stereoscopic image into a portal in the final 3D rendering world; An embodiment comprising: Embodiment 2. 2. The method of embodiment 1, further comprising the step of displaying the final rendered 3D rendering world to the user through an augmented reality device. Embodiment 3. 2. The method of embodiment 1, wherein the first 2D stereoscopic image is for the right eye and the second 2D stereoscopic image is for the left eye. Embodiment 4. 4. The method of embodiment 3, wherein the first 2D stereoscopic image and the second 2D stereoscopic image are provided from a conventional 3D video. Embodiment 5. 4. The method of embodiment 3, wherein the first 2D stereoscopic image and the second 2D stereoscopic image are provided from two virtual rendering cameras located within different portions of the 3D rendering world. Embodiment 6. 4. The method of embodiment 3, wherein the first 2D stereoscopic image and the second 2D stereoscopic image are provided from two cameras that capture real-world content. Embodiment 7. 4. The method of embodiment 3, wherein the first 2D stereoscopic image and the second 2D stereoscopic image are provided from different 3D rendering worlds. Embodiment 8. 8. The method of embodiment 7, wherein the different 3D rendering worlds are portal icons. EMBODIMENT 9. 2. The method of embodiment 1, wherein the portal is a virtual television screen. Embodiment 10. 1. A virtual and / or augmented reality display system, comprising: The final 3D rendering space, a module for processing data, the module being stored in a memory; identifying a portal within the final 3D rendered world, a portal being an opening within a surface of the final 3D rendered world; rendering the first 2D stereoscopic image and the second 2D stereoscopic image into a portal in the final 3D rendering world; A module configured to implement the A virtual and / or augmented reality display system comprising: Embodiment 11. A virtual and / or augmented reality display system as described in embodiment 10, further comprising a step of displaying the rendered final 3D rendering world to a user. Embodiment 12. 11. A virtual and / or augmented reality display system as described in embodiment 10, wherein the first 2D stereoscopic image is for the right eye and the second 2D stereoscopic image is for the left eye. Embodiment 13. 13. A virtual and / or augmented reality display system as described in embodiment 12, wherein the first 2D stereoscopic image and the second 2D stereoscopic image are supplied from a conventional 3D video. Embodiment 14. A virtual and / or augmented reality display system as described in embodiment 12, wherein the first 2D stereoscopic image and the second 2D stereoscopic image are provided from two virtual rendering cameras located within different parts of the 3D rendering world. EMBODIMENT 15. 13. A virtual and / or augmented reality display system as described in embodiment 12, wherein the first 2D stereoscopic image and the second 2D stereoscopic image are provided from two cameras that capture real-world content. Embodiment 16. 13. A virtual and / or augmented reality display system as described in embodiment 12, wherein the first 2D stereoscopic image and the second 2D stereoscopic image are supplied from a second 3D rendering world. EMBODIMENT 17. 17. A virtual and / or augmented reality display system as described in embodiment 16, wherein the second 3D rendering world is a portal icon. Embodiment 18. 11. A virtual and / or augmented reality display system as described in embodiment 10, wherein the portal is a virtual television screen. EMBODIMENT 19. 1. A method for displaying 3D video within a virtual and / or augmented reality environment, comprising: identifying a 3D video; identifying a volumetric space for displaying 3D video within a portion of the virtual and / or augmented reality environment; Rendering the 3D video in a volumetric space; A method comprising: EMBODIMENT 20. 20. The method of embodiment 19, wherein the 3D video is a stereoscopic 3D video. EMBODIMENT 21. 20. The method of embodiment 19, wherein the volumetric space is a portal view of a 3D object. EMBODIMENT 22. 22. The method of embodiment 21, wherein the 3D object is a virtual television having a flat-panel display. EMBODIMENT 23. 23. The method of embodiment 22, wherein the 3D video is rendered within a flat-view display of a virtual television. EMBODIMENT 24. The method of embodiment 19, wherein first depth information from the 3D video is added to second depth information from a first location of a portion of the virtual and / or augmented environment volume space to a second location of the user viewing the 3D video. EMBODIMENT 25. 1. A virtual and / or augmented reality display system, comprising: a camera for capturing a virtual and / or augmented reality environment; A module for processing data, the module being stored in a memory, the module comprising: Identify 3D videos, Identifying a volumetric space for displaying 3D video within a portion of the virtual and / or augmented reality environment; Rendering 3D video in volumetric space, A module, A virtual and / or augmented reality display system comprising: 26. 26. A virtual and / or augmented reality display system as described in embodiment 25, wherein the 3D video is stereoscopic 3D video. EMBODIMENT 27. A virtual and / or augmented reality display system as described in embodiment 25, wherein the volumetric space is a portal view of a 3D object. 28. 28. A virtual and / or augmented reality display system as described in embodiment 27, wherein the 3D object is a virtual television having a flat-panel display. 29. 29. A virtual and / or augmented reality display system as described in embodiment 28, wherein 3D video is rendered within a virtual television flat-panel display. EMBODIMENT 30. A virtual and / or augmented reality display system as described in embodiment 25, wherein first depth information from the 3D video is added to second depth information from a first location in a portion of the virtual and / or augmented environment volume space to a second location of the user viewing the 3D video. EMBODIMENT 31. 1. A method for displaying 3D video that extends beyond a surface of a display screen, comprising: identifying a 3D video; generating one or more 3D models corresponding to the 3D video; rendering one or more 3D models at appropriate trigger times along with the rendering of the 3D video; A method comprising: EMBODIMENT 32. 32. The method of embodiment 31, wherein the 3D video is stereoscopic 3D video. EMBODIMENT 33. 33. The method of embodiment 32, wherein the one or more 3D models are generated with animation. EMBODIMENT 34. 34. The method of embodiment 33, further comprising displaying animation of one or more 3D models outside the display screen of the 3D video. EMBODIMENT 35. 35. The method of embodiment 34, wherein animation of the one or more 3D models appears to move out of the plane of the 3D video and into the user's 3D environment. Embodiment 35a. 35. The method of embodiment 34, wherein animation of the one or more 3D models appears to exit the non-planar surface of the 3D video and enter into the user's 3D environment. EMBODIMENT 36. rendering one or more 3D models on one or more video panes; displaying one or more video panes simultaneously for one or more viewing orientations; 32. The method of embodiment 31, further comprising: 37. 33. The method of embodiment 32, wherein the stereoscopic 3D video is displayed accompanied by one or more 3D models. 38. The method of embodiment 31, wherein depth information is taken into account in the rendering of the 3D video and one or more 3D models, the depth information comprising a distance from a user of a virtual and / or augmented reality device to the 3D video, and the individual one or more 3D models are displayed within the virtual and / or augmented reality environment. 39. 32. The method of embodiment 31, wherein the 3D video comprises control data that instructs the video player to render one or more 3D models at appropriate trigger times along with rendering the 3D video. Embodiment 39a. 40. The method of embodiment 39, wherein the control data comprises a set of control commands for controlling objects and functions within and outside the 3D video. Embodiment 39b. The method of embodiment 39a, wherein the control command (e.g., the set of control commands) is at least one of extensible, updatable, or dynamic. Embodiment 39c. The method of embodiment 39b, wherein the control commands include at least one of the following steps: changing the speed of the 3D video, providing a user prompt to dim lights in the user's environment when the 3D video starts, calling a pizza restaurant when the 3D video starts, or playing the 3D video differently based at least in part on the user's line of sight. Embodiment 39d. The method of embodiment 31, wherein the 3D video is a 3D movie and comprises a programming language. EMBODIMENT 40. 32. The method of embodiment 31, wherein the rendering of one or more 3D models is based at least in part on a voxel-based video stream. EMBODIMENT 41. 1. A display system for displaying 3D video extending beyond a surface of a display screen, comprising: an augmented reality head mounted display system; 3D final rendering world and Two or more virtual cameras; one or more modules for processing data, the one or more modules being stored in one or more memories, the one or more modules comprising: identifying a 3D video; generating one or more 3D models corresponding to the 3D video; rendering one or more 3D models at appropriate trigger times along with the rendering of the 3D video; one or more modules configured to implement the A system comprising: EMBODIMENT 42. A virtual and / or augmented reality display system as described in embodiment 41, wherein the 3D video is stereoscopic 3D video. EMBODIMENT 43. A virtual and / or augmented reality display system as described in embodiment 42, wherein one or more 3D models are generated with animation. EMBODIMENT 44. A virtual and / or augmented reality display system as described in embodiment 43, wherein the module further includes a step of displaying animation of one or more 3D models outside the display screen of the 3D video. EMBODIMENT 45. A virtual and / or augmented reality display system as described in embodiment 44, wherein animation of one or more 3D models appears to move out of the plane of the 3D video and into the user's 3D environment. Embodiment 45a. A virtual and / or augmented reality display system as described in embodiment 44, wherein animation of one or more 3D models appears to exit the non-planar surface of the 3D video and enter into the user's 3D environment. EMBODIMENT 46. One or more of the modules may further include: rendering one or more 3D models on one or more video panes; displaying one or more video panes simultaneously for one or more viewing orientations; 42. A virtual and / or augmented reality display system as described in embodiment 41, configured to implement the above. EMBODIMENT 47. 43. A virtual and / or augmented reality display system as described in embodiment 42, wherein stereoscopic 3D video is displayed in conjunction with one or more 3D models. 48. A virtual and / or augmented reality display system as described in embodiment 41, wherein depth information is taken into account in the rendering of the 3D video and one or more 3D models, the depth information comprising a distance from a user of the virtual and / or augmented reality device to the 3D video, and the individual one or more 3D models are displayed within the virtual and / or augmented reality environment. 49. A virtual and / or augmented reality display system as described in embodiment 41, wherein the 3D video has a video file format comprising control data that instructs a video player to render one or more 3D models at appropriate trigger times along with rendering the 3D video. Embodiment 49a. A virtual and / or augmented reality display system as described in embodiment 49, wherein the control data comprises a set of control commands for controlling objects and functions within and outside the 3D video. Embodiment 49b. A virtual and / or augmented reality display system as described in embodiment 49a, wherein the control commands (e.g., the set of control commands) are at least one of extensible, updatable, or dynamic. Embodiment 49c. The virtual and / or augmented reality display system of embodiment 49b, wherein the control commands include at least one of the following: changing the speed of the 3D video, providing a user prompt to dim lights in the user's environment when the 3D video starts, calling a pizza restaurant when the 3D video starts, or playing the 3D video differently based at least in part on the user's line of sight. Embodiment 49d. A virtual and / or augmented reality display system as described in embodiment 31, wherein the 3D video is a 3D movie, the 3D movie comprising a programming language. EMBODIMENT 50. A virtual and / or augmented reality display system as described in embodiment 41, wherein the rendering of one or more 3D models is based at least in part on a voxel-based video stream. EMBODIMENT 51. 1. A method comprising: placing two first-stage cameras in a 3D environment; capturing one or more pairs of 2D images from two first stage cameras; placing one or more pairs of 2D images into locations within the final scene; rendering the end-stage scene from two end-stage cameras; A method comprising: EMBODIMENT 52. 52. The method of embodiment 51, further comprising the step of displaying the final stage scene via a virtual and / or augmented reality device. EMBODIMENT 53. 52. The method of embodiment 51, wherein the location is a portal. EMBODIMENT 54. The method of embodiment 51, wherein a first of the two first stage cameras captures a 2D image from a left eye perspective and a second of the two first stage cameras captures a 2D image from a right eye perspective. EMBODIMENT 55. The method of embodiment 51, wherein one or more other pairs of two 2D images are captured from two end-stage cameras corresponding to both eyes of the user. EMBODIMENT 56. 56. The method of embodiment 55, wherein the two first stage cameras are in different locations in the 3D environment than the two final stage rendering cameras. EMBODIMENT 57. 52. The method of embodiment 51, wherein the two first stage cameras are virtual cameras and the 3D environment is a 3D virtual world. EMBODIMENT 58. 52. The method of embodiment 51, wherein the 3D environment is the real world. EMBODIMENT 59. 1. A virtual and / or augmented reality display system, comprising: two first stage cameras positioned within the 3D environment, the two first stage cameras capturing one or more pairs of 2D images; two end-of-line cameras for capturing an end-of-line scene; a module for processing data, the module being stored in a memory; placing one or more pairs of 2D images into locations within the final scene; rendering the end-stage scene from one or more other pairs of 2D images captured from the two end-stage cameras; A module configured to implement the A virtual and / or augmented reality display system comprising: 60. 60. A virtual and / or augmented reality display system as described in embodiment 59, further comprising a step of displaying a final stage scene. 61. A virtual and / or augmented reality display system as described in embodiment 59, wherein the location is a portal. 62. A virtual and / or augmented reality display system as described in embodiment 59, wherein a first of the two first stage cameras captures a 2D image from a left eye perspective and a second of the two first stage cameras captures a 2D image from a right eye perspective. 63. A virtual and / or augmented reality display system as described in embodiment 59, wherein one or more other pairs of two 2D images are captured from two end-stage cameras corresponding to the user's two eyes. 64. A virtual and / or augmented reality display system as described in embodiment 63, wherein the two first stage cameras are in different locations in the 3D environment than the two final stage rendering cameras. 65. A virtual and / or augmented reality display system as described in embodiment 59, wherein the two first stage cameras are virtual cameras and the 3D environment is a 3D rendering world. 66. A virtual and / or augmented reality display system as described in embodiment 59, wherein the 3D environment is the real world. 67. A method comprising placing a first 3D content from a first source within a second 3D content from a second source. 68. 68. The method of embodiment 67, further comprising displaying the second 3D content via a virtual and / or augmented reality device. 69. 68. The method of embodiment 67, wherein the first 3D content is placed within a portal within the second 3D content. EMBODIMENT 70. 68. The method of embodiment 67, wherein a first camera of a first source captures a 2D image from a left eye perspective and a second camera of the first source captures a 2D image from a right eye perspective. EMBODIMENT 71. 68. The method of embodiment 67, wherein another pair of two 2D images is captured from two other cameras from a second source, the pair of two 2D images corresponding to both eyes of a user viewing the scene from the second source. EMBODIMENT 72. 68. The method of embodiment 67, wherein two cameras from a first source are at different locations in the 3D environment than two other cameras from a second source. EMBODIMENT 73. 68. The method of embodiment 67, wherein the first 3D content is captured from two virtual cameras and the first source is a 3D virtual world. 74. 68. The method of embodiment 67, wherein the first source is the real world. 75. 1. A virtual and / or augmented reality display system, comprising: a camera for capturing a virtual and / or augmented reality environment; a module for processing data, the module being stored in a memory, the module performing, when executed, a process of placing a first 3D content from a first source within a second 3D content from a second source; A virtual and / or augmented reality display system comprising: EMBODIMENT 76. A virtual and / or augmented reality display system as described in embodiment 75, wherein the module, when executed, further includes a step of displaying second 3D content via the virtual and / or augmented reality device. 77. 76. A virtual and / or augmented reality display system as described in embodiment 75, wherein the first 3D content is placed within a portal within the second 3D content. 78. A virtual and / or augmented reality display system as described in embodiment 75, wherein a first camera of a first source captures a 2D image from a left eye perspective and a second camera of the first source captures a 2D image from a right eye perspective. 79. A virtual and / or augmented reality display system as described in embodiment 75, wherein another pair of two 2D images is captured from two other cameras from a second source, the pair of two 2D images corresponding to both eyes of a user viewing the scene from the second source. 80. A virtual and / or augmented reality display system as described in embodiment 75, wherein two cameras from a first source are in different locations in the 3D environment than two other cameras from a second source. 81. 76. The virtual and / or augmented reality display system of embodiment 75, wherein the first 3D content is captured from two virtual cameras and the first source is a 3D virtual world. A virtual and / or augmented reality display system as described in embodiment 75, wherein the first source is the real world. 83. A method comprising placing a first set of stereoscopic images inside a second set of stereoscopic images. 84. 84. The method of embodiment 83, further comprising displaying a second set of stereoscopic images via a virtual and / or augmented reality device. 85. 84. The method of embodiment 83, wherein the first set of stereoscopic images is placed within a portal in the second set of stereoscopic images. 86. 84. The method of embodiment 83, wherein the first set of stereoscopic images is captured by a first camera that captures 2D images from a left eye perspective and a second camera that captures 2D images from a right eye perspective. 87. The method of embodiment 83, wherein a second set of stereoscopic images is captured by two other cameras, the two other cameras capturing the second set of stereoscopic images, the two other cameras corresponding to both eyes of a user viewing the scene showing the first set of stereoscopic images within the portal. 88. 88. The method of embodiment 87, wherein a first set of stereoscopic images is captured from two cameras from a different location than two other cameras that capture a second set of stereoscopic images. 89. 84. The method of embodiment 83, wherein a first set of stereo images is captured from two virtual cameras in a 3D virtual world. EMBODIMENT 90. 84. The method of embodiment 83, wherein the first set of stereo images is captured from two cameras in the real world. EMBODIMENT 91. 1. A virtual and / or augmented reality display system, comprising: a camera for capturing a virtual and / or augmented reality environment; a module for processing data, the module being stored in a memory, the module performing, when executed, a process of placing a first set of stereoscopic images inside a second set of stereoscopic images; A virtual and / or augmented reality display system comprising: EMBODIMENT 92. A virtual and / or augmented reality display system as described in embodiment 91, wherein the module, when executed, implements a process that further includes a step of displaying a second set of stereoscopic images via a virtual and / or augmented reality device. EMBODIMENT 93. A virtual and / or augmented reality display system as described in embodiment 91, wherein a first set of stereoscopic images is placed within a portal within a second set of stereoscopic images. EMBODIMENT 94. A virtual and / or augmented reality display system as described in embodiment 91, wherein a first set of stereoscopic images is captured by a first camera that captures 2D images from a left eye perspective and a second camera that captures 2D images from a right eye perspective. EMBODIMENT 95. A virtual and / or augmented reality display system as described in embodiment 91, wherein a second set of stereoscopic images is captured by two other cameras, the two other cameras capturing the second set of stereoscopic images, the two other cameras corresponding to both eyes of a user viewing the scene showing the first set of stereoscopic images within the portal. EMBODIMENT 96. A virtual and / or augmented reality display system as described in embodiment 95, wherein a first set of stereoscopic images is captured from two cameras from a different location than two other cameras that capture a second set of stereoscopic images. EMBODIMENT 97. A virtual and / or augmented reality display system as described in embodiment 91, wherein a first set of stereoscopic images is captured from two virtual cameras in a 3D virtual world. EMBODIMENT 98. A virtual and / or augmented reality display system as described in embodiment 91, wherein a first set of stereoscopic images is captured from two cameras in the real world. EMBODIMENT 99. 1. A method comprising: identifying two input images, one input image corresponding to a left eye viewpoint and a second input image corresponding to a right eye viewpoint; placing the two input images within defined locations within the final 3D rendering world. EMBODIMENT 100. 100. The method of embodiment 99, further comprising displaying the final 3D rendered world via a virtual and / or augmented reality device. EMBODIMENT 101. 100. The method of embodiment 99, wherein the specified location is a portal within the final 3D rendering world. EMBODIMENT 102. 100. The method of claim 99, wherein the two input images are captured by a first camera that captures a 2D image from a left eye viewpoint and a second camera that captures a 2D image from a right eye viewpoint. EMBODIMENT 103. The method of embodiment 99, wherein the final 3D rendering world is captured by two other cameras, the two other cameras corresponding to both eyes of a user viewing a scene showing two input images within a portal. EMBODIMENT 104. The method of embodiment 103, wherein the two input images are captured from two cameras from different locations than two other cameras that capture the final 3D rendering world. EMBODIMENT 105. 100. The method of embodiment 99, wherein the two input images are captured from two virtual cameras in a 3D virtual world. EMBODIMENT 106. 100. The method of embodiment 99, wherein the two input images are captured from two cameras in the real world. EMBODIMENT 107. 1. A virtual and / or augmented reality display system, comprising: a camera for capturing a virtual and / or augmented reality environment; A module for processing data, the module being stored in a memory, the module being executed to: a process of identifying two input images, one input image corresponding to a left eye viewpoint and a second input image corresponding to a right eye viewpoint; placing two input images into defined locations within the final 3D rendering world; A module for performing A virtual and / or augmented reality display system comprising: EMBODIMENT 108. A virtual and / or augmented reality display system as described in embodiment 107, wherein the module, when executed, implements a process that further includes a step of displaying the final 3D rendered world via a virtual and / or augmented reality device. EMBODIMENT 109. A virtual and / or augmented reality display system as described in embodiment 107, wherein the specified location is a portal within the final 3D rendering world. EMBODIMENT 110. A virtual and / or augmented reality display system as described in embodiment 107, wherein the two input images are captured by a first camera that captures a 2D image from a left eye perspective and a second camera that captures a 2D image from a right eye perspective. EMBODIMENT 111. A virtual and / or augmented reality display system as described in embodiment 107, wherein the final 3D rendered world is captured by two other cameras, which capture the final 3D rendered world and the two other cameras correspond to both eyes of a user viewing a scene showing the two input images within a portal. EMBODIMENT 112. A virtual and / or augmented reality display system as described in embodiment 111, wherein the two input images are captured from two cameras from different locations than two other cameras that capture the final 3D rendering world. EMBODIMENT 113. A virtual and / or augmented reality display system as described in embodiment 107, wherein the two input images are captured from two virtual cameras in a 3D virtual world. EMBODIMENT 114. A virtual and / or augmented reality display system as described in embodiment 107, wherein the two input images are captured from two cameras in the real world. EMBODIMENT 115. 1. A method comprising: identifying a 3D data input; placing the 3D data input into a location within the virtual final 3D rendering world; A method comprising: EMBODIMENT 116. The method of embodiment 115, further comprising a step of displaying the virtual final 3D rendering world via a virtual and / or augmented reality device. EMBODIMENT 117. The method of embodiment 115, wherein the location is a portal within the virtual final 3D rendering world. EMBODIMENT 118. The method of embodiment 115, wherein the 3D data input is captured by a first camera that captures a 2D image from a left eye perspective and a second camera that captures a 2D image from a right eye perspective. EMBODIMENT 119. The method described in embodiment 115, wherein the virtual final 3D rendering world is captured by two other cameras, the two other cameras capturing the virtual final 3D rendering world, and the two other cameras corresponding to both eyes of a user viewing a scene showing 3D data input within the portal. EMBODIMENT 120. The method of embodiment 119, wherein the 3D data input is captured from two cameras from locations different from two other cameras that capture the virtual final 3D rendering world. EMBODIMENT 121. The method of embodiment 115, wherein the 3D data input is captured from two virtual cameras in the 3D virtual world. EMBODIMENT 122. The method of embodiment 115, wherein the 3D data input is captured from two cameras in the real world. EMBODIMENT 123. 1. A virtual and / or augmented reality display system, comprising: a camera for capturing a virtual and / or augmented reality environment; A module for processing data, the module being stored in a memory, the module being executed to: A process for identifying 3D data input; a process of placing the 3D data input into a location within a virtual final 3D rendering world; A module for performing A virtual and / or augmented reality display system comprising: EMBODIMENT 124. A virtual and / or augmented reality display system as described in embodiment 123, wherein the module, when executed, implements a process that further includes a step of displaying a virtual final 3D rendering world via a virtual and / or augmented reality device. EMBODIMENT 125. A virtual and / or augmented reality display system as described in embodiment 123, wherein the location is a portal within a virtual final 3D rendering world. EMBODIMENT 126. A virtual and / or augmented reality display system as described in embodiment 123, wherein 3D data input is captured by a first camera that captures a 2D image from a left eye perspective and a second camera that captures a 2D image from a right eye perspective. EMBODIMENT 127. A virtual and / or augmented reality display system as described in embodiment 123, wherein the virtual final 3D rendering world is captured by two other cameras, which capture the virtual final 3D rendering world and the two other cameras correspond to both eyes of a user viewing a scene showing 3D data input within the portal. EMBODIMENT 128. A virtual and / or augmented reality display system as described in embodiment 127, wherein the two input images are captured from two cameras from different locations than two other cameras that capture the final 3D rendering world. EMBODIMENT 129. A virtual and / or augmented reality display system as described in embodiment 123, wherein 3D data input is captured from two virtual cameras in a 3D virtual world. EMBODIMENT 130. A virtual and / or augmented reality display system as described in embodiment 123, wherein 3D data input is captured from two cameras in the real world. EMBODIMENT 131. 1. A video file format, comprising: One or more animation streams; Data storage and at least one of a video stream or an audio stream; A video file format comprising: EMBODIMENT 132. The data storage is one or more pieces of control data; One or more 3D models; 132. The video file format of embodiment 131, comprising: Embodiment 132a. The data storage is One or more textures; One or more materials; 132. The video file format of embodiment 131, comprising: EMBODIMENT 133. A video file format as described in embodiment 132, wherein the control data is fully loaded into memory when an application that reads the video file format is first accessed. EMBODIMENT 134. A video file format as described in embodiment 133, in which the control stream is pre-loaded into memory from one or more control data from a data store when the control data is completely loaded into memory. Embodiment 134a. 135. A video file format as described in embodiment 134, wherein the control stream is a full scripting language. EMBODIMENT 135. A video file format as described in embodiment 134, wherein the control stream is configured to view a time corresponding to a timeline ahead of the video stream or audio stream. EMBODIMENT 136. A video file format as described in embodiment 134, in which the control stream comprises commands and simple logic gates that determine the combination of models and animations to play at the appropriate times within the video's timeline. EMBODIMENT 137. A video file format as described in embodiment 136, wherein appropriate times in the video timeline correspond to corresponding times in the video stream and audio stream. EMBODIMENT 138. A video file format as described in embodiment 133, wherein the control stream comprises key frames. EMBODIMENT 139. A video file format as described in embodiment 131, in which the animation stream, video stream, and audio stream are configured to show both ahead and behind a single key frame. EMBODIMENT 140. A video file format as described in embodiment 131, wherein one or more animation streams correspond to animation instructions associated with one or more 3D models. EMBODIMENT 141. A video file format as described in embodiment 131, wherein each animation stream of the one or more animation streams corresponds to at least one of a 3D model, a texture, or a material of the data storage. EMBODIMENT 142. 1. A method comprising: receiving a video file of a video, the video file comprising: One or more animation streams; a data store comprising control data; at least one of a video stream or an audio stream; and dynamically generating a control stream from the control data and a timeline controller; - loading a model of the 3D object received from a data store; attaching a 3D object to an animation stream of the one or more animation streams; A method comprising: EMBODIMENT 143. 143. The method of embodiment 142, wherein the one or more animation streams correspond to individual 3D models stored in the data store. EMBODIMENT 144. 143. The method of embodiment 142, wherein one or more animation streams control the movement, orientation, and position of a 3D object relative to the video. EMBODIMENT 145. The method of embodiment 142, wherein a model of the 3D object is loaded at least in part based on a control stream that looks ahead in time in the video stream to predict when the 3D object needs to be displayed. EMBODIMENT 146. The method of embodiment 145, further comprising determining a lead time for loading the model based on at least one of the size of the model, the network bandwidth, or the processing capabilities of the user's virtual and / or augmented reality system. EMBODIMENT 147. 143. The method of embodiment 142, further comprising the step of displaying the video via a virtual and / or augmented reality device. EMBODIMENT 148. 143. The method of embodiment 142, wherein the data store is loaded completely into memory when the video file is received. EMBODIMENT 149. 143. The method of embodiment 142, wherein the control stream is loaded completely into memory when the control stream is generated. 150. 143. The method of embodiment 142, wherein the video is a stereoscopic 3D video. 151. 1. A computer system implementing a mixed reality video player, comprising: a computer processor for executing a set of program code instructions; a memory for holding program code instructions, the program code instructions comprising: receiving a video file of a video, the video file comprising: One or more animation streams; a data store comprising control data; at least one of a video stream or an audio stream; and dynamically generating a control stream from the control data and a timeline controller; - loading a model of the 3D object received from a data store; attaching a 3D object to an animation stream of the one or more animation streams; comprising program code for carrying out Memory, A computer system comprising: 152. A computer system as described in embodiment 151, wherein one or more animation streams correspond to individual 3D models stored in the data store. 153. A computer system as described in embodiment 151, wherein one or more animation streams control the movement, orientation, and position of a 3D object relative to the video. 154. A computer system as described in embodiment 151, wherein a model of a 3D object is loaded based, at least in part, on a control stream that looks ahead in time in the video stream to predict when the 3D object needs to be displayed. 155. The computer system of embodiment 154, wherein the program code instructions further comprise program code for performing a step of determining a lead time for loading the model based on at least one of the size of the model, network bandwidth, or processing capabilities of the user's virtual and / or augmented reality system. 156. 152. The computer system of embodiment 151, wherein the program code instructions comprise program code for further performing the step of displaying the video via a virtual and / or augmented reality device. 157. 152. The computer system of embodiment 151, wherein the data store is fully loaded into memory when the video file is received. 158. 152. The computer system of embodiment 151, wherein the control stream is loaded completely into memory when the control stream is generated. 159. 152. The computer system of embodiment 151, wherein the video is a stereoscopic 3D video. 160. 1. A method comprising: receiving a video file of a video, the video file comprising: One or more animation streams; a data store comprising control data; at least one of a video stream or an audio stream; and dynamically generating a control stream from the control data and a timeline controller; requesting user interaction to answer a question that is displayed to the user at a point in the video; receiving an answer to the question via user interaction; notifying a timeline controller of the answer; skipping to the appropriate portion of the video that corresponds to the answer; displaying the video from an appropriate portion; A method comprising: 161. The method of embodiment 160, further comprising the step of displaying the video via a virtual and / or augmented reality device. 162. 161. The method of embodiment 160, wherein the data store is loaded completely into memory when the video file is received. 163. 161. The method of embodiment 160, wherein the control stream is loaded completely into memory when the control stream is generated. 164. The method of embodiment 160, wherein the video is a stereoscopic 3D video. 165. 161. The method of embodiment 160, wherein the control stream controls when questions are displayed to the user at specific points in the video based at least in part on data from the data store. 166. 161. The method of embodiment 160, wherein the user interaction includes at least one of head pose, eye tracking, gaze, user hand gestures, totem gestures, or an object recognition device. 167. 161. The method of embodiment 160, wherein the timeline controller controls the position of the video stream. 168. 161. The method of embodiment 160, further comprising storing a state of the video with the run-time interactivity data based at least in part on the received answer. 169. 161. The method of embodiment 160, wherein the questions displayed to the user correspond to questions that change the way the plot of the video may end. 170. 1. A computer system implementing a mixed reality video player, comprising: a computer processor for executing a set of program code instructions; a memory for holding program code instructions, the program code instructions comprising: receiving a video file of a video, the video file comprising: One or more animation streams; a data store comprising control data; at least one of a video stream or an audio stream; and dynamically generating a control stream from the control data and a timeline controller; requesting user interaction to answer a question that is displayed to the user at a point in the video; receiving an answer to the question via user interaction; notifying a timeline controller of the answer; skipping to the appropriate portion of the video that corresponds to the answer; displaying the video from an appropriate portion; a memory comprising program code for implementing the A computer system comprising: 171. 171. The computer system of embodiment 170, wherein the program code instructions comprise program code for further performing the step of displaying the video via a virtual and / or augmented reality device. 172. 171. The computer system of embodiment 170, wherein the data store is fully loaded into memory when the video file is received. 173. 171. The computer system of embodiment 170, wherein the control stream is loaded completely into memory when the control stream is generated. 174. 171. The computer system of embodiment 170, wherein the video is a stereoscopic 3D video. 175. 171. The computer system of embodiment 170, wherein the control stream controls when questions are displayed to the user at specific points in the video based at least in part on data from the data store. 176. 171. The computer system of embodiment 170, wherein the user interaction includes at least one of head pose, eye tracking, gaze, user hand gestures, totem gestures, or an object recognition device. 177. 171. The computer system of embodiment 170, wherein the timeline controller controls the position of the video stream. 178. 171. The computer system of embodiment 170, wherein the program code instructions further comprise program code for performing a step of storing a state of the video with the runtime interactivity data based at least in part on the received answer. 179. 171. The computer system of embodiment 170, wherein the questions displayed to the user correspond to questions that change the way the plot of the video may end. 180. 1. A computer-implemented method for displaying environmentally aware 3D video within a virtual and / or augmented reality environment, comprising: identifying a 3D video; receiving environmental information of a user's environment from one or more sensors, the environmental information identifying objects within the environment; Rendering a portion of the 3D video onto one or more objects identified from the environment; A method comprising: 181. The method of embodiment 180, wherein the 3D video is stereoscopic 3D video. 182. The method of embodiment 180, wherein the environment is the user's physical environment. 183. The method of embodiment 180, wherein the one or more sensors include one or more cameras for capturing scene information of the physical environment. 184. interpreting scene information captured from one or more sensors; Mapping one or more elements of the environment by discovering and registering one or more elements from the environment; 181. The method of embodiment 180, further comprising: 185. The method of embodiment 180, wherein the 3D video has a video file format comprising control data, the control data instructing a video player to display a portion of the 3D video on one or more objects identified from the environment. Embodiment 185a. 186. The method of embodiment 185, wherein the control data comprises a set of control commands for controlling objects and functions within and outside the 3D video. Embodiment 185b. The method of embodiment 185a, wherein the control command (e.g., the set of control commands) is at least one of extensible, updatable, or dynamic. Embodiment 185c. The method of embodiment 185b, wherein the control command includes at least one of the following steps: changing the speed of the 3D video, providing a user prompt to dim lights in the user's environment when the 3D video starts, calling a pizza restaurant when the 3D video starts, or playing the 3D video differently based at least in part on the user's line of sight. Embodiment 185d. The method of embodiment 180, wherein the 3D video is a 3D movie and is provided with a programming language. 186. 1. A virtual and / or augmented reality display system, comprising: a camera for capturing a virtual and / or augmented reality environment; A module for processing data, the module being stored in a memory, the module being executed to: A process for identifying 3D videos; a process for receiving environmental information of a user's environment from one or more sensors, the environmental information identifying objects within the environment; Rendering a portion of the 3D video onto one or more objects identified from the environment; A module for performing A virtual and / or augmented reality display system comprising: 187. A virtual and / or augmented reality display system as described in embodiment 186, wherein the 3D video is stereoscopic 3D video. 188. A virtual and / or augmented reality display system as described in embodiment 186, wherein the 3D video has a video file format comprising control data, the control data instructing a video player to render a portion of the 3D video based at least in part on a decision made by a user. 189. A virtual and / or augmented reality display system as described in embodiment 186, wherein the one or more interactions received from the user include at least one of a head pose, eye tracking, gesture, totem gesture, or object recognition device. 190. A virtual and / or augmented reality display system as described in embodiment 186, wherein interactions from one or more interactions received from a user cause the 3D video to jump to different parts of the 3D video. 191. 1. A computer-implemented method for displaying interactive 3D video within a virtual and / or augmented reality environment, comprising: identifying the 3D video as a stream of non-actionable data that is periodically interrupted by a decision node; receiving one or more interactions from a user as input corresponding to a decision to be taken by the user; Rendering the 3D video based at least on the decision selected by the user; A method comprising: 192. 192. The method of embodiment 191, wherein the 3D video is stereoscopic 3D video. 193. 192. The method of embodiment 191, wherein the 3D video has a video file format comprising control data, the control data instructing a video player to render a portion of the 3D video based at least in part on a decision made by a user. 194. 192. The method of embodiment 191, wherein the one or more interactions received from the user include at least one of a head pose, eye tracking, a gesture, a totem gesture, or an object recognition device. 195. The method of embodiment 191, wherein an interaction from one or more interactions received from a user causes the 3D video to jump to a different part of the 3D video. 196. 1. A virtual and / or augmented reality display system, comprising: a camera for capturing a virtual and / or augmented reality environment; A module for processing data, the module being stored in a memory, the module being executed to: a process for identifying 3D video as a stream of non-actionable data that is periodically interrupted by a decision node; a process for receiving one or more interactions from a user as input corresponding to a decision to be selected by the user; a process for rendering a 3D video based at least in part on a decision selected by a user; A module for performing A virtual and / or augmented reality display system comprising: 197. A virtual and / or augmented reality display system as described in embodiment 196, wherein the 3D video is stereoscopic 3D video. 198. A virtual and / or augmented reality display system as described in embodiment 196, wherein the 3D video has a video file format comprising control data, the control data instructing a video player to render a portion of the 3D video based at least in part on a decision made by a user. 199. A virtual and / or augmented reality display system as described in embodiment 196, wherein the one or more interactions received from the user include at least one of a head pose, eye tracking, gesture, totem gesture, or object recognition device. 200. A virtual and / or augmented reality display system as described in embodiment 196, wherein interactions from one or more interactions received from a user cause the 3D video to jump to different parts of the 3D video. EMBODIMENT 201. 1. A computer-implemented method for displaying a 3D object, comprising: storing in a database graphical data representing the 3D object; displaying a 2D opaque pane, the 2D opaque pane having a viewing aperture for viewing a 3D object located behind the 2D opaque pane; Rendering a first portion of the 3D object viewable through a viewing aperture from a first viewpoint; receiving a request to display a 3D object viewable through the viewing aperture from a second viewing viewpoint; rendering a second portion of the 3D object viewable through the viewing aperture from a second perspective; displaying a second portion of the 3D object from a second perspective; A method comprising: 202. The method of embodiment 201, wherein the graphical data representing the 3D object is a complete representation of the entire 3D object. EMBODIMENT 203. 202. The method of claim 201, wherein the rendering of the first portion of the 3D object is based, at least in part, on a viewing angle for a view of the 3D object from a first viewpoint. EMBODIMENT 204. The method of embodiment 201, wherein the rendering of the second portion of the 3D object is based, at least in part, on a viewing angle relative to a view of the 3D object from a second viewpoint. EMBODIMENT 205. The method of embodiment 201, wherein the opaque pane further comprises a plurality of viewing openings for viewing a plurality of 3D objects located behind the 2D opaque pane. EMBODIMENT 206. The method of embodiment 205, wherein each 3D object among the multiple 3D objects is located behind a respective viewing aperture. 207. The method of embodiment 201, wherein the 3D object is a 3D icon. 208. The method of embodiment 207, wherein the 2D opaque pane with the viewing opening is an icon grid for displaying multiple 3D icons. 209. The method of embodiment 201, wherein the 3D object moves through the viewing aperture as the user gazes at the 3D object. EMBODIMENT 210. 202. The method of embodiment 201, wherein the 3D object is displayed as a 2D image when the user is not focused on the 3D object. EMBODIMENT 211. 1. A computer system for displaying a 3D object, comprising: a computer processor for executing a set of program code instructions; a memory for holding program code instructions, the program code instructions comprising: storing in a database graphical data representing the 3D object; displaying a 2D opaque pane, the 2D opaque pane having a viewing aperture for viewing a 3D object located behind the 2D opaque pane; Rendering a first portion of the 3D object viewable through a viewing aperture from a first viewpoint; receiving a request to display a 3D object viewable through the viewing aperture from a second viewing viewpoint; rendering a second portion of the 3D object viewable through the viewing aperture from a second perspective; displaying a second portion of the 3D object from a second perspective; a memory comprising program code for implementing the A computer system comprising: EMBODIMENT 212. A computer system as described in embodiment 211, wherein the graphical data representing the 3D object is a complete representation of the entire 3D object. EMBODIMENT 213. 212. A computer system as described in embodiment 211, wherein the rendering of the first portion of the 3D object is based, at least in part, on a viewing angle for a view of the 3D object from a first viewpoint. EMBODIMENT 214. A computer system as described in embodiment 211, wherein the rendering of the second portion of the 3D object is based, at least in part, on a viewing angle for a view of the 3D object from the second viewpoint. EMBODIMENT 215. 212. The computer system of embodiment 211, wherein the opaque pane further comprises a plurality of viewing openings for viewing a plurality of 3D objects located behind the 2D opaque pane. EMBODIMENT 216. A computer system as described in embodiment 215, wherein each 3D object among the multiple 3D objects is located behind a separate viewing aperture. EMBODIMENT 217. 212. The computer system of embodiment 211, wherein the 3D object is a 3D icon. EMBODIMENT 218. A computer system as described in embodiment 217, wherein the 2D opaque pane with the viewing opening is an icon grid for displaying multiple 3D icons. 219. A computer system as described in embodiment 211, wherein the 3D object moves through the viewing aperture as the user gazes at the 3D object. 220. A computer system as described in embodiment 211, wherein a 3D object is displayed as a 2D image when a user is not focused on the 3D object. 221. A system, method, and computer program product for generating and displaying virtual content in a mixed reality system according to any of the inventive concepts disclosed herein. (System Architecture Overview)

[0175] 10 is a block diagram of an exemplary computing system 1400 suitable for implementing one or more of the embodiments of the present disclosure. The computing system 1400 includes a bus 1406 or other communication mechanism for communicating information that interconnects subsystems and devices such as a processor 1407, a main memory 1408 (e.g., RAM), a static storage device 1409 (e.g., ROM), a disk drive 1410 (e.g., magnetic or optical), a communication interface 1414 (e.g., a modem or Ethernet card), a display 1411 (e.g., CRT or LCD), an input device 1412 (e.g., a keyboard), and cursor control.

[0176] According to some embodiments, the computing system 1400 performs specific operations by the processor 1407 executing one or more sequences of one or more instructions contained in the main memory 1408. Such instructions may be read into the main memory 1408 from another computer-readable / usable medium, such as the static storage device 1409 or the disk drive 1410. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions to implement the present disclosure. Thus, the embodiments are not limited to any specific combination of hardware circuitry and / or software. In one embodiment, the term "logic" shall mean any combination of software or hardware used to implement all or a portion of the present disclosure.

[0177] The terms "computer-readable medium" or "computer usable medium," as used herein, refer to any medium that participates in providing instructions to the processor 1407 for execution. Such media may take many forms, including but not limited to, non-volatile media and volatile media. Non-volatile media include, for example, optical or magnetic disks, such as the disk drive 1410. Volatile media include dynamic memory, such as the main memory 1408.

[0178] Common forms of computer readable media include, for example, a floppy disk, a flexible disk, a hard disk, a magnetic tape, any other magnetic medium, a CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with a pattern of holes, RAM, PROM, EPROM, Flash-EPROM, any other memory chip or cartridge, or any other medium from which a computer can read.

[0179] In one embodiment, execution of the sequences of instructions for practicing the present disclosure is performed by a single computing system 1400. According to other embodiments, two or more computing systems 1400 coupled by a communications link 1415 (e.g., a LAN, PTSN, or wireless network) may cooperate with each other to perform the sequences of instructions required to practice the present disclosure.

[0180] The computing system 1400 may transmit and receive messages, data, and instructions, including programs, e.g., application code, through communications link 1415 via communications interface 1414. The received program code may be executed by processor 1407 as it is received, and / or stored in disk drive 1410 or other non-volatile storage for later execution. The computing system 1400 may communicate through data interface 1433 to a database 1432 on external storage device 1431.

[0181] In the foregoing specification, the present disclosure has been described with reference to specific embodiments thereof. However, it will be apparent that various modifications and changes can be made therein without departing from the broader spirit and scope of the present disclosure. For example, the process flows described above are described with reference to a particular order of process actions. However, the order of many of the described process actions can be changed without affecting the scope or operation of the present disclosure. The present specification and drawings are therefore to be regarded in an illustrative rather than restrictive sense.

Claims

1. A computer-implemented method for displaying environmentally aware 3D video within a virtual and / or augmented reality environment, the method comprising: Identifying the 3D video; receiving environmental information of a user's environment from one or more sensors, the environmental information identifying objects within the environment; Rendering a portion of the 3D video onto one or more objects identified from the environment; and A method comprising:

2. The method of claim 1, wherein the 3D video is stereoscopic 3D video.

3. The method of claim 1, wherein the environment is the user's physical environment.

4. The method of claim 1, wherein the one or more sensors include one or more cameras for capturing scene information of the physical environment.

5. interpreting scene information captured from the one or more sensors; and detecting and registering one or more elements of the environment from the environment, thereby mapping the one or more elements of the environment; The method of claim 1 further comprising:

6. The method of claim 1, wherein the 3D video has a video file format with control data, the control data instructing a video player to display the portion of the 3D video on the one or more objects identified from the environment.

7. The method of claim 6, wherein the control data comprises a set of control commands for controlling objects and functions within and outside of the 3D video.

8. The method of claim 7, wherein the control commands (e.g., the set of control commands) are at least one of extensible, updatable, or dynamic.

9. The method of claim 8, wherein the control commands include at least one of changing a speed of the 3D video, providing a user prompt to dim lights in a user's environment when the 3D video starts, calling a pizza restaurant when the 3D video starts, or playing the 3D video differently based at least in part on a user's line of sight.

10. The method of claim 1, wherein the 3D video is a 3D movie, the 3D movie comprising a programming language.

11. A virtual and / or augmented reality display system, comprising: a camera for capturing a virtual and / or augmented reality environment; A module for processing data, said module being stored in a memory, said module being capable of, when executed, A process for identifying a 3D video; a process for receiving environmental information of a user's environment from one or more sensors, the environmental information identifying objects within the environment; rendering a portion of the 3D video onto one or more objects identified from the environment; Module and A virtual and / or augmented reality display system comprising:

12. The virtual and / or augmented reality display system of claim 11, wherein the 3D video is stereoscopic 3D video.

13. The virtual and / or augmented reality display system of claim 11, wherein the 3D video has a video file format with control data, the control data instructing a video player to render a portion of the 3D video based at least in part on a decision selected by the user.

14. The virtual and / or augmented reality display system of claim 11, wherein the one or more interactions received from the user include at least one of information obtained by head pose, eye tracking, gestures, or an object recognition device.

15. A virtual and / or augmented reality display system as described in claim 11, wherein one of the one or more interactions received from the user causes the 3D video to jump to a different portion of the 3D video.

Images