Augmented reality content sharing using digital multimedia files

Abstract

The computer-implemented method includes capturing visual data of an environment using an image sensor of the electronic device and capturing non-visual data of the environment using one or more non-image sensors of the electronic device. Feature descriptors of one or more objects in the environment are generated using the visual data of the environment and the non-visual data of the environment. A map of the environment is generated using the feature descriptors of the one or more objects. One or more virtual objects are anchored to at least one of the objects using the map. The visual data, non-visual data, and map are combined in a digital multimedia container file. The digital multimedia container file is stored on the electronic device or on another electronic device connected to the electronic device.

Description

[Technical field]

[0001] REFERENCE TO RELATED APPLICATIONS This application is a continuation of and claims priority to U.S. non-provisional patent application 17 / 301,596, entitled "AUGMENTED REALITY CONTENT EXPERIENCE SHARING USING DIGITAL MULTIMEDIA FILES," filed on April 8, 2021, the disclosure of which is incorporated herein by reference in its entirety.

[0002] Technical Field This description relates to augmented reality (AR) content experience sharing using digital multimedia files. [Background technology]

[0003] background Augmented reality (AR) is an interactive experience of a real-world environment in which the real world is augmented by adding computer-generated objects and information, also referred to as virtual objects and information. Augmented reality is used to enhance a natural environment or situation and provide a perceptually enriched experience. These enriched experiences are desirably shared between users. Summary of the Invention

[0004] overview According to one general aspect, a computer-implemented method includes capturing visual data of an environment using an image sensor of an electronic device and capturing non-visual data of the environment using one or more non-image sensors of the electronic device. Feature descriptors of one or more objects in the environment are generated using the visual data of the environment and the non-visual data of the environment. A map of the environment is generated using the feature descriptors of the one or more objects. One or more virtual objects are anchored to at least one of the objects using the map. The visual data, non-visual data, and map are combined in a digital multimedia container file. The digital multimedia container file is stored on the electronic device or on another electronic device connected to the electronic device.

[0005] Implementations may include one or more of the following features, or any combination thereof. For example, in some implementations, the digital multimedia container file may be in Motion Picture Experts Group-4 (MPEG-4) format. In some implementations, the digital multimedia container file may be in QuickTime (MOV) format.

[0006] In some implementations, the method may further include combining the virtual object anchor information with the visual data, the non-visual data, and the map in a digital multimedia container file.

[0007] In some implementations, the method may further include playing the digital multimedia container file on an electronic device to view the visual data and one or more virtual objects anchored within the environment.

[0008] In some implementations, the method may further include exporting the digital multimedia container file to a different electronic device for playing the digital multimedia container file on the different electronic device.

[0009] In some implementations, the method may further include editing the digital multimedia container file, including anchoring one or more additional virtual objects to other ones of the objects using the map.

[0010] In some implementations, generating a map of the environment includes generating a mesh map of facial features of the face, anchoring the one or more virtual objects includes anchoring the one or more virtual objects to the facial features of the face using the mesh map, and combining the visual data, non-visual data, and the map in a digital multimedia container file includes combining the visual data, non-visual data, and the mesh map in a digital multimedia container file.

[0011] In another general aspect, a computer-implemented method includes receiving, at a first electronic device, a digital multimedia container file from a second electronic device, the digital multimedia container file including visual data of an environment, non-visual data of the environment, a map of the environment, and virtual object anchor information related to one or more virtual objects anchored to at least one object in the environment, the method further including, at the first electronic device, playing the digital multimedia container file to view the visual data, the non-visual data, and the one or more virtual objects; editing the digital multimedia container file including anchoring the one or more additional virtual objects to different objects in the environment; and saving the digital multimedia container file on the first electronic device.

[0012] Implementations may include one or more of the following features, or any combination thereof. For example, in some implementations, the digital multimedia container file is in a Motion Picture Experts Group-4 (MPEG-4) format. In some implementations, the digital multimedia container file is in a QuickTime (MOV) format.

[0013] In some implementations, the first electronic device playing the digital multimedia container file includes the first electronic device playing the digital multimedia container file using a live image sensor of the first electronic device.

[0014] In some implementations, the first electronic device playing the digital multimedia container file includes the first electronic device playing the digital multimedia container file using a live image sensor of the first electronic device in different positions viewing different objects.

[0015] In another general aspect, a computer program product is tangibly embodied on a non-transitory computer-readable medium and includes executable code that, when executed by at least one computing device, is configured to cause the at least one computing device to capture visual data of an environment using an image sensor of the electronic device, capture non-visual data of the environment using one or more non-image sensors of the electronic device, generate feature descriptors of one or more objects in the environment using the visual data of the environment and the non-visual data of the environment, generate a map of the environment using the feature descriptors of the one or more objects, anchor one or more virtual objects to at least one of the objects using the map, combine the visual data, the non-visual data, and the map in a digital multimedia container file, and store the digital multimedia container file on the electronic device or another electronic device connected to the electronic device.

[0016] In particular, the non-transitory storage medium may store instructions that, when executed by a processor, cause the processor to perform a method according to one of the aforementioned aspects.

[0017] Implementations may include one or more of the following features, or any combination thereof: For example, in some implementations, the digital multimedia container file is in the Moving Picture Experts Group-4 (MPEG-4) format.

[0018] In some implementations, the digital multimedia container file is in QuickTime (MOV) format.

[0019] In some implementations, the computer program product further includes executable code that, when executed by the at least one computing device, is configured to cause the at least one computing device to combine the virtual object anchor information with the visual data, the non-visual data, and the map within the digital multimedia container file.

[0020] In some implementations, the computer program product further includes executable code that, when executed by the at least one computing device, is configured to cause the at least one computing device to play the digital multimedia container file on the electronic device to view the visual data and one or more virtual objects anchored within the environment.

[0021] In some implementations, the computer program product further includes executable code that, when executed by the at least one computing device, is configured to cause the at least one computing device to export the digital multimedia container file to a different electronic device for playback on the different electronic device.

[0022] In some implementations, the computer program product further includes executable code that, when executed by the at least one computing device, is configured to cause the at least one computing device to edit the digital multimedia container file, including anchoring one or more additional virtual objects to other ones of the objects using the map.

[0023] In another general aspect, an electronic device includes an image sensor, one or more non-image sensors, a memory, and a processor adapted to perform the steps of a method according to one of the preceding aspects.

[0024] The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims. [Brief description of the drawings]

[0025] [Figure 1] 1 is a block diagram of a system in accordance with an implementation described herein. [Figure 2A] FIG. 2 is a front view of an exemplary user electronic device. [Figure 2B] FIG. 2 is a front view of an exemplary user electronic device. [Figure 2C] FIG. 2 is a front view of an exemplary user electronic device. [Diagram 3] FIG. 1 is a block diagram of a system configured to implement the concepts described herein. [Figure 4] FIG. 4 is a block diagram of the digital multimedia container file of FIG. 3. [Diagram 5] 4 is a flowchart illustrating an example operation of the system of FIG. 3. [Figure 6] 4 is a flowchart illustrating an example operation of the system of FIG. 3. [Figure 7] 1 is an exemplary scene of an AR virtual environment. [Figure 8] 1 illustrates an example of a general purpose computing device and a general purpose mobile computing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] Detailed Description This document describes systems and techniques for sharing augmented reality (AR) content experiences using digital multimedia container files. Augmented reality allows users to add virtual content to the real world. For example, a user may use an electronic device to capture visual data of a physical environment and overlay virtual content (e.g., virtual objects and information) onto the captured visual data of the physical environment to create an AR content experience. The user may then share the AR content experience with other users, which allows the other users to replay the same AR content experience. Technical issues with current solutions that provide for sharing AR content experiences with other users may arise because Internet connectivity may not be available for some or all of the duration of the shared AR content experience and / or electronic devices viewing the shared experience may have to be in physical proximity to each other and operating simultaneously when those conditions may not be possible.

[0027] This document describes a technical solution that solves technical problems encountered in current technology that enables sharing of AR content experiences. For example, the technical solution provides a mechanism for saving and storing information for an AR content experience in a single digital multimedia container file, which can be stored on a user's local electronic device and easily shared and used by other users on their electronic devices. The use of the digital multimedia container file allows users and others to edit the AR content, play the AR content in the same or different geographic locations, and edit the AR content while playing it. That is, a user can re-experience a live AR session using the stored AR content, and other users can also experience a live AR session using the stored AR content. The digital multimedia container file includes a generic file format that is compatible with players, editors, and online video sharing platforms that enable playback and editing capabilities. Furthermore, the digital multimedia container file provides a customizable track for storing AR-specific data used to play the same live AR session.

[0028] For example, as described in more detail below, user A can record an AR video of an object and annotate the AR video with AR annotations. User A can then save the AR video in a single digital multimedia format (e.g., a digital multimedia container format) on user A's local electronic device. User A can later play back the AR content from the stored digital multimedia format and edit the AR annotations using a live recording session of the same object, or edit the AR annotations using a live recording session of a similar object (but not the same exact object), all using the single digital multimedia format. One advantage is that Internet connectivity may not be required to perform these post-recording editing functions and updates. Furthermore, user A can share the digital multimedia format with user B, who can use the digital multimedia format to perform the same functions related to the object or a similar object in a different geographic location. In this way, user B can experience a live AR experience using the stored AR content, but with user B's own camera looking at the object. Again, the use of digital multimedia formats stored on the local device eliminates the need for an Internet connection by User B and also allows User B to have a live AR experience playback. User B can play (preview) the video with or without AR content (i.e., with or without AR annotations), from the same recorded camera view as recorded by User A, or from User B's camera view, to view from a different camera, allowing for User B's live AR experience.

[0029] As used herein, a digital multimedia container file is a type of digital file in a digital file format that holds audio, video, subtitles, and other information including customizable tracks for holding other information such as AR-specific data. The container can support a variety of audio and video compression methods and may not be tied to one particular audio or video codec. Examples of digital multimedia container files include, but are not limited to, MPEG-4 (MP4), QuickTime (MOV), AVI, RealMedia, and others. Digital multimedia container file may be used interchangeably with digital multimedia container format and digital multimedia container file format and digital multimedia container format file throughout this document.

[0030] 1 illustrates a user in relation to an exemplary system 100 for recording, storing, and sharing AR content using digital multimedia container files according to implementations described herein. The exemplary system 100 may access the stored digital multimedia container files and edit and / or recreate a live AR session using the AR content, which may be done without the use of network connectivity that may otherwise be required to access the stored AR content and data to play the live AR session.

[0031] In the example shown in FIG. 1, a user is wearing a first wearable device 10 and a second wearable device 20. The first wearable device 10 is a head mounted display (HMD) device 10 in the form of smart glasses 10, for purposes of explanation and illustration only. However, the principles described herein may be applied to other types of HMDs, such as goggles, headsets, etc., having visual and non-visual data capture capabilities, including, for example, annotating captured visual data with AR content. The second wearable device 20 is a wrist worn device 20 in the form of a smart watch 20, for purposes of explanation and illustration only. However, the principles described herein may be applied to other types of hand / wrist worn devices, such as, for example, bracelets, rings, etc. The user is holding a handheld device 30. The handheld device 30 may be, for purposes of explanation and illustration only, a smartphone 30, having visual and non-visual data capture capabilities, including, for example, annotating captured visual data with AR content, for example, in the form of a camera. However, the principles described herein may be applied to other types of electronic devices, such as handheld controllers, tablet devices, laptop computing devices, etc., including devices with visual and non-visual data capture capabilities, including annotating the captured visual data with AR content. Additionally, the first wearable device 10 and the second wearable device 20 may be capable of storing the captured and annotated content in a digital multimedia container file on the device itself or on the handheld device 30. Similarly, the handheld device 30 may store the captured and annotated content in a digital multimedia container file on the device itself.

[0032] The exemplary system 100 may include one or more computing and / or electronic devices that may exchange data over the network 190. The devices may communicate over the network 190 and / or over alternative networks and / or directly with each other. Exemplary client devices may include, for example, the exemplary wearable devices 10, 20, the exemplary handheld device 30, other electronic devices, such as laptop or netbook computing devices 150, tablet computing devices 160, desktop computing devices 170, and other such devices. The server 140 may be accessible to the devices over the network 190. The server 140 may provide access to the database 144. In this manner, the wearable devices 10, 20 and the handheld device 30 may also communicate stored digital multimedia container files over the network 190 to the server 140 and the database 144 for storage in a network-accessible location by them and other devices. Information stored in the database 144 may be used as a backup to locally stored AR content information and / or may supplement the locally stored AR content with AR-related information.

[0033] Fig. 2A is a front view of an exemplary first wearable device 10 (an exemplary HMD 10) worn by a user in Fig. 1. Fig. 2B is a front view of an exemplary second wearable device 20 (an exemplary smart watch 20) shown in Fig. 1. Fig. 2C is a front view of an exemplary handheld device 30 held by a user in Fig. 1.

[0034] A first wearable device 10 in the form of an HMD 10, or in this example, smart glasses 10, may include a frame 11, with a display device 12 coupled within the frame 11. In some implementations, an audio output device 13 may be coupled to the frame 11. The HMD 10 may include a sensing system 16 including various sensing system devices and a control system 17 including various control system devices for facilitating operation of the HMD 10. The control system 17 may include a processor 19 operably coupled to the components of the control system 17 and a communication module 15 providing communication with external devices and / or networks. The HMD 10 may also include an image sensor 18 (i.e., a camera 18). In some implementations, the image sensor 18 or camera 18 may be capable of capturing still and / or moving images, patterns, features, light, etc., and / or may be capable of scanning visual codes as described above. Further, in some implementations, the HMD 10 may include one or more non-image sensors (not shown) and / or the HMD 10 may use non-image sensor information obtained from the second wearable device 20 and / or the handheld device 30.

[0035] The first wearable device 10 may be connected to the second wearable device 20 and / or the handheld device 30. For example, the first wearable device 10 may stream information, including files and data, to and from the second wearable device 20 and / or the handheld device 30. For example, files electronically stored on the second wearable device 20 and / or the handheld device 30 may be streamed to the first wearable device 10 and played thereon. Similarly, information collected and / or processed by the first wearable device 10 may be stored on the second wearable device 20 and / or the handheld device 30.

[0036] The second wearable device 20, in this example in the form of a smart watch 20, may include an interface device 21. In some implementations, the interface device 21 may function as an output device including, for example, a display area 22 that may output information to a user. In some implementations, the interface device 21 may function as an input device including, for example, a touch surface 23 that allows the interface device 21 to receive touch input from a user. In some implementations, the interface device 21 may function as an input device and an output device. The second wearable device 20 may include a sensing system 26 including various sensing system devices. The second wearable device 20 may include a control system 27 including various control system devices, a communication module 25 that provides communication with external devices and / or networks, and a processor 29 to facilitate operation of the device 20. The second wearable device 20 may also include an image sensor 28 (i.e., a camera 28). In some implementations, the image sensor 28 or the camera 28 may be capable of capturing still and / or moving images and / or scanning visual codes as described above. Further, in some implementations, the second wearable device 20 may include one or more non-image sensors (not shown) and / or the second wearable device 20 may use non-image sensor information obtained from the HMD 10 and / or the handheld device 30.

[0037] The second wearable device 20 may be connected to the first wearable device 10 and / or the handheld device 30. For example, the second wearable device 20 may stream information, including files and data, to and from the first wearable device 10 and / or the handheld device 30. For example, files electronically stored on the first wearable device 10 and / or the handheld device 30 may be streamed to and played on the second wearable device 20. Similarly, information collected and / or processed by the second wearable device 20 may be stored on the first wearable device 10 and / or the handheld device 30.

[0038] The handheld device 30, in this example in the form of a smartphone 30, may include an interface device 31. In some implementations, the interface device 31 may function as an output device including, for example, a display area 32 that may output information to a user. In some implementations, the interface device 31 may function as an input device including, for example, a touch surface 33 that allows the interface device 31 to receive touch input from a user. In some implementations, the interface device 31 may function as both an input device and an output device. The handheld device 30 may include a sensing system 36 including various sensing system devices. The handheld device 30 may include a control system 37 including various control system devices, a communications module 35 that provides communications with external devices and / or networks, and a processor 39 to facilitate operation of the handheld device 30. The handheld device 30 may also include an image sensor 38 (i.e., a camera 38). In some implementations, the image sensor 38 or the camera 38 may be capable of capturing still and / or moving images. Additionally, in some implementations, the handheld device 30 may include one or more non-image sensors (not shown) and / or the handheld device 30 may use non-image sensor information obtained from the HMD 10 and / or the second wearable device 20. The handheld device 30 may store files and information that may then be streamed and played on the first wearable device 10 and / or the second wearable device 20.

[0039] FIG. 3 is a block diagram of an example electronic device 200, such as, for example, one of the example wearable devices 10, 20 shown in FIG. 1, the example handheld device 30 shown in FIG. 1, and / or other electronic devices that may be used to implement the principles described herein.

[0040] The electronic device 200 may include a sensing system 260 and a control system 270. The sensing system 260 may include one or more different types of sensors, imaging sensors, and non-imaging sensors. The imaging sensors may include an image sensor 261 (e.g., a camera) and an optical sensor 262. The non-imaging sensors may include, for example, an audio sensor 263, an inertial measurement unit (IMU) sensor 264, an optical sensor 265, a light source 266, a position sensor 267, and / or other sensors and / or different combinations of sensors. In some implementations, the sensing system 260 may include an image sensor 261 and / or an optical sensor 262 that can scan or read a visual code.

[0041] The control system 270 may include, for example, power / sleep control devices, audio and video control devices, optical control devices, and / or other such devices and / or different combinations of devices. The sensing system 260 and / or the control system 270 may include more or fewer devices, depending on the particular implementation.

[0042] The electronic device 200 may include at least one processor 290 in communication with the sensing system 260 and the control system 270. The processor 290 may process inputs received from the sensing system 260, such as, for example, images captured by an image sensor / optical sensor, including, but not limited to, still images and video. The electronic device 200 may include an input system 240 that can receive user input that is processed by the processor 290 under the control of the control system 270 and output by the output system 250. The input system 240 may include various types of input devices, including, for example, a touch input surface, an audio input device that can receive audio input (including, for example, an audio sensor or microphone included in the sensing system 260), a gesture recognition device (including, for example, an image captured by an image sensor of the sensing system 260 and processed by the processor 290), and other such input devices. The output system 250 may include various types of output devices, such as, for example, a display device, an audio output device, or a speaker, physical and / or tactile output devices, and other such output devices. The electronic device 200 may include at least one memory 280 and a communications module 295 that provides communications between the electronic device 200 and one or more other external devices, networks, servers, and the like.

[0043] The electronic device 200 may include one or more applications 275 that include executable code and / or instructions stored in memory 280 and executed by processor 290 to cooperate with components of the electronic device 200, such as sensing system 260 and control system 270, to perform certain functions and tasks. One application 275 includes an AR capture tool 277 configured to create and build AR experiences using the electronic device 200 using different application programming interfaces (APIs). The AR capture tool 277 enables the electronic device 200 to record and sense its environment recordings and create AR videos using the sensing system 260, including image sensors and non-image sensors. The AR capture tool 277 enables several core capabilities, including motion tracking, environment understanding, and light estimation. The AR capture tool 277 enables the motion tracking of the electronic device 200 to understand and track its position relative to the world in its current environment. The AR capture tool 277 enables the electronic device 200 to detect the size and position of all types of surfaces, including horizontal, vertical, and angled surfaces, such as, for example, the ground, tables, walls, and other surfaces. The AR capture tool 277 enables the electronic device to perform light estimation to estimate the current lighting conditions of the environment.

[0044] As the electronic device 200 moves through its environment, the AR capture tool 277 uses the sensing system 260 to enable the electronic device 200 to track its position as it moves and build its own understanding of the environment in which it is moving. The electronic device 200's understanding of the real world allows the user to place objects, annotations, or other information in a way that seamlessly integrates with the real world. For example, a user can place a napping kitten on the corner of a coffee table, or annotate a painting with biographical information about the artist. Motion tracking means that the user can move around and view these objects from any angle, and even if the user turns around and leaves the room, when the user comes back, the kitten or annotation will be exactly where the user left it.

[0045] More specifically, in some implementations, the AR capture tool 277 performs these motion tracking and environment understanding functions using a process called simultaneous localization and environment mapping (SLAM). The AR capture tool 277 uses SLAM and the sensing system 260 to detect visually distinguishable features, called feature descriptors, in the captured sensed images and uses these points to calculate the change in position of the electronic device 200. The visual information is combined with inertial measurements from the IMU 264 and other non-imaging sensed information to estimate the pose (i.e., position and orientation) of the electronic device 200 relative to the environment over time. The AR content can be rendered and aligned with the pose of the electronic device's image sensor 261 to be rendered from an accurate viewpoint. The rendered virtual image can be overlaid on top of the image obtained from the electronic device's image sensor 262, making it appear as if the virtual content were part of the real world. The AR capture tool 277 uses the feature descriptors to generate a map of the environment (also called a localization map). In some implementations, other processes may be used to perform the motion tracking and environment understanding functions. For example, in some implementations, AR capture tools 277 may use face detection and tracking algorithms to generate a face mesh on top of which AR annotations (e.g., facial configurations) can be added.

[0046] From an environmental understanding perspective, the AR capture tool 277 is constantly improving its understanding of the real-world environment by detecting feature points and planes. The AR capture tool 277 looks for clusters of feature points that appear to be on a common horizontal or vertical surface, such as a table or wall, and makes these surfaces available as planes. The AR capture tool 277 can also determine the boundaries of each plane and make that information available, which can then be used to position virtual objects that rest on the flat surface. The AR capture tool 277 can continuously and / or periodically update the localization map with additional and / or updated feature points and planes.

[0047] The AR capture tool 277 may also use the image sensor 261 to create a depth map, which is an image that contains data about the distance between surfaces from a given point. The AR capture tool 277 may use the information provided by the depth map to enable an immersive and realistic user experience, such as having virtual objects collide accurately with viewed surfaces or have them appear in front of or behind real-world objects. The AR capture tool 277 may detect information about the lighting of its environment and provide average brightness and color correction for a given image. This information may allow virtual objects to be lit in the same conditions as the surrounding environment, enhancing the sense of realism. The AR capture tool 277 may use hit testing to take (x,y) coordinates corresponding to the screen of the electronic device 200 (provided by a tap or other interaction), project a ray into the image sensor 261's view of the world, and return any planes or feature points where the ray intersects, along with the pose of that intersection in world space. This allows the user to select or otherwise interact with objects in the environment. Oriented points let the user place a virtual object on an angled surface. Once the hit test returns a feature point, the AR capture tool 277 looks at nearby feature points and tries to use them to estimate the angle of the surface at the given feature point. The AR capture tool 277 then returns a pose that takes that angle into account.

[0048] The pose may change as the AR capture tool 277 improves its understanding of the electronic device's own position and the environment. When a user wishes to place a virtual object, an anchor may be defined to ensure that the AR capture tool 277 tracks the object's position over time. In some implementations, the anchor may be created based on the pose returned by a hit test, as described above.

[0049] The fact that poses can change means that the AR capture tool 277 can update the positions of environmental objects such as planes and feature points over time. Planes and points can be referred to as a type of object called trackables. As the name suggests, these are the objects that the AR capture tool 277 tracks over time. AR content (e.g., virtual objects) can be anchored to specific trackables to ensure that the relationship between the AR content and the trackables remains stable as the electronic device 200 moves around. For example, if a user places a virtual object on a real object (e.g., a desk), if the AR capture tool 277 later adjusts the pose of a plane associated with the desk, the virtual object will still appear to stay on the table. The anchors ensure that the object appears to stay in the same position and orientation in space to maintain the illusion of a virtual object placed in the real world.

[0050] Anchors may use world space, which is the coordinate space in which the image sensor 261 (e.g., camera) and objects are positioned. The image sensor 261 and object positions are updated every frame in world space. The pose represents the object's position and orientation in world space. When an anchor is created, a pose is used that describes the position and orientation relative to the world space estimate for the current frame. One or more objects can be attached to the anchor. The anchor and the objects attached to it appear to stay where they are placed in the world. As the anchor pose adapts to the world space updates in each frame, the anchor updates the object's pose accordingly. Multiple objects can be attached to the same anchor to ensure that these objects maintain their relative position and orientation even when the anchor's pose is adjusted. Anchors may be used in AR scenes. Anchors may be created in the context of a trackable object (such as a plane) or position in world space.

[0051] The information captured by the AR capture tool 277 is formatted and stored locally on the electronic device 200 in a digital multimedia container file 285 in the memory 280. The digital multimedia container file 285 is a common file format that allows the application 275 to play and edit the information. The digital multimedia container file 285 also includes customizable tracks for storing information related to playing the same live AR video, including position estimation maps and anchor information. In some implementations, the digital multimedia container file 285 includes a Moving Picture Experts Group-4 (MPEG-4) format. In some implementations, the digital multimedia container file 285 includes a QuickTime (MOV) format. In some implementations, other digital multimedia container files may be used.

[0052] The digital multimedia container file 285 may be communicated from the electronic device 200 to another electronic device using the communications module 295, and the digital multimedia container file 285 is then stored locally on the other electronic device for use by that electronic device. For example, in some implementations, the digital multimedia container file 285 may be stored locally on the handheld device 30 for use by the first wearable device 10 and / or the second wearable device 20.

[0053] 4, the digital multimedia container file 285 may include the following information captured by the sensing system 260 and / or generated by the AR capture tool 277: For example, the digital multimedia container file 285 may include, but is not limited to, image information 405, image metadata 410, IMU sensor data 415, position sensor data 420, API call information 425, localization map 430, AR derived geometry 435, and audio data 440. This information and metadata 405-440 are stored locally so that they can be used to play and edit the AR video. In some implementations, the application 275 includes an AR authoring tool 279 that allows a user to play and edit the digital multimedia container file 285.

[0054] 5, an example process 500 illustrates an example operation of the electronic device 200 of FIG. 3 and the system 100 of FIG. 1. More specifically, the process 500 illustrates an example of a computer-implemented method for capturing, creating, and storing an annotated AR environment in a digital multimedia container file. Instructions for execution of the process 500 may be stored in the memory 280 and executed by the processor 290 on the electronic device 200. The instructions for execution of the process 500 may cause the processor 290 to implement the application 275 and one or more of its components.

[0055] Process 500 includes capturing visual data of the environment using an image sensor of the electronic device and capturing non-visual data of the environment using one or more non-image sensors of the electronic device (510). For example, electronic device 200 captures visual data of the environment using image sensor 261 and captures non-visual data of the environment using one or more non-image sensors 263, 264, 265, 266, and 267. In use, a user may capture visual data of the environment using the camera (image sensor 261) of electronic device 200 and capture non-visual data of the environment using the non-image sensors.

[0056] The process 500 includes generating (510) feature descriptors for one or more objects in the environment using the visual data of the environment and the non-visual data of the environment. For example, the application 275 including the AR capture tool 277 generates feature descriptors for one or more objects in the environment using the visual data of the environment and the non-visual data of the environment. The process 500 includes generating (530) a map of the environment using the feature descriptors of the one or more objects. For example, the application 275 including the AR capture tool 277 generates a map of the environment using the feature descriptors of the one or more objects.

[0057] The process 500 includes anchoring one or more virtual objects to at least one of the objects using the map (540). For example, an application 275 including an AR capture tool 277 anchors one or more virtual objects to at least one of the objects using the map. As described above, when an anchor is created, a pose is used that describes its position and orientation relative to the world space estimate for the current frame. One or more objects can be attached to the anchor. The anchor and the objects attached to it appear to stay where they are located in the world. As the anchor pose adapts to world space updates in each frame, the anchor updates the pose of the object accordingly. Multiple objects can be attached to the same anchor to ensure that the objects maintain their relative positions and orientations even when the pose of the anchor is adjusted.

[0058] The process 500 includes combining (550) the visual data, the non-visual data, and the map in a digital multimedia container file and storing (560) the digital multimedia container file on the electronic device. For example, an application 275 including an AR capture tool 277 combines the visual data, the non-visual data, and the map in a digital multimedia container file 285 and stores the digital multimedia container file 285 in memory 280 on the electronic device 200. The information in the digital multimedia container file 285 includes the information and data shown in FIG. 4. In some implementations, the digital multimedia container file is in MPEG-4 format. In some implementations, the digital multimedia container file 285 is in MOV format.

[0059] Optionally, the digital multimedia container file may be stored on another electronic device for use by another electronic device. For example, in some implementations, the first wearable device 10 and / or the second wearable device 20 may store the digital multimedia container file on the handheld device 30 for use (e.g., streaming and playback) by the first wearable device 10. That is, in some implementations, the first wearable device 10 and / or the second wearable device 20 may cooperate with the handheld device 30 to perform one or more of the functions of the process 500, including storing the digital multimedia container file on the handheld device 30.

[0060] Optionally, the process 500 may include combining the visual object anchor information with the visual data, non-visual data, and the map in the digital multimedia container file 285. The electronic device 200 may play the digital multimedia container file on the electronic device 200 to view the visual data and the one or more virtual objects anchored in the environment. The electronic device 200 may export the digital multimedia container file 285 to other electronic devices for playing the digital multimedia container file 285 on the other electronic devices. Additionally, the electronic device 200 and / or other electronic devices may edit the digital multimedia container file using the AR authoring tool 279, including anchoring one or more additional virtual objects to other ones of the objects using the map.

[0061] In another use case, the map generated as part of the process 500 may include generating a mesh map of the facial features of the face. In this manner, one or more virtual objects may be anchored to the facial features of the face using the mesh map. The visual data, non-visual data, and mesh map of the face may be combined in a digital multimedia container file and stored in the memory 280 of the electronic device 200.

[0062] Referring to Figure 6, an example process 600 illustrates an example operation of the electronic device 200 of Figure 3 and the system 100 of Figure 1. More specifically, the process 600 illustrates an example of a computer-implemented method for receiving a digital multimedia container file and playing, editing, and saving formats on the electronic device. Instructions for execution of the process 600 may be stored in the memory 280 and executed by the processor 290 on the electronic device 200. The instructions for execution of the process 600 may cause the processor 290 to implement the application 275 and one or more of its components.

[0063] The process 600 includes receiving, at the first electronic device, a digital multimedia container file from a second electronic device, the digital multimedia container file including visual data of the environment, non-visual data of the environment, a map of the environment, and virtual object anchor information associated with one or more virtual objects anchored to at least one object in the environment (610). For example, the electronic device 200 can receive the digital multimedia container file 285 from the second electronic device, the digital multimedia container file 285 including visual data of the environment, non-visual data of the environment, a map of the environment, and virtual object anchor information associated with one or more virtual objects anchored to at least one object in the environment.

[0064] The process 600 may include the first electronic device playing the digital multimedia container file to view the visual data, the non-visual data, and the one or more virtual objects (620). For example, the electronic device 200 may play the digital multimedia container file to view the visual data, the non-visual data, and the one or more virtual objects.

[0065] The process 600 includes editing the digital multimedia container file, including anchoring the one or more additional virtual objects to different objects in the environment (630) and saving the digital multimedia container file on the first electronic device (640). For example, the electronic device 200 can use an application 275 including an AR authoring tool 279 to edit the digital multimedia container file 285, including anchoring the one or more additional virtual objects to different objects in the environment, and save the digital multimedia container file 285 to a memory 280 on the electronic device 200. The digital multimedia container file may be in a Moving Picture Experts Group-4 (MPEG-4) format.

[0066] Process 600 may optionally include the first electronic device playing the digital multimedia container file using a live image sensor of the first electronic device. Additionally, process 600 may include the first electronic device playing the digital multimedia container file using a live image sensor of the first electronic device at different positions viewing different objects.

[0067] Referring to FIG. 7, an exemplary AR scene 700 according to the principles described herein is shown. The AR scene 700 may be live recorded and created using the system 100, the electronic device 200, and the processes 500 and 600. In one exemplary use case, a user A wants to record an AR video to show a customer B how to use a new model coffee machine. The user A uses the electronic device 200, turns on the image sensor 261 (i.e., the camera), and records the coffee machine along with the AR capture tool 277 and the AR authoring tool 279. The user A adds AR objects 705, 710, 715, 720, and 725, which are AR annotations in this example. The electronic device 200 generates a map using the feature descriptors of the coffee machine object and anchors the AR objects 705-725 to positions around the coffee machine selected by the user A. The electronic device 200 stores the visual data, non-visual data, and the map in a digital multimedia container file 285, such as an MP4 format file, in the memory 280 of the electronic device 200. The virtual object information may also be stored as part of the digital multimedia container file. If desired, User A can add additional AR objects to the digital multimedia container file using a post-editing tool, such as an AR authoring tool 279, and re-save the file.

[0068] User A can communicate the digital multimedia format file to User B, for example, by posting the digital multimedia format file 285 on a network site for User B to download to his electronic device. In some implementations, User B can preview the video with or without AR annotations. User B, who is in a different geographic location with a different coffee machine, the same model as one User A, used to create the video, can play the digital multimedia container file on his electronic device 200 and turn on the AR mode of User B's camera. This allows User B to study his own coffee machine and see the same AR annotations 705-725 all from different viewing angles without needing Internet connectivity, since the required information is included within the digital multimedia container file received on User B's electronic device. User B can also add his own AR objects to the file using an authoring tool.

[0069] Similarly, in other use cases, the above-mentioned systems and techniques may be used for an augmented face painting of user A, which may be saved in a digital multimedia container file and shared with friends who may try on the same virtual face painting on their own faces. As described above, the AR capture tool 277 and the AR authoring tool 279 may use face detection and tracking algorithms to generate a face mesh to which AR content (e.g., AR face configuration) may be applied. The face mesh and related sensor data may be saved in a digital multimedia container file, which may then be shared with other users so that they may experience the AR face configuration on their own faces. The same use may be applied not only to faces, but also to human bodies, using body detection and tracking algorithms to generate a body mesh and / or body map to which AR content (e.g., AR clothing) may be applied. The body mesh and / or body map and related sensor data may be saved in a digital multimedia container file, which may then be shared with other users so that they may experience the AR clothing on their own bodies. User A can also record customized notes about where and how home decorations should be moved and positioned, and send the AR video to User B for playback. User A can also create AR guidance of a scenic location and share it with others using a digital multimedia container file, allowing others to follow the AR annotations using their own electronic devices when physically visiting the same scenic location.

[0070] FIG. 10 illustrates examples of a general-purpose computing device 2000 and a general-purpose mobile computing device 2050 that may be used with the techniques described herein. The computing device 2000 is intended to represent various forms of digital computers, such as laptops, desktops, tablets, workstations, personal digital assistants, televisions, servers, blade servers, mainframes, and other suitable computing devices. For example, the computing device 2000 may be and / or may be used as the server 140 of FIG. 1. The computing device 2050 is intended to represent various forms of mobile devices, such as personal digital assistants, mobile phones, smartphones, and other similar computing devices. The components, their connections and relationships, and their functions illustrated herein are merely exemplary and are not intended to limit the implementation of the invention described and / or claimed in this document.

[0071] The computing device 2000 includes a processor 2002, a memory 2004, a storage device 2006, a high-speed interface 2008 that connects to the memory 2004 and a high-speed expansion port 2010, and a low-speed interface 2012 that connects to a low-speed bus 2014 and the storage device 2006. The processor 2002 may be a semiconductor-based processor. The memory 2004 may be a semiconductor-based memory. Each of the components 2002, 2004, 2006, 2008, 2010, and 2012 may be interconnected using various buses and implemented on a common motherboard or in other manners as appropriate. The processor 2002 may process instructions for execution within the computing device 2000, including instructions stored in the memory 2004 or the storage device 2006, to display graphical information for a GUI on an external input / output device, such as a display 2016 coupled to the high-speed interface 2008. In other implementations, multiple processors and / or multiple buses may be used, along with multiple memories and types of memory, as appropriate. Also, multiple computing devices 2000 may be connected, each providing a portion of the required operations (e.g., as a bank of servers, as a group of blade servers, or as a multiprocessor system).

[0072] The memory 2004 stores information within the computing device 2000. In one implementation, the memory 2004 is one or more volatile memory units. In another implementation, the memory 2004 is one or more non-volatile memory units. The memory 2004 may also be another form of computer-readable medium, such as a magnetic disk or optical disk.

[0073] The storage device 2006 can provide mass storage for the computing device 2000. In one implementation, the storage device 2006 can be or include a computer-readable medium such as a floppy disk drive, a hard disk drive, an optical disk drive, or a tape drive, a flash memory or other similar solid-state memory device, or an array of devices including devices in a storage area network or other configuration. The computer program product can be tangibly embodied in an information carrier. The computer program product can also include instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as the memory 2004, the storage device 2006, or a memory on the processor 2002.

[0074] The high-speed controller 2008 manages bandwidth-intensive operations for the computing device 2000, and the low-speed controller 2012 manages less bandwidth-intensive operations. Such an allocation of functions is merely exemplary. In one implementation, the high-speed controller 2008 is coupled to the memory 2004 (e.g., via a graphics processor or accelerator), to a display 2016, and to a high-speed expansion port 2010 that can accept various expansion cards (not shown). In this implementation, the low-speed controller 2012 is coupled to the storage device 2006 and to a low-speed expansion port 2014. The low-speed expansion port may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), and may be coupled, e.g., via a network adapter, to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router.

[0075] The computing device 2000 may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server 2020, or multiple times within a group of such servers. It may be implemented as part of a rack server system 2024. In addition, it may be implemented in a personal computer, such as a laptop computer 2022. Alternatively, components from the computing device 2000 may be combined with other components in a mobile device (not shown), such as device 2050. Each such device may include one or more of the computing devices 2000, 2050, and the entire system may be composed of multiple computing devices 2000, 2050 in communication with each other.

[0076] Computing device 2050 includes, among other components, a processor 2052, a memory 2064, input / output devices such as a display 2054, a communication interface 2066, and a transceiver 2068. Device 2050 may also be provided with a storage device, such as a microdrive or other device, to provide additional storage. Each of components 2050, 2052, 2064, 2054, 2066, and 2068 are interconnected using various buses, and some of the components may be implemented on a common motherboard or in other manners as desired.

[0077] The processor 2052 may execute instructions within the computing device 2050, including instructions stored in the memory 2064. The processor may be implemented as a chipset of chips including separate analog and digital processors. The processor may provide coordination of other components of the device 2050, such as, for example, control of a user interface, applications executed by the device 2050, and wireless communications by the device 2050.

[0078] The processor 2052 may communicate with a user via a control interface 2058 and a display interface 2056 coupled to a display 2054. The display 2054 may be, for example, a TFT LCD (thin film transistor liquid crystal display) or an OLED (organic light emitting diode) display, or other suitable display technology. The display interface 2056 may comprise suitable circuitry for driving the display 2054 to present graphical and other information to the user. The control interface 2058 may receive commands from the user and convert them for submission to the processor 2052. Additionally, an external interface 2062 may be provided in communication with the processor 2052 to enable short-range communication between the device 2050 and other devices. The external interface 2062 may, for example, provide wired communication in some implementations and wireless communication in other implementations, and multiple interfaces may also be used.

[0079] The memory 2064 stores information within the computing device 2050. The memory 2064 may be realized as one or more of one or more computer-readable media, one or more volatile memory units, or one or more non-volatile memory units. An expansion memory 2074 may also be provided and connected to the device 2050 via an expansion interface 2072, which may include, for example, a SIMM (single in-line memory module) card interface. Such expansion memory 2074 may provide additional storage space for the device 2050 or may store applications or other information for the device 2050. In particular, the expansion memory 2074 may include instructions for performing or supplementing the processes described above, and may also include secured information. Thus, for example, the expansion memory 2074 may be provided as a security module for the device 2050 and may be programmed with instructions that enable secured use of the device 2050. Additionally, secured applications may be provided via a SIMM card along with additional information, such as placing identification information on the SIMM card in an unhackable manner.

[0080] The memory may include, for example, flash memory and / or NVRAM memory, as described below. In one implementation, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as the methods described above. The information carrier is a computer-readable or machine-readable medium, such as memory 2064, expansion memory 2074, or memory on processor 2052, and may be received, for example, via transceiver 2068 or external interface 2062.

[0081] The device 2050 may communicate wirelessly via a communication interface 2066, which may include digital signal processing circuitry as necessary. The communication interface 2066 may provide for communication under various modes or protocols, such as GSM voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others. Such communication may occur, for example, via a radio frequency transceiver 2068. Further, short-range communication may occur using Bluetooth, WiFi, or other such transceivers (not shown), or the like. In addition, a GPS (Global Positioning System) receiver module 2070 may provide the device 2050 with additional navigation- and location-related wireless data that may be used as appropriate by applications executing on the device 2050.

[0082] Device 2050 may also communicate audibly using voice codec 2060, which may receive spoken information from a user and convert it into usable digital information. Voice codec 2060 may similarly generate audible sounds for the user, such as through a speaker in a handset of device 2050. Such sounds may include sounds from a voice telephone call, may include recorded sounds (e.g., voice messages, music files, etc.), and may also include sounds generated by applications running on device 2050.

[0083] The computing device 2050 may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a mobile phone 2080. It may be implemented as part of a smartphone 2082, a personal digital assistant, or other similar mobile device.

[0084] A computer program such as the computer program described above may be written in any type of programming language, including compiled or interpreted languages, and may be deployed in any form as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may be deployed to be executed on one computer or multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0085] The method steps may be performed by one or more programmable processors executing computer programs to perform functions by operating on input data and generating output. Performance of the method steps and apparatus implementations may be performed in special purpose logic circuitry such as, for example, an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit).

[0086] Processors suitable for executing computer programs may include, by way of example, both general-purpose and special-purpose microprocessors, as well as any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory, or both. Elements of a computer may include at least one processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer may further include one or more mass storage devices for storing data, such as, for example, magnetic disks, magneto-optical disks, or optical disks, and may be operatively coupled to receive data from or transfer data to the one or more mass storage devices, or both. Information carriers suitable for carrying computer program instructions and data include, by way of example, all forms of non-volatile memory, including, for example, EPROM, EEPROM, and flash memory devices; magnetic disks, such as, for example, internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0087] To provide for interaction with a user, implementations may be implemented on a computer having a display device, e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor, for displaying information to the user, and a keyboard and pointing device, e.g., a mouse, trackball, by which the user can provide input to the computer. Other types of devices may be used to provide for interaction with a user as well; for example, feedback provided to the user may be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; input from the user may be received in any form, including acoustic input, speech input, or tactile input.

[0088] An implementation may be implemented in a computing system including back-end components such as, for example, a data server, or a computing system including middleware components such as an application server, or a computing system including front-end components such as, for example, a client computer having a graphical user interface or a web browser through which a user can interact with the implementation, or any combination of such back-end, middleware, or front-end components. The components may be interconnected by any form or medium of digital data communication, such as, for example, a communications network. Examples of communications networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0089] While certain features of the described implementations have been shown as described herein, many modifications, substitutions, changes and equivalents will now occur to those skilled in the art, and it is therefore to be understood that the claims are intended to cover all such modifications and changes that are within the scope of the embodiments.

Claims

1. 1. A computer-implemented method comprising: capturing visual data of an environment using an image sensor of an electronic device and capturing non-visual data of the environment using one or more non-image sensors of said electronic device; generating feature descriptors for one or more objects in the environment using the visual data of the environment and the non-visual data of the environment; generating a map of the environment using the feature descriptors of the one or more objects; and anchoring one or more virtual objects to at least one of the one or more objects using the map; combining said visual data, said non-visual data and said map in a digital multimedia container file; storing said digital multimedia container file on said electronic device or on another electronic device connected to said electronic device; and receiving edits to the digital multimedia container file including anchoring one or more additional virtual objects to other ones of the one or more objects using the map.

2. 2. The computer-implemented method of claim 1, wherein the digital multimedia container file is in the Motion Picture Experts Group-4 (MPEG-4) format.

3. The computer-implemented method of claim 1 , wherein the digital multimedia container file is in QuickTime (MOV) format.

4. The computer-implemented method of claim 1 , further comprising combining virtual object anchor information with the visual data, the non-visual data, and the map in the digital multimedia container file.

5. 2. The computer-implemented method of claim 1, further comprising playing the digital multimedia container file on the electronic device to view the visual data and the one or more virtual objects anchored within the environment.

6. The computer-implemented method of claim 1 , further comprising exporting the digital multimedia container file to a different electronic device for playback on the different electronic device.

7. generating the map of the environment includes generating a mesh map of facial features of a face; Anchoring the one or more virtual objects includes anchoring the one or more virtual objects to the facial features of the face using the mesh map; 2. The computer-implemented method of claim 1, wherein combining the visual data, the non-visual data, and the map in the digital multimedia container file comprises combining the visual data, the non-visual data, and the mesh map in the digital multimedia container file.

8. 1. A computer-implemented method comprising: and receiving, at a first electronic device, a digital multimedia container file from a second electronic device, the digital multimedia container file including visual data of an environment, non-visual data of the environment, a map of the environment, and virtual object anchor information relating to one or more virtual objects anchored to at least one object in the environment, the method further comprising: the first electronic device playing the digital multimedia container file to view the visual data, the non-visual data, and the one or more virtual objects; receiving edits to the digital multimedia container file, including anchoring one or more additional virtual objects to different objects within the environment; storing the digital multimedia container file on the first electronic device.

9. 9. The computer-implemented method of claim 8, wherein the digital multimedia container file is in the Motion Picture Experts Group-4 (MPEG-4) format.

10. 9. The computer-implemented method of claim 8, wherein the digital multimedia container file is in QuickTime (MOV) format.

11. 9. The computer-implemented method of claim 8, wherein the first electronic device playing the digital multimedia container file includes the first electronic device playing the digital multimedia container file using a live image sensor of the first electronic device.

12. 9. The computer-implemented method of claim 8, wherein the first electronic device playing the digital multimedia container file includes the first electronic device playing the digital multimedia container file using a live image sensor of the first electronic device in different positions viewing different objects.

13. A computer program comprising executable code which, when executed by at least one computing device, causes the at least one computing device to perform a method according to any one of claims 1 to 12.

14. A computer-implemented method comprising: Capturing visual data of an environment using an image sensor of an electronic device and capturing non-visual data of the environment using one or more non-image sensors of the electronic device; generating feature descriptors for one or more objects in the environment using the visual data of the environment and the non-visual data of the environment; generating a map of the environment using the feature descriptors of the one or more objects, including generating a mesh map of facial features of a face; and anchoring one or more virtual objects to the facial features of the face using the mesh map; combining the visual data, the non-visual data, and the mesh map in a digital multimedia container file; storing the digital multimedia container file on the electronic device or another electronic device connected to the electronic device.

15. A computer-implemented method comprising: recording an augmented reality session of an environment using an image sensor of an electronic device, the augmented reality session including one or more virtual objects inserted into the environment, the environment being located at a first geographic location, the method further comprising: storing the augmented reality session in a digital multimedia container file on the electronic device; and playing the augmented reality session of the environment on the electronic device from the digital multimedia container file at a second geographic location without using the image sensor, the second geographic location being a different location than the first geographic location.

16. 16. The computer-implemented method of claim 15, wherein the digital multimedia container file is in the Motion Picture Experts Group-4 (MPEG-4) format.

17. 16. The computer-implemented method of claim 15, wherein the digital multimedia container file is in QuickTime (MOV) format.

18. Recording the augmented reality session includes capturing visual data of the environment using the image sensor of the electronic device and capturing non-visual data of the environment using one or more non-image sensors of the electronic device; 16. The computer-implemented method of claim 15, wherein storing the augmented reality session includes storing the visual data of the environment and the non-visual data of the environment using one or more customizable tracks in the digital multimedia container file.

19. The electronic device is a first electronic device, and the computer-implemented method further comprises: transferring said digital multimedia container file to a second electronic device; and playing the augmented reality session of the environment on the second electronic device from the digital multimedia container file at a third geographic location, the third geographic location being a different location than the first geographic location.

20. A computer program comprising executable code configured, when executed by an electronic device, to cause the electronic device to perform a method according to any one of claims 15 to 19.

21. A device comprising: An image sensor; one or more non-imaging sensors; a memory having instructions; a processor, the processor being configured to: and recording an augmented reality session of an environment including visual data using the image sensor and non-visual data using the one or more non-image sensors, the augmented reality session including one or more virtual objects inserted into the environment, the environment being located at a first geographic location, the processor further configured by the instructions to: configured to store the augmented reality session in a digital multimedia container file in the memory; A device configured to play back the augmented reality session of the environment from the digital multimedia container file at a second geographic location without using the image sensor, the second geographic location being a different location than the first geographic location.

22. The device of claim 21, wherein the digital multimedia container file is in Motion Picture Experts Group-4 (MPEG-4) format.

23. The device of claim 21, wherein the digital multimedia container file is in QuickTime (MOV) format.

24. The device of claim 21, wherein the processor is configured by the instructions to store the visual data of the environment and the non-visual data of the environment using one or more customizable tracks in the digital multimedia container file.

Images