Multi-dimensional experience presentation using augmented reality

Abstract

The present disclosure relates to methods and systems for providing a presentation of an experience (e.g., travel) to a user using augmented reality (AR). During a capture phase, a person in travel may take videos or photos using their smartphones, GoPros, and / or smart glasses. The drone may also take videos or photos during travel. In the experience phase, AR terrain rendering of the real-world environment of the travel may be rendered on the desktop, highlighting / animating the path that a person walks during the travel. A person may be rendered as a miniature avatar / doll superimposed on a representation of a real-world environment. When a user clicks a point in the presentation of the travel, a viewing angle (e.g., a video or photo) at the point is presented.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This patent application claims the benefit of priority to U.S. application serial number 18 / 169,631, filed on February 15, 2023, which claims the benefit of priority to U.S. provisional patent application 63 / 402,897, entitled “Multi-dimensional Experience Presentation Using Augmented Reality,” filed on August 31, 2022, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates generally to mobile and wearable computing technologies. In particular, example embodiments of the present disclosure relate to systems, methods, and user interfaces for presenting past experiences to a user using augmented reality (AR). Background Art

[0004] Augmented reality (AR) experience includes: whether by presenting virtual content through a transparent display through which the real-world environment can be seen, or by enhancing image data to include virtual content superimposed on the real-world environment depicted therein, to apply virtual content to the real-world environment. Virtual content may include one or more AR content items. AR content items may include audio content, visual content, or visual effects. Examples of audio and visual content include photos, text, logos, animations, and sound effects. Devices that support AR experience in any of these methods are referred to herein as "AR devices."

[0005] For some example AR devices, audio and visual content or visual effects are applied to media data, such as a real-time image stream. Other example AR devices include head-mounted devices that can be implemented with a transparent or translucent display, through which the user of the head-mounted device can view the surrounding environment. Such a device enables the user to view the surrounding environment through the transparent or translucent display, and can also see objects (e.g., virtual objects such as 3D renderings, images, videos, text, etc.) generated for display that appear to be part of the surrounding environment and / or superimposed on the surrounding environment. The user of the head-mounted device can access and use computer software applications to perform various tasks or participate in entertainment activities. In order to use the computer software application, the user interacts with the 3D user interface provided by the head-mounted device.

[0006] The so-called "Internet of Things" or "IoT" is a network of physical objects (called "IoT devices") embedded with sensors, software, and other technologies to connect and exchange data with other devices over the Internet. For example, IoT devices are used for home automation to control lighting, heating and air conditioning, media and security systems, and camera systems. Several IoT-enabled devices have been provided that serve as smart home hubs to connect different smart home products. IoT devices have also been used for several other applications. Application layer protocols and supporting frameworks for implementing such IoT applications have been provided. Artificial intelligence has also been combined with the IoT infrastructure to enable more efficient IoT operations, improve human-computer interaction, and enhance data management and analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] To easily identify the discussion of any particular element or act, the most significant digit(s) in a reference number refers to the figure number in which the element is first introduced.

[0008] Figure 1 is a diagrammatic representation of a network environment in which a volumetric content rendering system may be deployed, according to some example embodiments.

[0009] Figure 2A is a perspective view of a head mounted device according to some example embodiments.

[0010] Figure 2B According to some example embodiments, Figure 2A Additional views of the headset.

[0011] Figure 3 is a diagram showing a method according to some example embodiments including Figure 1 A block diagram of the networking system with details of the head mounted device.

[0012] Figure 4 is a diagrammatic representation of a volumetric content rendering system having both client-side and server-side functionality according to some examples.

[0013] Figure 5 is a diagrammatic representation of data structures maintained in a database according to some example implementations.

[0014] Figure 6 is a conceptual diagram illustrating an example presentation of a visual representation of an experience conducted through a head mounted device according to some example embodiments.

[0015] Figure 7 is a conceptual diagram illustrating an example presentation of recorded perspectives associated with points of interest, according to some example embodiments.

[0016] Figure 8is a conceptual diagram illustrating an example 2-dimensional (2D) presentation of a visual representation of an experience according to some example embodiments.

[0017] Figures 9 to 12 is a flow chart illustrating an example method for presenting an experience using AR, according to some example implementations.

[0018] Fig.13 is a block diagram illustrating a representative software architecture that may be used in conjunction with the various hardware architectures described herein, according to some example implementations.

[0019] Fig.14 is a block diagram illustrating components of a machine capable of reading instructions from a machine-readable medium (eg, a machine-readable storage medium) and performing any one or more of the methodologies discussed herein, according to some example embodiments. DETAILED DESCRIPTION

[0020] The following description includes systems, methods, techniques, instruction sequences and computer program products that implement the illustrative embodiments of the present disclosure. In the following description, for the purpose of illustration, many specific details are set forth to provide an understanding of the various embodiments of the subject of the present invention. However, it will be apparent to those skilled in the art that the embodiments of the subject of the present invention can be put into practice without these specific details. Usually, it is not necessary to show in detail known instruction examples, protocols, structures and techniques.

[0021] Volumetric content is an example of an augmented reality (AR) experience. Volumetric content may include volumetric video and three-dimensional space images captured in 3-D form (as well as audio signals recorded with the volumetric video and volumetric images). Recording of volumetric content includes capturing elements of a three-dimensional space, such as objects and people, from a volume using a combination of a camera and a sensor. Volumetric content includes a volumetric representation of one or more three-dimensional elements (e.g., an object or a person) of a three-dimensional space. A volumetric representation of an element (e.g., an AR content item) refers to a visual representation of a three-dimensional element in three dimensions. Presentation of volumetric content may include displaying one or more AR content items superimposed on a real-world space, which may be the same space as the three-dimensional space in which the volumetric video was captured, or a different space. Presentation of volumetric content may include: displaying one or more content items in motion; displaying one or more content items performing a move or other action; displaying one or more content items positioned statically; or a combination thereof. Content items may be displayed during presentation of volumetric content or portions thereof.

[0022] The presentation of volumetric content may include tracking the user's position and movement in their physical real-world environment and using the tracked position and movement of the user to allow the user to move around in and interact with the presentation of volumetric content. Thus, the presentation of volumetric content may include displaying content items from multiple perspectives based on the user's movement and position changes. In this way, the presentation of volumetric content provides an immersive AR experience to the user.

[0023] Conventional presentation of volumetric content is typically limited to a visual representation of a background scene and cannot provide a presentation of a foreground experience (e.g., a journey) in the scene. Aspects of the present disclosure include systems, methods, techniques, instruction sequences, and computer program products that provide a multidimensional presentation of a recorded user experience using AR. The multidimensional presentation of the experience may include people participating in the experience, the real-world environment of the experience, and multiple points of interest associated with the experience. In some examples, the multidimensional presentation of the experience may also include multiple perspectives recorded at some or all of the multiple points of interest. When a user interacts with (e.g., selects) the multiple points of interest presented, the display device may present the corresponding perspectives.

[0024] In the aforementioned example, a hiking trip between two friends is recorded and presented. In the capture phase, both friends wear or carry smart devices while hiking. Throughout the hiking trip, the two friends can use smart devices to take videos and photos and hike through the peaks at will. Smart devices may include, but are not limited to, smart phones, smart bracelets, smart watches, smart glasses, cameras mounted on handheld poles, smart helmets, etc. The embedded camera from the smart device can capture multiple perspectives of the two friends at different points of interest. The smart device may also include a drone that captures multiple perspectives at different points of interest or along the hiking trip. After the hike, the presentation system presents an AR terrain rendering of the mountain on the desktop, where the path that the friends took to reach the top of the mountain is visually highlighted. The presentation may also include a representation of each of the two friends (e.g., a miniature avatar). Viewing a user clicking anywhere on the path can view a video shot from a smart device (e.g., a drone) at that point. Viewing a user clicking on an avatar triggers the presentation of a video from that person's perspective (captured using the embedded camera of a smart device). Therefore, using such a presentation method, users can experience the first person, the second person, and / or the third person (in the air) from the perspective of their experience. The presentation of the experience may also include a visualization of a friend's heart rate (captured using a smartwatch) at a specific moment. Images and videos generated during the hike may be highlighted by individual markers on the trail. Additionally, the presentation may include a representation of weather conditions (clouds above a mountain in AR indicating a cloudy day, or animated wind in AR indicating a windy day). By presenting the experience in this way, the system allows the user to relive the experience as if they were watching a dollhouse story play out.

[0025] Figure 1 1 is a block diagram illustrating an example volumetric content rendering system 100 for rendering volumetric content. The volumetric content rendering system 100 includes a client device 102. The client device 102 hosts several applications including a rendering client 104. Each rendering client 104 is communicatively coupled to a rendering server system 108 via a network 106 (e.g., the Internet). In an example, the client device 102 is a wearable device (e.g., smart glasses) worn by a user 103, the wearable device including a camera and an optical element including a transparent display through which the user 103 can see a real-world environment.

[0026] The presence client 104 is able to communicate and exchange data with another presence client 104 and a presence server system 108 via the network 106. The data exchanged between the presentation clients 104 and between another presentation client 104 and the presentation server system 108 includes functionality (e.g., commands to enable functionality) and payload data (e.g., text, audio, video, or other multimedia data).

[0027] The presentation server system 108 provides server-side functionality to a particular presentation client 104 via the network 106. Although certain functionality of the volumetric content presentation system 100 is described herein as being performed by either the presentation client 104 or by the presentation server system 108, the location of certain functionality within the presentation client 104 or within the presentation server system 108 is a design choice. For example, it may be technically preferable to initially deploy certain technologies and functionality within the presentation server system 108, but later migrate the technologies and functionality to the presentation client 104 where the client device (102) has sufficient processing power.

[0028] The presentation server system 108 supports various services and operations provided to the presentation clients 104. Such operations include sending data to the presentation clients 104, receiving data from the presentation clients 104, and processing data generated by the presentation clients 104. Specifically, turning now to the communication server system 108, an application program interface (API) server 110 is coupled to and provides a programming interface to the application server 112. As examples, the data can include volumetric content (e.g., volumetric video), message content, device information, geo-location information, media annotations and overlays, message content persistence conditions, social network information, and real-time event information. The data exchange within the volumetric content presentation system 100 is invoked and controlled by available functions via a user interface (UI) and the presentation clients 104.

[0029] Specifically, turning now to the presence server system 108, an application program interface (API) server 110 is coupled to and provides a programming interface to an application server 112. The application server 112 is communicatively coupled to a database server 118, which facilitates access to a database 120 in which data associated with messages processed by the application server 112 is stored.

[0030] An application program interface (API) server 110 receives and sends message data (e.g., commands and message payloads) between a client device 102 and an application server 112. Specifically, the API server 110 provides a set of interfaces (e.g., routines and protocols) that can be called or queried by a presence client 104 to invoke functionality of the application server 112. The API server 110 exposes various functions supported by the application server 112, including: account registration, login functionality, sending messages from a particular presence client 104 to another presence client 104 via the application server 112, sending media files (e.g., volumetric video) to a presence client 104, setting a collection of media data (e.g., a story), retrieving a friend list of a user of the client device 102, retrieving such a collection, retrieving messages and content, adding and removing friends from a social graph, the location of friends within a social graph, and opening application events (e.g., associated with a presence client 104).

[0031] The application server 112 hosts several server applications and subsystems, including a rendering server 114, an image processing server 116, and a social networking server 122. The rendering server 114 is generally responsible for managing volumetric content and facilitating the rendering of volumetric content by the client device 102. The image processing server 116 is dedicated to performing various image processing operations, generally related to images or videos generated and displayed by the client device 102. The rendering server 114 and the image processing server 116 can work in conjunction to provide one or more AR experiences to the user 103. For example, the rendering server 114 and the image processing server 116 can work in conjunction to support the rendering of volumetric content by the client device 102. Further details regarding the rendering of volumetric content are discussed below.

[0032] The social network server 122 supports various social networking functions and services and makes these functions and services available to the presence server 114. To this end, the social network server 122 maintains and accesses an entity graph within the database 120. Examples of functions and services supported by the social network server 122 include identification of other users of the volumetric content presentation system 100 who have a relationship with a particular user or who the particular user is "following", and also includes identification of interests and other entities of a particular user.

[0033] The application server 112 is communicatively coupled to a database server 118 , which facilitates access to a database 120 in which data associated with content rendered by the rendering server 114 and the image processing server 116 is stored.

[0034] Figure 2A is a perspective view of a head mounted display device (eg, glasses 200) according to some examples. Glasses 200 are Figure 1The glasses 200 are examples of client devices 102. The glasses 200 are capable of displaying content and are therefore examples of display devices, which will be referred to below. In addition, the display capabilities of the glasses 200 support AR experiences, and therefore the glasses 200 are examples of AR devices. As described above, the AR experience includes applying virtual content to the real-world environment, whether by presenting the virtual content through a transparent display through which the real-world environment can be seen, or by augmenting the image data to include virtual content superimposed on the real-world environment depicted therein.

[0035] The glasses 200 may include a frame 202 made of any suitable material, such as plastic or metal, including any suitable shape memory alloy. In one or more examples, the frame 202 includes a first optical element holder or a left optical element holder 204 (e.g., a display or lens holder) and a second optical element holder or a right optical element holder 206 connected by a bridge 212. A first optical element or a left optical element 208 and a second optical element or a right optical element 210 may be disposed in the left optical element holder 204 and the right optical element holder 206, respectively. The right optical element 210 and the left optical element 208 may be lenses, displays, display components, or a combination of the foregoing. Any suitable display component may be disposed in the glasses 200.

[0036] Frame 202 additionally includes a left arm or temple piece 222 and a right arm or temple piece 224. In some examples, frame 202 can be formed from a single piece of material to have a unitary or integrated construction.

[0037] The glasses 200 may include a computing device such as a computer 220, which may be of any suitable type so as to be carried by the frame 202, and in one or more examples, the computing device may be of a suitable size and shape to be partially disposed in one of the temple pieces 222 or temple pieces 224. The computer 220 may include one or more processors as well as memory, wireless communication circuitry, and a power supply. As discussed below, the computer 220 includes a low-power circuitry, a high-speed circuitry, and a display processor. Various other examples may include these elements in different configurations or integrated together in different ways. Additional details of various aspects of the computer 220 may be implemented as shown in the data processor 302 discussed below.

[0038] The computer 220 additionally includes a battery 218 or other suitable portable power source. In some examples, the battery 218 is disposed in the left temple piece 222 and is electrically coupled to the computer 220 disposed in the right temple piece 224. The glasses 200 may include a connector or port (not shown) suitable for charging the battery 218, a wireless receiver, transmitter or transceiver (not shown), or a combination of such devices.

[0039] The glasses 200 include a first or left camera 214 and a second or right camera 216. Although two cameras are depicted, other examples contemplate the use of a single or additional (i.e., more than two) cameras. In one or more examples, the glasses 200 include any number of input sensors or other input / output devices in addition to the left camera 214 and the right camera 216. Such sensors or input / output devices may additionally include biometric sensors, position sensors, motion sensors, etc.

[0040] In some examples, left camera 214 and right camera 216 provide video frame data for glasses 200 to use to extract 3D information from a real-world scene.

[0041] The glasses 200 may also include a touchpad 226 mounted to or integrated with one or both of the left temple piece 222 and the right temple piece 224. The touchpad 226 is typically arranged vertically, with the touchpad 226 being approximately parallel to the temple of the user in some examples. As used herein, generally vertical alignment means that the touchpad is more vertical than horizontal, but may be more vertical than the vertical described above. Additional user input may be provided by one or more buttons 228, which in the example shown are disposed on the outer upper edges of the left optical element holder 204 and the right optical element holder 206. The one or more touchpads 226 and buttons 228 provide a means by which the glasses 200 can receive input from the user of the glasses 200.

[0042] Figure 2B The glasses 200 are shown from the user's perspective. Figure 2A Several elements shown in FIG. have been omitted. Figure 2A As described in Figure 2B The illustrated eyeglasses 200 include a left optical element 208 and a right optical element 210 secured within a left optical element holder 204 and a right optical element holder 206, respectively.

[0043] The glasses 200 include a forward optical assembly 230 including a right projector 232 and a right near-eye display 234 , and a forward optical assembly 238 including a left projector 240 and a left near-eye display 244 .

[0044] In some examples, the near-eye display is a waveguide. The waveguide includes a reflective structure or a diffractive structure (e.g., a grating and / or an optical element such as a mirror, lens, or prism). Light 236 emitted by projector 232 encounters the diffractive structure of the waveguide of near-eye display 234, which directs the light toward the right eye of the user to provide an image on or in the right optical element 210 that is superimposed on the view of the real world seen by the user. Similarly, light 242 emitted by projector 240 encounters the diffractive structure of the waveguide of near-eye display 244, which directs the light toward the left eye of the user to provide an image on or in the left optical element 208 that is superimposed on the view of the real world seen by the user. The combination of the GPU, the front optical assembly 230, the left optical element 208, and the right optical element 210 provides an optical engine of the glasses 200. The glasses 200 use the optical engine to generate an overlay of the user's real-world view, including displaying a 3D user interface to the user of the glasses 200.

[0045] However, it should be understood that other display technologies or configurations may be used within the optical engine to display images to the user in the user's field of view. For example, instead of the projector 232 and waveguide, an LCD, LED or other display panel or surface may be provided.

[0046] In use, the user of the glasses 200 will be presented with information, content, and various 3D user interfaces on the near-eye display. As described in more detail herein, the user can then use the touch pad 226 and / or buttons 228, associated devices (e.g., Figure 3 The user can interact with the glasses 200 by voice input or touch input on a client device 328 shown in FIG. 1 and / or hand movements, position, and orientation detected by the glasses 200 .

[0047] Figure 3is a block diagram showing a networked system 300 including details of the glasses 200 according to some examples. The networked system 300 includes the glasses 200, a client device 328, and a server system 332. The client device 328 may be a smartphone, a tablet computer, a tablet phone, a laptop computer, an access point, or any other such device capable of connecting to the glasses 200 using both a low power wireless connection 336 and / or a high speed wireless connection 334. The client device 328 is connected to the server system 332 via a network 330. The network 330 may include any combination of wired and wireless connections. The server system 332 may be one or more computing devices that are part of a service or network computing system. The server system 332 may be a server system 332 that is a networked computing system that may be used separately. Fig.13 and Fig.14 The details of the software architecture 1306 or machine 1400 described in the implementation of the client device 328 and any elements of the server system 332 and network 330.

[0048] The glasses 200 include a data processor 302, a display 310, one or more cameras 308, and additional input / output elements 316. The input / output elements 316 may include a microphone, an audio speaker, a biometric sensor, an additional sensor, or an additional display element integrated with the data processor 302. Fig.13 and Fig.14 Examples of input / output elements 316 are further discussed. For example, input / output elements 316 may include any I / O components 1418, including output components 1426, motion components 1434, etc. Figure 2B Examples of display 310 are discussed in . In the particular examples described herein, display 310 includes displays for the left and right eyes of a user.

[0049] Data processor 302 includes image processor 306 (eg, video processor), GPU and display driver 338, tracking module 340, interface 312, low power circuitry 304, and high speed circuitry 320. Components of data processor 302 are interconnected by bus 342.

[0050] The interface 312 refers to any source of user commands provided to the data processor 302. In one or more examples, the interface 312 is a physical button that, when pressed, sends a user input signal from the interface 312 to the low-power processor 314. The low-power processor 314 can process pressing such a button and then immediately releasing it as a request to capture a single image, and vice versa. The low-power processor 314 can process pressing such a button for a first period of time as a request to capture video data when the button is pressed and stop video capture when the button is released, wherein the video captured when the button is pressed is stored as a single video file. Alternatively, pressing the button for a long period of time can capture a still image. In some examples, the interface 312 can be any mechanical switch or physical interface capable of accepting user input associated with requesting data from the camera 308. In other examples, the interface 312 can have a software component, or can be associated with a command received wirelessly from another source, such as from the client device 328.

[0051] Image processor 306 includes circuitry for receiving signals from camera 308 and processing those signals from camera 308 into a format suitable for storage in memory 324 or for transmission to client device 328. In one or more examples, image processor 306 (e.g., a video processor) includes a microprocessor integrated circuit (IC) customized for processing sensor data from camera 308, and volatile memory used by the microprocessor in operation.

[0052] The low power circuit system 304 includes a low power processor 314 and a low power wireless circuit system 318. These elements of the low power circuit system 304 can be implemented as separate elements or can be implemented on a single IC as part of a single system on a chip. The low power processor 314 includes logic for managing other elements of the glasses 200. As described above, for example, the low power processor 314 can accept user input signals from the interface 312. The low power processor 314 can also be configured to receive input signals or command communications from a client device 328 via a low power wireless connection 336. The low power wireless circuit system 318 includes circuit elements for implementing a low power wireless communication system. Bluetooth TM Smart, also known as Bluetooth TM Low power consumption is a standard implementation of a low power wireless communication system that may be used to implement the low power wireless circuitry 318. In other examples, other low power communication systems may be used.

[0053] High-speed circuit system 320 includes a high-speed processor 322, a memory 324, and a high-speed wireless circuit system 326. High-speed processor 322 can be any processor capable of managing high-speed communications and operations of any general-purpose computing system used by data processor 302. High-speed processor 322 includes processing resources used to manage high-speed data transmission on high-speed wireless connection 334 using high-speed wireless circuit system 326. In some examples, high-speed processor 322 executes an operating system such as a LINUX operating system or other such operating system. In addition to any other duties, high-speed processor 322 using the software architecture of executing data processor 302 manages data transmission with high-speed wireless circuit system 326. In some examples, high-speed wireless circuit system 326 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, which is also referred to as Wi-Fi herein. In other examples, high-speed wireless circuit system 326 can implement other high-speed communication standards.

[0054] Memory 324 includes any storage device capable of storing camera data generated by camera 308 and image processor 306. Although memory 324 is shown as being integrated with high-speed circuitry 320, in other examples, memory 324 may be a separate, independent element of data processor 302. In some such examples, electrical wiring may provide a connection from image processor 306 or low-power processor 314 to memory 324 through a chip including high-speed processor 322. In other examples, high-speed processor 322 may manage addressing of memory 324 so that low-power processor 314 will direct high-speed processor 322 whenever a read or write operation involving memory 324 is desired.

[0055] The tracking module 340 estimates the pose of the glasses 200. For example, the tracking module 340 uses image data and corresponding inertial data from the camera 308 and the positioning component 1438 and GPS data to track the position and determine the pose of the glasses 200 relative to a reference frame (e.g., a real-world environment). The tracking module 340 continuously collects and uses updated sensor data describing the movement of the glasses 200 to determine an updated three-dimensional pose of the glasses 200, which indicates changes in relative position and orientation relative to physical objects in the real-world environment. The tracking module 340 allows the glasses 200 to visually place virtual objects relative to physical objects within the user's field of view via the display 310.

[0056] The GPU and display driver 338 can use the pose of the glasses 200 to generate frames of virtual content or other content to be presented on the display 310 when the glasses 200 are operating in a conventional augmented reality mode. In this mode, the GPU and display driver 338 generate updated frames of virtual content based on the updated three-dimensional pose of the glasses 200, which reflects changes in the user's position and orientation relative to physical objects in the user's real-world environment.

[0057] One or more functions or operations described herein may also be performed in an application resident on the glasses 200 or on the client device 328 or on a remote server. The glasses 200 may be a stand-alone client device capable of independent operation, or may be a companion device that works with a host device to offload intensive processing and / or exchange data with the presentation server system 108 over the network 106. The glasses 200 may also be communicatively coupled to a companion device such as a smartwatch and may be configured to exchange data with the companion device. The glasses 200 may also include various components common to mobile electronic devices such as smart glasses or smart phones (e.g., including a display controller for controlling the display of visual media on a display mechanism incorporated in the device).

[0058] Figure 4 1 is a block diagram showing more details about the volumetric content rendering system 100 according to some examples. Specifically, the volumetric content rendering system 100 is shown to include a rendering client 104 and an application server 112. The volumetric content rendering system 100 contains several subsystems, which are supported on the client side by the rendering client 104 and on the server side by the application server 112. For example, these subsystems include: a collection management system 402, a rendering control system 404, an enhancement system 406, and an experience rendering system 408.

[0059] The collection management system 402 is responsible for managing sets or collections of content (e.g., collections of text, images, video, and audio data). Collections of content may be organized into "event libraries" or "event stories." Such collections may be made available for a specified period of time (e.g., for the duration of an event to which the content relates). For example, content related to a concert may be made available as a "story" for the duration of the concert. The collection management system 402 may also be responsible for publishing an icon to the user interface of the presentation client 104 that provides notification that a particular collection exists.

[0060] In addition, the collection management system 402 includes a curation interface 412 that allows a collection manager to manage and curate a specific content collection. For example, the curation interface 412 enables an event organizer to curate a collection of content related to a specific event (e.g., to remove inappropriate content or redundant messages). In addition, the collection management system 402 uses machine vision (or image recognition technology) and content rules to automatically curate content collections.

[0061] The presentation control system 404 is responsible for facilitating and controlling the volumetric content presentation. Thus, the presentation control system 404 provides a mechanism that allows a user to specify control operations for controlling the volumetric content presentation. For example, the control operations may include: a stop operation that stops the presentation; a pause operation that pauses the presentation; a fast-forward operation that advances the presentation at a higher speed; a rewind operation that rewinds the presentation; a zoom-in operation that increases the zoom level of the presentation; a zoom-out operation that decreases the zoom level of the presentation; and a play speed modification operation that changes the speed of the presentation (e.g., to produce a slow motion presentation of the volumetric video).

[0062] For some embodiments, a user may access the system through one or more I / O components (examples of I / O components are described below). Fig.14 Detailed description further) provides one or more inputs to specify an input indicating a control operation for controlling the presentation of the volumetric content. For some embodiments, the presentation control system 404 may provide an interactive control interface including one or more interactive elements (e.g., virtual buttons) to trigger the control operation, and the presentation control system 404 monitors interactions with the interactive interface to detect input indicating the control operation. For some embodiments, the user may trigger the control operation using a gesture (e.g., a hand gesture or a head gesture) that may be associated with a particular control operation.

[0063] The enhancement system 406 provides various functions that enable a user to enhance (e.g., annotate or otherwise modify or edit) media content. For example, the enhancement system 406 provides functions related to the generation, publication, and application of enhancement data, such as media overlays (e.g., image filters) to volumetric content. Media overlays can include audio and visual content and visual effects. Examples of audio and visual content include photos, text, logos, animations, and sound effects. Examples of visual effects include color overlays. Audio and visual content or visual effects can be applied to media content items (e.g., photos) at the client device 102. For example, a media overlay can include text or images that can be superimposed on top of a photo taken by the client device 102. The enhancement system 406 provides one or more media overlays to the presentation client 104 when in operation based on the geographic location of the client device 102 or based on other information such as social network information of a user of the client device 102. The media overlays can be stored in the database 120 and accessed through the database server 118.

[0064] A filter is an example of a media overlay that is displayed as an overlay on an image or video during presentation to a user. Filters can be of various types, including filters selected by a user from a set of filters presented to a user by presentation client 104. Other types of filters include geolocation filters (also referred to as geofilters), which can be presented to a user based on a geographic location. For example, presentation client 104 can present geolocation filters specific to nearby or special locations within a user interface based on geographic location information determined by a global positioning system (GPS) unit of client device 102.

[0065] Another type of filter is a data filter, which can be selectively presented to the user by the presentation client 104 based on other input or information collected by the client device 102. Examples of data filters include the current temperature at a particular location, the current speed at which the sending user is traveling, the battery life of the client device 102, or the current time.

[0066] AR content items are another example of a media overlay. AR content items may be real-time special effects and / or sounds that may be added to images or videos, including volumetric images and volumetric videos.

[0067] In general, AR content items, overlays, image transformations, images, and similar terms refer to modifications that can be applied to image data (e.g., video or images) including volumetric content. This includes real-time modifications that modify images as they are captured using a device sensor (e.g., one or more cameras) of the client device 102 and then displayed and modified by a display device (e.g., an embedded display of the client device) of the client device 102. This also includes modifications to stored content, such as volumetric videos in a gallery or collection that can be modified. For example, in a client device 102 that has access to multiple AR content items, a user can use a single video with multiple AR content items to see how different AR content items will modify the stored video. For example, by selecting different AR content items for the content, multiple augmented reality content items that apply different pseudo-random movement models can be applied to the same content. Similarly, real-time video capture can be used with the modifications shown to show how the video image currently captured by the sensor of the client device 102 will modify the captured data. Such data can be simply displayed on the screen without being stored in memory, or the content captured by the device sensor can be recorded and stored in memory with or without modification (or both). In some systems, the preview feature can show how different augmented reality content items will be displayed in different windows of the display at the same time. For example, this can enable viewing multiple windows with different pseudo-random animations on the display at the same time.

[0068] Thus, data and various systems using augmented reality content items or other such augmented systems using augmented data to modify content may involve detection of objects (e.g., faces, hands, bodies, cats, dogs, surfaces, objects, etc.) in video frames, tracking such objects as they leave, enter, and move around in the field of view, and modifying or transforming such objects as they are tracked. In various examples, different methods for implementing such transformations may be used. Some examples may involve: generating a three-dimensional mesh model of one or more objects; and using transformations and animated textures of the models within a video to implement the transformations. In other examples, tracking of points on an object may be used to place an image or texture (which may be two-dimensional or three-dimensional) at a tracked location. In yet another example, neural network analysis of a video frame may be used to place an image, model, or texture in content (e.g., a frame of an image or video). Thus, an AR content item refers to both images, models, and textures used to create transformations in content, as well as additional modeling and analysis information required to implement such transformations using object detection, tracking, and placement.

[0069] Real-time video processing can be performed with any kind of video data (e.g., video streams, video files, volumetric video, etc.) stored in the memory of any kind of computerized system. For example, a user can load video files and store them in the memory of the device, or a sensor of the device can be used to generate a video stream. In addition, any object, such as a human face and parts of a human body, an animal, or a non-living thing (e.g., a chair, a car, or other objects) can be processed using computer animation models.

[0070] In some examples, when a specific modification is selected together with the content to be enhanced (e.g., edited), the element to be transformed is identified by the computing device, and then if the element to be transformed is present in the frame of the video, the element to be transformed is detected and tracked. The elements of the object are modified according to the request for modification, so the frame of the video stream is transformed. For different types of transformations, the frame transformation of the video stream can be performed by different methods. For example, for frame transformations that mainly involve changing the form of the elements of the object, characteristic points for each element of the object are calculated (e.g., using active shape models (ASM) or other known methods). Then, for each element of at least one element of the object, a grid based on the characteristic points is generated. The grid can be used for subsequent stages of tracking the elements of the object in the video stream. In the tracking process, the mentioned grid for each element is aligned with the position of each element. Then, additional points are generated on the grid. A set of first points is generated for each element based on the modification request, and a set of second points is generated for each element based on the set of first points and the modification request. Then, the frame of the video stream can be transformed by modifying the elements of the object based on the set of first points and the set of second points and the grid. In such methods, the background of the modified object may also be changed or distorted by tracking and modifying the background.

[0071] In some examples, a transformation that changes certain areas of an object using elements of an object can be performed by calculating characteristic points of each element of the object and generating a grid based on the calculated characteristic points. Points are generated on the grid, and then various areas based on the points are generated. The elements of the object are then tracked by aligning the area of ​​each element with the position of each element of at least one element, and the properties of the area can be modified based on the modification request, thereby transforming the frame of the video stream. Depending on the specific modification request, the properties of the mentioned area can be transformed in different ways. Such modifications may involve: changing the color of the area; removing at least some parts of the area from the frame of the video stream; including one or more new objects into the area based on the modification request; and modifying or distorting the elements of the area or object. In various examples, any combination of such modifications or other similar modifications may be used. For certain models to be animated, some characteristic points can be selected as control points to determine the entire state space of options for model animation.

[0072] In some examples of computer animation models that use face detection to transform image data, faces are detected on an image using a specific face detection algorithm (e.g., Viola-Jones). An active shape model (ASM) algorithm is then applied to the face region of the image to detect facial feature reference points.

[0073] Other methods and algorithms suitable for face detection can be used. For example, in some examples, landmarks are used to locate features, and landmarks represent distinguishable points that are present in most of the images considered. For example, for facial landmark points, the location of the left eye pupil can be used. If the initial landmark is not recognizable (for example, if the person has an eye mask), secondary landmarks can be used. Such a boundary recognition process can be used for any such object. In some examples, a set of landmarks forms a shape. The coordinates of the points in the shape can be used to represent the shape as a vector. A shape is aligned with another shape using a similarity transformation (allowing translation, scaling, and rotation) that minimizes the average Euclidean distance between the shape points. The mean shape (meanshape) is the average value of the aligned training shapes.

[0074] In some examples, the landmark search begins with an average shape that aligns with the position and size of the face determined by a global face detector. Such a search then repeats the steps of suggesting tentative shapes by adjusting the positions of shape points by template matching of the image texture around each point, and then conforming the tentative shapes to a global shape model until convergence occurs. In some systems, template matching alone is unreliable, and the shape model pools the results of weak template matches to form a stronger overall classifier. The entire search is repeated at each level of the image pyramid from coarse to fine resolution.

[0075] The augmentation system 406 can capture an image or video stream on a client device (e.g., client device 102) and perform complex image manipulations locally on the client device 102 while maintaining an appropriate user experience, computation time, and power consumption. Complex image manipulations can include size and shape changes, emotion conversions (e.g., changing a face from a frown to a smile), state transitions (e.g., aging an object, reducing apparent age, changing gender), style transfer, application of graphical elements, and any other suitable image or video manipulations enabled by a convolutional neural network that has been configured to execute efficiently on the client device 102.

[0076] In some examples, the enhancement system 406 may use a computer animation model for transforming video and image content, where a neural network operates as part of a rendering client 104 operating on a client device 102. The enhancement system 406 determines the presence of a face within an image or video stream and provides an interactive modification element (e.g., an icon) associated with a computer animation model for transforming image data, or the computer animation model may exist as associated with an interface described herein. The interactive modification element includes a change that can modify the basis of the user's face within the image or video content as part of a modification operation. Once the interactive modification element is selected, the transformation system initiates a process of transforming the user's image to reflect the selected interactive modification element (e.g., generating a smiley face on the user). Once the image or video stream is captured and the specified modification is selected, the modified image or video content can be presented in a graphical user interface displayed on the client device 102. The enhancement system 406 can implement a complex convolutional neural network on a portion of the image or video content to generate and apply the selected modification. That is, the modified content can be presented to the user in real time or near real time. In addition, the modification can persist while the content is being presented. Machine learning neural networks can be used to implement such modifications.

[0077] The experience presentation system 408 is responsible for providing a presentation of the recorded experience (e.g., a trip). In some examples, the experience presentation system 408 generates and maintains experience data related to the user's experience. The experience data may describe the real-world environment of the experience and multiple points of interest during the experience. The experience data may also include weather data describing the weather conditions of the experience, and the user's biometrics generated during the experience. Then, the experience presentation system 408 may cause the display device to present a visual representation of the experience based on the experience data related to the user's experience. The visual representation of the experience may include a multi-dimensional representation of the real-world environment and a representation of multiple points of interest superimposed on the multi-dimensional representation of the real-world environment. The real-world environment may be two-dimensional or three-dimensional. The visual representation of the experience may also include a representation of weather conditions and a representation of biometric data. In some examples, the data may also include a perspective at some or all of the multiple points of interest. The experience presentation system 408 may detect a point of interest selected by the user among the multiple points of interest. In response to the detection, the experience presentation system 408 may cause the display device to switch to a presentation of the corresponding perspective.

[0078] Figure 5 is a diagrammatic representation of data structures maintained in database 120 according to some examples. Although the contents of database 120 are shown as including several tables, it should be understood that data may be stored in other types of data structures (eg, as an object-oriented database).

[0079] The entity table 504 stores entity data and is linked (e.g., by reference) to the entity graph 506 and profile data 508. The entities whose records are maintained within the entity table 504 may include individuals, corporate entities, organizations, objects, places, events, etc. Regardless of the entity type, any entity for which the presence server system 108 stores data may be an identified entity. Each entity is provided with a unique identifier as well as an entity type identifier (not shown). The entity table 504 may associate various enhancements from the enhancement table 510 with various images and videos stored in the image table 516 and the video table 514.

[0080] The entity graph 506 stores relationships and associated information between entities. Such relationships may be, for example, social, professional (e.g., working in a common company or organization), interest-based, or activity-based.

[0081] The profile data 508 stores various types of profile data about a particular entity. Based on the privacy settings specified by the particular entity, the profile data 508 can be selectively used and presented to other users of the volumetric content presentation system 100. In the case where the entity is a person, the profile data 508 includes, for example, a user name, a phone number, an address, settings (e.g., notification and privacy settings), and an avatar representation (or a collection of such avatar representations) selected by the user.

[0082] The database 120 also stores augmentation data (e.g., including AR content items and overlays of filters) in augmentation table 510. The augmentation data is associated with and applied to videos (whose data is stored in video table 514) and images (whose data is stored in image table 516), including volumetric videos and volumetric images.

[0083] The story table 512 stores data about collections of content including associated images, video, or audio data that are compiled into collections (e.g., stories or libraries). The creation of a particular collection can be initiated by a particular user (e.g., each user whose record is maintained in the entity table 504). A user can create a "personal story" in the form of a collection of content that has been created and sent / broadcasted by the user. To this end, the user interface of the presentation client 104 may include an icon that is user-selectable to enable the sending user to add specific content to his or her personal story.

[0084] As mentioned above, the video table 514 stores video data including volumetric videos. Similarly, the image table 516 stores image data including volumetric images.

[0085] Figure 6is a conceptual diagram illustrating an example presentation of a visual representation of an experience through a head mounted display device according to some example embodiments. Figure 6 As shown in , user 602 can wear head mounted display device 604 (e.g., glasses 200). In some embodiments, head mounted display device 604 supports AR experience. AR experience includes: applying virtual content to real world environment by presenting virtual content through a transparent display through which the real world environment can be seen. For example, head mounted display device 604 superimposes the presentation of virtual content 606 on real world objects (e.g., table 608). User 602 can see both the presentation of virtual content 606 and the presentation of real world table 608 through head mounted display device 604.

[0086] Virtual content 606 may correspond to an experience (e.g., a hiking trip) between two friends. During the capture phase, the two friends may wear or carry smart devices (e.g., glasses 200, smart phones) while hiking. During the hike, the two friends may use the smart devices to take videos and photos and casually hike across the mountain. Embedded cameras from the smart devices may capture multiple perspectives of the two friends at different points of interest. The smart devices may also include drones that capture multiple perspectives at different points of interest or along the hike. Back Figure 6 , after the hike, the presentation of virtual content 606 may include an AR terrain rendering of mountain 610, with the path the friend took to reach the top of the mountain visually presented or highlighted. The presentation of virtual content 606 may also include representations of the two friends (e.g., miniature avatar 612 and miniature avatar 614).

[0087] In some embodiments, the presentation of the virtual content 606 also includes the presentation of multiple points of interest 616. The multiple points of interest 616 may be determined evenly along the path of the hike, or may be determined based on videos and photos taken by the two friends. Alternatively or additionally, the points of interest 616 may be determined based on the biometric data of the friends during the hike (e.g., blood pressure, heart rate, body temperature). For example, any special moment during the hike (e.g., seeing an endangered flower or animal, seeing a beautiful view, facing danger) may cause the friend's heart rate to increase dramatically. The friend's perspective at the special moment may be recorded, and the relative position and / or latitude at which the special moment occurred may be recorded and marked as a point of interest 616 on the presentation of the virtual content 606. When the user 602 interacts with (e.g., touches, clicks) a representation of one of the points of interest 616, the head mounted display device 604 may switch to the presentation of the recorded perspective associated with the point of interest. Fig.11 Details on the methods used to switch presentation can be found in and its description. Figure 7An example presentation of recorded perspectives associated with a point of interest is found in the map and its description. In some embodiments, user 602 can click anywhere on the path to view the video taken at that point from any smart device (e.g., drone) during the hike.

[0088] It should be noted that the presentation of virtual content 606 is merely an example and should not be limited thereto. Any other components disclosed elsewhere in this application may be integrated into the presentation of virtual content 606. For example, the presentation of virtual content 606 may include a virtual control panel (e.g., virtual control panel 706). In response to detecting an input indicating interaction with the control panel, the head mounted display device 604 may present a visual representation of the user (e.g., miniature avatar 612 and miniature avatar 614) at a position corresponding to the desired progress of the experience, and the desired progress of the experience corresponds to the input indicating the interaction.

[0089] Figure 7 6 is a conceptual diagram illustrating an example presentation 700 of recorded perspectives associated with a point of interest, according to some example embodiments. Presentation 700 corresponds to a perspective captured by a smart device of one of two hikers (e.g., the hiker of miniature avatar 612 walking behind). Hiker 702 in presentation 700 corresponds to Figure 6704 is an example of a mountain 610. The recorded perspective in the presentation 700 may be a still photo or a captured video. The presentation 700 may also include a virtual control panel 706. When the display device (e.g., the head mounted display device 604) or its sensor device detects an input indicating a user's interaction with the control panel 706, the display device may control or change the presentation 700 based on the detected input indicating the user's interaction with the control panel 706. The virtual control panel 706 may include a plurality of virtual buttons, such as a virtual play button 708, a virtual pause button 710, a virtual stop button 712, a virtual record button 714, a virtual rewind button 716, a virtual previous button 718, a virtual next button 720, and a virtual fast forward button. In response to an input indicating a user's interaction with the virtual play button 708 (e.g., touching, pressing, or clicking), the display device may switch to a presentation of a captured video associated with the presentation 700 of the recorded perspective. In response to input indicating user interaction with virtual pause button 710, the display device may pause the presentation of the captured video. In response to input indicating user interaction with virtual stop button 712, the display device may switch back to the default presentation 700 (e.g., a still photo) of the recording perspective. In response to input indicating user interaction with virtual record button 714, the display device may record the currently displayed still photo or captured video. The recording may last for a predetermined duration, for example, 10 seconds, 30 seconds, 1 minute, 5 minutes, or until the user interacts with virtual stop button 712. When the recording is finished, the display device may transmit the recording to the user's user device or server, or store the recording on a storage medium. In response to input indicating user interaction with virtual rewind button 716, the display device may rewind the presentation of the captured video. The captured video may be rewound for a preset duration, for example, 5 seconds, 10 seconds, or 20 seconds. Alternatively, the presentation of the captured video may be rewound until the user stops interacting with virtual rewind button 716 (released from virtual rewind button 716). In response to input indicating user interaction with virtual previous button 718, the display device may switch to presentation of a recorded perspective of a previous point of interest. Similarly, in response to input indicating user interaction with virtual next button 720, the display device may switch to presentation of a recorded perspective of a next point of interest. In response to input indicating user interaction with virtual fast forward button 722, the display device may fast forward presentation of captured video.

[0090] It should be noted that virtual buttons 708 to 722 are examples and should not be limited thereto. On-site personnel can modify, add or delete buttons without departing from the spirit of the present application. Such modifications, additions or deletions are within the scope of protection of the present application. In the case where the recorded perspective is associated only with a static photo, virtual buttons 708 to 716 and virtual button 722 can be omitted. In some embodiments, a smart device (e.g., a drone) can take static photos or capture videos during an experience (e.g., a hiking trip between two friends). The presentation 700 of the recorded perspective can also include a virtual symbol 724 and a virtual interactive icon 726 of the smart device. When the user interacts with the virtual interactive icon 726, the display device can switch to the presentation of the recorded perspective taken by the smart device.

[0091] Figure 8 is a conceptual diagram illustrating an example 2-dimensional (2D) presentation 800 of a visual representation of an experience according to some example implementations. Figure 6Similar to the presentation of virtual content 606 in , 2D presentation 800 can be presented by a head mounted display device (e.g., head mounted display device 604, glasses 200) as an AR terrain rendering of an experience between two people (e.g., a hiking trip between two friends). In examples where there is no altitude change during the experience, the head mounted display device can present the experience through a 2D representation. In other examples, the head mounted display device can present the experience in a 3D representation. Alternatively or additionally, the user can specify whether the head mounted display device presents the experience through a 2D representation or a 3D representation. Presentation 800 may include: a path of the experience, representations of two people (e.g., miniature avatars 806 and miniature avatars 808), other terrain information (e.g., trees, roads, lakes), a virtual symbol 810 of a smart device, a representation 812 corresponding to the perspective of the smart device, and multiple representations 814 corresponding to points of interest during the experience. Smart devices may include smart phones, smart bracelets, smart watches, smart glasses, camera devices mounted on handheld poles, smart helmets, drones, etc. The following description uses drones as examples of smart devices, but should not be limited to this. When the user interacts with the representation 814 of one of the points of interest (e.g., touches, clicks), the head mounted display device may switch to a presentation of a recorded perspective associated with the point of interest. When the user interacts with the representation 812 corresponding to the perspective of the drone, the head mounted display device may switch to a presentation of an associated recorded perspective taken by the drone. The 2D presentation 800 may also include a virtual control panel 802. In response to detecting an input indicating an interaction with the virtual control panel 802, the display device may present a visual representation of the user at a location corresponding to the desired progress of the experience in the real world environment. The detailed functions of the virtual buttons on the virtual control panel 802 are similar to the virtual buttons 708 to 722 on the virtual control panel 706, except that the virtual control panel 802 is used to control the progress of the experience (e.g., the position of the miniature avatar 806 and the miniature avatar 808). For example, when the user interacts with the virtual play button (e.g., presses), the display device causes the presentation to include a slow and continuous change in the position of the miniature avatar 806 and the miniature avatar 808 to reflect the movement of two people during the real world experience. When the user interacts with (eg, presses) the virtual fast forward button, the display device causes the presentation to include a larger change in the position of micro-avatar 806 and micro-avatar 808 (moving faster).

[0092] Figures 9 to 12 is a flow chart illustrating an example method for presenting an experience using AR, according to some example implementations.

[0093] The method 900 may be implemented in computer-readable instructions for execution by one or more processors, such that the operations of the method 900 may be performed in part or in whole by functional components of the volumetric content rendering system 100; therefore, the method 900 is described below by way of example with reference thereto. However, it should be understood that at least some of the operations of the method 900 may be deployed on various other hardware configurations besides the volumetric content rendering system 100.

[0094] At operation 902, the volumetric content presentation system 100 generates experience data associated with a recorded experience of a user. The experience data describes multiple aspects of the recorded experience, the multiple aspects of the experience including a first real-world environment of the experience. The experience data also identifies one or more points of interest within the first real-world environment determined based on device data (e.g., location data) obtained from one or more devices of the user. Typically, generating the experience data includes obtaining and aggregating data describing one or more aspects of the experience from one or more devices. The data obtained from the one or more devices may include input data, biometric data, motion data, environmental data, and positioning data, examples of which are described below with reference to Fig.11 Have a discussion.

[0095] One or more devices of the user may include a camera device, and the generation of the experience data may include: acquiring image data from a camera device including one or more images (e.g., a video) describing the experience. In an example, the image data may be acquired from an embedded camera of a drone for capturing at least a portion of the experience. In another example, the image data may be acquired from a wearable device (e.g., glasses 200) including one or more images that depict the experience from the perspective of a user of the wearable device. In yet another example, the image data may be acquired from a camera device including one or more images that depict the experience from the perspective of the point of interest or depict a point of interest.

[0096] For some embodiments, performing volumetric capture of an experience, and generating experience data may include acquiring or generating volumetric content based on the volumetric capture. For example, the volumetric content rendering system 100 may generate the volumetric content by stitching together various video, image, audio, and sensor data generated via known volumetric capture techniques. For example, the video, image, audio, and sensor data may be provided by a user to the volumetric content rendering system 100 as part of a request to generate volumetric content.

[0097] The experience data may also include data generated from other devices and third-party servers. In an example, data such as weather data, map data, traffic data, and topology data may be obtained from a third-party server and included in the experience data.

[0098] Depending on the type and location of the experience, generating experience data may also include acquiring data from one or more network-connected devices (eg, IoT devices).

[0099] At operation 904, the volumetric content presentation system 100 stores the experience data in a network accessible data store (e.g., database 120). The network accessible data store may include a collection of experience data that may include the user's experience data and experience data of other users to whom the user has been permitted to access.

[0100] At operation 906, the volumetric content rendering system 100 provides an interactive interface to a display device (e.g., glasses 200), which allows a user to select a recorded experience for rendering. At operation 908, the volumetric content rendering system 100 receives input indicating a selection of an experience for rendering. At operation 910, in response to receiving the input, the volumetric content rendering system 100 accesses the experience data.

[0101] At operation 912, the volumetric content presentation system 100 causes presentation of a visual representation of the experience via a display device of the user. The visual representation is rendered based on the experience data. For some embodiments, the experience data may include the visual representation. For some embodiments, the visual representation is generated based on the experience data.

[0102] The visual representation of the experience includes a multi-dimensional representation of a first real-world environment and representations of one or more points of interest presented with the multi-dimensional representation of the first real-world environment. The visual representation of the experience may also include a visual representation of a user superimposed on the multi-dimensional representation of the first real-world environment. For some embodiments, the visual representation of the experience includes a terrain rendering of the first real-world environment. The terrain rendering of the first real-world environment may be generated based on geospatial information, architectural information, or an existing model of the experience. Presentation of the visual representation of the experience includes presenting the multi-dimensional representation of the first real-world environment superimposed on a second real-world environment, the second real-world environment being within a field of view of a user of a display device.

[0103] As described above, in some examples, the experience data may include weather data. The weather data describes the weather conditions of the experience (e.g., weather types include: rainy, sunny, windy, cloudy, stormy, etc., and weather parameters include temperature, humidity, wind level, visibility). In these examples, the visual representation of the experience may include a representation of the weather conditions superimposed on the multi-dimensional representation of the first real-world environment.

[0104] As described above, in some examples, the experience data includes a recorded perspective associated with a point of interest. Consistent with these examples, the volumetric content presentation system 100 can detect input indicating a user's interaction with a representation of a point of interest within a presentation of a visual representation of the experience. The user's interaction with the representation of the point of interest can correspond to a manual selection of the point of interest. Detecting the interaction can include: detecting a user's gesture pointing to the representation of the point of interest, detecting a user's gaze pointing to the representation of the point of interest, or detecting a user's motion or movement in the direction of the representation of the point of interest. In response to detecting the input indicating the user's interaction with the representation of the point of interest, the volumetric content presentation system 100 causes the display device to switch to presentation of the recorded perspective associated with the point of interest. The perspective can be recorded by a smart device at the point of interest. The smart device can include a smart phone, a smart bracelet, a smart watch, smart glasses, a camera mounted on a handheld pole, a smart helmet, a drone, etc.

[0105] The volumetric content presentation system 100 can detect an input indicative of a control operation associated with the visual representation of the experience. The control operation can indicate a desired progress of the presentation of the experience. For some embodiments, the input corresponds to one or more I / O components of any one or more of the display devices (examples of which are described below with reference to Fig.14 In some embodiments, receiving input indicating a control operation may include receiving one or more input signals based on user interaction with one or more I / O components.

[0106] For some embodiments, presentation of a visual representation of an experience may include an interactive control interface including one or more interactive elements (e.g., virtual buttons) to trigger control operations, and receiving input may include detecting interaction of a user of the display device with the interactive elements of the interactive interface.

[0107] For some embodiments, receiving input indicating a control operation may include detecting a gesture of a user of the display device. For example, a body movement of the user (e.g., a hand gesture or a head gesture) may be associated with the control operation and used by the user to trigger the control operation. For example, detecting the user's gesture may include performing image processing on image data generated by a camera coupled to the display device in communication. Performing image processing to detect hand gestures may include utilizing one or more machine learning models.

[0108] In response to detecting the input indicating the control operation, the volumetric content presentation system 100 can cause the display device to control the presentation of the visual representation of the experience. In an example, controlling the presentation includes presenting the visual representation of the user at a location in the real-world environment corresponding to the desired progress of the experience.

[0109] In some examples, the experience data includes biometric data of the user generated during the experience (e.g., blood pressure, heart rate, body temperature). The visual representation of the experience may include a representation of the biometric data of the user generated during the experience.

[0110] In some examples, the volumetric content presentation system 100 may determine a region of interest in a multi-dimensional representation of the real-world environment based on the biometric data.The visual representation of the experience may include visual enhancement of the region of interest in the multi-dimensional representation of the real-world environment.

[0111] In some examples, the experience may be at a constant height, and the multi-dimensional representation of the scene may include a two-dimensional representation. Alternatively, the experience may include a change in height, and the multi-dimensional representation of the scene may include a three-dimensional representation.

[0112] Depending on the implementation, the operations of method 900 may be repeated in different ways or involve operations not shown occurring in between. Although the operations of method 900 may be depicted and described in a certain order, the order in which the operations are performed may vary between implementations, including performing certain operations in parallel.

[0113] In some embodiments, Fig.10 As shown, method 900 may include operations 1002, 1008, and 1010. Consistent with these embodiments, operation 1002 may be performed as part of the volumetric content rendering system 100 generating experience data in operation 902. At operation 1002, the volumetric content rendering system 100 determines the state of the network-connected device during the volumetric capture of the user's experience. In other words, the volumetric content rendering system 100 determines the state of the network-connected device when the volumetric content is captured. For some devices, the state of the device corresponds to a specific state of the device. For example, the device state data may include one or more device settings of the network-connected device. For some devices, the device state data may include map or location data, such as data defining a path or route between two locations. The volumetric content rendering system 100 may determine the real-time state when performing the volumetric capture or after the volumetric capture. For some devices, the device state data may include the height, direction, temperature, humidity, sound of the device.

[0114] As shown, in determining the state of the network-connected device, the volumetric content presentation system 100 may perform any one or more of operations 1004 and 1006. At operation 1004, the volumetric content presentation system 100 sends a request for device state data to the network-connected device or a network server associated with the network-connected device (e.g., through an API of the device or the network server) over a network. At operation 1006, the volumetric content presentation system 100 collects sensor data from one or more sensors associated with the real-world three-dimensional space. For example, the one or more sensors may include: an embedded sensor of the network-connected device, a sensor communicatively coupled to the network-connected device, or a sensor in the real-world three-dimensional space or a sensor in a real-world environment that is capable of sensing one or more aspects of the operation of the network-connected device.

[0115] Consistent with these embodiments, operations 1008 and 1010 may be performed as part of the volumetric content rendering system 100 causing the rendering of the volumetric content in operation 912. At operation 1008, the volumetric content rendering system 100 causes the rendering of a volumetric representation of one or more elements superimposed on a real-world space, which may be the same space as the three-dimensional space in which the volumetric video was captured, or a different space. The rendering of the volumetric content may include: displaying one or more content items in motion; displaying one or more content items performing movement or other actions; displaying one or more content items in a static position; or a combination thereof. Content items may be displayed during the rendering of the volumetric content or a portion thereof.

[0116] The presentation of volumetric content may include tracking the user's position and movement in their physical real-world environment and using the tracked position and movement of the user to allow the user to move around in and interact with the presentation of volumetric content. Thus, the presentation of volumetric content may include displaying content items from multiple perspectives based on the user's movement and position changes. In this way, the presentation of volumetric content provides an immersive AR experience to the user.

[0117] At operation 1010, the volumetric content presentation system 100 configures one or more devices using the device state data. As described above, the device state data may include the state of the network-connected device during the volumetric capture of the user's experience. Thus, in some instances, configuring the one or more network-connected devices includes: configuring the network-connected device according to the state of the network-connected device during the volumetric capture of the user's experience so that the device is in the same state as when the user's experience occurred. Examples of configuring the network-connected device using the device state data include: enabling or adjusting the presentation of a 2D or 3D representation of the experience based on the altitude and orientation recorded during the experience; simulating weather events (using sound and light effects) based on the temperature, humidity, and sound recorded during the experience; setting the temperature of a smart thermostat; causing a navigation device to display a navigation path; and causing a smart speaker to play a song.

[0118] In some embodiments, based on device state data (e.g., whether there is a change in altitude during the experience), the volumetric content presentation system 100 can determine whether to present the experience through a 2D representation or a 3D representation. Alternatively or additionally, the user can select whether the volumetric content presentation system presents the experience through a 2D representation or a 3D representation.

[0119] Consistent with these embodiments, operation 1012 may be performed as part of the volumetric content rendering system 100 configuring one or more network-connected devices in operation 1012. At operation 1012, the volumetric content rendering system 100 sends a command to a network-connected device or a network server associated with the network-connected device over a network to configure the device according to the state of the device when the volumetric content was captured, so that the network-connected device is configured to the same state as when the user experience occurred. As described above, the state of the device may include one or more device settings, and thus configuring the network-connected device may include applying the one or more device settings to the network-connected device.

[0120] In some embodiments, Fig.11 As shown in , method 900 may also include operations 1102 and 1104. As described above, the presentation of the visual representation of the experience may include multiple representations of the point of interest captured by the user device or the drone during the experience (e.g., representation 616, representation 726, representation 812, and representation 814). At operation 1102, the volumetric content presentation system 100 detects a user interaction (touch, hold, click) with the representation of the point of interest. At operation 1104, in response to detecting the user interaction with the representation of the point of interest, the volumetric content presentation system 100 switches to presentation of a recorded perspective associated with the point of interest. The recorded perspective includes image data depicting a portion of the real-world environment visible from the point of interest during the experience.

[0121] In some embodiments, Fig.12 As shown in , method 900 may also include operations 1202 and 1204. As described above, the presentation of the visual representation of the experience may include the presentation of a control panel (e.g., virtual control panel 802). At operation 1202, the volumetric content presentation system 100 detects an interaction with the control panel indicating a target progress of the experience. The interaction may include a direct interaction (e.g., touching, pressing, clicking) with a virtual button on the control panel to move the user's visual representation to a desired location in the visual representation of the experience. In some embodiments, the interaction may include directly interacting with the user's visual representation (e.g., dragging or moving) to a desired location in the visual representation of the experience. At operation 1204, in response to detecting the interaction with the control panel, the volumetric content presentation system 100 causes the display device to present the user's visual representation at a location corresponding to the desired progress of the experience in the real world environment. For example, if the user clicks the play button on the control panel, the volumetric content presentation system 100 causes the visual representation of the two hikers (e.g., miniature avatar 806 and miniature avatar 808) to move from the starting point (the foot of the mountain) to the end point (the top of the mountain). The user may interact with (eg, click) a rewind button on the control panel to move the visual representations of the two hikers backward.

[0122] Software Architecture

[0123] Fig.13 is a block diagram illustrating an example software architecture 1306 that may be used in conjunction with the various hardware architectures described herein. Fig.13 is a non-limiting example of a software architecture, and it will be appreciated that many other architectures may be implemented to facilitate the functionality described herein. Software architecture 1306 may be implemented in a system such as Fig.14 1400, which includes a processor 1404, a memory / storage device 1406, and an I / O component 1418, among others. A representative hardware layer 1352 is shown and may represent, for example, Fig.14 The representative hardware layer 1352 layer includes a processing unit 1354 having associated executable instructions 1304. The executable instructions 1304 represent executable instructions of the software architecture 1306, including implementations of the methods, components, etc. described herein. The hardware layer 1352 also includes a memory and / or storage module 1356 also having executable instructions 1304. The hardware layer 1352 may also include other hardware 1358.

[0124] exist Fig.13In the exemplary architecture shown, software architecture 1306 can be conceptualized as a layer stack, wherein each layer provides specific functionality. For example, software architecture 1306 may include layers such as operating system 1302, library 1320, framework / middleware 1318, application 1316, and presentation layer 1314. In operation, application 1316 and / or other components within a layer may call API call 1308 through a software stack and receive a response to API call 1308 as message 1312. The layers shown are representative in nature, and not all software architectures have all layers. For example, some mobile or dedicated operating systems may not provide framework / middleware 1318, while other operating systems may provide such layers. Other software architectures may include additional layers or different layers.

[0125] The operating system 1302 can manage hardware resources and provide public services. For example, the operating system 1302 can include a core 1322, a server 1324, and a driver 1326. The core 1322 can serve as an abstraction layer between the hardware and other software layers. For example, the core 1322 can be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, etc. The server 1324 can provide other public services to other software layers. The driver 1326 is responsible for controlling or connecting to the underlying hardware. For example, depending on the hardware configuration, the driver 1326 includes a display driver, a camera driver, drivers, flash memory drivers, serial communication drivers (e.g., Universal Serial Bus (USB) drivers), Drivers, audio drivers, power management drivers, etc.

[0126] The library 1320 provides a common infrastructure used by the application 1316 and / or other components and / or layers. The library 1320 provides functions that allow other software components to perform tasks in an easier manner than directly interfacing with the underlying operating system 1302 functions (e.g., the core 1322, the server 1324, and / or the driver 1326). The library 1320 may include a system library 1344 (e.g., a C standard library), which may provide functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. In addition, the library 1320 may include an API library 1346, such as a media library (e.g., a library that supports the rendering and manipulation of various media formats such as MPEG4, H.2104, MP3, AAC, AMR, JPG, and PNG), a graphics library (e.g., an OpenGL framework that can be used to render 2D and 3D graphics content on a display), a database library (e.g., SQLite that can provide various relational database functions), a web library (e.g., WebKit that can provide web browsing functions), etc. The library 1320 may also include various other libraries 1348 to provide numerous other APIs to the applications 1316 and other software components / modules.

[0127] The framework / middleware 1318 provides a higher level common infrastructure that can be used by the applications 1316 and / or other software components / modules. For example, the framework / middleware 1318 can provide various GUI functions, advanced resource management, advanced location services, etc. The framework / middleware 1318 can provide a wide range of other APIs that can be utilized by the applications 1316 and / or other software components / modules, some of which can be specific to a particular operating system 1302 or platform.

[0128] Applications 1316 include built-in applications 1338 and / or third-party applications 1340. Examples of representative built-in applications 1338 may include, but are not limited to, contact applications, browser applications, book reader applications, location applications, media applications, messaging applications, and / or game applications. Third-party applications 1340 may include applications that are run by entities other than the vendor of a particular platform using ANDROID TM or IOS TM Software Development Kit (SDK) for developing applications and can be used on platforms such as IOS TM ANDROID TM , The third-party application 1340 may call an API call 1308 provided by the mobile operating system (eg, operating system 1302) to facilitate the functionality described herein.

[0129] Applications 1316 may create user interfaces to interact with users of the system using built-in operating system functionality (e.g., kernels 1322, servers 1324, and / or drivers 1326), libraries 1320, and frameworks / middleware 1318. Alternatively or additionally, in some systems, interaction with the user may occur through a presentation layer such as presentation layer 1314. In these systems, the application / component "logic" may be separated from aspects of the application / component that interact with the user.

[0130] Fig.14 1 is a block diagram illustrating components of a machine 1400 capable of reading instructions from a machine-readable medium (e.g., a machine-readable storage medium) and performing any one or more of the methods discussed herein, according to some example embodiments. Specifically, Fig.14 A graphical representation of a machine 1400 in the example form of a computer system is shown, within which instructions 1410 (e.g., software, programs, applications, applet, app, or other executable code) for causing the machine 1400 to perform any one or more of the methods discussed herein may be executed. Thus, the instructions 1410 may be used to implement the modules or components described herein. The instructions 1410 convert a general-purpose, unprogrammed machine 1400 into a specific machine 1400 that is programmed to perform the functions described and shown in the described manner. In alternative embodiments, the machine 1400 operates as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine 1400 may operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 1400 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a PDA, an entertainment media system, a cellular phone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web device, a network router, a network switch, a network bridge, or any machine capable of executing instructions 1410 that specify actions to be taken by the machine 1400, either sequentially or otherwise. In addition, while only a single machine 1400 is shown, the term "machine" should also be taken to include a collection of machines that individually or jointly execute instructions 1410 to perform any one or more of the methodologies discussed herein.

[0131] The machine 1400 may include a processor 1404, a memory / storage device 1406, and an I / O component 1418, which may be configured to communicate with each other, for example, via a bus 1402. In an example embodiment, the processor 1404 (e.g., a CPU, a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a GPU, a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, a processor 1408 and a processor 1409 that may execute instructions 1410. Although Fig.14 Multiple processors 1404 are shown, but the machine 1400 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0132] The memory / storage 1406 may include a memory 1412, such as a main memory or other storage device, and a storage unit 1414, both of which are accessible to the processor 1404, for example, via the bus 1402. The storage unit 1414 and the memory 1412 store instructions 1410 that embody any one or more of the methods or functions described herein. During execution of the instructions 1410 by the machine 1400, the instructions 1410 may also reside, completely or partially, within the memory 1412, within the storage unit 1414, within at least one of the processors 1404 (e.g., within a cache memory of a processor), or within any suitable combination thereof. Thus, the memory 1412, the storage unit 1414, and the memory of the processor 1404 are examples of machine-readable media.

[0133] I / O components 1418 may include a variety of components that receive input, provide output, generate output, send information, exchange information, capture measurements, etc. The specific I / O components 1418 included in a particular machine 1400 will depend on the type of machine. For example, a portable machine such as a mobile phone will likely include a touch input device or other such input mechanism, while a headless server machine will likely not include such a touch input device. It will be appreciated that I / O components 1418 may include Fig.141420. The I / O components 1418 are grouped according to function only to simplify the following discussion, and the grouping is by no means limiting. In various example embodiments, the I / O components 1418 may include output components 1426 and input components 1428. The output components 1426 may include visual components (e.g., displays such as plasma display panels (PDPs), light emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), auditory components (e.g., speakers), tactile components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. The input components 1428 may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touch pads, trackballs, joysticks, motion sensors, or other pointing instruments), touch input components (e.g., physical buttons, touch screen displays or other touch input components that provide the location and / or force of touch or touch gestures), audio input components (e.g., microphones), etc.

[0134] In other example embodiments, the I / O component 1418 may include a biometric component 1430, a motion component 1434, an environmental component 1436, or a positioning component 1438, as well as various other components. For example, the biometric component 1430 may include components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, perspiration, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. Expressions and recognition may include user data, and therefore may only be stored locally on the user device unless otherwise requested by the user. User data may be deleted periodically or at the request of the user. Alternatively, the user data is deleted from the user device immediately after a processing result is generated based on the user data. The motion component 1434 may include: an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. The environment component 1436 may include, for example, an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas sensor that detects concentrations of hazardous gases for safety or measures pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. The positioning component 1438 may include a position sensor component (e.g., a global positioning system (GPS) receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure that can provide altitude), an orientation sensor component (e.g., a magnetometer), etc.

[0135] A variety of technologies may be used to achieve communication. I / O components 1418 may include communication components 1440 that are operable to couple machine 1400 to network 1432 or device 1420 via coupling 1424 and coupling 1422, respectively. For example, communication components 1440 may include a network interface component or other suitable device to interface with network 1432. In other examples, communication components 1440 may include wired communication components, wireless communication components, cellular communication components, near field communication (NFC) components, Bluetooth components (e.g., Bluetooth low energy), Wi-Fi components, and other communication components that provide communication via other modalities. Device 1420 may be another machine or any of a variety of peripheral devices (e.g., a peripheral device coupled via USB).

[0136] In addition, the communication component 1440 can detect the identifier or can include a component that can be operated to detect the identifier. For example, the communication component 1440 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting the following items: a one-dimensional barcode, such as a universal product code (UPC) barcode; a multi-dimensional barcode, such as a quick response (QR) code, an Aztec code, a data matrix, a data glyph, a MaxiCode, a PDF4114, a super code, a UCC RSS-2D barcode, and other optical codes), or an acoustic detection component (e.g., a microphone for identifying an audio signal of a tag). In addition, various information can be obtained via the communication component 1440, such as a location obtained via Internet Protocol (IP) geolocation, a location obtained via Wi-Fi signal triangulation, a location obtained via detecting an NFC beacon signal that can indicate a specific location, etc.

[0137] Glossary

[0138] "Carrier signal" in this context refers to any intangible medium capable of storing, encoding or carrying instructions for execution by a machine, and includes digital or analog communications signals or other intangible media to facilitate communication of such instructions. Instructions may be sent or received over a network using a transmission medium via a network interface device and using any of a number of well-known transmission protocols.

[0139] A "client device" in this context refers to any machine that interfaces with a communication network to obtain resources from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a PDA, a smart phone, a tablet computer, an ultrabook, a netbook, a laptop computer, a multiprocessor system, a microprocessor-based or programmable consumer electronics system, a game console, a set-top box, or any other communication device that a user may use to access a network.

[0140] A "communication network" in this context refers to one or more parts of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, A network, another type of network, or a combination of two or more such networks. For example, the network or a portion of the network may include a wireless or cellular network, and the coupling to the network may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or another type of cellular or wireless coupling. In this example, the coupling may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rate for GSM evolution (EDGE) technology, including the third generation partnership project (3GPP) of 3G, fourth generation wireless (4G) network, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standard, other standards defined by various standard setting organizations, other remote protocols or other data transmission technologies.

[0141] "Machine-readable medium" in this context refers to a component, device, or other tangible medium capable of temporarily or permanently storing instructions and data, and may include, but is not limited to, random access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical media, magnetic media, cache memory, other types of storage devices (e.g., erasable programmable read-only memory (EPROM)), and / or any suitable combination thereof. The term "machine-readable medium" should be deemed to include a single medium or multiple media capable of storing instructions (e.g., centralized or distributed databases or associated caches and servers). The term "machine-readable medium" should also be deemed to include any medium or combination of multiple media capable of storing instructions (e.g., code) for machine execution, such that the instructions, when executed by one or more processors of the machine, cause the machine to perform any one or more of the methods described herein. Thus, "machine-readable medium" refers to a single storage device or device as well as a "cloud-based" storage system or storage network comprising multiple storage devices or devices. The term "machine-readable medium" does not include the signal itself.

[0142] A "component" in this context refers to a logical or device, physical entity having boundaries defined by function or subroutine calls, branch points, APIs, or other techniques that provide partitioning or modularization of specific processing or control functions. Components may interface with other components via their interfaces to perform machine processes. A component may be a packaged functional hardware unit designed for use with other components, and is part of a program that generally performs specific functions of related functions. A component may constitute a software component (e.g., code embodied on a machine-readable medium) or a hardware component.

[0143] A "hardware component" is a tangible unit that can perform certain operations and can be configured or arranged in a physical manner. In various example embodiments, one or more computer systems (e.g., an independent computer system, a client computer system, or a server computer system) or one or more hardware components (e.g., a processor or a processor group) of a computer system can be configured by software (e.g., an application or an application part) to operate to perform certain operations as described herein. Hardware components can also be implemented mechanically, electronically, or any suitable combination thereof. For example, a hardware component may include a dedicated circuit or logic that is permanently configured to perform certain operations. A hardware component may be a dedicated processor, such as a field programmable gate array (FPGA) or an ASIC. A hardware component may also include a programmable logic or circuit system that is temporarily configured to perform certain operations by software. For example, a hardware component may include software executed by a general-purpose processor or other programmable processor.

[0144] Once configured by such software, the hardware component becomes a specific machine (or specific component of a machine) that is uniquely customized to perform the configured function, rather than a general-purpose processor. It will be understood that the decision to implement the hardware component mechanically in a dedicated and permanently configured circuit system or in a temporarily configured (e.g., configured by software) circuit system can be made for cost and time considerations. Therefore, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity, that is, an entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate in a certain manner or perform certain operations described herein.

[0145] Considering implementations where hardware components are temporarily configured (e.g., programmed), it is not necessary to configure or instantiate each of the hardware components at any one time in a timely manner. For example, where the hardware components include a general-purpose processor that is configured by software to be a special-purpose processor, the general-purpose processor can be configured as a respective different special-purpose processor (e.g., including different hardware components) at different times. The software configures one or more specific processors accordingly, for example, to constitute a specific hardware component at one time and to constitute a different hardware component at a different time.

[0146] Hardware components can provide information to other hardware components and receive information from other hardware components. Accordingly, the described hardware components can be considered to be communicatively coupled. In the case of multiple hardware components simultaneously, communication can be achieved by signal transmission between two or more hardware components in the hardware components (e.g., by appropriate circuits and buses). In an embodiment in which multiple hardware components are configured or instantiated at different times, communication between such hardware components can be achieved, for example, by storing information in a memory structure accessed by multiple hardware components and retrieving information in the memory structure. For example, a hardware component can perform an operation, and the output of the operation is stored in a memory device coupled to the communication. Then, other hardware components can access the memory device at a subsequent time to retrieve and process the stored output. The hardware component can also initiate communication with an input or output device, and can operate on resources (e.g., the collection of information).

[0147] The various operations of the example methods described herein may be performed at least in part by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the related operations. Whether temporarily configured or permanently configured, such a processor may constitute a processor-implemented component that operates to perform one or more operations or functions described herein. As used herein, "processor-implemented components" refer to hardware components implemented using one or more processors. Similarly, the methods described herein may be implemented at least in part by a processor, wherein one or more specific processors are examples of hardware. For example, at least some of the operations of the method may be performed by one or more processors or processor-implemented components.

[0148] In addition, one or more processors may also operate to support the execution of related operations in a "cloud computing" environment or operate as "software as a service" (SaaS). For example, at least some operations may be performed by a group of computers (as an example of a machine including a processor), wherein these operations can be accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., application programming interface (API)). The execution of certain operations in the operation may be distributed between processors, not just resident in a single machine but deployed across multiple machines. In some example embodiments, a processor or a component implemented by a processor may be located in a single geographic location (e.g., in a home environment, an office environment, or a server cluster). In other example embodiments, a processor or a component implemented by a processor may be distributed across multiple geographic locations.

[0149] A "processor" in this context refers to any circuit or virtual circuit (a physical circuit simulated by logic executed on an actual processor) that manipulates data values ​​according to control signals (e.g., "commands," "opcodes," "machine codes," etc.) and produces corresponding output signals that are applied to operate a machine. For example, a processor may be a CPU, a RISC processor, a CISC processor, a GPU, a DSP, an ASIC, an RFIC, or any combination thereof. A processor may also be a multi-core processor having two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously.

[0150] A "timestamp" in this context refers to a sequence of characters or encoded information that identifies when some event occurred (e.g., giving a date and time of day), sometimes to within a fraction of a second.

Claims

1. A method comprising: generating experience data for a recorded experience of a user, the experience data describing a first real-world environment of the recorded experience and identifying one or more points of interest associated with the first real-world environment determined based on device data from one or more devices of the user; receiving input indicating a selection of the recorded experience for presentation; Based on the input, access the experience data; as well as Causing presentation, via a display device of the user, of a visual representation of the recorded experience superimposed on a second real-world environment within a field of view of the user, the visual representation of the experience comprising a representation of the first real-world environment and representations of the one or more points of interest displayed together with the representation of the first real-world environment.

2. The method according to claim 1, wherein: The experience data also includes a recorded perspective associated with a point of interest of the one or more points of interest, the recorded perspective including image data generated by a camera, the image data depicting a portion of the first real-world environment visible from the point of interest; as well as The method further comprises: detecting input indicative of interaction of the user with a representation of the point of interest within the presentation of the visual representation of the experience; as well as In response to detecting the input indicative of the user's interaction with the representation of the point of interest, causing the display device to switch to presentation of the recorded perspective associated with the point of interest.

3. The method according to claim 2, wherein: The image data includes one or more images generated by a drone at the point of interest.

4. The method according to claim 2, wherein: The image data includes one or more images generated by the user's mobile device at the point of interest.

5. The method according to claim 2, wherein: The one or more points of interest are determined based in part on location data from a device associated with the experience.

6. The method according to claim 1, wherein: The visual representation of the experience includes a representation of a control panel, the method further comprising: detecting input indicative of an interaction with the control panel, the interaction with the control panel indicative of a desired progress of the experience; and In response to detecting the input indicative of the interaction with the control panel, causing the display device to present the visual representation of the user at a location in the first real-world environment corresponding to the desired progress of the experience.

7. The method according to claim 1, wherein: The experience data also includes weather data describing weather conditions of the experience; and The visual representation of the experience includes a representation of the weather condition.

8. The method of claim 1, wherein: The experience data includes biometric data generated during the experience; and The visual representation of the experience includes a representation of the biometric data generated during the experience.

9. The method according to claim 1, wherein: The experience data includes biometric data generated during the experience; The method further includes determining a region of interest in the representation of the first real-world environment based on the biometric data; as well as The visual representation of the experience includes a visual enhancement of the area of ​​interest in the representation of the first real-world environment.

10. The method of claim 1, wherein: The experience is at a constant altitude; and The representation of the first real-world environment comprises a two-dimensional representation.

11. The method according to claim 1, wherein: The experience includes a change in altitude; and The representation of the first real-world environment comprises a three-dimensional representation.

12. The method of claim 1, wherein: The experience data also includes volumetric content generated based on volumetric capture of the recorded experience; and The visual representation of the experience is generated based on the volumetric content.

13. The method of claim 1, wherein: The generation of the experience data includes aggregating data from one or more devices of the user, The one or more devices include at least one camera device, the data from the one or more devices includes image data from the at least one camera device, and The image data depicts at least a portion of the first real-world environment.

14. A system comprising: one or more processors, and at least a memory storing instructions that, when executed by the one or more processors, configure the system to perform operations comprising: generating experience data for a recorded experience of a user, the experience data describing a first real-world environment of the recorded experience and identifying one or more points of interest associated with the first real-world environment determined based on device data from one or more devices of the user; receiving input indicating a selection of the recorded experience for presentation; Based on the input, accessing the experience data; and Causing presentation, via a display device of the user, of a visual representation of the recorded experience superimposed on a second real-world environment within a field of view of the user, the visual representation of the experience comprising a representation of the first real-world environment and virtual representations of the one or more points of interest displayed together with the representation of the first real-world environment.

15. The system of claim 14, wherein: The experience data also includes a recorded perspective associated with a point of interest of the one or more points of interest, the recorded perspective including image data generated by a camera, the image data depicting a portion of the first real-world environment visible from the point of interest; as well as The operations also include: detecting input indicative of interaction of the user with a representation of the point of interest within the presentation of the visual representation of the experience; as well as In response to detecting the input indicative of the user's interaction with the representation of the point of interest, causing the display device to switch to presentation of the recorded perspective associated with the point of interest.

16. The system of claim 14, wherein: The visual representation of the experience includes a representation of a control panel, the operations further comprising: detecting input indicative of an interaction with the control panel, the interaction with the control panel indicative of a desired progress of the experience; and In response to detecting the input indicative of the interaction with the control panel, causing the display device to present the visual representation of the user at a location in the first real-world environment corresponding to the desired progress of the experience.

17. The system of claim 14, wherein: The experience data also includes weather data describing weather conditions of the experience; and The visual representation of the experience includes a representation of the weather condition.

18. The system of claim 14, wherein: The one or more points of interest are determined based in part on location data from a device associated with the experience.

19. The system of claim 14, wherein: The experience data includes biometric data generated during the experience; and The visual representation of the experience includes a representation of the biometric data generated during the experience.

20. A non-transitory computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform operations comprising: generating experience data for a recorded experience of a user, the experience data describing a first real-world environment of the recorded experience and identifying one or more points of interest associated with the first real-world environment determined based on device data from one or more devices of the user; receiving input indicating a selection of the recorded experience for presentation; Based on the input, access the experience data; as well as Causing presentation, via a display device of the user, of a visual representation of the recorded experience superimposed on a second real-world environment within a field of view of the user, the visual representation of the experience comprising a representation of the first real-world environment and virtual representations of the one or more points of interest displayed together with the representation of the first real-world environment.

Images