Multi-view augmented reality experience

Abstract

The present disclosure relates to methods and systems for providing a multi-view augmented reality experience. A volumetric video of a three-dimensional space is captured. The volume video of the three-dimensional space includes a volume representation of a first user within the three-dimensional space. The volumetric video is displayed by a display device worn by a second user, and the second user sees a volumetric representation of the first user within the three-dimensional space. An input indicative of an interaction (e.g., entry or exit) of a second user with the volumetric representation of the first user is detected. Based on detecting the input indicative of the interaction, the display device switches to display of the recorded view angle to the first user. Thus, the second user views the perspective of the first user in the three-dimensional space by interacting with the volumetric representation of the first user in the volumetric video.

Description

[0001] Priority claim

[0002] This patent application claims the benefit of priority to U.S. application serial number 17 / 900,200, filed on August 31, 2022, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates generally to mobile and wearable computing technologies. In particular, example embodiments of the present disclosure provide systems, methods, and user interfaces for providing a multi-view augmented reality (AR) experience that allows a user to enter and exit a volumetric representation of a human body. Background Art

[0004] An augmented reality (AR) experience includes applying virtual content to a real-world environment, whether by presenting the virtual content through a transparent display through which the real-world environment is visible, or by enhancing image data to include virtual content superimposed on the real-world environment depicted therein. The virtual content may include one or more AR content items. AR content items may include audio content, visual content, or visual effects. A device that supports an AR experience in any of these methods is referred to herein as an "AR device." BRIEF DESCRIPTION OF THE DRAWINGS

[0005] To easily identify the discussion of any particular element or action, the most significant digit(s) in a reference number refers to the figure number in which the element or action is first introduced.

[0006] Figure 1 is a diagrammatic representation of a networked environment in which a volumetric content rendering system may be deployed, according to some example implementations.

[0007] Figure 2A is a perspective view of a head mounted device according to some example embodiments.

[0008] Figure 2B According to some example embodiments, Figure 2A Additional views of the head-mounted device.

[0009] Figure 3 is a diagram showing a method according to some example embodiments including Figure 1 A block diagram of the networking system with details of the head mounted device.

[0010] Figure 4 is a diagrammatic representation of a volumetric content rendering system having both client-side and server-side functionality, according to some examples.

[0011] Figure 5is a diagrammatic representation of a data structure as maintained in a database, according to some example implementations.

[0012] FIG. 6A to FIG. 6D is a conceptual diagram illustrating an example presentation of volumetric video as part of a multi-view augmented reality experience in accordance with some example embodiments.

[0013] Figure 7 is a flow chart illustrating an example method for providing a multi-perspective experience by entering a volumetric AR body, according to some example embodiments.

[0014] Figure 8 is a flow chart illustrating an example method for determining whether a second user enters a volume of a first user, according to some example embodiments.

[0015] Fig. 9 is a block diagram illustrating an example software architecture according to some example implementations.

[0016] Fig.10 is a block diagram illustrating components of a machine according to some example embodiments. DETAILED DESCRIPTION

[0017] The following description includes the system, method, technology, instruction sequence and computer program product of the illustrative embodiment of the present disclosure. In the following description, for the purpose of illustration, many specific details are set forth, so as to provide an understanding of the various embodiments of the subject of the present invention. However, it will be apparent to those skilled in the art that the embodiment of the subject of the present invention can be put into practice without these specific details. Usually, it is not necessary to show in detail known instruction examples, protocols, structures and technologies.

[0018] Volumetric content is an example of an augmented reality (AR) experience. Volumetric content may include volumetric video and images of a three-dimensional space captured in three dimensions (as well as audio signals recorded with the volumetric video and images). Recording of volumetric content includes volumetric capture of elements of the three-dimensional space, such as objects and humans, using a combination of a camera and a sensor. Volumetric content includes a volumetric representation of one or more three-dimensional elements (e.g., objects or humans) of the three-dimensional space. The volumetric representation of an element (e.g., an AR content item) refers to a visual representation of the three-dimensional element in three dimensions. The presentation of volumetric content may include displaying one or more AR content items superimposed on a real-world space, which may be the same or a different space from the three-dimensional space in which the volumetric video was captured. The presentation of volumetric content may include displaying one or more content items in motion, displaying one or more content items performing movement or other actions, displaying one or more content items in a static position, or a combination thereof. The content items may be displayed for the duration of the presentation of the volumetric content or a portion of the volumetric content.

[0019] The presentation of volumetric content may include tracking the user's position and movement within their physical real-world environment, and using the tracked user's position and movement to allow the user to move around and interact with the presentation of volumetric content. Thus, the presentation of volumetric content may include displaying content items from multiple perspectives based on changes in the user's movement and position. In this manner, the presentation of volumetric content provides an immersive AR experience to the user.

[0020] Conventional volumetric content systems use volumetric recorders (cameras or sensors) to capture volumetric content. Although volumetric content can include volumetric representations of one or more three-dimensional elements of a three-dimensional scene (e.g., one or more people), the presentation of volumetric content through conventional volumetric presentation is typically limited to a single perspective. Aspects of the present disclosure include systems, methods, techniques, instruction sequences, and computer programs for providing an AR experience in which multiple perspectives of a three-dimensional scene can be presented to a user of a volumetric content system. During the presentation of volumetric content, when a user interacts with (e.g., enters) a volumetric representation of a person, the presentation of multiple perspectives can be triggered.

[0021] In the above example, volumetric content corresponding to a basketball game is presented. The volumetric content includes volumetric representations (e.g., AR content items) of elements of the basketball game, such as basketballs, players, scoreboards, baskets, etc. The volumetric content may also include the perspective of a player of the basketball game. During the presentation of the volumetric content, a user may enter the volumetric representation of the player to view the player's perspective during the basketball game. The display may include: playing a video of the player's recorded perspective on the user's display device, or superimposing real-life items (e.g., scoreboards, baskets) with the player's recorded perspective using AR or virtual reality (VR) technology.

[0022] The volumetric content including the player's perspective can be stored in the sharing platform. An authorized person (or anyone) can access (e.g., download or stream) the volumetric content from the sharing platform and view the player's perspective by entering the player's volumetric representation. Two users can view the player's perspective together and share their viewing perspectives with each other or with any other person. Referring back to the example mentioned above, the two users can be players of a basketball game. In this case, the present disclosure provides a way for the two players to relieve their memories of the basketball game.

[0023] Some illustrative examples of the foregoing include: determining that the real volume of the second user enters the volume representation of the first user. The determination is made based on multiple conditions, including any one or all of the following three conditions:

[0024] 1. The real volumetric volume of the second user at least partially overlaps with the volumetric representation of the first user;

[0025] 2. The second user's real volumetric body is within a predefined distance of the first user's volumetric representation; and

[0026] 3. The second user's gaze is directed toward the volume representation of the first user.

[0027] It should be noted that these conditions are merely exemplary and should not be limiting. Other similar or different conditions may also be used when making a determination.

[0028] Figure 1 1 is a block diagram illustrating an example volumetric content rendering system 100 for rendering volumetric content. The volumetric content rendering system 100 includes a client device 102. The client device 102 hosts several applications including a rendering client 104. Each rendering client 104 is communicatively coupled to a rendering server system 108 via a network 106 (e.g., the Internet). In an example, the client device 102 is a wearable device (e.g., smart glasses) worn by a user 103, the wearable device including a camera and an optical element including a transparent display through which a real world environment is visible to the user 103.

[0029] The presence client 104 is able to communicate and exchange data with another presence client 104 and with a presence server system 108 via the network 106. The data exchanged between the presence clients 104 and between another presence client 104 and the presence server system 108 includes functions (e.g., commands for activating functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0030] The presentation server system 108 provides server-side functionality to a particular presentation client 104 via the network 106. Although certain functionality of the volumetric content presentation system 100 is described herein as being performed by either the presentation client 104 or by the presentation server system 108, the location of certain functionality within the presentation client 104 or within the presentation server system 108 is a design choice. For example, it may be technically preferable to initially deploy certain technologies and functionality within the presentation server system 108, but later migrate the technologies and functionality to a presentation client 104 where the client device (102) has sufficient processing power.

[0031] The presence server system 108 supports various services and operations provided to the presence clients 104. Such operations include sending data to the presence clients 104, receiving data from the presence clients 104, and processing data generated by the presence clients 104. As examples, the data may include volumetric content (e.g., volumetric video), message content, device information, geolocation information, media annotations and overlays, message content persistence conditions, social network information, and live event information. The exchange of data within the volumetric content presentation system 100 is activated and controlled by functionality available via a user interface (UI) of the presence clients 104.

[0032] Turning now specifically to the presence server system 108, an application program interface (API) server 110 is coupled to and provides a programming interface to an application server 112. The application server 112 is communicatively coupled to a database server 118, which facilitates access to a database 120 in which data associated with messages processed by the application server 112 is stored.

[0033] An application program interface (API) server 110 receives and sends message data (e.g., commands and message payloads) between a client device 102 and an application server 112. Specifically, the API server 110 provides a set of interfaces (e.g., routines and protocols) that can be called or queried by a presence client 104 to activate functionality of the application server 112. The API server 110 exposes various functions supported by the application server 112, including: account registration, login functionality, sending messages from a particular presence client 104 to another presence client 104 via the application server 112, sending media files (e.g., volumetric video) to a presence client 104, setting a collection of media data (e.g., a story), retrieving a friend list of a user of the client device 102, retrieving such a collection, retrieving messages and content, adding and removing friends from a social graph, locating friends within a social graph, and opening application events (e.g., related to a presence client 104).

[0034] The application server 112 hosts several applications and subsystems, including a rendering server 114, an image processing server 116, and a social network server 122. The rendering server 114 is generally responsible for managing volumetric content and facilitating the rendering of volumetric content by the client device 102. The image processing server 116 is dedicated to performing various image processing operations, generally with respect to images or videos generated and displayed by the client device 102. The rendering server 114 and the image processing server 116 can work together to provide one or more AR experiences for the user 103. For example, the rendering server 114 and the image processing server 116 can work together to support the rendering of volumetric content by the client device 102. Further details regarding the rendering of volumetric content are discussed below.

[0035] The social network server 122 supports various social networking functions and services and makes these functions and services available to the presence server 114. To this end, the social network server 122 maintains and accesses an entity graph within the database 120. Examples of functions and services supported by the social network server 122 include identifying other users of the volumetric content presentation system 100 that have a relationship with a particular user or that the particular user is "following", and also includes identifying interests and other entities of a particular user.

[0036] The application server 112 is communicatively coupled to a database server 118 , which facilitates access to a database 120 in which data associated with content rendered by the rendering server 114 and the image processing server 116 is stored.

[0037] Figure 2A is a perspective view of a head mounted display device (eg, glasses 200) according to some examples. Glasses 200 are Figure 1 The glasses 200 are examples of client devices 102. The glasses 200 are capable of displaying content, and are therefore examples of display devices referenced below. In addition, the display capabilities of the glasses 200 support AR experiences, and are therefore examples of AR devices. As described above, AR experiences include applying virtual content to a real-world environment, whether by presenting the virtual content through a transparent display through which the real-world environment is visible, or by augmenting image data to include virtual content superimposed on the real-world environment depicted therein.

[0038] The glasses 200 may include a frame 202 made of any suitable material, such as plastic or metal, including any suitable shape memory alloy. In one or more examples, the frame 202 includes a first optical element holder or left optical element holder 204 (e.g., a display or lens holder) and a second optical element holder or right optical element holder 206 connected by a bridge 212. A first optical element or left optical element 208 and a second optical element or right optical element 210 may be disposed in the respective left optical element holder 204 and right optical element holder 206. The right optical element 210 and the left optical element 208 may be lenses, displays, display components, or a combination of the foregoing. Any suitable display component may be disposed in the glasses 200.

[0039] Frame 202 additionally includes a left arm or temple piece 222 and a right arm or temple piece 224. In some examples, frame 202 can be formed from a single piece of material to have a unitary or unitary construction.

[0040] The glasses 200 may include a computing device such as a computer 220, which may be of any suitable type so as to be carried by the frame 202, and in one or more examples, the computing device may be of a suitable size and shape so as to be partially disposed in one of the temple pieces 222 or temple pieces 224. The computer 220 may include one or more processors with memory, wireless communication circuitry, and a power supply. As discussed below, the computer 220 includes a low-power circuitry, a high-speed circuitry, and a display processor. Various other examples may include these elements in different configurations or these elements integrated together in different ways. As shown in the data processor 302 discussed below, additional details of various aspects of the computer 220 may be implemented.

[0041] The computer 220 additionally includes a battery 218 or other suitable portable power source. In some examples, the battery 218 is disposed in the left temple piece 222 and is electrically coupled to the computer 220 disposed in the right temple piece 224. The glasses 200 may include a connector or port (not shown) suitable for charging the battery 218, a wireless receiver, transmitter or transceiver (not shown), or a combination of such devices.

[0042] The glasses 200 include a first camera or left camera 214 and a second camera or right camera 216. Although two cameras are depicted, other examples contemplate the use of a single or additional (i.e., more than two) cameras. In one or more examples, the glasses 200 include any number of input sensors or other input / output devices in addition to the left camera 214 and the right camera 216. Such sensors or input / output devices may additionally include biometric sensors, positioning sensors, motion sensors, etc.

[0043] In some examples, left camera 214 and right camera 216 provide video frame data for use by glasses 200 to extract 3D information from a real-world scene.

[0044] The glasses 200 may also include a touchpad 226 mounted to or integrated with one or both of the left temple piece 222 and the right temple piece 224. In some examples, the touchpad 226 is generally arranged vertically, approximately parallel to the temple of the user. As used herein, generally vertically aligned means that the touchpad is more vertical than horizontal, although potentially more vertical than the vertical. Additional user input can be provided by one or more buttons 228, which in the example shown are disposed on the outer upper edges of the left optical element holder 204 and the right optical element holder 206. One or more touchpads 226 and buttons 228 provide a device by which the glasses 200 can receive input from the user of the glasses 200.

[0045] Figure 2B The glasses 200 are shown from the user's perspective. For clarity, the Figure 2A Several elements shown in FIG. Figure 2A As described, Figure 2B The illustrated eyeglasses 200 include a left optical element 208 and a right optical element 210 secured in a left optical element holder 204 and a right optical element holder 206, respectively.

[0046] The glasses 200 include a forward optical assembly 230 including a right projector 232 and a right near-eye display 234 , and a forward optical assembly 238 including a left projector 240 and a left near-eye display 244 .

[0047] In some examples, the near eye display is a waveguide. The waveguide includes a reflective structure or a diffractive structure (e.g., a grating and / or an optical element such as a mirror, lens, or prism). Light 236 emitted by projector 232 encounters the diffractive structure of the waveguide of near eye display 234, which directs the light toward the right eye of the user to provide an image superimposed on or in the right optical element 210 with a view of the real world seen by the user. Similarly, light 242 emitted by projector 240 encounters the diffractive structure of the waveguide of near eye display 244, which directs the light toward the left eye of the user to provide an image superimposed on or in the left optical element 208 with a view of the real world seen by the user. The combination of the GPU, front optical assembly 230, left optical element 208, and right optical element 210 provides an optical engine of glasses 200. Glasses 200 use the optical engine to generate an overlay of the user's real world view, including displaying a 3D user interface to the user of glasses 200.

[0048] However, it will be appreciated that other display technologies or configurations may be utilized within the optical engine to display images to the user in the user's field of view. For example, instead of the projector 232 and waveguide, an LCD, LED or other display panel or surface may be provided.

[0049] When in use, the user of the glasses 200 will be presented with information, content, and various 3D user interfaces on the near-eye display. As described in more detail herein, the user can then use the touch pad 226 and / or buttons 228, associated devices (e.g., Figure 3 The user may interact with the glasses 200 through voice input or touch input on a client device 328 shown in FIG. 1 , and / or hand movements, positions, and locations detected by the glasses 200 .

[0050] Figure 3 is a block diagram showing a networked system 300 including details of the glasses 200 according to some examples. The networked system 300 includes the glasses 200, a client device 328, and a server system 332. The client device 328 may be a smartphone, a tablet computer, a tablet phone, a laptop computer, an access point, or any other such device capable of connecting to the glasses 200 using a low power wireless connection 336 and / or a high speed wireless connection 334. The client device 328 is connected to the server system 332 via a network 330. The network 330 may include any combination of wired and wireless connections. The server system 332 may be one or more computing devices that are part of a service or network computing system. The client device 328 and any elements of the server system 332 and the network 330 may use Fig. 9 and Fig.10 The details of the software architecture 906 or machine 1000 described respectively are implemented.

[0051] The glasses 200 include a data processor 302, a display 310, one or more cameras 308, and additional input / output elements 316. The input / output elements 316 may include a microphone, an audio speaker, a biometric sensor, an additional sensor, or an additional display element integrated with the data processor 302. Fig. 9 and Fig.10 Examples of input / output elements 316 are further discussed. For example, input / output elements 316 may include any I / O component 818, including output component 826, motion component 834, etc. Figure 2B Examples of display 310 are discussed in . In the particular examples described herein, display 310 includes displays for the user's left eye and right eye.

[0052] Data processor 302 includes image processor 306 (eg, video processor), GPU and display driver 338, tracking module 340, interface 312, low power circuitry 304, and high speed circuitry 320. Components of data processor 302 are interconnected by bus 342.

[0053] Interface 312 refers to any source of user commands provided to data processor 302. In one or more examples, interface 312 is a physical button that sends a user input signal from interface 312 to low-power processor 314 when pressed. Pressing such a button and then releasing it immediately can be processed by low-power processor 314 as a request to capture a single image, and vice versa. Pressing such a button for a first period of time can be processed by low-power processor 314 as a request to capture video data when the button is pressed and end video capture when the button is released, wherein the video captured when the button is pressed is stored as a single video file. Alternatively, pressing the button for a long period of time can capture a still image. In some examples, interface 312 can be any mechanical switch or physical interface capable of accepting user input associated with requesting data from camera 308. In other examples, interface 312 can have a software component, or can be associated with a command received wirelessly from another source such as from client device 328.

[0054] Image processor 306 includes circuitry for receiving signals from camera 308 and processing those signals from camera 308 into a format suitable for storage in memory 324 or for transmission to client device 328. In one or more examples, image processor 306 (e.g., a video processor) includes a microprocessor integrated circuit (IC) customized for processing sensor data from camera 308, and volatile memory used by the microprocessor in operation.

[0055] The low power circuit system 304 includes a low power processor 314 and a low power wireless circuit system 318. These elements of the low power circuit system 304 can be implemented as separate elements or can be implemented on a single IC as part of a single system on a chip. The low power processor 314 includes logic for managing other elements of the glasses 200. As described above, for example, the low power processor 314 can accept user input signals from the interface 312. The low power processor 314 can also be configured to receive input signals or command communications from a client device 328 via a low power wireless connection 336. The low power wireless circuit system 318 includes circuit elements for implementing a low power wireless communication system. Bluetooth TM Smart, also known as Bluetooth TMLow power consumption is a standard implementation of a low power wireless communication system that may be used to implement the low power wireless circuitry 318. In other examples, other low power communication systems may be used.

[0056] High-speed circuit system 320 includes a high-speed processor 322, a memory 324, and a high-speed wireless circuit system 326. High-speed processor 322 can be any processor capable of managing high-speed communications and operations of any general-purpose computing system for data processor 302. High-speed processor 322 includes processing resources for managing high-speed data transmission on high-speed wireless connection 334 using high-speed wireless circuit system 326. In some examples, high-speed processor 322 executes an operating system such as a LINUX operating system or other such operating system. In addition to any other duties, high-speed processor 322 executing the software architecture of data processor 302 can be used to manage data transmission with high-speed wireless circuit system 326. In some examples, high-speed wireless circuit system 326 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, which is also referred to as Wi-Fi herein. In other examples, other high-speed communication standards can be implemented by high-speed wireless circuit system 326.

[0057] Memory 324 includes any storage device capable of storing camera data generated by camera 308 and image processor 306. Although memory 324 is shown as being integrated with high-speed circuitry 320, in other examples, memory 324 may be a separate, stand-alone element of data processor 302. In some such examples, electrical wiring may provide a connection from image processor 306 or low-power processor 314 to memory 324 through a chip including high-speed processor 322. In other examples, high-speed processor 322 may manage addressing of memory 324 so that low-power processor 314 will initiate high-speed processor 322 any time a read or write operation involving memory 324 is needed.

[0058] The tracking module 340 estimates the pose of the glasses 200. For example, the tracking module 340 uses image data and corresponding inertial data from the camera 308 and the position component 838 and GPS data to track the location and determine the pose of the glasses 200 relative to a reference frame (e.g., a real-world environment). The tracking module 340 continuously collects and uses updated sensor data describing the movement of the glasses 200 to determine an updated three-dimensional pose of the glasses 200, which indicates changes in relative position and orientation relative to physical objects in the real-world environment. The tracking module 340 allows for visual placement of virtual objects relative to physical objects within the user's field of view through the glasses 200 via the display 310.

[0059] The GPU and display driver 338 may use the pose of the glasses 200 to generate frames of virtual content or other content to be presented on the display 310 when the glasses 200 are operating in a conventional augmented reality mode. In this mode, the GPU and display driver 338 generate updated frames of virtual content based on the updated three-dimensional pose of the glasses 200, which reflects changes in the position and orientation of the user relative to physical objects in the user's real-world environment.

[0060] One or more functions or operations described herein may also be performed in an application resident on the glasses 200 or on the client device 328 or on a remote server. The glasses 200 may be a stand-alone client device capable of independent operation, or may be a companion device that works with a host device to offload intensive processing and / or exchange data with the presentation server system 108 over the network 106. The glasses 200 may also be communicatively coupled to a companion device such as a smart watch and may be configured to exchange data with the companion device. The glasses 200 may also include various components common to mobile electronic devices such as smart glasses or smart phones (e.g., including a display controller for controlling the display of visual media on a display mechanism incorporated in the device).

[0061] Figure 4 1 is a block diagram showing additional details about a volumetric content rendering system 100 according to some examples. Specifically, the volumetric content rendering system 100 is shown to include a rendering client 104 and an application server 110. The volumetric content rendering system 100 includes several subsystems, which are supported on the client side by the rendering client 104 and on the server side by the application server 110. These subsystems include, for example, a collection management system 402, a rendering control system 404, an enhancement system 406, and a multi-view experience generation system 408.

[0062] The collection management system 402 is responsible for managing sets or collections of content (e.g., collections of text, images, video, and audio data). Collections of content may be organized into "event galleries" or "event stories." Such collections may be made available for a specified period of time, such as the duration of an event to which the content is related. For example, content related to a concert may be available as a "story" for the duration of the concert. The collection management system 402 may also be responsible for publishing an icon to the user interface of the presentation client 104 that provides notification of the existence of a particular collection.

[0063] In addition, the collection management system 402 also includes a curation interface 410, which allows collection managers to manage and curate specific content collections. For example, the curation interface 410 enables event organizers to curate content collections related to specific events (e.g., remove inappropriate content or redundant messages). In addition, the collection management system 402 uses machine vision (or image recognition technology) and content rules to automatically curate content collections.

[0064] The presentation control system 404 is responsible for facilitating and controlling the volumetric content presentation. Thus, the presentation control system 404 provides a mechanism that allows a user to specify control operations for controlling the volumetric content presentation. For example, the control operations may include: a stop operation for stopping the presentation; a pause operation for pausing the presentation; a fast forward operation for advancing the presentation at a higher speed; a rewind operation for rewinding the presentation; a zoom in operation for increasing the zoom level of the presentation; a zoom out operation for decreasing the zoom level of the presentation; and a playback speed modification operation for changing the speed of the presentation (e.g., for producing a slow motion presentation of volumetric video).

[0065] For some embodiments, a user may specify an input indicative of a control operation for controlling presentation of volumetric content by providing one or more inputs via one or more I / O components (see below). Fig.10 Examples of one or more I / O components are described in further detail). For some embodiments, the presentation control system 404 may provide an interactive control interface including one or more interactive elements (e.g., virtual buttons) to trigger a control operation, and the presentation control system 404 monitors interactions with the interactive interface to detect input indicating a control operation. For some embodiments, a user may trigger a control operation using a gesture such as a hand gesture or a head gesture that may be associated with a particular control operation.

[0066] The enhancement system 406 provides various functions that enable a user to enhance (e.g., annotate or otherwise modify or edit) media content. For example, the enhancement system 406 provides functions related to the generation, publication, and application of enhancement data, such as media overlays (e.g., image filters) to volumetric content. Media overlays may include audio content and visual content and visual effects. Examples of audio content and visual content include pictures, text, logos, animations, and sound effects. Examples of visual effects include color overlays. Audio content and visual content or visual effects may be applied to media content items (e.g., photos) at the client device 102. For example, a media overlay may include text or images that may be superimposed on a photo taken by the client device 102. The enhancement system 406 may be operable to provide one or more media overlays to the presentation client 104 based on the geographic location of the client device 102 or based on other information such as social network information of a user of the client device 102. The media overlays may be stored in the database 120 and accessed through the database server 118.

[0067] A filter is an example of a media overlay that is displayed as an overlay on an image or video during presentation to a user. Filters can be of various types, including filters selected by a user from a set of filters presented to a user by presentation client 104. Other types of filters include geolocation filters (also referred to as geofilters), which can be presented to a user based on a geographic location. For example, presentation client 104 can present a geolocation filter specific to a nearby or particular location within a user interface based on geographic location information determined by a global positioning system (GPS) unit of client device 102.

[0068] Another type of filter is a data filter, which can be selectively presented to the user by the presentation client 104 based on other input or information collected by the client device 102. Examples of data filters include the current temperature at a particular location, the current speed the sending user is traveling, the battery life of the client device 102, or the current time.

[0069] AR content items are another example of a media overlay. AR content items may be real-time special effects and / or sounds that can be added to images or videos, including volumetric images and videos.

[0070] In general, AR content items, overlays, image transformations, images, and the like refer to modifications that can be applied to image data (e.g., video or images) that include volumetric content. This includes real-time modifications that modify an image as it is captured using a device sensor (e.g., one or more cameras) of the client device 102, and then the image with the modifications is displayed by a display device (e.g., an embedded display of the client device) of the client device 102. This also includes modifications to stored content such as volumetric videos in a gallery or collection that can be modified. For example, in a client device 102 that has access to multiple AR content items, a user can use a single volumetric video with multiple AR content items to see how different AR content items will modify the stored content. For example, by selecting different AR content items for the content, multiple augmented reality content items that apply different pseudo-random movement models can be applied to the same content. Similarly, real-time video capture can be used with the modifications shown to show how the video image currently captured by the sensor of the client device 102 will modify the captured data. Such data may be displayed only on the screen and not stored in memory, or content captured by the device sensors may be recorded and stored in memory with or without modification (or both). In some systems, a preview feature may show how different augmented reality content items will look in different windows in the display simultaneously. For example, this may enable multiple windows with different pseudo-random animations to be viewed on the display simultaneously.

[0071] Thus, data and various systems using augmented reality content items or other such augmented systems using augmented data to modify content may involve: detection of objects (e.g., faces, hands, bodies, cats, dogs, surfaces, objects, etc.) in video frames, tracking such objects as they leave the field of view, enter the field of view, and move around in the field of view, and modifying or transforming such objects as they are tracked. In various examples, different methods for implementing such transformations may be used. Some examples may involve generating a three-dimensional mesh model of one or more objects, and implementing the transformation using transformations and animated textures of the models within the video. In other examples, tracking of points on an object may be used to place an image or texture (which may be two-dimensional or three-dimensional) at a tracked location. In yet another example, neural network analysis of video frames may be used to place images, models, or textures in content (e.g., frames of an image or video). Thus, AR content items refer to both images, models, and textures used to create transformations in content, and additional modeling and analysis information required to implement such transformations using object detection, tracking, and placement.

[0072] Real-time video processing can be performed with any kind of video data (e.g., video streams, video files, volumetric video, etc.) stored in the memory of any kind of computerized system. For example, a user can load video files and store them in the memory of the device, or a sensor of the device can be used to generate a video stream. Additionally, any object can be processed using computer animation models, such as human faces and parts of human bodies, animals, or non-living objects such as chairs, cars, or other objects.

[0073] In some examples, when a specific modification is selected together with the content to be enhanced (e.g., edited), the elements to be transformed are identified by a computing device, and then if the elements to be transformed are present in a frame of a video, the elements to be transformed are detected and tracked. The elements of the object are modified according to the request for modification, so the frames of the video stream are transformed. For different kinds of transformations, the transformation of the frames of the video stream can be performed by different methods. For example, for the transformation of the frame that mainly involves changing the form of the elements of the object, the characteristic points of each element of the object are calculated (e.g., using an active shape model (ASM) or other known methods). Then, for each of at least one element of the object, a grid based on characteristic points is generated. The grid is used in the subsequent stage of tracking the elements of the object in the video stream. In the process of tracking, the grid mentioned for each element is aligned with the positioning of each element. Then, additional points are generated on the grid. A first group of first points is generated for each element based on the request for modification, and a group of second points is generated for each element based on the request for the group of first points and the modification. Then, the frame of the video stream can be transformed by modifying the elements of the object based on the group of first points and the group of second points and the grid. In such methods, the background of the modified object may also be changed or distorted by tracking and modifying the background.

[0074] In some examples, a transformation that changes some areas of an object using elements of an object can be performed by calculating characteristic points of each element of the object and generating a grid based on the calculated characteristic points. Points are generated on the grid, and then various areas are generated based on these points. Then, the elements of the object are tracked by aligning the area of ​​each element with the position of each element of at least one element, and the properties of the area can be modified based on the modification request, thereby transforming the frame of the video stream. Depending on the specific modification request, the properties of the mentioned area can be transformed in different ways. Such modifications may involve: changing the color of the area; removing at least part of the area from the frame of the video stream; including one or more new objects in the area based on the modification request; and modifying or distorting the elements of the area or object. In various examples, any combination of such modifications or other similar modifications may be used. For certain models to be animated, some feature points can be selected as control points to determine the entire state space of options for model animation.

[0075] In some examples of computer animation models that use face detection to transform image data, faces are detected on an image using a specific face detection algorithm (e.g., Viola-Jones). An active shape model (ASM) algorithm is then applied to the face region of the image to detect facial feature reference points.

[0076] Other methods and algorithms suitable for face detection can be used. For example, in some examples, landmarks are used to locate features, which represent distinguishable points that are present in most of the images considered. For example, for facial landmarks, the location of the left eye pupil can be used. If the initial landmarks are not identifiable (for example, if the person is wearing an eye patch), secondary landmarks can be used. Such a landmark recognition process can be used for any such object. In some examples, a set of landmarks forms a shape. The shape can be represented as a vector using the coordinates of the points in the shape. A similarity transformation (which allows translation, scaling, and rotation) that minimizes the average Euclidean distance between the shape points is used to align one shape with another. The mean shape is the average of the aligned training shapes.

[0077] In some examples, the landmark search begins with an average shape that aligns with the position and size of the face determined by a global face detector. Such a search then repeats the steps of suggesting tentative shapes by adjusting the positions of shape points by template matching of the image texture around each point, and then conforming the tentative shapes to a global shape model until convergence occurs. In some systems, individual template matches are unreliable, and the shape model pools the results of weak template matches to form a stronger overall classifier. The entire search is repeated at each level in the image pyramid, from coarse to fine resolution.

[0078] The augmentation system 406 can capture an image or video stream on a client device (e.g., client device 102) and perform complex image manipulations locally on the client device 102 while maintaining a suitable user experience, computational time, and power consumption. Complex image manipulations can include size and shape changes, emotional transitions (e.g., changing a face from a frown to a smile), state transitions (e.g., aging a subject, reducing apparent age, changing gender), style transitions, application of graphical elements, and any other suitable image or video manipulations enabled by a convolutional neural network that has been configured to execute efficiently on the client device 102.

[0079] In some examples, a computer animation model for transforming video and image content can be used by the enhancement system 406, where the neural network operates as part of the rendering client 104 operating on the client device 102. The enhancement system 406 determines the presence of a face within an image or video stream and provides an interactive modification element (e.g., an icon) associated with the computer animation model for transforming image data, or the computer animation model can exist in association with the interface described herein. The interactive modification element includes the following changes, which can be the basis for modifying the user's face within the image or video content as part of the modification operation. Once the modification icon is selected, the transformation system initiates the process of transforming the user's image to reflect the selected modification icon (e.g., generating a smiley face on the user). Once the image or video stream is captured and the specified modification is selected, the modified image or video content can be presented in a graphical user interface displayed on the client device 102. The enhancement system 406 can implement a complex convolutional neural network on a portion of the image or video stream to generate and apply the selected modification. That is, the modified content can be presented to the user in real time or near real time. In addition, the modification can be persistent when the content is presented. Machine-taught neural networks can be used to implement such modifications.

[0080] The multi-perspective experience generation system 408 is responsible for providing a multi-perspective experience to the user. The multi-perspective experience generation system 408 can cause the user's display device to present a volumetric video. The presentation of the volumetric video can include the presentation of a volumetric representation of one or more people in a three-dimensional space. The perspective of each of the one or more people can be pre-recorded by the multi-perspective experience generation system 408 or any other system of the volumetric content presentation system 100. When the multi-perspective experience generation system 408 detects that the user interacts with the volumetric representation of the one or more people in the three-dimensional space (e.g., enters, leaves), the multi-perspective experience generation system 408 can cause the user's display device to switch to the presentation of the perspective of the three-dimensional space corresponding to the person with whom the user interacted. When the user enters / leaves a different volumetric representation of one or more people, the multi-perspective experience generation system 408 causes the display device to switch between the presentation of the volumetric video and the presentation of the different perspectives, thereby providing the user with the ability to experience multiple perspectives of different people in the three-dimensional space.

[0081] Figure 5 is a diagrammatic representation of a data structure 500 as maintained in database 120, according to some examples. While the contents of database 120 are shown as including several tables, it will be appreciated that data may be stored in other types of data structures (eg, as an object-oriented database).

[0082] The entity table 504 stores entity data and is linked (e.g., by reference) to the entity graph 506 and profile data 508. Entities whose records are maintained within the entity table 504 may include individuals, corporate entities, organizations, objects, places, events, etc. Regardless of the entity type, any entity about which the presence server system 108 stores data may be an identified entity. Each entity is provided with a unique identifier as well as an entity type identifier (not shown). The entity table 504 may associate various enhancements from the enhancement table 510 with various images and videos stored in the image table 516 and the video table 514.

[0083] Entity graph 506 stores information about relationships and associations between entities. Such relationships may be social, professional (e.g., working in a common company or organization), interest-based, or activity-based, by way of example only.

[0084] The profile data 508 stores various types of profile data about a particular entity. Based on the privacy settings specified by the particular entity, the profile data 508 can be selectively used and presented to other users of the volumetric content presentation system 100. In the case where the entity is a person, the profile data 508 includes, for example, a user name, a phone number, an address, settings (e.g., notification and privacy settings), and an avatar representation (or a collection of such avatar representations) selected by the user.

[0085] The database 120 also stores augmentation data, such as overlays including AR content items and filters, in augmentation table 510. The augmentation data is associated with and applied to videos (whose data is stored in video table 514) and images (whose data is stored in image table 516), including volumetric videos and images.

[0086] The story table 512 stores data about a collection of content including associated images, video or audio data that is compiled into a collection (e.g., a story or gallery). The creation of a particular collection can be initiated by a particular user (e.g., each user whose record is saved in the entity table 504). A user can create a "personal story" in the form of a collection of content that has been created and sent / broadcasted by the user. To this end, the user interface of the presentation client 104 may include an icon that is user-selectable to enable the sending user to add specific content to his or her personal story.

[0087] As mentioned above, the video table 514 stores video data including volumetric videos. Similarly, the image table 516 stores image data including volumetric images.

[0088] Reference Fig. 6A, an example volumetric capture scene 602 from a volumetric video is shown. The volumetric capture scene 602 includes a volumetric representation of a real-world environment. In the example scene 602, two players 606 and 608 play basketball together on a basketball court 610. The basketball court includes a basketball hoop 612. At least a portion of the volumetric video is based on a video captured by a volumetric recorder 614, which includes the basketball court and the players 606 and 608. In some examples, either or both of the players 606 and 608 can wear a wearable device (e.g., glasses 200). A camera of the wearable device (e.g., camera 216) can record the player's perspective occasionally or continuously before, during, or after a basketball game.

[0089] In some examples, a scoreboard 616 can record the real-time score of two players 606 and 608. For example, the score is one to two. Music 618 can be played in scene 602. Conversations between players 606 and 608 or conducted by players 606 and / or 608 (e.g., in the form of volumetric signals) can also be recorded and presented as part of the presentation of the volumetric video. The volumetric recorder 614 and / or the wearable devices of the two players 606 and 608 can record the music 618 playing in the background and the conversations between or conducted by the two players.

[0090] Reference Figure 6B , an example environment 604 is shown in which a user 620 is located. User 620 can be the same as or different from player 606 or player 608. In some examples, environment 604 can be at the same environment depicted in example scene 602. In this case, user 620 is accessing a game that is related to Fig. 6A Alternatively, environment 604 may be located at a different location than example scene 602. For example, user 620 may visit a different basketball court. In some examples, environment 604 may be a virtual scene presented by an augmented reality (AR) or virtual reality (VR) device of user 620, and the user may be at any location, not limited to a basketball court.

[0091] Assume that user 620 is accessing Fig. 6A If the example environment 604 includes the same basketball court as the example scene 602 in FIG. 6 , the example environment 604 may include the same basketball hoop 612. In some examples, the example environment 604 also includes a scoreboard 622; however, the score on the scoreboard 622 may be different from the score on the scoreboard 616. For example, the score is zero to zero.

[0092] like Figure 6CAs shown, user 620 may wear wearable device 624. Wearable device 624 may be augmented reality (AR) glasses or virtual reality (VR) glasses. For example, wearable device 624 may be glasses 200. A presentation 636 of volumetric video may be provided to user 620 by wearable device 624. In some examples, presentation 636 may correspond to Fig. 6A 6. A volumetric video of a basketball court captured by volume recorder 614 in example scene 602 in FIG. 636 may include representations of two players 626 and 628 corresponding to players 606 and 608, a representation of basketball 630 corresponding to basketball 610, and a representation of scoreboard 632 corresponding to scoreboard 616. In some examples, the representations of the two players 626 and 628 may be volumetric. In some examples, as part of the presentation of the volumetric video, scoreboard 632 may be superimposed on scoreboard 622 so that scoreboard 632 appears to have a score similar to that of scoreboard 616. For example, "zero to zero" in scoreboard 622 is superimposed with "one to two" in scoreboard 616. For some presentations of the volumetric video, music 634 may be played. Music 634 may be the same as music 618. If music is playing in environment 604 (not shown), the music will be replaced by music 634 / music 618. For example, the wearable device 624 may send a control signal to a speaker (not shown) to play the music 634 or switch to the music 634. The speaker may be mounted on the wearable device 624 or on the basketball court.

[0093] Fig.6D The first user's recorded perspective of the three-dimensional space when the second user enters the first user's volume is presented. Figure 6C As discussed, presentation 636 may include a volumetric representation of player 626. When user 620 "enters" the representation of player 626, wearable device 624 may switch from presentation 636 to presentation 638. Presentation 638 may correspond to Fig. 6A 606 in scene 602. Presentation 638 may include basketball hoop 612 from the perspective of player 606, which is a front view of basketball hoop 612. Presentation 638 may also include player 628 corresponding to player 608. In some examples, presentation 638 may include the same scoreboard 632, and the same music 634 may be played in or with presentation 638. Detailed description of how to determine whether user 620 enters the representation of player 626 may be provided elsewhere in this disclosure, such as in Figure 8 and the corresponding description.

[0094] In some examples, user 620 may be the same as player 608. By entering the volume representation of player 606, user 620 (player 608) may have the opportunity to view himself or herself from the perspective of the other player. In some examples, player 606 and player 608 may come to a basketball court together and view each perspective or both perspectives together. Player 606 and player 608 may also view the perspective of volume recorder 614.

[0095] It should be noted that FIG. 6A to FIG. 6D The method and system of the present application can be used in other scenarios in the same or similar manner, and FIG. 6A to FIG. 6D The examples in are non-limiting.

[0096] Figure 7 is a flow chart illustrating an example method 700 for providing a multi-perspective experience by entering a volumetric AR human body according to some examples. The method 700 may be embodied in computer-readable instructions for execution by one or more processors, such that the operations of the method 700 may be performed in part or in whole by functional components of the volumetric content rendering system 100; therefore, the method 700 is described below by way of example with reference thereto. However, it should be understood that at least some of the operations of the method 700 may be deployed on various other hardware configurations besides the volumetric content rendering system 100.

[0097] In operation 702, the volumetric content presentation system 100 accesses volumetric content. The volumetric content includes a volumetric video that includes a volumetric representation of a first user within a three-dimensional space. The volumetric video may be based on video captured by a wearable device of the first user (e.g., glasses 200), one or more volume recorders (e.g., volume recorder 614), or various combinations thereof. The captured volumetric video may be stored in a storage device or database (e.g., database 120) and later accessed from the storage device or database. The volumetric video may be stored in a storage device or database (e.g., database 120). Fig. 6A An example volumetric video can be found in . The 3D space can be a football field, basketball court, baseball field, music stadium, or any other indoor or outdoor scene. Fig. 6A An example three-dimensional space is found in . Fig. 6A An example first user is found as the first player 606 .

[0098] In operation 704, the volumetric content presentation system 100 causes a volumetric video to be presented by a display device of a second user. Presenting the volumetric video may include presenting a volumetric representation of the first user within a three-dimensional space. The display device may be a wearable device (e.g., glasses 200, wearable device 624) or a portion thereof. The second user may be the same as or different from the first user. Figure 6C An example second user is found as user 620 .

[0099] In determining operation 706, the volumetric content presentation system 100 determines whether input indicating an interaction of the second user with the volumetric representation of the first user within the presentation of the volumetric video is detected. If it is determined that input indicating an interaction of the second user with the volumetric representation of the first user within the presentation of the volumetric video is detected, the method 700 may proceed to operation 708; otherwise, the method 700 may return to operation 704. In some examples, the input indicating an interaction of the second user with the volumetric representation of the first user may include detecting that the second user enters the volumetric representation of the first user. Detailed description of determining whether the second user enters the volumetric representation of the first user may be provided elsewhere in this disclosure, such as in Figure 8 In some examples, the second user's interaction with the volumetric representation may include other types of interactions, including but not limited to, the second person touching, hugging, or talking to the first user's volumetric representation or a portion thereof.

[0100] In an example, the three-dimensional space corresponds to a real-world location, and the display device is worn by a second user at the same real-world location. The rendering of the volumetric video includes rendering a volumetric representation of the first user superimposed on the real-world location at a location in the real-world location corresponding to the location of the first user in the real-world location during the capture of the volumetric video. In this example, rendering the volumetric video includes rendering the volumetric representation of the first user superimposed on one or more elements of the real-world location.

[0101] In another example, the three-dimensional space corresponds to a first real-world location, and the display device is worn by a second user at a second (different) real-world location. In this example, presenting the volumetric video includes presenting a volumetric representation of the first user at a location in the three-dimensional space corresponding to the location of the first user in the real-world location during capture of the volumetric video. Presenting the volumetric video includes presenting one or more AR content items superimposed on one or more elements of the real-world location.

[0102] In operation 708, the volumetric content presentation system 100 accesses a recorded perspective of a first user of the three-dimensional space. The perspective of the first user of the three-dimensional space may be recorded by a capture device (e.g., the camera 216 of the glasses 200). The perspective of the first user refers to the field of view of the first user of the three-dimensional space. The recorded perspective of the first user may include an audio signal including audio provided by the first user during the capture of the volumetric video. The audio signal may also include a conversation between the first user and another user (e.g., the second user or the third user).

[0103] In operation 710, the volumetric content presentation system 100 causes the display device to switch to the presentation of the recorded perspective of the first user in the three-dimensional space. Presenting the recorded perspective may include: playing a video of the recorded perspective of the first user to the display device of the second user. Alternatively, or in addition, the display device may use AR or VR technology to superimpose the real world environment with the recorded perspective of the first user. In some examples, the volumetric video may also include a volumetric representation of the second user, and the recorded perspective of the first user includes the volumetric representation of the second user. The first user may wear a similar display device, and the method 700 may be similarly performed by the display device of the first user. In this way, the first user and the second user can freely view their own or each other's recorded perspectives by interacting with their own volumetric representations. Alternatively, or in addition, a third user may view the recorded perspectives of the first user and / or the second user by interacting with the volumetric representations of the first user and / or the second user.

[0104] Figure 8 8 is a flow chart illustrating an example method 800 for determining whether a second user has entered a volumetric volume of a first user according to some examples. The method 800 may be embodied in computer-readable instructions for execution by one or more processors, such that the operations of the method 800 may be performed in part or in whole by functional components of the volumetric content presentation system 100; therefore, the method 800 is described below by way of example with reference thereto. However, it should be understood that at least some of the operations of the method 800 may be deployed on various other hardware configurations besides the volumetric content presentation system 100.

[0105] For some implementations, method 800 may be performed in conjunction with Figure 7 8. For example, the result of method 800 may be a determination that the second user has entered the volume (or volume representation) of the first user. Figure 7 As described in operation 706 of process 700 in , such a determination corresponds to detection of an input indicative of an interaction of the second user with the volumetric representation of the first user within the presentation of the volumetric video.

[0106] In operation 802, the volumetric content presentation system 100 determines a spatial relationship between a real volumetric body of a second user and a volumetric representation of a first user. In some examples, an initial spatial relationship between the real volumetric body of the second user and the volumetric representation of the first user may be determined based on an initial spatial pose (e.g., position, volume, and orientation) of the real volumetric body of the second user and the volumetric representation of the first user. Operation 802 may also include continuously or periodically updating the initial spatial relationship based on the initial spatial pose of the volumetric representation of the first user and the updated spatial pose of the real volumetric body of the second user. Fig.6DAn exemplary real volumetric volume of a second user as user 620 is found in . Fig.6D An exemplary volume representation of a first user who is a representation of player 626 is found in .

[0107] In determination operation 804, the volumetric content presentation system 100 determines whether the real volumetric volume of the second user at least partially overlaps with the volumetric representation of the first user. If it is determined that the real volumetric volume of the second user at least partially overlaps with the volumetric representation of the first user, the method 800 can proceed to determination operation 806; otherwise, the method 800 can return to operation 802. In some examples, the volume of the real volumetric volume of the second user and the volumetric representation of the first user can be obtained. Determination operation 804 may include: determining whether there is an overlap between the two volumes. In some examples, "partial overlap" requires that any overlap is more than no overlap. Alternatively, a minimum overlap may be required. The minimum overlap may be 0.1%, 0.2%, 0.5%, 1%, 2%, 5%, etc. Determination operation 804 may only determine that when the overlap of the real volumetric volume of the second user with the volumetric representation of the first user is greater than or equal to the minimum overlap, the real volumetric volume of the second user at least partially overlaps with the volumetric representation of the first user.

[0108] In determination operation 806, the volumetric content presentation system 100 determines whether the real volumetric body of the second user is within a predefined distance of the volumetric representation of the first user. If it is determined that the real volumetric body of the second user is within a predefined distance of the volumetric representation of the first user, the method 800 may proceed to determination operation 808; otherwise, the method 800 may return to operation 802. In some examples, the distance from the middle point (or center of gravity) of the real volumetric body of the second user to the middle point (or center of gravity) of the volumetric representation of the first user may be measured. For example, if the hand of the second user touches the volumetric representation of the first user, while most of the body of the second user is away from the volumetric representation of the first user (e.g., the distance between the middle point of the first user and the second user is greater than the predefined distance), the determination may be "no" and the method 800 may return to operation 802. The spatial relationship is updated and the method 800 is performed again. Alternatively, the distance from a part of the real volumetric body of the second user (e.g., head, neck, chest, abdomen, arms, legs, feet) to the same part of the volumetric representation of the first user may be measured. The predefined distance may be 1 millimeter (mm), 2 mm, 5 mm, 1 centimeter (cm), 2 cm, 5 cm, 1 decimeter (dm), 2 dm, 5 dm, etc.

[0109] In determination operation 808, the volumetric content presentation system 100 determines whether the second user's line of sight is directed toward the volumetric representation of the first user. If it is determined that the second user's line of sight is directed toward the volumetric representation of the first user, the method 800 may proceed to operation 810; otherwise, the method 800 may return to operation 802. In some examples, the determination operation 808 may include determining whether the second user's line of sight passes through the volumetric representation of the first user. The line of sight of the second user may be determined based on the orientation of the eyes or head of the second user. In some examples, the determination operation 808 may include measuring the angle or distance between the second user's line of sight and a point or portion of the volumetric representation of the first user. For example, if the second user's line of sight is directed toward the outstretched arm of the first user rather than the torso of the second user, the determination may be "no" and the method 800 may return to operation 802. The spatial relationship is updated and the method 800 is performed again.

[0110] In operation 810, the volumetric content presentation system 100 determines that the second user enters the volumetric volume of the first user. It should be noted that the determination operations 804, 806, and 808 may be performed in any order. In addition, any of the determination operations 804, 806, and 808 may be omitted. In some examples, the method 800 may be modified so that when any of the determination operations 804, 806, and 808 is "yes", it is determined that the second user enters the volumetric volume of the first user. In some examples, when all of the determination operations 804, 806, and 808 in the method 800 are "yes" or any of the determination operations 804, 806, and 808 in the modified method 800 is "yes", the spatial relationship may be referred to as a target spatial relationship. As mentioned above, the determination that the second user enters the volumetric volume of the first user in operation 810 may be related to Figure 7 706 of process 700 corresponds to an example of the second user interacting with the volume representation of the first user.

[0111] Software Architecture

[0112] Fig. 9 is a block diagram illustrating an example software architecture 906 that may be used in conjunction with the various hardware architectures described herein. Fig. 9 is a non-limiting example of a software architecture, and it should be understood that many other architectures may be implemented to facilitate the functionality described herein. Software architecture 906 may be implemented in a system such as Fig.10 1000, which includes a processor 1004, a memory / storage device 1006, and an I / O component 1018. A representative hardware layer 952 is shown and can represent, for example Fig.10The machine 1000 of the embodiment of the present invention. The representative hardware layer 952 includes a processing unit 954 having associated executable instructions 904. The executable instructions 904 represent executable instructions of the software architecture 906, including implementations of the methods, components, etc. described herein. The hardware layer 952 also includes a memory and / or storage module 956 also having executable instructions 904. The hardware layer 952 may also include other hardware 958.

[0113] exist Fig. 9 In the example architecture of , software architecture 906 can be conceptualized as a stack of layers in which each layer provides specific functionality. For example, software architecture 906 may include layers such as operating system 902, library 920, framework / middleware 918, application 916, and presentation layer 914. In operation, application 916 and / or other components within a layer may activate API call 908 through the software stack and receive a response to API call 908 as message 912. The layers shown are representative in nature, and not all software architectures have all layers. For example, some mobile or dedicated operating systems may not provide framework / middleware 918, while other operating systems may provide such a layer. Other software architectures may include additional or different layers.

[0114] The operating system 902 can manage hardware resources and provide common services. For example, the operating system 902 may include a kernel 922, a service 924, and a driver 926. The kernel 922 may be used as an abstraction layer between the hardware layer and other software layers. For example, the kernel 922 may be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, etc. The service 924 may provide other public services for other software layers. The driver 926 is responsible for controlling or interfacing with the underlying hardware. For example, depending on the hardware configuration, the driver 926 includes a display driver, a camera driver, drivers, flash memory drivers, serial communication drivers (e.g., Universal Serial Bus (USB) drivers), Drivers, audio drivers, power management drivers, etc.

[0115] The library 920 provides a common infrastructure used by the application 916 and / or other components and / or layers. The library 920 provides functions that allow other software components to perform tasks in an easier manner than directly docking with the underlying operating system 902 functions (e.g., core 922, service 924, and / or driver 926). The library 920 may include a system library 944 (e.g., a C standard library), which may provide functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. In addition, the library 920 may include an API library 946, such as a media library (e.g., a library for supporting the presentation and manipulation of various media formats such as MPEG4, H.294, MP3, AAC, AMR, JPG, and PNG), a graphics library (e.g., an OpenGL framework that can be used to render 2D and 3D graphics content on a display), a database library (e.g., SQLite that can provide various relational database functions), a web library (e.g., WebKit that can provide web browsing functions), etc. The library 920 may also include various other libraries 948 to provide many other APIs to the application 916 and other software components / modules.

[0116] The framework / middleware 918 provides a higher level common infrastructure that can be used by the applications 916 and / or other software components / modules. For example, the framework / middleware 918 can provide various GUI functions, advanced resource management, advanced location services, etc. The framework / middleware 918 can provide a wide range of other APIs that can be utilized by the applications 916 and / or other software components / modules, some of which can be specific to a particular operating system 902 or platform.

[0117] Applications 916 include built-in applications 938 and / or third-party applications 940. Examples of representative built-in applications 938 may include, but are not limited to, contact applications, browser applications, book reader applications, location applications, media applications, messaging applications, and / or game applications. Third-party applications 940 may include applications that are developed by entities other than the vendor of a particular platform using ANDROID TM or IOS TM Software Development Kit (SDK) for developing applications and can be used on platforms such as IOS TM ANDROID TM , Mobile software running on the mobile operating system of the Phone or other mobile operating systems. The third-party application 940 can activate API calls 908 provided by the mobile operating system (eg, operating system 902) to facilitate the functions described herein.

[0118] Applications 916 may create user interfaces to interact with users of the system using built-in operating system functionality (e.g., kernels 922, services 924, and / or drivers 926), libraries 920, and framework / middleware 918. Alternatively, or in addition, in some systems, interaction with the user may occur through a presentation layer such as presentation layer 914. In these systems, application / component "logic" may be separated from aspects of the application / component that interact with the user.

[0119] Fig.10 1 is a block diagram illustrating components of a machine 1000 capable of reading instructions from a machine-readable medium (e.g., a machine-readable storage medium) and performing any one or more of the methods discussed herein, according to some example embodiments. Specifically, Fig.10 A graphical representation of a machine 1000 in the example form of a computer system is shown, within which instructions 1010 (e.g., software, programs, applications, applet, app, or other executable code) for causing the machine 1000 to perform any one or more of the methods discussed herein may be executed. Thus, the instructions 1010 may be used to implement the modules or components described herein. The instructions 1010 transform a general unprogrammed machine 1000 into a specific machine 1000 that is programmed to perform the functions described and shown in the described manner. In alternative embodiments, the machine 1000 operates as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine 1000 may operate with the performance of a server machine or a client machine in a server-client network environment, or operate as a peer machine in a peer-to-peer (or distributed) network environment. The machine 1000 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a PDA, an entertainment media system, a cellular phone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web device, a network router, a network switch, a network bridge, or any machine capable of executing instructions 1010, either sequentially or otherwise, that specify actions to be taken by the machine 1000. Furthermore, while only a single machine 1000 is shown, the term “machine” should also be taken to include a collection of machines that individually or jointly execute instructions 1010 to perform any one or more of the methodologies discussed herein.

[0120] The machine 1000 may include a processor 1004, a memory / storage device 1006, and an I / O component 1018, which may be configured to communicate with each other, for example, via a bus 1002. In an example embodiment, the processor 1004 (e.g., a CPU, a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a GPU, a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, a processor 1008 and a processor 1009 that may execute instructions 1010. Although Fig.10 Multiple processors 1004 are shown, but the machine 1000 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0121] The memory / storage 1006 may include a memory 1012, such as a main memory or other storage device, and a storage unit 1014, both of which are accessible to the processor 1004, for example, via the bus 1002. The storage unit 1014 and the memory 1012 store instructions 1010 that embody any one or more of the methods or functions described herein. During execution of the instructions 1010 by the machine 1000, the instructions 1010 may also reside, completely or partially, within the memory 1012, within the storage unit 1014, within at least one of the processors 1004 (e.g., within a cache memory of a processor), or within any suitable combination thereof. Thus, the memory 1012, the storage unit 1014, and the memory of the processor 1004 are examples of machine-readable media.

[0122] The I / O components 1018 may include various components for receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurements, etc. The specific I / O components 1018 included in a particular machine 1000 will depend on the type of machine. For example, a portable machine such as a mobile phone will likely include a touch input device or other such input mechanism, while a headless server machine will likely not include such a touch input device. It should be understood that the I / O components 1018 may include Fig.101028. The I / O components 1018 are grouped according to function only to simplify the following discussion, and the grouping is by no means limiting. In various example embodiments, the I / O components 1018 may include output components 1026 and input components 1028. The output components 1026 may include visual components (e.g., displays such as plasma display panels (PDPs), light emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), acoustic components (e.g., speakers), tactile components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. The input components 1028 may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touch pads, trackballs, joysticks, motion sensors, or other pointing instruments), touch input components (e.g., physical buttons, touch screen displays or other touch input components that provide location and / or force of touch or touch gestures), audio input components (e.g., microphones), etc.

[0123] In other example embodiments, the I / O component 1018 may include a biometric component 1030, a motion component 1034, an environmental component 1036, or a positioning component 1038, as well as various other components. For example, the biometric component 1030 may include a component for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying a person (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component 1034 may include an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. The environment component 1036 may include, for example, an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an auditory sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas sensor that detects concentrations of hazardous gases for safety reasons or measures pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. The positioning component 1038 may include a position sensor component (e.g., a global positioning system (GPS) receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure, from which the altitude can be obtained), an orientation sensor component (e.g., a magnetometer), etc.

[0124] Various technologies may be used to implement communications. I / O components 1018 may include communications components 1040 operable to couple machine 1000 to network 1032 or device 1020 via coupling 1024 and coupling 1022, respectively. For example, communications components 1040 may include a network interface component or other suitable device to interface with network 1032. In other examples, communications components 1040 may include wired communications components, wireless communications components, cellular communications components, near field communications (NFC) components, Components (e.g. Low energy consumption), Device 1020 may be another machine or any of a variety of peripheral devices (eg, a peripheral device coupled via USB).

[0125] In addition, the communication component 1040 can detect the identifier or can include a component that is operable to detect the identifier. For example, the communication component 1040 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional bar codes such as universal product codes (UPC) bar codes, multi-dimensional bar codes such as quick response (QR) codes, Aztec codes, data matrix, data symbols (Dataglyph), MaxiCode, PDF4114, Ultra Code, UCC RSS-2D bar codes and other optical codes) or an acoustic detection component (e.g., a microphone for identifying tagged audio signals). In addition, various information can be obtained via the communication component 1040, such as location via Internet Protocol (IP) geolocation, location via Signal triangulation to obtain location, location obtained via detection of NFC beacon signals that can indicate a specific location, etc.

[0126] Glossary

[0127] "Carrier signal" in this context refers to any intangible medium capable of storing, encoding or carrying instructions for execution by a machine, and includes digital or analog communications signals or other intangible media to facilitate communication of such instructions. Instructions may be sent or received over a network using a transmission medium via a network interface device and using any of a number of well-known transmission protocols.

[0128] A "client device" in this context refers to any machine that interfaces with a communication network to obtain resources from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a PDA, a smart phone, a tablet computer, an ultrabook, a netbook, a laptop computer, a multiprocessor system, a microprocessor-based or programmable consumer electronics system, a game console, a set-top box, or any other communication device that a user may use to access a network.

[0129] A "communication network" in this context refers to one or more parts of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, A network, another type of network, or a combination of two or more such networks. For example, the network or a portion of the network may include a wireless or cellular network, and the coupling to the network may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or another type of cellular or wireless coupling. In this example, the coupling may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rate for GSM evolution (EDGE) technology, including the third generation partnership project (3GPP) of 3G, fourth generation wireless (4G) network, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standard, other standards defined by various standard setting organizations, other remote protocols or other data transmission technologies.

[0130] "Machine-readable medium" in this context refers to a component, device, or other tangible medium capable of temporarily or permanently storing instructions and data, and may include, but is not limited to, random access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical media, magnetic media, cache memory, other types of storage devices (e.g., erasable programmable read-only memory (EPROM)), and / or any suitable combination thereof. The term "machine-readable medium" should be deemed to include a single medium or multiple media capable of storing instructions (e.g., centralized or distributed databases or associated caches and servers). The term "machine-readable medium" should also be deemed to include any medium or combination of multiple media capable of storing instructions (e.g., code) for machine execution, such that the instructions, when executed by one or more processors of the machine, cause the machine to perform any one or more of the methods described herein. Thus, "machine-readable medium" refers to a single storage device or device as well as a "cloud-based" storage system or storage network comprising multiple storage devices or devices. The term "machine-readable medium" does not include the signal itself.

[0131] A "component" in this context refers to a logical or device, physical entity having boundaries defined by function or subroutine calls, branch points, APIs, or other techniques that provide partitioning or modularization of a particular processing or control function. A component may be combined with other components via its interfaces to perform a machine process. A component may be a packaged functional hardware unit designed for use with other components and as part of a program that typically performs a specific function of related functions. A component may constitute a software component (e.g., code implemented on a machine-readable medium) or a hardware component.

[0132] A "hardware component" is a tangible unit that can perform certain operations and can be configured or arranged in a physical manner. In various example embodiments, one or more computer systems (e.g., an independent computer system, a client computer system, or a server computer system) or one or more hardware components (e.g., a processor or a processor group) of a computer system can be configured by software (e.g., an application or an application part) to be a hardware component that operates to perform certain operations as described herein. The hardware components can also be implemented mechanically, electronically, or in any suitable combination thereof. For example, a hardware component may include a dedicated circuit system or logic that is permanently configured to perform certain operations. The hardware component may be a dedicated processor, such as a field programmable gate array (FPGA) or an ASIC. The hardware component may also include a programmable logic or circuit system that is temporarily configured to perform certain operations by software. For example, a hardware component may include software executed by a general-purpose processor or other programmable processor.

[0133] Once configured by such software, the hardware component becomes a specific machine (or specific component of a machine) that is uniquely customized to perform the configured functions and is no longer a general-purpose processor. It should be understood that it may be decided for cost and time considerations whether to mechanically implement the hardware component in a dedicated and permanently configured circuit system or in a temporarily configured (e.g., configured by software) circuit system. Therefore, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity, that is, an entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate in a certain manner or perform certain operations described herein.

[0134] Considering implementations where hardware components are temporarily configured (e.g., programmed), it is not necessary to configure or instantiate each of the hardware components at any one time in a timely manner. For example, where the hardware components include a general-purpose processor that is configured by software to be a special-purpose processor, the general-purpose processor can be configured as a different special-purpose processor (e.g., including different hardware components) at different times. For example, software configures one or more specific processors accordingly to constitute a specific hardware component at one time and to constitute different hardware components at different times.

[0135] Hardware components can provide information to other hardware components and can receive information from other hardware components. Therefore, the described hardware components can be considered to be communicatively coupled. In the case of multiple hardware components simultaneously, communication can be achieved by signal transmission between or among two or more hardware components in the hardware components (e.g., by appropriate circuits and buses). In an embodiment in which multiple hardware components are configured or instantiated at different times, communication between such hardware components can be achieved, for example, by storing information in a memory structure accessed by multiple hardware components and retrieving information in the memory structure. For example, a hardware component can perform an operation and store the output of the operation in a memory device to which it is communicatively coupled. Then, other hardware components can access the memory device at a subsequent time to retrieve the stored output and process it. Hardware components can also initiate communication with input or output devices, and can operate resources (e.g., the collection of information).

[0136] The various operations of the example methods described herein may be performed at least in part by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the related operations. Whether it is a temporary configuration or a permanent configuration, such a processor may constitute a processor-implemented component that operates to perform one or more operations or functions described herein. As used herein, "processor-implemented components" refer to hardware components implemented using one or more processors. Similarly, the methods described herein may be implemented at least in part by a processor, wherein a specific one or more processors are examples of hardware. For example, at least some operations in the operation of the method may be performed by one or more processors or processor-implemented components.

[0137] In addition, one or more processors may also operate to support the execution of related operations in a "cloud computing" environment or as a "software as a service" (SaaS) operation. For example, at least some operations may be performed by a group of computers (as an example of a machine including a processor), where these operations can be accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., application programming interface (API)). The execution of certain operations may be distributed among the processors, not just resident in a single machine, but deployed across multiple machines. In some example embodiments, the processor or the component implemented by the processor may be located in a single geographic location (e.g., in a home environment, an office environment, or a server cluster). In other example embodiments, the processor or the component implemented by the processor may be distributed across several geographic locations.

[0138] A "processor" in this context refers to any circuit or virtual circuit (a physical circuit simulated by logic executed on an actual processor) that manipulates data values ​​according to control signals (e.g., "commands," "opcodes," "machine codes," etc.) and produces corresponding output signals that are applied to operate a machine. For example, a processor may be a CPU, a RISC processor, a CISC processor, a GPU, a DSP, an ASIC, an RFIC, or any combination thereof. A processor may also be a multi-core processor having two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously.

[0139] A "timestamp" in this context refers to a sequence of characters or coded information that identifies when some event occurred (eg, thereby giving a date and time of day), sometimes to within a fraction of a second.

Claims

1. A method comprising: accessing volumetric content including a volumetric video including a volumetric representation of a first user within a three-dimensional space; causing presentation of the volumetric video by a display device of a second user, such that presenting the volumetric video includes causing presentation of a volumetric representation of the first user within the three-dimensional space; detecting input indicative of an interaction of the second user with a volumetric representation of the first user within a presentation of the volumetric video; as well as In response to detecting an input indicative of an interaction by the second user with the volume representation of the first user, a recorded perspective of the first user accessing the three-dimensional space; as well as The display device is caused to switch to presenting the recorded perspective of the first user in the three-dimensional space.

2. The method according to claim 1, wherein: Detecting input indicative of interaction of the second user with the volume representation includes detecting entry of the second user into the volume representation of the first user.

3. The method according to claim 2, wherein: Detecting that the second user enters the volume representation of the first user includes: A target spatial relationship between the volumetric representation of the first user and a real volumetric body of the second user is detected.

4. The method according to claim 3, wherein: Detecting the target spatial relationship includes at least one of the following: determining that a real volumetric volume of the second user at least partially overlaps a volumetric representation of the first user; determining that a real volumetric volume of the second user is within a predefined distance of the volumetric representation of the first user; or It is determined that the line of sight of the second user is directed toward the volumetric representation of the first user.

5. The method according to claim 1, wherein: The recorded perspective of the first user also includes an audio signal including audio provided by the first user during capturing of the volumetric video, wherein causing presentation of the recorded perspective of the first user includes causing presentation of the audio signal.

6. The method according to claim 1, wherein: The three-dimensional space corresponds to a real-world location; The display device is worn by the second user at the real-world location; as well as Rendering the volumetric video includes rendering a volumetric representation of the first user superimposed on the real-world location at a location in the real-world location corresponding to a location of the first user in the real-world location during capturing of the volumetric video.

7. The method according to claim 1, wherein: The three-dimensional space corresponds to a first real-world location; the display device being worn by the second user at a second real-world location; as well as Rendering the volumetric video includes rendering a volumetric representation of the first user at a location in the three-dimensional space corresponding to a location of the first user in the real-world location during capture of the volumetric video.

8. The method according to claim 7, wherein: Causing presentation of the volumetric video by the display device includes causing presentation of one or more AR content items superimposed on one or more elements of the real-world location.

9. The method according to claim 1, wherein: The three-dimensional space corresponds to a real-world location; The display device is worn by the second user at the real-world location; as well as Causing presentation of the volumetric video by the display device includes causing presentation of a volumetric representation of the first user superimposed on one or more elements of the real-world location.

10. The method of claim 1, wherein: The volumetric video also includes a volumetric representation of the second user; and The recorded perspective of the first user includes a volumetric representation of the second user.

11. The method according to claim 1, wherein: The three-dimensional space includes the second user, and the method further includes: accessing the volumetric content including the volumetric video, the volumetric video including a volumetric representation of a second user within the three-dimensional space; causing a volumetric representation of the second user within the three-dimensional space to be presented by a display device of the first user; detecting input indicative of an interaction of the first user with a volumetric representation of the second user within a presentation of the volumetric video; and In response to detecting an input indicative of an interaction represented by the first user and the volume of the second user, a recorded perspective of the second user accessing the three-dimensional space; and The display device of the first user is switched to presenting the recorded perspective of the second user in the three-dimensional space.

12. A computing device comprising: processor; as well as a memory storing instructions that, when executed by the processor, configure the apparatus to: accessing volumetric content including a volumetric video including a volumetric representation of a first user within a three-dimensional space; causing presentation of the volumetric video by a display device of a second user, such that presenting the volumetric video includes causing presentation of a volumetric representation of the first user within the three-dimensional space; detecting input indicative of an interaction of the second user with a volumetric representation of the first user within a presentation of the volumetric video; as well as In response to detecting an input indicative of an interaction by the second user with the volume representation of the first user, a recorded perspective of the first user accessing the three-dimensional space; as well as The display device is caused to switch to presenting the recorded perspective of the first user in the three-dimensional space.

13. The computing device of claim 12, wherein: To detect input indicative of an interaction of the second user with the volume representation, the instructions further configure the apparatus to: A target spatial relationship between the volumetric representation of the first user and a real volumetric body of the second user is detected.

14. The computing device of claim 13, wherein: To detect the target spatial relationship, the instructions further configure the apparatus to: determining that a real volumetric volume of the second user at least partially overlaps a volumetric representation of the first user; determining that a real volumetric volume of the second user is within a predefined distance of the volumetric representation of the first user; or It is determined that the line of sight of the second user is directed toward the volumetric representation of the first user.

15. The computing device of claim 12, wherein: The three-dimensional space corresponds to a real-world location; The display device is worn by the second user at the real-world location; as well as Rendering the volumetric video includes rendering a volumetric representation of the first user superimposed on the real-world location at a location in the real-world location corresponding to a location of the first user in the real-world location during capturing of the volumetric video.

16. The computing device of claim 12, wherein: The three-dimensional space corresponds to a first real-world location; the display device being worn by the second user at a second real-world location; as well as Rendering the volumetric video includes rendering a volumetric representation of the first user at a location in the three-dimensional space corresponding to a location of the first user in the real-world location during capture of the volumetric video.

17. The computing device of claim 16, wherein: Causing presentation of the volumetric video by the display device includes causing presentation of one or more AR content items superimposed on one or more elements of the real-world location.

18. The computing device of claim 12, wherein: The three-dimensional space corresponds to a real-world location; The display device is worn by the second user at the real-world location; as well as Causing presentation of the volumetric video by the display device includes causing presentation of a volumetric representation of the first user superimposed on one or more elements of the real-world location.

19. The computing device of claim 12, wherein: The volumetric video also includes a volumetric representation of the second user; and The recorded perspective of the first user includes a volumetric representation of the second user.

20. A non-transitory computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to: accessing volumetric content including a volumetric video including a volumetric representation of a first user within a three-dimensional space; causing presentation of the volumetric video by a display device of a second user, such that presenting the volumetric video includes causing presentation of a volumetric representation of the first user within the three-dimensional space; detecting input indicative of an interaction of the second user with a volumetric representation of the first user within a presentation of the volumetric video; as well as In response to detecting an input indicative of an interaction by the second user with the volume representation of the first user, a recorded perspective of the first user accessing the three-dimensional space; as well as The display device is caused to switch to presenting the recorded perspective of the first user in the three-dimensional space.

Images