Controlling and editing presentation of volumetric content

Abstract

A display device presents volumetric content including a volumetric video. The volumetric video includes a volumetric representation of one or more elements of a three-dimensional space. Input indicating a control operation associated with presentation of the volumetric video is received. Presentation of volumetric video by a display device is controlled by performing a control operation. While a control operation is being performed, a volumetric representation of one or more elements of a three-dimensional space is displayed from a plurality of perspectives based on a user's movement.

Description

[0001] Priority claim

[0002] This patent application claims the benefit of priority to U.S. application serial number 17 / 900,407, filed on August 31, 2022, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates generally to mobile and wearable computing technologies. In particular, example implementations of the present disclosure relate to systems, methods, and user interfaces for controlling and editing the presentation of volumetric content to support augmented reality experiences. Background Art

[0004] Augmented reality (AR) experience includes: whether by presenting virtual content by a transparent display (through which the real world environment can be seen), or by enhancing image data to include virtual content superimposed on the real world environment depicted therein, to apply virtual content to the real world environment. Virtual content may include one or more AR content items. AR content items may include audio content, visual content, or visual effects. Devices that support AR experience in any of these methods are referred to herein as "AR devices". As an example, some electronically enabled eyewear devices (e.g., so-called smart glasses) facilitate AR experience because users are allowed to interact with AR content items while the user is participating in some activities. The user wears the eyewear device and can view the real world environment through the eyewear device while interacting with the AR content displayed by the eyewear device. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] To easily identify the discussion of any particular element or action, the most significant digit(s) in a reference number refers to the figure number in which the element or action is first introduced.

[0006] Figure 1 is a diagrammatic representation of a networked environment in which a volumetric content rendering system may be deployed, according to some example implementations.

[0007] Figure 2A is a perspective view of a head mounted device according to some example embodiments.

[0008] Figure 2B According to some example embodiments, Figure 2A Additional views of the head-mounted device.

[0009] Figure 3 is a block diagram illustrating aspects of a wearable device according to some example embodiments.

[0010] Figure 4is a diagrammatic representation of a rendering system having both client-side and server-side functionality, according to some examples.

[0011] Figure 5 is a diagrammatic representation of data structures as maintained in a database according to some examples.

[0012] Figure 6 and Figure 7 is a flow chart illustrating the operation of a volumetric content presentation system in executing a method for controlling presentation of volumetric content according to an example implementation.

[0013] Figure 8 is a flow chart illustrating the operation of a volumetric content presentation system in executing a method for editing presentation of volumetric content according to an example implementation.

[0014] Fig. 9 is a conceptual diagram illustrating an example interactive control interface for controlling presentation of volumetric content according to an example implementation.

[0015] Fig.10 is a block diagram illustrating a representative software architecture that may be used in conjunction with the various hardware architectures described herein, according to an example implementation.

[0016] Fig.11 is a block diagram illustrating components of a machine capable of reading instructions from a machine-readable medium (eg, a machine-readable storage medium) and performing any one or more of the methodologies discussed herein, according to example embodiments. DETAILED DESCRIPTION

[0017] The following description includes the system, method, technology, instruction sequence and computing machine program product of the illustrative embodiment of the present disclosure. In the following description, for the purpose of illustration, many specific details are set forth, so as to provide an understanding of the various embodiments of the subject of the present invention. However, it will be apparent to those skilled in the art that the embodiment of the subject of the present invention can be put into practice without these specific details. Usually, it is not necessary to show in detail known instruction examples, protocols, structures and technologies.

[0018] Volumetric content is an example of an augmented reality (AR) experience. Volumetric content may include volumetric video and images of a three-dimensional space captured in three dimensions (as well as audio signals recorded with the volumetric video and images). The recording of volumetric content includes: using a combination of a camera and a sensor to volumetrically capture (volumetrically capturing) one or more elements of the three-dimensional space such as objects and humans. The resulting volumetric content includes a volumetric representation of one or more three-dimensional elements (e.g., objects or people) of the three-dimensional space. The volumetric representation of an element, which is usually in the form of an AR content item, refers to a visual representation of the three-dimensional element in three dimensions. The presentation of volumetric content may include displaying one or more AR content items superimposed on a real-world space, which may be the same as or a different space from the three-dimensional space in which the volumetric video is captured. The presentation of volumetric content may include playback of a volumetric video, which may include displaying one or more content items in motion, displaying one or more content items performing a move or other action, displaying one or more content items in a static position, or a combination thereof. The content item may be displayed for the duration of the presentation of the volumetric content or a portion of the volumetric content.

[0019] The presentation of volumetric content may include tracking the user's position and movement within their physical real-world environment, and using the tracked user's position and movement to allow the user to move around and interact with the presentation of volumetric content. Thus, the presentation of volumetric content may include displaying content items from multiple perspectives based on changes in the user's movement and position. In this manner, the presentation of volumetric content provides an immersive AR experience to the user.

[0020] Conventional volumetric content systems do not include mechanisms that allow a user to easily control and edit the presentation of volumetric content. Aspects of the present disclosure include systems, methods, techniques, instruction sequences, and computing machine program products for controlling and editing the presentation of volumetric content. The volumetric content presentation system provides an interface that allows a user to specify control operations regarding the presentation of volumetric content, and the system controls the presentation accordingly. Example control operations include: a stop operation for stopping the presentation; a pause operation for pausing the presentation; a fast forward operation for advancing the presentation at a higher speed; a rewind operation for rewinding the presentation; a zoom in operation for increasing the zoom level of the presentation; a zoom out operation for decreasing the zoom level of the presentation; and a playback speed modification operation for changing the speed of the presentation (e.g., for producing a slow motion presentation of volumetric video).

[0021] In the above example, volumetric content corresponding to a basketball game is presented. The volumetric content includes volumetric representations (e.g., AR content items) of elements of a basketball game such as a basketball, a player, a scoreboard, a hoop, etc. During the presentation of the volumetric content, the control system of the volumetric content presentation system allows the user to pause the presentation so that the volumetric representation of the player freezes in mid-air, running, dribbling, etc., and while paused, the user can observe and interact with the three-dimensional scene. The user can provide input indicating a play operation to cancel the pause presentation and continue to watch the volumetric representation of the player's game. The user can also rewind the presentation of the volumetric representation of the basketball game, thereby producing a presentation of the volumetric representation of the player moving backwards. In addition, the user can zoom in or out on certain elements of the presentation. The user can also fast forward the presentation to speed up the movement of the volumetric representation, or the user can put the presentation in slow motion to slow down the movement of the player to capture key moments like their expressions when the player is about to shoot, all as part of the augmented reality experience. By experiencing the presentation of content in AR, the user becomes part of the scene instead of just watching the scene.

[0022] In addition, the volumetric content rendering system provides an edit mode that allows the user to edit the volumetric content and create modified volumetric content therefrom. For example, the enhancement system of the volumetric content rendering system provides such an edit mode that allows the user to specify edits to the volumetric content, such as: modifications to content items (e.g., changes to size, shape, color, shading, etc.); removal of content items; addition of content items (e.g., selected from a library or other volumetric video); and adding filters to content items or real-world objects visible to the user via the rendering of the volumetric content.

[0023] Some illustrative examples of the foregoing include: applying an echo "filter" to volumetric content to make sounds more memorable, applying music effects to make content more dramatic, fading certain audio signals (e.g., background noise) while amplifying others (e.g., laughter or screaming) to draw focus to specific moments, highlighting sound effects such as a heartbeat to convey intensity, adding narration to provide greater context to content, applying a glow effect on legs while running, enhancing stars from star gazing content, adding a glitter effect to the ball during a soccer game, applying a sparkle effect to a ring during a marriage proposal, or adding visual indications to a person's gaze to show where their attention is focused. Allowing users to enhance volumetric content in this manner can make the volumetric content more engaging and otherwise enhance the overall AR experience.

[0024] Figure 11 is a block diagram illustrating an example volumetric content rendering system 100 for rendering volumetric content. The volumetric content rendering system 100 includes a client device 102. The client device 102 hosts several applications including a rendering client application 104. Each rendering client application 104 is communicatively coupled to a rendering server system 108 via a network 106 (e.g., the Internet). In an example, the client device 102 is a wearable device (e.g., smart glasses) worn by a user 103, the wearable device including a camera and optical elements including a transparent display through which a real-world environment is visible to the user 103.

[0025] The presence client application 104 is capable of communicating and exchanging data with another presence client application 104 and a presence server system 108 via the network 106. The data exchanged between the presence client applications 104 and between another presence client application 104 and the presence server system 108 includes functions (e.g., commands for activating functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0026] The presentation server system 108 provides server-side functionality to a particular presentation client application 104 via the network 106. Although certain functionality of the volumetric content presentation system 100 is described herein as being performed by either the presentation client application 104 or by the presentation server system 108, the location of certain functionality within the presentation client application 104 or within the presentation server system 108 is a design choice. For example, it may be technically preferable to initially deploy certain technologies and functionality within the presentation server system 108, but later migrate the technologies and functionality to the presentation client application 104 where the client device (102) has sufficient processing power.

[0027] The presence server system 108 supports various services and operations provided to the presence client applications 104. Such operations include sending data to the presence client applications 104, receiving data from the presence client applications 104, and processing data generated by the presence client applications 104. As examples, the data may include volumetric content (e.g., volumetric video), message content, device information, geo-location information, media annotations and overlays, message content persistence conditions, social network information, and live event information. The exchange of data within the volumetric content presentation system 100 is activated and controlled by functionality available via a user interface (UI) of the presence client applications 104.

[0028] Turning now specifically to the presence server system 108, an application program interface (API) server 110 is coupled to and provides a programming interface to an application server 112. The application server 112 is communicatively coupled to a database server 118, which facilitates access to a database 120 in which data associated with messages processed by the application server 112 is stored.

[0029] An application program interface (API) server 110 receives and sends message data (e.g., commands and message payloads) between a client device 102 and an application server 112. Specifically, the API server 110 provides a collection of interfaces (e.g., routines and protocols) that can be called or queried by a presence client application 104 in order to activate functionality of the application server 112. The API server 110 exposes various functionality supported by the application server 112, including: account registration, login functionality, sending messages from a particular presence client application 104 to another presence client application 104 via the application server 112, sending media files (e.g., volumetric video) to a presence client application 104, setting a collection of media data (e.g., a story), retrieving a friend list of a user of the client device 102, retrieving such a collection, retrieving messages and content, adding and removing friends from a social graph, locating friends within a social graph, and opening application events (e.g., related to a presence client application 104).

[0030] The application server 112 hosts several applications and subsystems, including a rendering server 114, an image processing server 116, and a social network server 122. The rendering server 114 is generally responsible for managing volumetric content and facilitating the rendering of volumetric content by the client device 102. The image processing server 116 is dedicated to performing various image processing operations, generally with respect to images or videos generated and displayed by the client device 102. The rendering server 114 and the image processing server 116 can work together to provide one or more AR experiences for the user 103. For example, the rendering server 114 and the image processing server 116 can work together to support the rendering of volumetric content by the client device 102. Further details regarding the rendering of volumetric content are discussed below.

[0031] The social network server 122 supports various social networking functions and services and makes these functions and services available to the presence server 114. To this end, the social network server 122 maintains and accesses an entity graph within the database 120. Examples of functions and services supported by the social network server 122 include identifying other users of the volumetric content presentation system 100 that have a relationship with a particular user or that the particular user is "following", and also includes identifying interests and other entities of a particular user.

[0032] The application server 112 is communicatively coupled to a database server 118 , which facilitates access to a database 120 in which data associated with content processed by the rendering server 114 and the image processing server 116 is stored.

[0033] Figure 2A is a perspective view of a head mounted display device (eg, glasses 200) according to some examples. Glasses 200 are Figure 1 200 is an example of a client device 102. Glasses 200 are capable of displaying content and are therefore examples of display devices referenced below. In addition, the display capabilities of glasses 200 support AR experiences and are therefore examples of AR devices. As described above, AR experiences include applying virtual content to a real-world environment, whether by presenting the virtual content through a transparent display through which the real-world environment can be seen, or by augmenting image data to include virtual content superimposed on the real-world environment depicted therein.

[0034] The glasses 200 may include a frame 202 made of any suitable material, such as plastic or metal, including any suitable shape memory alloy. In one or more examples, the frame 202 includes a first optical element holder or a left optical element holder 204 (e.g., a display or lens holder) and a second optical element holder or a right optical element holder 206 connected by a bridge 212. A first optical element or a left optical element 208 and a second optical element or a right optical element 210 may be disposed in the respective left optical element holder 204 and the right optical element holder 206. The right optical element 210 and the left optical element 208 may be lenses, displays, display components, or a combination of the foregoing. Any suitable display component may be disposed in the glasses 200.

[0035] Frame 202 additionally includes a left arm or temple piece 222 and a right arm or temple piece 224. In some examples, frame 202 can be formed from a single piece of material to have a unitary or integrated construction.

[0036] The glasses 200 may include a computing device such as a computer 220, which may be of any suitable type so as to be carried by the frame 202, and in one or more examples, the computing device may be of a suitable size and shape to be partially disposed in one of the temple pieces 222 or temple pieces 224. The computer 220 may include one or more processors and memory, wireless communication circuitry, and a power supply. As discussed below, the computer 220 includes a low-power circuitry, a high-speed circuitry, and a display processor. Various other examples may include these elements in different configurations or integrated together in different ways. As shown in the data processor 302 discussed below, additional details of various aspects of the computer 220 may be implemented.

[0037] The computer 220 additionally includes a battery 218 or other suitable portable power supply. In some examples, the battery 218 is disposed in the left temple piece 222 and is electrically coupled to the computer 220 disposed in the right temple piece 224. The glasses 200 may include a connector or port (not shown) suitable for charging the battery 218, a wireless receiver, transmitter or transceiver (not shown), or a combination of such devices.

[0038] The glasses 200 include a first camera or left camera 214 and a second camera or right camera 216. Although two cameras are depicted, other examples contemplate the use of a single or additional (i.e., more than two) cameras. In one or more examples, the glasses 200 include any number of input sensors or other input / output devices in addition to the left camera 214 and the right camera 216. Such sensors or input / output devices may additionally include biometric sensors, position sensors, motion sensors, etc.

[0039] In some examples, left camera 214 and right camera 216 provide video frame data for use by glasses 200 to extract 3D information from a real-world scene.

[0040] The glasses 200 may also include a touchpad 226 mounted to or integrated with one or both of the left temple piece 222 and the right temple piece 224. In some examples, the touchpad 226 is generally arranged vertically, approximately parallel to the temple of the user. As used herein, generally vertical alignment means that the touchpad is more vertical than horizontal, although potentially more vertical than this. Additional user input can be provided by one or more buttons 228, which, in the example shown, are disposed on the outer upper edges of the left optical element holder 204 and the right optical element holder 206. One or more touchpads 226 and buttons 228 provide a path through which the glasses 200 can receive input from the user of the glasses 200.

[0041] Figure 2B The glasses 200 are shown from the user's perspective. Figure 2A Several elements shown in FIG. Figure 2A As described, Figure 2B The illustrated eyeglasses 200 include a left optical element 208 and a right optical element 210 secured in a left optical element holder 204 and a right optical element holder 206, respectively.

[0042] The glasses 200 include a forward optical assembly 230 including a right projector 232 and a right near-eye display 234 , and a forward optical assembly 238 including a left projector 240 and a left near-eye display 244 .

[0043] In some examples, the near-eye display is a waveguide. The waveguide includes a reflective structure or a diffractive structure (e.g., a grating and / or an optical element such as a mirror, lens, or prism). Light 236 emitted by projector 232 encounters the diffractive structure of the waveguide of near-eye display 234, which directs the light toward the right eye of the user to provide an image superimposed on or in the right optical element 210 with a view of the real world seen by the user. Similarly, light 242 emitted by projector 240 encounters the diffractive structure of the waveguide of near-eye display 244, which directs the light toward the left eye of the user to provide an image superimposed on or in the left optical element 208 with a view of the real world seen by the user. The combination of the GPU, the forward optical assembly 230, the left optical element 208, and the right optical element 210 provides an optical engine of glasses 200. Glasses 200 use the optical engine to generate an overlay of the user's real-world view, including displaying a 3D user interface to the user of glasses 200.

[0044] However, it will be appreciated that other display technologies or configurations may be utilized within the optical engine to display images to the user in the user's field of view. For example, instead of the projector 232 and waveguide, an LCD, LED or other display panel or surface may be provided.

[0045] When in use, the user of the glasses 200 will be presented with information, content, and various 3D user interfaces on the near-eye display. As described in more detail herein, the user can then use the touch pad 226 and / or buttons 228, associated devices (e.g., Figure 3 The user may interact with the glasses 200 through voice input or touch input on a client device 328 shown in FIG. 1 , and / or hand movements, positions, and locations detected by the glasses 200 .

[0046] Figure 3 is a block diagram showing a networked system 300 including details of the glasses 200 according to some examples. The networked system 300 includes the glasses 200, a client device 328, and a server system 332. The client device 328 may be a smartphone, a tablet computer, a tablet phone, a laptop computer, an access point, or any other such device capable of connecting to the glasses 200 using a low power wireless connection 336 and / or a high speed wireless connection 334. The client device 328 is connected to the server system 332 via a network 330. The network 330 may include any combination of wired and wireless connections. The server system 332 may be one or more computing devices that are part of a service or network computing system. The client device 328 and any elements of the server system 332 and the network 330 may use Fig. 9 and Fig.10 The details of the software architecture 1006 or machine 1100 described respectively are implemented.

[0047] The glasses 200 include a data processor 302, a display 310, one or more cameras 308, and additional input / output elements 316. The input / output elements 316 may include a microphone, an audio speaker, a biometric sensor, an additional sensor, or an additional display element integrated with the data processor 302. Fig. 9 and Fig.10 Examples of input / output elements 316 are further discussed. For example, input / output elements 316 may include any I / O component 1118, including output component 1126, motion component 1134, etc. Figure 2B Examples of display 310 are discussed in . In the particular examples described herein, display 310 includes displays for the user's left eye and right eye.

[0048] Data processor 302 includes image processor 306 (eg, video processor), GPU and display driver 338, tracking module 340, interface 312, low power circuitry 304, and high speed circuitry 320. Components of data processor 302 are interconnected by bus 342.

[0049] The interface 312 refers to any source of user commands provided to the data processor 302. In one or more examples, the interface 312 is a physical button that sends a user input signal from the interface 312 to the low-power processor 314 when pressed. Pressing such a button and then releasing it immediately can be processed by the low-power processor 314 as a request to capture a single image, and vice versa. Pressing such a button for a first period of time can be processed by the low-power processor 314 as a request to capture video data when the button is pressed and end video capture when the button is released, wherein the video captured when the button is pressed is stored as a single video file. Alternatively, pressing the button for a long period of time can capture a still image. In some examples, the interface 312 can be any mechanical switch or physical interface capable of accepting user input associated with a request for data from the camera 308. In other examples, the interface 312 can have a software component, or can be associated with a command received wirelessly from another source such as from the client device 328.

[0050] Image processor 306 includes circuitry for receiving signals from camera 308 and processing those signals from camera 308 into a format suitable for storage in memory 324 or for transmission to client device 328. In one or more examples, image processor 306 (e.g., a video processor) includes a microprocessor integrated circuit (IC) customized for processing sensor data from camera 308, and volatile memory used by the microprocessor in operation.

[0051] The low power circuit system 304 includes a low power processor 314 and a low power wireless circuit system 318. These elements of the low power circuit system 304 can be implemented as separate elements or can be implemented on a single IC as part of a single system on a chip. The low power processor 314 includes logic for managing other elements of the glasses 200. As described above, for example, the low power processor 314 can accept user input signals from the interface 312. The low power processor 314 can also be configured to receive input signals or command communications from a client device 328 via a low power wireless connection 336. The low power wireless circuit system 318 includes circuit elements for implementing a low power wireless communication system. Bluetooth TM Smart (also known as Bluetooth TMLow power wireless communication (LPW) is one standard implementation of a low power wireless communication system that may be used to implement low power wireless circuitry 318. In other examples, other low power communication systems may be used.

[0052] High-speed circuit system 320 includes a high-speed processor 322, a memory 324, and a high-speed wireless circuit system 326. High-speed processor 322 can be any processor capable of managing high-speed communications and operations for any general-purpose computing system for data processor 302. High-speed processor 322 includes processing resources for managing high-speed data transmission over high-speed wireless connection 334 using high-speed wireless circuit system 326. In some examples, high-speed processor 322 executes an operating system such as a LINUX operating system or a program such as a UNIX operating system. Fig.10 The high-speed processor 322, which executes the software architecture of the data processor 302, manages data transmission with the high-speed wireless circuit system 326, in addition to any other duties. In some examples, the high-speed wireless circuit system 326 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, which is also referred to herein as Wi-Fi. In other examples, the high-speed wireless circuit system 326 can implement other high-speed communication standards.

[0053] Memory 324 includes any storage device capable of storing camera data generated by camera 308 and image processor 306. Although memory 324 is shown as being integrated with high-speed circuitry 320, in other examples, memory 324 may be a separate, stand-alone element of data processor 302. In some such examples, electrical wiring may provide a connection from image processor 306 or low-power processor 314 to memory 324 through a chip that includes high-speed processor 322. In other examples, high-speed processor 322 may manage addressing of memory 324 so that low-power processor 314 will initiate high-speed processor 322 any time a read or write operation involving memory 324 is needed.

[0054] The tracking module 340 estimates the pose of the glasses 200. For example, the tracking module 340 uses image data and corresponding inertial data from the camera 308 and the positioning component 1138, as well as GPS data, to track the position and determine the pose of the glasses 200 relative to a reference frame (e.g., a real-world environment). The tracking module 340 continuously collects and uses updated sensor data describing the movement of the glasses 200 to determine an updated three-dimensional pose of the glasses 200, which indicates changes in relative position and orientation relative to physical objects in the real-world environment. The tracking module 340 allows the glasses 200 to visually place virtual objects relative to physical objects within the user's field of view via the display 310.

[0055] The GPU and display driver 338 can use the pose of the glasses 200 to generate frames of virtual content or other content to be presented on the display 310 when the glasses 200 are operating in a conventional augmented reality mode. In this mode, the GPU and display driver 338 generate updated frames of virtual content based on the updated three-dimensional pose of the glasses 200, which reflects the changes in the position and orientation of the user relative to physical objects in the user's real-world environment.

[0056] One or more functions or operations described herein may also be performed in an application resident on the glasses 200 or on the client device 328 or on a remote server. The glasses 200 may be a stand-alone client device capable of independent operation, or may be a companion device that works with a host device to offload intensive processing and / or exchange data with the presentation server system 108 over the network 106. The glasses 200 may also be communicatively coupled to a companion device such as a smart watch and may be configured to exchange data with the companion device. The glasses 200 may also include various components common to mobile electronic devices such as smart glasses or smart phones (e.g., including a display controller for controlling the display of visual media on a display mechanism incorporated in the device).

[0057] Figure 4 1 is a block diagram showing additional details about a volumetric content rendering system 100 according to some examples. Specifically, the volumetric content rendering system 100 is shown to include a rendering client application 104 and an application server 112. The volumetric content rendering system 100 contains several subsystems that are supported on the client side by the rendering client application 104 and on the server side by the application server 112. These subsystems include, for example, a collection management system 402, a rendering control system 404, and an enhancement system 406.

[0058] The collection management system 402 is responsible for managing groups or collections of content (e.g., collections of text, images, video, and audio data). Collections of content may be organized into "event galleries" or "event stories." Such collections may be made available for specified time periods, such as for the duration of an event to which the content relates. For example, content related to a concert may be available as a "story" for the duration of the concert. The collection management system 402 may also be responsible for publishing an icon that provides notification of the existence of a particular collection to the user interface of the presentation client application 104.

[0059] In addition, the collection management system 402 also includes a curation interface 408, which allows collection managers to manage and curate specific content collections. For example, the curation interface 408 enables event organizers to curate a collection of content related to a specific event (e.g., to remove inappropriate content or redundant messages). Additionally, the collection management system 402 uses machine vision (or image recognition technology) and content rules to automatically curate content collections.

[0060] The presentation control system 404 is responsible for facilitating and controlling the volumetric content presentation. Thus, the presentation control system 404 provides a mechanism that allows a user to specify control operations for controlling the volumetric content presentation. The control operations may include, for example: a stop operation for stopping the presentation; a pause operation for pausing the presentation; a fast forward operation for advancing the presentation at a higher speed; a rewind operation for rewinding the presentation; a zoom in operation for increasing the zoom level of the presentation; a zoom out operation for decreasing the zoom level of the presentation; and a playback speed modification operation for changing the speed of the presentation (e.g., for producing a slow motion presentation of volumetric video).

[0061] For some embodiments, a user may specify an input indicative of a control operation for controlling presentation of volumetric content by providing one or more inputs via one or more I / O components (see below). Fig.10 Examples of one or more I / O components are described in further detail. For some embodiments, the presentation control system 404 may provide an interactive control interface that includes one or more interactive elements (e.g., virtual buttons) for triggering control operations, and the presentation control system 404 monitors interactions with the interactive interface to detect input indicating a control operation. For some embodiments, a user may trigger a control operation using a gesture, such as a hand gesture or a head gesture, that may be associated with a particular control operation.

[0062] The enhancement system 406 provides various functions that enable a user to enhance (e.g., annotate or otherwise modify or edit) media content. For example, the enhancement system 406 provides functions related to the generation, publication, and application of enhancement data, such as media overlays (e.g., image filters) to volumetric content. Media overlays can include audio and visual content and visual effects. Examples of audio and visual content include pictures, text, logos, animations, and sound effects. Examples of visual effects include color overlays. Audio and visual content or visual effects can be applied to media content items (e.g., photos) at the client device 102. For example, a media overlay can include text or images that can be superimposed on a photo taken by the client device 102. The enhancement system 406 is operable to provide one or more media overlays to the presentation client application 104 based on the geographic location of the client device 102 or based on other information such as social network information of a user of the client device 102. The media overlays can be stored in the database 120 and accessed through the database server 118.

[0063] A filter is an example of a media overlay that is displayed as an overlay on an image or video during presentation to a user. Filters can be of various types, including filters selected by a user from a set of filters presented to a user by presentation client application 104. Other types of filters include geolocation filters (also referred to as geofilters), which can be presented to a user based on a geographic location. For example, a geolocation filter specific to a nearby or special location can be presented within a user interface by presentation client application 104 based on geographic location information determined by a global positioning system (GPS) unit of client device 102.

[0064] Another type of filter is a data filter, which can be selectively presented to the user by the presentation client application 104 based on other input or information collected by the client device 102. Examples of data filters include the current temperature at a particular location, the current speed at which the user is traveling, the battery life of the client device 102, or the current time.

[0065] AR content items are another example of a media overlay. AR content items may be real-time special effects and / or sounds that may be added to images or videos, including volumetric images and videos.

[0066] In general, AR content items, overlays, image transformations, images, and the like refer to modifications that can be applied to image data (e.g., videos or images) that include volumetric content. This includes real-time modifications that modify an image as it is captured using a device sensor (e.g., one or more cameras) of the client device 102, and then the image with the modifications is displayed by a display device (e.g., an embedded display of the client device) of the client device 102. This also includes modifications to stored content (e.g., volumetric videos in a gallery or collection that can be modified). For example, in a client device 102 that accesses multiple AR content items, a user can use a single volumetric video with multiple AR content items to see how different AR content items will modify the stored content. For example, by selecting different AR content items for the content, multiple augmented reality content items that apply different pseudo-random movement models can be applied to the same content. Similarly, real-time video capture can be used with the modifications shown to show how the video image currently captured by the sensor of the client device 102 will modify the captured data. Such data may simply be displayed on the screen without being stored in memory, or content captured by the device sensors may be recorded and stored in memory with or without modification (or both). In some systems, a preview feature may show how different augmented reality content items will look in different windows of the display simultaneously. For example, this may enable viewing multiple windows with different pseudo-random animations on the display simultaneously.

[0067] Thus, data and various systems using augmented reality content items or other such augmented systems using augmented data to modify content may involve detection of objects (e.g., faces, hands, bodies, cats, dogs, surfaces, objects, etc.) in video frames, tracking such objects as they leave, enter, and move around in the field of view, and modifying or transforming such objects as they are tracked. In various examples, different methods for implementing such transformations may be used. Some examples may involve: generating a three-dimensional mesh model of one or more objects; and using transformations of the model and animated textures within a video to implement the transformations. In other examples, tracking of points on an object may be used to place an image or texture (which may be two-dimensional or three-dimensional) at the tracked location. In yet another example, neural network analysis of a video frame may be used to place an image, model, or texture in content (e.g., a frame of an image or video). Thus, an AR content item refers to both images, models, and textures used to create transformations in the content, as well as additional modeling and analysis information required to implement such transformations using object detection, tracking, and placement.

[0068] Real-time video processing can be performed with any kind of video data (e.g., video streams, video files, volumetric video, etc.) stored in the memory of any kind of computerized system. For example, a user can load video files and store them in the memory of the device, or a sensor of the device can be used to generate a video stream. In addition, any object, such as a human face and parts of a human body, an animal, or a non-living thing (e.g., a chair, a car, or other objects) can be processed using computer animation models.

[0069] In some examples, when a specific modification is selected together with the content to be enhanced (e.g., edited), the element to be transformed is identified by a computing device, and then if the element to be transformed is present in a frame of a video, the element to be transformed is detected and tracked. The elements of the object are modified according to a request for modification, so the frame of the video stream is transformed. For different types of transformations, the transformation of the frame of the video stream can be performed by different methods. For example, for the transformation of the frame that mainly involves changing the form of the elements of the object, the characteristic point (e.g., using an active shape model (ASM) or other known methods) is calculated for each element of the object. Then, a grid based on the characteristic point is generated for each element of at least one element of the object. The grid can be used in a subsequent stage of tracking the elements of the object in the video stream. During the tracking process, the grid mentioned for each element is aligned with the positioning of each element. Then, additional points are generated on the grid. A set of first points is generated for each element based on the request for modification, and a set of second points is generated for each element based on the set of first points and the request for modification. Then, the frame of the video stream can be transformed by modifying the elements of the object based on the set of first points, the set of second points and the grid. In such an approach, the background of the modified object may also be changed or distorted by tracking and modifying the background.

[0070] In some examples, a transformation that changes some areas of an object using elements of an object can be performed by calculating characteristic points of each element of the object and generating a grid based on the calculated characteristic points. Points are generated on the grid, and then various areas based on the points are generated. The elements of the object are then tracked by aligning the area for each element with the positioning for each element of at least one element, and the properties of the area can be modified based on a request for modification, thereby transforming the frame of the video stream. Depending on the specific request for modification, the properties of the mentioned area can be transformed in different ways. Such modifications may involve: changing the color of the area; removing at least some parts of the area from the frame of the video stream; including one or more new objects into the area based on the request for modification; and modifying or distorting the elements of the area or object. In various examples, any combination of such modifications or other similar modifications may be used. For certain models to be animated, some characteristic points may be selected as control points for the entire state space to be used to determine options for model animation.

[0071] In some examples of computer animation models for transforming image data using face detection, faces are detected on an image using a specific face detection algorithm (e.g., Viola-Jones). An active shape model (ASM) algorithm is then applied to the face region of the image to detect facial feature reference points.

[0072] Other methods and algorithms suitable for face detection may be used. For example, in some examples, features are located using landmarks that represent distinguishable points that are present in most of the images considered. For example, for facial landmarks, the location of the left pupil may be used. If the initial landmarks are not recognizable (e.g., if the person has an eye mask), secondary landmarks may be used. Such a landmark identification process may be used for any such object. In some examples, a set of landmarks forms a shape. The shape may be represented as a vector using the coordinates of the points in the shape. A similarity transform (which allows translation, scaling, and rotation) that minimizes the average Euclidean distance between the shape points is used to align one shape with another. The mean shape is the mean of the aligned training shapes.

[0073] In some examples, the landmark search begins with a mean shape that is aligned with the location and size of the face as determined by a global face detector. Such a search then repeats the steps of proposing tentative shapes by adjusting the positions of shape points via template matching of the image texture around each point, and then conforming the tentative shapes to a global shape model until convergence occurs. In some systems, individual template matches are unreliable, and the shape model pools the results of weak template matches to form a stronger overall classifier. The entire search is repeated at each level of the image pyramid from coarse to fine resolution.

[0074] The augmentation system 406 can capture images or video streams on a client device (e.g., client device 102) and perform complex image manipulations locally on the client device 102 while maintaining a suitable user experience, computational time, and power consumption. Complex image manipulations can include size and shape changes, emotional transitions (e.g., changing a face from a frown to a smile), state transitions (e.g., aging a subject, reducing apparent age, changing gender), style transitions, application of graphical elements, and any other suitable image or video manipulations enabled by a machine learning model that has been configured to execute efficiently on the client device 102. In some examples, a computer animation model for transforming video and image content can be used by the augmentation system 406, where a neural network operates as part of a rendering client application 104 operating on the client device 102.

[0075] The machine learning model can be trained using labeled or unlabeled training data that includes ground-truth information. For example, one or more images depicting a portion of a person's body and / or the entire body can be included in the training data along with ground-truth segmentation information that defines the current positioning of the limbs and joints of the user's entire body (e.g., the positioning of arms, torso, legs, head, shoulders, etc.). The machine learning model can be applied to a subset of the training data (e.g., one or more images in the training data), and an estimate or prediction of the ground-truth segmentation information about the current positioning of the limbs and joints of the user's entire body (e.g., the positioning of arms, torso, legs, head, shoulders, etc.) can be generated. The estimate or prediction can be compared with the corresponding ground-truth information to calculate the deviation. The deviation can then be used to update one or more parameters of the machine learning model. After updating one or more parameters, the machine learning model is applied to another subset of the training data, and these operations are repeated until the stopping criteria are reached.

[0076] Machine learning is a field of study that gives computers the ability to learn without explicit programming. Machine learning explores the study and construction of algorithms (also referred to herein as tools) that can learn from existing data and make predictions about new data. Such machine learning tools operate by building models based on example training data to make data-driven predictions or decisions that are represented as outputs or evaluations. Although examples are given for several machine learning tools, the principles given herein can be applied to other machine learning tools.

[0077] In some examples, different machine learning tools can be used. For example, logistic regression (LR), naive Bayes, random forest (RF), neural network (NN), matrix factorization, and support vector machine (SVM) tools can be used to classify or score job positions.

[0078] Two common types of problems in machine learning are classification problems and regression problems. Classification problems (also called categorization problems) aim to classify an item into one of several class values ​​(e.g., is this object an apple or an orange?). Regression algorithms aim to quantify some item (e.g., by providing values ​​that are real numbers).

[0079] Machine learning algorithms use features to analyze data to generate estimates. Each of the features is a separate measurable property of the phenomenon being observed. The concept of a feature is related to the concept of explanatory variables used in statistical techniques such as linear regression. Selecting features that are informative, discriminative, and independent is important for the effective operation of MLPs in pattern recognition, classification, and regression. Features can be of various types, such as numeric features, strings, and graphs.

[0080] In one example, by way of example only, features may be of different types and may include one or more of content, concepts, attributes, historical data, and / or user data.

[0081] Machine learning algorithms use training data to find correlations between identified features that influence an outcome or evaluation. In some examples, the training data includes labeled data, which is known data about one or more identified features and one or more outcomes, such as detecting communication patterns, detecting the meaning of a message, generating a summary of a message, detecting action items in a message, detecting urgency in a message, detecting a user's relationship to a sender, calculating a score attribute, calculating a message score, etc.

[0082] Using the training data and the identified features, the machine learning tool is trained at the machine learning program training. The machine learning tool evaluates the value of the feature when the feature is associated with the training data. The result of the training is a trained machine learning program.

[0083] When performing an evaluation using a trained machine learning program, new data is provided as input to the trained machine learning program, and the trained machine learning program generates an evaluation as output.

[0084] The machine learning program supports two types of phases, namely, a training phase and a prediction phase. In the training phase, supervised learning, unsupervised learning, or reinforcement learning can be used. For example, the machine learning program (1) receives features (e.g., as structured data or labeled data in supervised learning) and / or (2) identifies features in the training data (e.g., unstructured data or unlabeled data for unsupervised learning). In the prediction phase, the machine learning program uses the features to analyze the query data to generate a result or prediction, as an example of evaluation.

[0085] In the training phase, feature engineering is used to identify features and can include identifying informative, discriminative, and independent features for effective operation of machine learning programs in pattern recognition, classification, and regression. In some examples, the training data includes labeled data, which is known data of pre-identified features and one or more results. Each of the features can be a variable or attribute, such as a measurable property of a process, item, system, or phenomenon represented by a data set (e.g., training data).

[0086] During the training phase, the machine learning program uses the training data to find correlations between features that influence the predicted outcome or assessment.

[0087] The machine learning program is trained during a training phase at which the machine learning program is trained using the training data and the identified features. The machine learning program evaluates the values ​​of the features when the features are associated with the training data. The result of the training is a trained machine learning program (e.g., a trained or learned model).

[0088] In addition, the training phase may involve machine learning, in which the training data is structured (e.g., labeled during a preprocessing operation), and the trained machine learning program implements a relatively simple neural network that can, for example, perform classification and clustering operations. In other examples, the training phase may involve deep learning, in which the training data is unstructured, and the trained machine learning program implements a deep neural network that can perform both feature extraction and classification / clustering operations.

[0089] The neural network generated during the training phase and implemented within the trained machine learning program can include a hierarchical (e.g., layered) organization of neurons. For example, neurons (or nodes) can be hierarchically arranged into several layers, including an input layer, an output layer, and multiple hidden layers. Each of the layers within the neural network can have one or many neurons, and each of these neurons operationally computes a small function (e.g., an activation function). For example, if the activation function generates a result that exceeds a certain threshold, the output can be transmitted from the neuron (e.g., a sending neuron) to a connected neuron (e.g., a receiving neuron) in a consecutive layer. The connection between neurons also has an associated weight that defines the influence of the input from the sending neuron to the receiving neuron.

[0090] In some examples, the neural network may also be one of several different types of neural networks, including, by way of example only, a single-layer feedforward network, an artificial neural network (ANN), a recurrent neural network (RNN), a symmetrically connected neural network, and an unsupervised pre-trained network, a convolutional neural network (CNN), or a recurrent neural network (RNN).

[0091] During the prediction phase, the trained machine learning program is used to perform the evaluation. The query data is provided as input to the trained machine learning program, and the trained machine learning program generates the evaluation as output in response to receiving the query data.

[0092] Figure 5 is a diagrammatic representation of a data structure 500 as maintained in database 120, according to some examples. Although the contents of database 120 are shown as including a plurality of tables, it will be appreciated that data may be stored in other types of data structures (eg, as an object-oriented database).

[0093] The entity table 504 stores entity data and is linked (e.g., by reference) to the entity graph 506 and profile data 508. The entities whose records are maintained within the entity table 504 may include individuals, corporate entities, organizations, objects, places, events, etc. Regardless of the entity type, any entity about which the presence server system 108 stores data may be an identified entity. Each entity is provided with a unique identifier and an entity type identifier (not shown).

[0094] Entity graph 506 stores information about relationships and associations between entities. Such relationships may be social, professional (e.g., working in a common company or organization), interest-based, or activity-based, by way of example only.

[0095] The profile data 508 stores various types of profile data about a particular entity. Based on the privacy settings specified by the particular entity, the profile data 508 can be selectively used and presented to other users of the volumetric content presentation system 100. In the case where the entity is a person, the profile data 508 includes, for example, a user name, a phone number, an address, settings (e.g., notification and privacy settings), and an avatar representation (or a collection of such avatar representations) selected by the user.

[0096] The database 120 also stores augmentation data, such as overlays including AR content items and filters, in augmentation table 510. The augmentation data is associated with and applied to videos (whose data is stored in video table 514) and images (whose data is stored in image table 516), including volumetric videos and images.

[0097] The story table 512 stores data about a collection of messages and associated images, video, or audio data, which are compiled into a collection (e.g., a story or gallery). The creation of a particular collection can be initiated by a particular user (e.g., each user for whom a record is maintained in the entity table 504). A user can create a "personal story" in the form of a collection of content that has been created and sent / broadcasted by the user. To this end, the user interface presenting the client application 104 may include an icon that the user can select to enable the sending user to add specific content to his or her personal story.

[0098] Entity table 504 may associate various enhancements from enhancement table 510 with various images and videos stored in image table 516 and video table 514 .

[0099] Figure 6 and Figure 7 6 is a flow chart illustrating the operation of a volumetric content rendering system according to an example embodiment when performing a method 600 for rendering volumetric video. The method 600 may be embodied in computer-readable instructions for execution by one or more processors, such that the operations of the method 600 may be performed in part or in whole by functional components of the volumetric content rendering system 100; therefore, the method 600 is described below by way of example with reference thereto. However, it should be understood that at least some of the operations of the method 600 may be deployed on various other hardware configurations besides the volumetric content rendering system 100.

[0100] At operation 605, the volumetric content presentation system 100 causes the volumetric content to be presented by a display device. The volumetric content includes a volumetric video that includes a volumetric representation of one or more three-dimensional elements of a three-dimensional space. That is, the volumetric representation includes at least one content item (e.g., an AR content item) corresponding to a person or object in the three-dimensional space. For some embodiments, the display device is an augmented reality device (e.g., glasses 200), and the presentation of the volumetric content may include: displaying one or more AR content items superimposed on a real-world space that is visible to a user of the display device, which real-world space may be the same as or a different space from the three-dimensional space in which the volumetric video is captured. The real-world space may be visible to the user through a transparent portion of the display device or based on image data presented by the display device. The presentation of the volumetric content may include displaying one or more content items in motion, displaying one or more content items performing a movement or other action, displaying one or more content items in a static position, or a combination thereof. The content items may be displayed for the duration of the presentation of the volumetric content or a portion of the volumetric content.

[0101] The presentation of volumetric content may include tracking the position and movement of a user within their physical real world environment, and using the tracked position and movement of the user to allow the user to move within and interact with the presentation of the volumetric content. Thus, the presentation of volumetric content may include displaying content items from multiple perspectives based on the user's movement and changes in position. The presentation of volumetric content may be based on motion and positioning components from any one or more of the display device, a host device to which the display device is a complementary device, or a complementary device (e.g., a smartwatch) that communicates with the display device (examples of which are described below with reference to Fig.10 further described in detail) to track the user's location and movements.

[0102] The volumetric content may also include audio data including one or more audio signals, and thus, rendering of the volumetric content may include rendering the one or more audio signals.

[0103] At operation 610, the volumetric content presentation system 100 receives input indicating a control operation associated with the presentation of the volumetric video. For example, the control operation may include: a stop operation for stopping the presentation; a pause operation for pausing the presentation; a fast forward operation for advancing the presentation at a higher speed; a rewind operation for rewinding the presentation; a zoom in operation for increasing the zoom level of the presentation; a zoom out operation for decreasing the zoom level of the presentation; and a playback speed modification operation for changing the speed of the presentation (e.g., for producing a slow motion presentation of the volumetric video).

[0104] For some embodiments, the input corresponds to one or more I / O components of any one or more of a slave display device, a master device of a complementary device to which the display device is a master, or a complementary device (e.g., headphones) that communicates with the display device (examples of which are described below with reference to Fig.10 Consistent with these embodiments, receiving input indicative of a control operation may include receiving one or more input signals based on user interaction with one or more I / O components.

[0105] For some embodiments, the presentation of the volumetric video may include an interactive control interface that includes one or more interactive elements (e.g., virtual buttons) for triggering control operations, and receiving input may include: detecting interaction with an interactive element of the interactive interface by a user of the display device.

[0106] For some embodiments, receiving input indicating a control operation may include detecting a gesture of a user of the display device. For example, a body movement of the user, such as a hand gesture or a head gesture, may be associated with the control operation and used by the user to trigger the control operation. For example, detecting the user's gesture may include performing image processing on image data generated by a camera (e.g., camera 216 of glasses 200) communicatively coupled to the display device. Performing image processing to detect gestures may include utilizing one or more machine learning models.

[0107] For some embodiments, receiving input indicating a control operation may include: detecting a user interaction with a content item (e.g., a representation of a person or object) within the volumetric video. Detecting the interaction may include: detecting a user's gesture pointing at a virtual content item; detecting a user's line of sight pointing at a virtual content item; detecting a user's motion or movement in the direction of the content item; or determining that a distance between a tracked user's position and a position of a virtual content item within the volumetric representation of the three-dimensional space satisfies a threshold distance constraint.

[0108] At operation 615, the volumetric content presentation system 100 controls the presentation of the volumetric content by the display device by performing a control operation. Depending on the control operation, performing the control operation may include any of the following: stopping the presentation of the volumetric video; pausing the presentation of the volumetric video; rewinding the presentation of the volumetric video; fast forwarding the presentation of the volumetric video; increasing the zoom level of the presentation of the volumetric video; decreasing the zoom level of the presentation of the volumetric video; increasing the playback speed of the presentation of the volumetric video; and decreasing the playback speed of the presentation of the volumetric video. As part of controlling the presentation of the volumetric content, the volumetric content presentation system allows the user to move around in the three-dimensional space of the volumetric content to observe the performance of the control operation for the presentation of the volumetric content from multiple perspectives. That is, while the control operation is being performed, the volumetric content presentation system 100, according to the control operation, causes the display device to present one or more volumetric representations of the element from multiple perspectives according to the determined positioning of the user in the three-dimensional space (based on, for example, the position of the user in their real-world environment). For example, while the presentation of the volumetric content is paused, the user may move around to observe the volumetric representations of the element in the volumetric content from multiple perspectives while the movement of the element is frozen.

[0109] like Figure 7 As shown, in some embodiments, method 600 may further include operations 705 and 710. Consistent with these embodiments, operation 705 may be performed before operation 610 in which the volumetric content presentation system 100 receives an input indicating a control operation. At operation 705, the volumetric content presentation system 100 causes the display device to display an interactive control interface for controlling the presentation of the volumetric video. The interactive control interface includes one or more interactive elements (e.g., buttons) for triggering the control operation.

[0110] Consistent with these embodiments, operation 710 may be performed as part of operation 610. At operation 710, the volumetric content presentation system 100 detects an interaction by a user of a display device with an interactive element of an interactive interface that corresponds to a control operation.

[0111] Figure 8 800 is a flowchart illustrating the operation of the volumetric content rendering system 100 in accordance with an example embodiment when performing a method 800 for editing a presentation of volumetric content. The method 800 may be embodied in computer-readable instructions for execution by one or more processors, such that the operations of the method 800 may be performed in part or in whole by functional components of the volumetric content rendering system 100; therefore, the method 800 is described below by way of example with reference thereto. However, it should be understood that at least some of the operations of the method 800 may be deployed on various other hardware configurations other than the volumetric content rendering system 100.

[0112] For some embodiments, the operations of method 800 may be performed as part of method 600. For example, the operations of method 800 may be performed at any time after operation 605 in which volumetric content rendering system 100 causes rendering of volumetric content, including after controlling rendering of the volumetric content according to a control operation specified by a detected input.

[0113] At operation 805, the volumetric content rendering system 100 initiates an edit mode associated with the rendering of volumetric video. While in the edit mode, the user can modify aspects of the rendering of the volumetric content, and generate the modified volumetric content for subsequent rendering. That is, the edit mode allows the user to temporarily modify how the volumetric content is rendered, as well as create and save modifications that can be rendered at a later time. The volumetric content rendering system 100 initiates the edit mode in response to a user input indicating a request to initiate the edit mode (e.g., input received from an I / O component of the display device, a host device to which the display device is companion, or a companion device of the display device).

[0114] For some embodiments, when the editing mode is initiated, the volumetric content presentation system 100 causes the display device to display an interactive editing interface for editing the volumetric video. Therefore, the interactive editing interface includes one or more interactive elements (e.g., buttons, toggle keys, switches, drop-down menus, etc.) to provide editing of the volumetric video.

[0115] For some embodiments, method 800 includes operation 810, at which volumetric content presentation system 100 detects a user selection of a volumetric representation of an element from the volumetric content (e.g., an AR content item). Detecting the user selection of the element may include: detecting a user interaction with the AR content item included in the presentation of the volumetric content. Detecting the interaction may include: detecting a gesture of a user pointing at a virtual content item; detecting a line of sight of a user pointing at a virtual content item; or determining that a distance between a tracked user's location and a location of the virtual content item within the volumetric representation of the three-dimensional space satisfies a threshold distance constraint.

[0116] At operation 815, the volumetric content presentation system 100 receives edits to the volumetric video. For some embodiments, the edits to the volumetric video may be provided by a user via an interactive editing interface. For some embodiments, the edits are specific to a content item selected by the user.

[0117] Editing can include any one or more of the following: modification of content items (e.g., changes to size, shape, color, shading, etc.), removal of content items; addition of content items (e.g., images, video, and / or audio content); and adding filters to content items or real-world objects visible to a user via the presentation of volumetric content, for example by applying filters. Content items or enhancements to be added to the volumetric video can be selected from a library, or can be selected from another volumetric video. When editing includes the addition of content items, the editing can also specify aspects such as the positioning and orientation of the content items within the three-dimensional space of the volumetric video. Editing can also specify configurable characteristics of the content items such as size, shape, color, shading, movement, etc.

[0118] At operation 820, the volumetric content rendering system 100 edits the volumetric video based on the edits, which produces an edited volumetric video. When editing the volumetric video, the volumetric content rendering system 100 modifies the volumetric content based on the edits, thereby producing the modified volumetric content. Depending on the nature of the edits, the volumetric content rendering system 100 can edit the volumetric video by performing any one or more of the following: modifying content items in the three-dimensional space (e.g., by changing the size, shape, color, shading, etc.); removing content items in the three-dimensional space; adding content items to the three-dimensional space (e.g., adding images, video, or audio content); and applying filters to content items in the three-dimensional space or real-world objects visible within the rendering of the volumetric content. In an example, the volumetric video includes a volumetric representation of at least an element of a basketball game, and the edits include a filter applied to the volumetric representation of the basketball to make the basketball appear to glow.

[0119] At operation 825, the volumetric content presentation system 100 causes the edited volumetric video to be presented by the display device. Returning to the example of the basketball game, the presentation of the edited volumetric video includes presentation of a glowing volumetric representation of a basketball based on the filter included in the edit. The presentation of the glowing volumetric representation of the basketball game makes the presentation of the volumetric video more engaging and immersive.

[0120] For some implementations, an option may be provided to a user of the display device to save the edited volumetric video, and in response to receiving input indicating a request to save the edited volumetric video, at operation 830 the volumetric content presentation system 100 stores the modified volumetric content in the database 120 .

[0121] Fig. 9 906 is a conceptual diagram illustrating an example interactive control interface 900 for controlling the presentation of volumetric content within a display area 902 of a display device according to an example implementation. As shown, the interactive control interface 900 is displayed in conjunction with the presentation of volumetric content. The volumetric content includes a volumetric video that includes volumetric representations of two people interacting with each other. The volumetric representations are in the form of AR content items 904 and 906. As part of the presentation of the volumetric content, the AR content items 904 and 906 are displayed as an overlay on a real-world environment 908 visible within the display area 902 of the display device.

[0122] In this example, an interactive control interface 900 is also displayed as an overlay on the real-world environment 908. The interactive control interface 900 includes a plurality of interactive elements to trigger control operations for controlling the presentation of the volumetric content. In particular, the interactive control interface 900 includes: a fast forward element 910 for advancing the presentation at a higher speed; a pause element 911 for pausing the presentation; a slow motion element 912 for reducing the speed of the presentation; a play element 913 for resuming the presentation; and a rewind element 914 for rewinding the presentation. Upon detecting a user interaction (e.g., selection) with one of the elements 910 to 914, the volumetric content presentation system 100 controls the presentation of the volumetric video according to the control operation corresponding to the element for which the interaction was detected.

[0123] Software Architecture

[0124] Fig.10 is a block diagram illustrating an example software architecture 1006 that may be used in conjunction with the various hardware architectures described herein. Fig.10 is a non-limiting example of a software architecture, and it should be understood that many other architectures may be implemented to facilitate the functionality described herein. Software architecture 1006 may be implemented in a variety of environments such as Fig.101100, which includes a processor 1104, a memory / storage device 1106, and an I / O component 1118. A representative hardware layer 1052 is shown and may represent, for example, Fig.11 The representative hardware layer 1052 includes a processor 1104 with associated executable instructions 1110. The executable instructions 1110 represent executable instructions of the software architecture 1006, including implementations of the methods, components, etc. described herein. The hardware layer 1052 also includes a memory and / or storage device 1106 also having executable instructions 1110. The hardware layer 1052 may also include other hardware 1058.

[0125] exist Fig.10 In the example architecture of , software architecture 1006 can be conceptualized as a stack of layers in which each layer provides specific functionality. For example, software architecture 1016 can include layers such as operating system 1002, library 1020, framework / middleware 1018, application 1016, and presentation layer 1014. In operation, application 1016 and / or other components within a layer can activate API call 1008 through the software stack and receive a response to API call 1008 as message 1012. The layers shown are representative in nature, and not all software architectures have all layers. For example, some mobile or dedicated operating systems may not provide framework / middleware 1018, while other operating systems may provide such a layer. Other software architectures may include additional or different layers.

[0126] The operating system 1002 can manage hardware resources and provide public services. For example, the operating system 1002 can include a core 1022, services 1024, and drivers 1026. The core 1022 can be used as an abstraction layer between the hardware and other software layers. For example, the core 1022 can be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, etc. The services 1024 can provide other public services to other software layers. The drivers 1026 are responsible for controlling or interfacing with the underlying hardware. For example, depending on the hardware configuration, the drivers 1026 include display drivers, camera drivers, drivers, flash memory drivers, serial communication drivers (e.g., Universal Serial Bus (USB) drivers), Drivers, audio drivers, power management drivers, etc.

[0127] The library 1020 provides a common infrastructure used by the application 1016 and / or other components and / or layers. The library 1020 provides functions that allow other software components to perform tasks in an easier manner than directly interfacing with the underlying operating system 1002 functions (e.g., the kernel 1022, the service 1024, and / or the driver 1026). The library 1020 may include a system library 1044 (e.g., a C standard library), which may provide functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. In addition, the library 1020 may include an API library 1046, such as a media library (e.g., a library for supporting the presentation and manipulation of various media formats such as MPEG4, H.294, MP3, AAC, AMR, JPG, and PNG), a graphics library (e.g., an OpenGL framework that can be used to render 2D and 3D graphics content on a display), a database library (e.g., SQLite that can provide various relational database functions), a web library (e.g., WebKit that can provide web browsing functions), etc. The library 1020 may also include various other libraries 1048 to provide numerous other APIs to the applications 1016 and other software components / modules.

[0128] The framework / middleware 1018 provides a higher level common infrastructure that can be used by the applications 1016 and / or other software components / modules. For example, the framework / middleware 1018 can provide various GUI functions, advanced resource management, advanced location services, etc. The framework / middleware 1018 can provide a wide range of other APIs that can be utilized by the applications 1016 and / or other software components / modules, some of which can be specific to a particular operating system 1002 or platform.

[0129] Applications 1016 include built-in applications 1038 and / or third-party applications 1040. Examples of representative built-in applications 1038 may include, but are not limited to, contact applications, browser applications, book reader applications, location applications, media applications, messaging applications, and / or game applications. Third-party applications 1040 may include applications that are developed by entities other than the vendor of a particular platform using ANDROID TM or IOS TM Software Development Kit (SDK) for developing applications and can be used on platforms such as IOS TM ANDROID TM , Mobile software running on the mobile operating system of the phone or other mobile operating systems. The third-party application 1040 can call API calls 1008 provided by the mobile operating system (such as the operating system 1002) to facilitate the functions described herein.

[0130] Applications 1016 may create user interfaces to interact with users of the system using built-in operating system functionality (e.g., kernel 1022, services 1024, and / or drivers 1026), libraries 1020, and framework / middleware 1018. Alternatively or additionally, in some systems, interaction with the user may occur through a presentation layer such as presentation layer 1014. In these systems, the application / component "logic" may be separated from aspects of the application / component that interact with the user.

[0131] Fig.11 1 is a block diagram illustrating components of a machine 1100 capable of reading instructions from a machine-readable medium (e.g., a machine-readable storage medium) and performing any one or more of the methods discussed herein, according to some example embodiments. Specifically, Fig.11 A graphical representation of a machine 1100 is shown in the form of an example of a computer system, within which instructions 1110 (e.g., software, programs, applications, applet, application (app) or other executable code) for causing the machine 1100 to perform any one or more of the methods discussed herein can be executed. Thus, the instructions 1110 can be used to implement the modules or components described herein. The instructions 1110 convert a general-purpose, unprogrammed machine 1100 into a specific machine 1100 that is programmed to perform the functions described and shown in the manner described. In alternative embodiments, the machine 1100 operates as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, the machine 1100 can operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 1100 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a PDA, an entertainment media system, a cellular phone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web device, a network router, a network switch, a network bridge, or any machine capable of executing instructions 1110 that specify actions to be taken by the machine 1100, either sequentially or otherwise. In addition, while only a single machine 1100 is shown, the term "machine" should also be taken to include a collection of machines that individually or jointly execute instructions 1110 to perform any one or more of the methodologies discussed herein.

[0132] The machine 1100 may include a processor 1104, a memory / storage device 1106, and an I / O component 1118, which may be configured to communicate with each other, for example, via a bus 1102. In an example embodiment, the processor 1104 (e.g., a CPU, a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a GPU, a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, a processor 1108 and a processor 1109 that may execute instructions 1110. Although Fig.11 Multiple processors 1104 are shown, but the machine 1100 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0133] The memory / storage 1106 may include a memory 1112, such as a main memory or other memory storage, and a storage unit 1114, both of which are accessed by the processor 1104, such as via the bus 1102. The storage unit 1114 and the memory 1112 store instructions 1110 that embody any one or more of the methods or functions described herein. During execution of the instructions 1110 by the machine 1100, the instructions 1110 may also reside, completely or partially, within the memory 1112, within the storage unit 1114, within at least one of the processors 1104 (e.g., within a cache memory of the processor), or any suitable combination thereof. Thus, the memory 1112, the storage unit 1114, and the memory of the processor 1104 are examples of machine-readable media.

[0134] The I / O components 1118 may include various components for receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurements, etc. The specific I / O components 1118 included in a particular machine 1100 will depend on the type of machine. For example, a portable machine such as a mobile phone will likely include a touch input device or other such input mechanism, while a headless server machine will likely not include such a touch input device. It should be understood that the I / O components 1118 may include Fig.1111 and 12. The I / O components 1118 are grouped according to function to simplify the following discussion, and the grouping is by no means limiting. In various example embodiments, the I / O components 1118 may include output components 1126 and input components 1128. The output components 1126 may include visual components (e.g., displays such as plasma display panels (PDPs), light emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), auditory components (e.g., speakers), tactile components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. The input components 1128 may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touch pads, trackballs, joysticks, motion sensors, or other pointing instruments), touch input components (e.g., physical buttons, touch screen displays or other touch input components that provide the location and / or force of touch or touch gestures), audio input components (e.g., microphones), etc.

[0135] In further example embodiments, the I / O component 1118 may include a biometric component 1130, a motion component 1134, an environmental component 1136, or a positioning component 1138, as well as various other components. For example, the biometric component 1130 may include a component for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, perspiration, or brain waves), identifying a person (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component 1134 may include an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. The environment component 1136 may include, for example, an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an auditory sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas sensor that detects concentrations of hazardous gases for safety reasons or measures pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. The positioning component 1138 may include a position sensor component (e.g., a global positioning system (GPS) receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure that can obtain altitude), an orientation sensor component (e.g., a magnetometer), etc.

[0136] A variety of technologies may be used to implement communications. I / O components 1118 may include communications components 1140 operable to couple machine 1100 to network 1132 or device 1120 via coupling 1124 and coupling 1122, respectively. For example, communications components 1140 may include a network interface component or other suitable device to interface with network 1132. In other examples, communications components 1140 may include wired communications components, wireless communications components, cellular communications components, near field communications (NFC) components, Parts (e.g. Low power consumption), Device 1120 may be another machine or any of a variety of peripheral devices (eg, a peripheral device coupled via USB).

[0137] In addition, the communication component 1140 can detect the identifier or can include a component that can be operated to detect the identifier. For example, the communication component 1140 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting the following items: a one-dimensional barcode, such as a universal product code (UPC) barcode; a multi-dimensional barcode, such as a quick response (QR) code, an Aztec code, a data matrix, a data glyph, a MaxiCode, a PDF4114, a super code, a UCC RSS-2D barcode, and other optical codes), or an acoustic detection component (e.g., a microphone for identifying an audio signal of a tag). In addition, various information can be obtained via the communication component 1140, such as a location obtained via Internet Protocol (IP) geolocation, a location obtained via Wi-Fi signal triangulation, a location obtained via detecting an NFC beacon signal that can indicate a specific location, etc.

[0138] Glossary

[0139] "Carrier signal" in this context refers to any intangible medium capable of storing, encoding or carrying instructions for execution by a machine, and includes digital or analog communications signals or other intangible media to facilitate communication of such instructions. Instructions may be sent or received over a network using a transmission medium via a network interface device and using any of a number of well-known transmission protocols.

[0140] A "client device" in this context refers to any machine that interfaces with a communication network to obtain resources from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a PDA, a smart phone, a tablet computer, an ultrabook, a netbook, a laptop computer, a multiprocessor system, a microprocessor-based or programmable consumer electronics system, a game console, a set-top box, or any other communication device that a user may use to access a network.

[0141] A "communication network" in this context refers to one or more parts of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, A network, another type of network, or a combination of two or more such networks. For example, the network or a portion of the network may include a wireless or cellular network, and the coupling to the network may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or another type of cellular or wireless coupling. In this example, the coupling may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rate for GSM evolution (EDGE) technology, including the third generation partnership project (3GPP) of 3G, fourth generation wireless (4G) network, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standard, other standards defined by various standard setting organizations, other remote protocols or other data transmission technologies.

[0142] "Machine-readable medium" in this context refers to a component, device, or other tangible medium capable of temporarily or permanently storing instructions and data, and may include, but is not limited to, random access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical media, magnetic media, cache memory, other types of storage devices (e.g., erasable programmable read-only memory (EPROM)), and / or any suitable combination thereof. The term "machine-readable medium" should be deemed to include a single medium or multiple media capable of storing instructions (e.g., centralized or distributed databases or associated caches and servers). The term "machine-readable medium" should also be deemed to include any medium or combination of multiple media capable of storing instructions (e.g., code) for machine execution, such that the instructions, when executed by one or more processors of the machine, cause the machine to perform any one or more of the methods described herein. Thus, "machine-readable medium" refers to a single storage device or device as well as a "cloud-based" storage system or storage network comprising multiple storage devices or devices. The term "machine-readable medium" does not include the signal itself.

[0143] A "component" in this context refers to a logic or device, physical entity having boundaries defined by function or subroutine calls, branch points, APIs, or other techniques that provide partitioning or modularization of a particular processing or control function. A component can be combined with other components via its interface to perform a machine process. A component can be a packaged functional hardware unit designed for use with other components, and is part of a program that generally performs a specific function among related functions. A component can constitute a software component (e.g., code implemented on a machine-readable medium) or a hardware component.

[0144] A "hardware component" is a tangible unit that can perform certain operations and can be configured or arranged in a physical manner. In various example embodiments, one or more computer systems (e.g., an independent computer system, a client computer system, or a server computer system) or one or more hardware components of a computer system (e.g., a processor or a processor group) can be configured by software (e.g., an application or an application part) to be a hardware component that operates to perform certain operations as described herein. The hardware component can also be implemented mechanically, electronically, or any suitable combination thereof. For example, a hardware component may include a dedicated circuit or logic that is permanently configured to perform certain operations. A hardware component may be a dedicated processor, such as a field programmable gate array (FPGA) or an ASIC. A hardware component may also include a programmable logic or circuit system that is temporarily configured to perform certain operations by software. For example, a hardware component may include software executed by a general-purpose processor or other programmable processor.

[0145] Once configured by such software, the hardware component becomes a specific machine (or specific component of a machine) that is uniquely customized to perform the configured functions and is no longer a general-purpose processor. It should be understood that it may be decided for cost and time considerations whether to mechanically implement the hardware component in a dedicated and permanently configured circuit system or in a temporarily configured (e.g., configured by software) circuit system. Therefore, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity, that is, an entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate in a certain manner or perform certain operations described herein.

[0146] Considering embodiments where hardware components are temporarily configured (e.g., programmed), it is not necessary to configure or instantiate each of the hardware components at any one time in a timely manner. For example, where the hardware components include a general-purpose processor that is configured by software to be a special-purpose processor, the general-purpose processor can be configured to be a different special-purpose processor (e.g., including different hardware components) at different times. Software accordingly configures one or more specific processors to, for example, constitute a specific hardware component at one time and to constitute different hardware components at different times.

[0147] Hardware components can provide information to other hardware components and receive information from other hardware components. Therefore, the described hardware components can be considered to be communicatively coupled. In the case of multiple hardware components simultaneously, communication can be realized by signal transmission between two or more hardware components in the hardware components (for example, by appropriate circuits and buses). In the embodiment in which multiple hardware components are configured or instantiated at different times, the communication between such hardware components can be realized, for example, by storing information in a memory structure accessed by multiple hardware components and retrieving information in the memory structure. For example, a hardware component can perform an operation and store the output of the operation in a memory device to which it is communicatively coupled. Then, other hardware components can access the memory device at a subsequent time to retrieve the stored output and process it. The hardware component can also initiate communication with an input device or an output device, and can operate on resources (for example, a collection of information).

[0148] The various operations of the example methods described herein may be performed at least in part by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the related operations. Whether temporarily configured or permanently configured, such a processor may constitute a processor-implemented component that operates to perform one or more operations or functions described herein. As used herein, "processor-implemented components" refer to hardware components implemented using one or more processors. Similarly, the methods described herein may be implemented at least in part by a processor, wherein a specific one or more processors are examples of hardware. For example, at least some of the operations of the method may be performed by one or more processors or processor-implemented components.

[0149] In addition, one or more processors may also operate to support the execution of related operations in a "cloud computing" environment or operate as "software as a service" (SaaS). For example, at least some operations may be performed by a group of computers (as an example of a machine including a processor), where these operations can be accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., application programming interface (API)). The execution of certain operations in the operation may be distributed between processors, not just resident in a single machine, but deployed across several machines. In some example embodiments, a processor or a component implemented by a processor may be located in a single geographic location (e.g., in a home environment, an office environment, or a server cluster). In other example embodiments, a processor or a component implemented by a processor may be distributed across multiple geographic locations.

[0150] A "processor" in this context refers to any circuit or virtual circuit (a physical circuit simulated by logic executed on an actual processor) that manipulates data values ​​according to control signals (e.g., "commands," "opcodes," "machine codes," etc.) and produces corresponding output signals that are applied to operate a machine. For example, a processor may be a CPU, a RISC processor, a CISC processor, a GPU, a DSP, an ASIC, an RFIC, or any combination thereof. A processor may also be a multi-core processor having two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously.

[0151] A "timestamp" in this context refers to a sequence of characters or coded information that identifies when some event occurred, for example, thereby giving a date and time of day, sometimes accurate to a fraction of a second.

Claims

1. A method comprising: causing presentation by a display device of volumetric content comprising a volumetric video, the volumetric video comprising a volumetric representation of one or more elements of a three-dimensional space, such that presenting the volumetric content comprises causing the display device to present the volumetric representation of the one or more elements superimposed on a real-world environment visible to a user of the display device; receiving input indicative of a control operation associated with presentation of the volumetric content; as well as By performing the control operation, the presentation of the volumetric content by the display device is controlled, and controlling the presentation of the volumetric content includes: when the control operation is being performed based on the movement of the user of the display device, the volumetric representation of the one or more elements is presented by the display device from multiple perspectives.

2. The method according to claim 1, further comprising: receiving an edit of the volumetric video; editing the volumetric video based on the editing, the editing of the volumetric video generating an edited volumetric video; as well as The edited volumetric video is caused to be presented by the display device.

3. The method according to claim 2, wherein: Receiving the edits includes receiving input indicating at least one or more of: modification of a content item included in the volumetric video; removal of the content item included in the volumetric video; adding a new content item to the volumetric video; as well as A filter is applied to the content item in the volumetric video or a real-world object visible as part of the presentation of the volumetric content.

4. The method according to claim 3, wherein: The volumetric video is a first volumetric video; Receiving the input indicating adding the new content item to the first volumetric video comprises one of: receiving a selection of the new content item from the library; and A selection of the new content item from the second volumetric video is received.

5. The method according to claim 2, wherein: Editing the volumetric video includes at least one of: modifying a content item included in the volumetric video; removing the content item included in the volumetric video; adding a new content item to the volumetric video; as well as A filter is applied to the content item in the volumetric video or a real-world object visible as part of the presentation of the volumetric content.

6. The method according to claim 2, further comprising: A selection of a content item from the presentation of the volumetric content is detected, wherein the edit corresponds to the content item.

7. The method according to claim 2, further comprising: An editing interface is caused to be displayed within the presentation of the volumetric video, wherein receiving the edit comprises detecting a user interaction with the editing interface.

8. The method according to claim 1, wherein: Receiving the input includes detecting a gesture by a user of the display device.

9. The method according to claim 1, wherein: The operations further include causing display of an interactive interface within the presentation of the volumetric video, the interactive interface comprising an interactive element for triggering the control operation; and Receiving the input includes detecting interaction with the interactive element of the interactive interface by a user of the display device.

10. The method according to claim 1, wherein: The control operation includes one of the following: a stop operation, a pause operation, a fast forward operation, a rewind operation, a zoom-in operation, a zoom-out operation, and a playback speed modification.

11. The method according to claim 1, wherein: Controlling presentation of the volumetric content by the display device according to the control operation comprises one of: stopping presentation of the volumetric video; pausing presentation of the volumetric video; rewinding the presentation of the volumetric video; fast forwarding the presentation of the volumetric video; increasing a zoom level of the presentation of the volumetric video; reducing a zoom level of a presentation of the volumetric video; increasing a playback speed of the presentation of the volumetric video; as well as A playback speed of the presentation of the volumetric video is reduced.

12. A system comprising: one or more hardware processors; as well as At least one memory storing instructions that cause the one or more hardware processors to perform operations comprising: causing presentation by a display device of volumetric content comprising a volumetric video, the volumetric video comprising a volumetric representation of one or more elements of a three-dimensional space, such that presenting the volumetric content comprises causing the display device to present the volumetric representation of the one or more elements superimposed on a real-world environment visible to a user of the display device; receiving input indicative of a control operation associated with presentation of the volumetric content; and By performing the control operation, the presentation of the volumetric content by the display device is controlled, and controlling the presentation of the volumetric content includes: when the control operation is being performed based on the movement of the user of the display device, the volumetric representation of the one or more elements is presented by the display device from multiple perspectives.

13. The system of claim 12, further comprising: receiving an edit of the volumetric video; editing the volumetric video based on the editing, the editing of the volumetric video generating an edited volumetric video; as well as The edited volumetric video is caused to be presented by the display device.

14. The system according to claim 13, wherein: Receiving the edit includes receiving input indicating at least one or more of: modification of a content item included in the volumetric video; removal of the content item included in the volumetric video; adding a new content item to the volumetric video; and A filter is applied to the content item in the volumetric video or a real-world object visible as part of the presentation of the volumetric content.

15. The system of claim 13, wherein: Editing the volumetric video includes at least one of: modifying a content item included in the volumetric video; removing the content item included in the volumetric video; adding a new content item to the volumetric video; as well as A filter is applied to the content item in the volumetric video or a real-world object visible as part of the presentation of the volumetric content.

16. The system of claim 13, further comprising: A selection of a content item from the presentation of the volumetric content is received, wherein the edit corresponds to the content item.

17. The system of claim 12, wherein: Receiving the input includes at least one of: detecting a gesture performed by a user of the display device; detecting interaction by a user of the display device with an interactive element of an interactive interface provided with the presentation of the volumetric video; as well as An input signal is received from a physical input element communicatively coupled to the display device.

18. The system of claim 12, wherein: Controlling presentation of the volumetric content by the display device according to the control operation comprises one of: stopping presentation of the volumetric video; pausing presentation of the volumetric video; rewinding the presentation of the volumetric video; fast forwarding the presentation of the volumetric video; increasing a zoom level of the presentation of the volumetric video; reducing a zoom level of a presentation of the volumetric video; increasing a playback speed of the presentation of the volumetric video; as well as A playback speed of the presentation of the volumetric video is reduced.

19. A machine-readable medium storing instructions that, when executed by a computer system, cause the computer system to perform operations comprising: causing presentation by a display device of volumetric content comprising a volumetric video, the volumetric video comprising a volumetric representation of one or more elements of a three-dimensional space, such that presenting the volumetric content comprises causing the display device to present the volumetric representation of the one or more elements superimposed on a real-world environment visible to a user of the display device; receiving input indicative of a control operation associated with presentation of the volumetric content; as well as By performing the control operation, the presentation of the volumetric content by the display device is controlled, and controlling the presentation of the volumetric content includes: when the control operation is being performed based on the movement of the user of the display device, the volumetric representation of the one or more elements is presented by the display device from multiple perspectives.

20. The machine-readable medium of claim 19, further comprising: receiving an edit of the volumetric video; editing the volumetric video based on the editing, the editing of the volumetric video generating an edited volumetric video; as well as The edited volumetric video is caused to be presented by the display device.

Images