Mixing and matching volumetric content to obtain new augmented reality experience

Abstract

A volumetric content presentation system includes a head-mounted display device including one or more processors and a memory storing instructions that, when executed by the one or more processors, configure the display device to access an AR content item corresponding to a real-world object or a virtual object, the AR content items are mixed and matched and volume content is presented that includes the mixed and matched AR content items superimposed on a real-world environment to create a new AR scene that a user may experience.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This patent application claims the benefit of priority to U.S. application No. 18 / 058,175, filed on November 22, 2022, which claims the benefit of U.S. Provisional Patent Application No. 63 / 402,604, filed on August 31, 2022, entitled “MIXING AND MATCHING VOLUMETRIC CONTENTS FOR NEW AUGMENTED REALITY EXPERIENCES,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates generally to mobile and wearable computing technologies. In particular, example implementations of the present disclosure present systems, methods, and user interfaces for mixing and matching volumetric content items to create new augmented reality experiences. Background Art

[0004] An augmented reality (AR) experience includes applying virtual content to a real-world environment, whether by presenting the virtual content through a transparent or translucent display through which the real-world environment can be seen, or by augmenting image data to include virtual content superimposed on the real-world environment depicted therein. The virtual content may include one or more AR content items. AR content items may include audio content, visual content, visual effects, or objects (e.g., virtual objects such as 3D renderings, images, videos, text, etc.) generated for display to appear as part of and / or superimposed on the surrounding environment. A device that supports an AR experience in any of these methods is referred to herein as a "display device."

[0005] For some example AR devices, audio and visual content or visual effects are applied to media data such as live image streams. Other example AR devices include head-mounted devices that can be implemented with transparent or translucent displays, through which users of the head-mounted devices can view the surrounding environment. Such devices enable users to view the surrounding environment through a transparent or translucent display, and can also see objects (e.g., virtual objects such as 3D renderings, images, videos, text, etc.) generated for display to appear as part of the surrounding environment and / or superimposed on the surrounding environment to provide users with an augmented reality experience. Users of head-mounted devices can access and use computer software applications to perform various tasks or participate in entertainment activities. In order to use the computer software application, the user interacts with the 3D user interface provided by the head-mounted device. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] In the accompanying drawings (not necessarily drawn to scale), similar reference numerals may describe similar components in different views. To easily identify the discussion of any particular element or action, the most significant digit or digits in a reference numeral refer to the figure number in which the element is first introduced. Some non-limiting examples are shown in the figures of the accompanying drawings, in which:

[0007] Figure 1 is a diagrammatic representation of a networked environment in which a volumetric content rendering system may be deployed, according to some example implementations.

[0008] Figure 2A is a perspective view of a head mounted device according to some examples.

[0009] Figure 2B According to some examples Figure 2A Another view of the headset.

[0010] Figure 3 is a diagram showing a method according to some example embodiments including Figure 1 A block diagram of the networking system with details of the head mounted device.

[0011] Figure 4 is a diagrammatic representation of a volumetric content rendering system having both client-side and server-side functionality according to some examples.

[0012] Figure 5 is a diagrammatic representation of data structures maintained in a database according to some examples.

[0013] Fig. 6A is a conceptual diagram illustrating the use of volumetric capture in creating volumetric content according to an example implementation.

[0014] Figure 6B is a conceptual diagram illustrating example volumetric content items generated based on volumetric capture according to example implementations.

[0015] Figure 6C is a conceptual diagram illustrating an example for presenting volumetric content items according to example implementations.

[0016] Fig.6D is a conceptual diagram illustrating an example for presenting volumetric content items according to example implementations.

[0017] Fig. 6E is a conceptual diagram illustrating an example for presenting volumetric content items according to example implementations.

[0018] Fig. 6F is a conceptual diagram illustrating an example for presenting volumetric content items according to example implementations.

[0019] Figure 6G is a conceptual diagram illustrating an example for presenting volumetric content items according to example implementations.

[0020] Figure 6H is a conceptual diagram illustrating an example for presenting volumetric content items according to example implementations.

[0021] Fig. 7A is a flow chart illustrating example operation of a volumetric content rendering system in executing a method for rendering volumetric content according to some examples.

[0022] Figure 7B is a flow chart illustrating example operation of a volumetric content rendering system in executing a method for rendering volumetric content according to some examples.

[0023] Figure 8 is a block diagram illustrating a representative software architecture that may be used in conjunction with the various hardware architectures described herein, according to an example implementation.

[0024] Fig. 9 is a block diagram illustrating components of a machine capable of reading instructions from a machine-readable medium (eg, a machine-readable storage medium) and performing any one or more of the methodologies discussed herein, according to example embodiments. DETAILED DESCRIPTION

[0025] The following description includes systems, methods, techniques, instruction sequences and computer program products that implement the illustrative embodiments of the present disclosure. In the following description, for the purpose of explanation, many specific details are set forth to provide an understanding of the various embodiments of the subject of the present invention. However, it will be apparent to those skilled in the art that the embodiments of the subject of the present invention can be put into practice without these specific details. Usually, it is not necessary to show in detail known instruction examples, protocols, structures and techniques.

[0026] Volumetric content is an example of an augmented reality (AR) experience. Volumetric content may include volumetric video and images of a three-dimensional space captured in three dimensions (as well as audio signals recorded with the volumetric video and images). Recording volumetric content includes capturing elements of the three-dimensional space, such as objects and humans, in volume using a combination of a camera and a sensor. Volumetric content includes a volumetric representation of one or more three-dimensional elements (e.g., objects or people) of the three-dimensional space. The volumetric representation of an element (e.g., an AR content item) refers to a visual representation of a three-dimensional element in three dimensions. The presentation of volumetric content may include displaying one or more AR content items superimposed on a real-world space, which may be the same as the three-dimensional space in which the volumetric video was captured, or may be a different space. Presenting volumetric content may include displaying one or more content items in motion, displaying one or more content items performing movement or other actions, displaying one or more content items in a static position, or a combination thereof. The content item may be displayed for the duration of the presentation of the volumetric content or a portion thereof.

[0027] The presentation of volumetric content may include tracking the position and movement of a user in their physical real-world environment, and using the tracked position and movement of the user to allow the user to move around in and interact with the presentation of the volumetric content. Thus, the presentation of volumetric content may include displaying content items from multiple perspectives based on changes in the user's movement and position. In this way, the presentation of volumetric content provides an immersive AR experience to the user. Conventional volumetric content systems lack mechanisms that can create new volumetric content using existing elements of volumetric content.

[0028] Aspects of the present disclosure include systems, methods, techniques, instruction sequences, and computer program product systems to enable a user to mix and match volumetric content items from existing volumetric content to create a new augmented reality experience. That is, a volumetric content rendering system can obtain one or more volumetric content items from an existing volumetric video or image and render them as an overlay on a real-world environment as part of a new augmented reality experience. In addition, the volumetric content rendering system can create new volumetric content based on a combination of volumetric content items obtained from existing volumetric content.

[0029] In the example, volumetric capture technology is used to generate volumetric content depicting a motion scene of a person sitting in a chair. A machine learning model is used to segment people and chairs from image data generated during volumetric capture. That is, the chair is segmented out using a first model and the person is segmented out using a second model. Together, these two models create a volumetric scene of a person sitting in a chair in AR. The volumetric content presentation system described herein enables a user to decompose an AR scene such as building blocks (e.g., a person and a chair) and create a new augmented experience. In a first illustrative example, a user can create an augmented reality experience in which a first volumetric content item corresponding to a chair (e.g., a volumetric representation of a chair) and a second volumetric content item corresponding to a person are presented as an overlay on a real-world environment, which can correspond to the location where the volumetric capture is performed or any other location such as a park, a table, or the top of a building. This enables users to participate in an augmented reality experience anywhere.

[0030] In a second illustrative example, a user can create an AR experience in which a volumetric representation of a person is superimposed on a real-world chair. The volumetric content rendering system can use object-based triggers to render a volumetric representation of the person superimposed on a chair. The user can place the volumetric representation of the person on the same real-world chair that captured the volumetric scene or on any chair, allowing for flexible and creative use of the volumetric representation of the person. The AR content seamlessly and semantically connected to the physical world in this way highlights the experience and makes it more situational. Placing the volumetric representation of the person sitting on the volumetric representation of the chair at a random location loses the original context of the volumetric content, but placing the volumetric representation of the person on the real-world chair adds context again. In this way, the volumetric content rendering system provides people with a variety of options to use the scene captured volumetrically in any way they want.

[0031] In a third illustrative example, a user can use the volumetric representation of a chair as a collectible or create new content using it. A real person can pose next to the volumetric representation of the chair and create a new experience that is different from the experience depicted in the original volumetric content.

[0032] In a fourth illustrative example, a user can place a volumetric representation of a person on a table, and a volumetric representation of a chair in a garden. In this way, the AR components (i.e., the volumetric representations of the person and the chair) can be separated and placed wherever the user wants. Taking pictures next to the AR scene can create new memories and fill in the gaps even when the real person is not present (as if they were there). For example, in a group photo with a missing person, the user can add the volumetric representation of the user to complete the group photo.

[0033] As used herein, the phrase "augmented reality experience" includes or refers to various image processing operations corresponding to image modification, filtering, media overlay, transformation, etc., as further described herein. In some examples, these image processing operations provide an interactive experience of a real-world environment, wherein objects, surfaces, backgrounds, lighting, etc. in the real world are enhanced by computer-generated perceptual information. In this context, an "augmented reality effect" includes a collection of data, parameters, and other assets for applying a selected augmented reality experience to an image or video feed. In some examples, augmented reality effects are provided by Snap Inc. under the registered trademark LENSES.

[0034] In some examples, a 3D user interface is used to enable user interaction with a software application executed on an AR device. The 3D user interface includes virtual objects displayed to the user by the AR device in a 3D rendering displayed to the user. In the case of AR, when wearing the AR device, the user perceives the virtual objects as objects within the user's field of view superimposed on the real world. In the case of VR, when wearing the AR device, the user perceives the virtual objects as objects within the virtual world as viewed by the user. In order to allow the user to interact with the virtual objects, the AR device detects the user's hand positions and movements, and uses these hand positions and movements to determine the user's intent in manipulating the virtual objects.

[0035] The generation of 3D user interfaces and the detection of user interactions with virtual objects may also include: detection of real-world objects (e.g., faces, hands, bodies, cats, dogs, surfaces, objects); tracking these real-world objects in video frames as they leave, enter, and move around the field of view; and modifying or transforming such real-world objects when they are tracked. In various examples, different methods for detecting real-world objects and implementing such transformations may be used. For example, some examples may involve generating a 3D mesh model of one or more real-world objects, and implementing the transformation using the transformation of the model and animated textures within a video frame. In other examples, tracking of points on real-world objects may be used to place an image or texture at the tracked location, which image and texture may be two-dimensional or three-dimensional. In yet another example, neural network analysis of video frames may be used to place images, models, or textures in content (e.g., images or frames of a video). Therefore, AR effect data may include both: images, models, and textures used to create transformations in content, and additional modeling and analysis information for implementing such transformations by detecting, tracking, and placing real-world objects.

[0036] Figure 11 is a block diagram illustrating an example volumetric content rendering system 100 for rendering volumetric content. The volumetric content rendering system 100 includes a user device 102. The user device 102 hosts a plurality of applications including a client application 104. Each client application 104 is communicatively coupled to a rendering server system 108 via a network 106 (e.g., the Internet). In an example, the user device 102 is a wearable device (e.g., smart glasses) worn by a user 124 that includes a camera and optical elements including a transparent display through which the user 124 can view a real-world environment.

[0037] The client application 104 can communicate and exchange data with another client application 104 and a presence server system 108 via a network 106. The data exchanged between client applications 104 and between another client application 104 and the presence server system 108 includes functions (e.g., commands to invoke functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0038] The presentation server system 108 provides server-side functionality to the client application 104 via the network 106. Although certain functionality of the volumetric content presentation system 100 is described herein as being performed by the client application 104 or by the presentation server system 108, the location of certain functionality within the client application 104 or within the presentation server 108 is a design choice. For example, it may be technically preferable to initially deploy certain techniques and functionality within the presentation server system 108 but later migrate the techniques and functionality to the client application 104 where the client device (102) has sufficient processing power.

[0039] The presentation server system 108 supports various services and operations provided to the client applications 104. Such operations include sending data to the client applications 104, receiving data from the client applications 104, and processing data generated by the client applications 104. As examples, the data may include volumetric content (e.g., volumetric video), message content, device information, geolocation information, media annotations and overlays, message content persistence conditions, social network information, and live event information. The exchange of data within the volumetric content presentation system 100 is invoked and controlled by the functionality available via a user interface (UI) and the client applications 104.

[0040] Turning now specifically to the presentation server system 108, an application program interface (API) server 110 is coupled to and provides a programming interface to an application server 112. The application server 112 is communicatively coupled to a database server 114, which facilitates access to a database 116 in which data associated with messages processed by the application server 112 is stored.

[0041] The application program interface (API) server 110 receives and sends message data (e.g., commands and message payloads) between the user device 102 and the application server 112. Specifically, the application program interface (API) server 110 provides a set of interfaces (e.g., routines and protocols) that can be called or queried by the client application 104 to invoke the functions of the application server 112. The API application program interface (API) server 110 exposes various functions supported by the application server 112, including: account registration, login functionality, sending messages from a client application 104 to another client application 104 via the application server 112, sending media files (e.g., volumetric videos) to a client application 104, setting a collection of media data (e.g., a story), retrieving a friend list of a user 124 of the user device 102, retrieving such a collection, retrieving messages and content, adding friends to and removing friends from a social graph, locating friends within a social graph, and opening application events (e.g., related to a client application 104).

[0042] The application server 112 hosts a number of applications and subsystems, including a rendering server 118, an image processing server 120, and a social networking server 122. The application server 112 is generally responsible for managing volumetric content and facilitating the rendering of volumetric content by the user device 102. The image processing server 120 is dedicated to performing various image processing operations, typically on images or videos generated and displayed by the user device 102. The application server 112 and the image processing server 120 can work together to provide one or more AR experiences to the user 124. For example, the application server 112 and the image processing server 120 can work together to support the rendering of volumetric content by the client device 102. More details on the rendering of volumetric content are discussed below.

[0043] The social network server 122 supports various social network functions and services and makes these functions and services available to the application server 112. To this end, the social network server 122 maintains and accesses an entity graph within the database 116. Examples of functions and services supported by the social network server 122 include identifying other users of the volumetric content presentation system 100 who have a relationship with a particular user or who the particular user is "following," and identifying interests and other entities of a particular user.

[0044] The application server 112 is communicatively coupled to a database server 114 , which facilitates access to a database 116 in which data associated with content rendered by the rendering server 112 and the image processing server 120 is stored.

[0045] Figure 2A 2 is a diagram showing a wearable device in the example form of glasses 200 according to some example embodiments. Figure 1 The glasses 200 are examples of user devices 102. The glasses 200 are capable of displaying content and are therefore examples of display devices referred to below. In addition, the display capabilities of the glasses 200 support AR experiences, and therefore the glasses 200 are examples of AR devices. As mentioned above, the AR experience includes: applying virtual content to the real world environment, whether by presenting the virtual content through a transparent display through which the real world environment can be seen, or by enhancing the image data to include virtual content superimposed on the real world environment depicted therein. More details about AR content items are discussed below.

[0046] The eyeglasses 200 may include a frame 202 made of any suitable material, such as plastic or metal, including any suitable shape memory alloy. In one or more examples, the frame 202 includes a first optical element or left optical element holder 204 (e.g., a display or lens holder) and a second optical element or right optical element holder 206 connected by a bridge 212. The first optical element or left optical element 208 and the second optical element or right optical element 210 may be disposed within the respective left optical element holder 204 and right optical element holder 206. The right optical element 210 and the left optical element 208 may be lenses, displays, display components, or combinations of the foregoing.

[0047] For example, the glasses 200 may include an inward-facing display component that includes an integrated near-eye display mechanism that is capable of, for example, displaying to the user a preview image of visual media captured by the left camera 214 and / or the right camera 216 of the glasses 200. The integrated near-eye display mechanism also facilitates the AR experience, for example, by displaying AR content to present volumetric video so that the volumetric video is superimposed on the real-world environment visible through the left optical element 208 and the right optical element 210. AR content items may include audio content, visual content, or visual effects. Examples of audio and video content include pictures, text, logos, animations, and sound effects. Audio and video content or visual effects can be applied to media data such as a live image stream.

[0048] The frame 202 additionally includes a left arm or temple piece 222 coupled to a corresponding left optic holder 204 and a right arm or temple piece 224 coupled to a right optic holder 206 by any suitable means, such as one or more hinges 228. In some examples, the frame 202 can be formed from a single piece of material to have a uniform or unitary structure.

[0049] The glasses 200 may include a computing device such as a computer 220, which may be of any suitable type to be carried by the frame 202, and in one or more examples, may be of a suitable size and shape to be partially disposed in one of the left temple piece 222 or the right temple piece 224. The computer 220 may include one or more processors with memory, wireless communication circuitry, and a power source. The computer 220 includes low-power circuits, high-speed circuits, and a display processor. Various other examples may include these elements in different configurations or integrated together in different ways.

[0050] The computer 220 additionally includes a battery 218 or other suitable portable power source. In some examples, the battery 218 is disposed in the left temple piece 222 and is electrically coupled to the computer 220 disposed in the right temple piece 224. The glasses 200 may include a connector or port (not shown) suitable for charging the battery 218, a wireless receiver, transmitter or transceiver (not shown), or a combination of such devices.

[0051] The glasses 200 include a first camera or left camera 214 and a second camera or right camera 216. Although two cameras are depicted, other examples contemplate the use of a single or additional (i.e., more than two) cameras. In one or more examples, the glasses 200 include any number of input sensors or other input / output devices in addition to the left camera 214 and the right camera 216. Such sensors or input / output devices may additionally include biometric sensors, position sensors, motion sensors, etc.

[0052] In some examples, left camera 214 and right camera 216 provide volumetric content for use by glasses 200 or application server 112 to extract volumetric content items from a real-world three-dimensional environment.

[0053] The glasses 200 may also include a touchpad 226 mounted to or integrated with one or both of the left temple piece 222 and the right temple piece 224. The touchpad 226 is generally arranged vertically, approximately parallel to the temple of the user in some examples. As used herein, generally vertically aligned means that the touchpad is more vertical than horizontal, although potentially more vertical than the vertical. One or more touchpads 226 provide a device by which the glasses 200 can receive input from the user 124 of the glasses 200.

[0054] In various examples, the glasses 200 may include any number of input sensors or peripherals in addition to the left camera 214 and the right camera 216. Such sensors may include inward-facing video sensors or digital imaging modules, such as those that may be mounted or disposed on an inner surface of the frame 202 or elsewhere to face the camera of the user, and outward-facing video sensors or digital imaging modules, such as those that may be mounted or disposed on an outer surface of the frame 202 or elsewhere to face away from the user. Such sensors, peripherals, or peripherals may additionally include biometric sensors, position sensors, accelerometers, or any other such sensors.

[0055] Figure 2B The glasses 200 are shown from the user's perspective. Figure 2A Several components shown in FIG. Figure 2A As described, Figure 2B The illustrated eyeglasses 200 include a left optical element 208 and a right optical element 210 secured within a left optical element holder 204 and a right optical element holder 206, respectively.

[0056] The glasses 200 include a forward optical assembly 230 including a right projector 232 and a right near-eye display 234 , and a forward optical assembly 238 including a left projector 240 and a left near-eye display 244 .

[0057] In some examples, the near eye display is a waveguide. The waveguide includes a reflective or diffractive structure (e.g., a grating and / or an optical element such as a mirror, lens, or prism). Light 236 emitted by the right projector 232 encounters the diffractive structure of the waveguide of the near eye display 234, which directs the light toward the right eye of the user to provide an image on or in the right optical element 210 superimposed on the view of the real world seen by the user. Similarly, light 242 emitted by the projector 240 encounters the diffractive structure of the waveguide of the near eye display 244, which directs the light toward the left eye of the user to provide an image on or in the left optical element 208 superimposed on the view of the real world seen by the user. The combination of the GPU, the forward optical assembly 230, the left optical element 208, and the right optical element 210 provides an optical engine of the glasses 200. The glasses 200 use the optical engine to generate an overlay of the user's real world view, including displaying a volumetric user interface to the user of the glasses 200.

[0058] However, it should be understood that other display technologies or configurations may be utilized within the optical engine to display images to the user in the user's volumetric content 622. For example, instead of the projector 232 and waveguide, an LCD, LED, or other display panel or surface may be provided.

[0059] In use, information, various volumetric content, and user interfaces will be presented on the near-eye display to the user of the glasses 200. The user can then interact with the glasses 200 using the touchpad 226, voice input or touch input, and / or hand motions, positions, and orientations detected by the glasses 200, as described in more detail herein.

[0060] Figure 3 is a block diagram showing a networked system 300 including details of the glasses 200 according to some examples. The networked system 300 includes the glasses 200, a client device 328, and a server system 332. The client device 328 may be a smartphone, a tablet, a slate phone, a laptop, an access point, or any other such device capable of connecting to the glasses 200 using a low power wireless connection 336 and / or a high speed wireless connection 334. The client device 328 is connected to the server system 332 via a network 330. The network 330 may include any combination of wired and wireless connections. The server system 332 may be one or more computing devices that are part of a service or network computing system. The client device 328 and any elements of the server system 332 and the network 330 may be connected using a wireless network, respectively. Figure 8 and Figure 8 The details of the software architecture 802 or machine 900 described in the embodiment of the present invention are implemented.

[0061] The glasses 200 include a data processor 302, a display 310, one or more cameras 308, and additional input / output elements 316. The input / output elements 316 may include a microphone, an audio speaker, a biometric sensor, an additional sensor, or an additional display element integrated with the data processor 302. Fig. 7A and Figure 8 Examples of input / output elements 316 are discussed further. For example, input / output elements 316 may include any of I / O components 810 including an output component, a motion component 930, and the like. Figure 2B Examples of display 310 are discussed in . In the particular examples described herein, display 310 includes displays for the left and right eyes of a user.

[0062] Data processor 302 includes image processor 306 (eg, video processor), GPU and display driver 338, tracking module 340, interface 312, low power circuitry 304, and high speed circuitry 320. Components of data processor 302 are interconnected by bus 342.

[0063] The interface 312 refers to any source of user commands provided to the data processor 302. In one or more examples, the interface 312 is a physical button that, when pressed, sends a user input signal from the interface 312 to the low-power processor 314. The low-power processor 314 can process the pressing of such a button and then releasing it immediately as a request to capture a single image, and vice versa. The low-power processor 314 can process the pressing of such a button for a first period of time as a request to capture video data when the button is pressed and stop video capture when the button is released, wherein the video captured when the button is pressed is stored as a single video file. Alternatively, pressing the button for a long period of time can capture a still image. In some examples, the interface 312 can be any mechanical switch or physical interface capable of accepting user input related to requesting data from the camera 308. In other examples, the interface 312 can have a software component, or can be associated with a command received wirelessly from another source, such as from the client device 328.

[0064] Image processor 306 includes circuitry that receives signals from camera 308 and processes those signals from camera 308 into a format suitable for storage in memory 324 or transmission to client device 328. In one or more examples, image processor 306 (e.g., a video processor) includes a microprocessor integrated circuit (IC) customized for processing sensor data from camera 308 and volatile memory used by the microprocessor in operation.

[0065] The low power circuit system 304 includes a low power processor 314 and a low power wireless circuit system 318. These elements of the low power circuit system 304 can be implemented as separate elements, or can be implemented on a single IC as part of a single system on a chip. The low power processor 314 includes logic for managing other elements of the glasses 200. As described above, for example, the low power processor 314 can accept user input signals from the interface 312. The low power processor 314 can also be configured to receive input signals or command communications from a client device 328 via a low power wireless connection 336. The low power wireless circuit system 318 includes circuit elements for implementing a low power wireless communication system. Bluetooth TM Smart, also known as Bluetooth TM Low energy is a standard implementation of a low power wireless communication system that may be used to implement low power wireless circuitry 318. In other examples, other low power communication systems may be used.

[0066] High-speed circuit system 320 includes a high-speed processor 322, a memory 324, and a high-speed wireless circuit system 326. High-speed processor 322 can be any processor capable of managing high-speed communications and operations for any general-purpose computing system for data processor 302. High-speed processor 322 includes processing resources for managing high-speed data transmission on high-speed wireless connection 334 using high-speed wireless circuit system 326. In some examples, high-speed processor 322 executes an operating system such as LINUX operating system or other such operating system. In addition to any other duties, high-speed processor 322 using the software architecture of executing data processor 302 is used to manage data transmission with high-speed wireless circuit system 326. In some examples, high-speed wireless circuit system 326 is configured to implement the communication standard of Institute of Electrical and Electronics Engineers (IEEE) 802.11, which is also referred to as Wi-Fi herein. In other examples, high-speed wireless circuit system 326 can implement other high-speed communication standards.

[0067] Memory 324 includes any storage device capable of storing camera data generated by camera 308 and image processor 306. Although memory 324 is shown as being integrated with high-speed circuitry 320, in other examples, memory 324 may be a separate, stand-alone element of data processor 302. In some such examples, electrical wiring lines may provide a connection from image processor 306 or low-power processor 314 to memory 324 through a chip including high-speed processor 322. In other examples, high-speed processor 322 may manage addressing of memory 324 so that low-power processor 314 will initiate high-speed processor 322 whenever a read or write operation involving memory 324 is desired.

[0068] The tracking module 340 estimates the pose of the glasses 200. For example, the tracking module 340 uses image data and corresponding inertial data from the camera 308 and the positioning component 934 and GPS data to track the position and determine the pose of the glasses 200 relative to a reference frame (e.g., a real-world environment). The tracking module 340 continuously collects and uses updated sensor data describing the movement of the glasses 200 to determine an updated three-dimensional pose of the glasses 200, which indicates changes in relative position and orientation relative to physical objects in the real-world environment. The tracking module 340 allows the glasses 200 to visually place virtual objects relative to physical objects within the user's field of view via the display 310.

[0069] The GPU and display driver 338 may use the pose of the glasses 200 to generate frames of virtual content or other content to be presented on the display 310 when the glasses 200 are operating in a conventional augmented reality mode. In this mode, the GPU and display driver 338 generate updated frames of virtual content based on the updated three-dimensional pose of the glasses 200, which reflects changes in the user's position and orientation relative to physical objects in the user's real-world environment.

[0070] One or more functions or operations described herein may also be performed in an application resident on the glasses 200 or on the client device 328 or on a remote server. The glasses 200 may be a stand-alone client device capable of independent operation, or may be a companion device that works with a host device to offload intensive processing and / or exchange data with the presentation server system 108 over the network 106. The glasses 200 may also be communicatively coupled to a companion device, such as a smartwatch, and may be configured to exchange data with the companion device. The glasses 200 may also include various components common to mobile electronic devices such as smart glasses or smart phones (e.g., including a display controller for controlling the display of visual media on a display mechanism incorporated into the device).

[0071] Figure 4 1 is a block diagram showing further details regarding a volumetric content rendering system 100 according to some examples. Specifically, the volumetric content rendering system 100 is shown to include a client application 104 and an application server 112. The volumetric content rendering system 100 includes a number of subsystems supported on the client side by the client application 104 and on the server side by the application server 112. Example subsystems are discussed below.

[0072] The image processing system 402 provides various functions that enable a user to capture and enhance (eg, annotate or otherwise modify or edit) volumetric content or volumetric content items stored in the database 116 , the user device 102 .

[0073] The camera system 404 includes control software (e.g., in a camera application) that interacts with and controls the hardware camera hardware of the user device 102 (e.g., directly or via operating system control) to modify and enhance real-time images captured and displayed via the client application 104.

[0074] The enhancement system 406 provides various functions that enable a user to enhance (e.g., annotate or otherwise modify or edit) media content. For example, the enhancement system 406 provides functions related to the generation, publication, and application of enhancement data, such as media overlays (e.g., image filters) to volumetric content. Media overlays can include audio and visual content and visual effects. Examples of audio and video content include pictures, text, logos, animations, and sound effects. Examples of visual effects include color overlays. Audio and visual content or visual effects can be applied to media content items (e.g., photos) at the user device 102. For example, a media overlay can include text or images that can be superimposed on top of a photo taken by the user device 102. The enhancement system 406 is operable to supply one or more media overlays to the client application 104 based on the geographic location of the user device 102 or based on other information, such as social network information of a user of the user device 102. The media overlays can be stored in the database 116 and accessed through the database server 114.

[0075] A filter is an example of a media overlay that is displayed as an overlay on an image or video during presentation to a user. Filters can be of various types, including user-selected filters from a set of filters presented to the user by client application 104. Other types of filters include geolocation filters (also referred to as geofilters), which can be presented to a user based on a geographic location. For example, based on geolocation information determined by a global positioning system (GPS) unit of user device 102, a geolocation filter specific to a nearby or special location can be presented by client application 104 within a user interface.

[0076] Another type of filter may be a data filter that is selectively presented to the user by the client application 104 based on other input or information collected by the user device 102. Examples of data filters include the current temperature at a particular location, the current speed at which the sending user is traveling, the battery life of the user device 102, or the current time.

[0077] Volumetric content items are another example of a media overlay. Volumetric content items may be real-time special effects and / or sounds that can be added to images or videos, including volumetric images and videos.

[0078] In general, volumetric content items, overlays, image transforms, images, and similar terms refer to modifications that can be applied to image data (e.g., video or images) that include volumetric content. This includes real-time modifications that modify what a user sees in his or her volumetric content 622, or modify images as they are captured using a device sensor (e.g., one or more cameras) of a user device 102, and then displayed by the user device 102 (e.g., an embedded display of the user device 102) with the modifications. This also includes modifications to stored content, such as modifications to volumetric videos in a library or collection that can be modified. For example, in a user device 102 that accesses multiple volumetric content items, a user can use a single volumetric video with multiple volumetric content items to see how different volumetric content items will modify the stored content. For example, by selecting different volumetric content items for the content, multiple volumetric content items that apply different pseudo-random motion models can be applied to the same content. Similarly, real-time video capture can be used with the modifications shown to show how the video image currently captured by the sensor of the user device 102 will modify the captured data. Such data may simply be displayed on the screen without being stored in memory, or content captured by the device sensors may be recorded and stored in memory with or without modification (or both). In some systems, a preview feature may show how different volumetric content items will appear simultaneously within different windows of the display. For example, this may enable multiple windows with different pseudo-random animations to be viewed simultaneously on the display.

[0079] Thus, data and various systems using volumetric content items or other such augmentation systems that modify content using augmentation data may involve: detection of objects (e.g., faces, hands, bodies, cats, dogs, surfaces, objects, etc.); tracking of such objects as they leave, enter, and move around the field of view in a video frame; and modifying or transforming such objects as they are tracked. In various examples, different methods for implementing such transformations may be used. Some examples may involve generating a three-dimensional mesh model of one or more objects, and using transformations of the models and animated textures within a video to implement the transformations. In other examples, points on tracked objects may be used to place images or textures (which may be two-dimensional or three-dimensional) at tracked locations. In yet another example, neural network analysis of video frames may be used to place images, models, or textures in content (e.g., frames of an image or video). Thus, volumetric content items refer both to images, models, and textures used to create transformations in content, and to additional modeling and analysis information required to implement such transformations using object detection, tracking, and placement.

[0080] Real-time video processing can be performed with any kind of video data (e.g., video streams, video files, volumetric video, etc.) stored in the memory of any kind of computerized system. For example, a user can load a video file and save it in the memory of the device, or a sensor of the device can be used to generate a video stream. In addition, any object can be processed using computer animation models, such as human faces and parts of human bodies, animals, or non-living things such as chairs, cars, or other objects.

[0081] In some examples, in the case of selecting a specific modification together with the content to be enhanced (e.g., edited), the elements to be transformed are identified by a computing device, and then if the elements to be transformed are present in the frame of the video, the elements to be transformed are detected and tracked. The elements of the object are modified according to the request for modification, thereby converting the frame of the video stream. For different types of transformations, the transformation of the frame of the video stream can be performed by different methods. For example, for frame transformations that mainly involve changing the form of the elements of the object, characteristic points for each element of the object are calculated (e.g., using an active shape model (ASM) or other known methods). Then, a grid based on characteristic points is generated for each element of at least one element of the object. The grid can be used for subsequent stages of tracking the elements of the object in the video stream. During the tracking process, the grid mentioned for each element is aligned with the position of each element. Then, additional points are generated on the grid. Based on the modification request, a set of first points are generated for each element, and a set of second points are generated for each element based on the set of first points and the modification request. Then, the frame of the video stream can be transformed by modifying the elements of the object based on the set of first points and the set of second points and the grid. In such methods, the background of the modified object may also be changed or distorted by tracking and modifying the background.

[0082] In some examples, a transformation that changes some areas of an object using elements of an object can be performed by calculating characteristic points of each element of the object and generating a grid based on the calculated characteristic points. Points are generated on the grid, and then various areas are generated based on these points. Then, the elements of the object are tracked by aligning the area of ​​each element with the position of each element of at least one element, and the properties of the area can be modified based on the modification request, thereby transforming the frame of the video stream. Depending on the specific modification request, the properties of the mentioned area can be transformed in different ways. Such modifications may involve: changing the color of the area; removing at least some parts of the area from the frame of the video stream; including one or more new objects in the area based on the modification request; and modifying or distorting the elements of the area or object. In various examples, any combination of such modifications or other similar modifications can be used. For certain models to be animated, some characteristic points can be selected as control points for determining the entire state space of options for model animation.

[0083] In some examples of computer animation models that use face detection to transform image data, faces are detected on an image using a specific face detection algorithm (e.g., Viola-Jones). An active shape model (ASM) algorithm is then applied to the face region of the image to detect facial feature reference points.

[0084] Other methods and algorithms suitable for face detection may be used. For example, in some examples, features are located using landmarks that represent distinguishable points that are present in most of the images considered. For example, for facial landmarks, the location of the left eye pupil may be used. If the initial landmarks are not recognizable (e.g., if the person has an eye mask), secondary landmarks may be used. This landmark recognition procedure may be used for any such object. In some examples, a set of landmarks forms a shape. The shape may be represented as a vector using the coordinates of the points in the shape. A shape is aligned to another shape using a similarity transformation (allowing for translation, scaling, and rotation) that minimizes the average Euclidean distance between the shape points. The average shape is the average of the aligned training shapes.

[0085] In some examples, the search for landmarks begins with an average shape aligned with the face position and size determined by a global face detector. Such a search then repeats the steps of proposing tentative shapes by adjusting the positions of shape points by template matching the image texture around each point, and then conforming the tentative shapes to a global shape model until convergence occurs. In some systems, individual template matches are unreliable, and the shape model aggregates the results of weak template matches to form a stronger overall classifier. The entire search is repeated at each level of the image pyramid from coarse to fine resolution.

[0086] The augmentation system 406 may capture an image or video stream on a client device (e.g., user device 102) and perform complex image manipulations locally on the user device 102 while maintaining an appropriate user experience, computational time, and power consumption. Complex image manipulations may include size and shape changes, emotion transfer (e.g., changing a face from a frown to a smile), state transfer (e.g., aging a subject, reducing apparent age, changing gender), style transfer, application of graphical elements, and any other suitable image or video manipulation enabled by a machine learning model that has been configured to execute efficiently on the user device 102. In some examples, the augmentation system 406 may use a computer animation model for transforming video and image content, where a neural network operates as part of a client application 104 operating on the user device 102.

[0087] The machine learning model can be trained using labeled or unlabeled training data containing ground-truth information. For example, one or more images depicting parts and / or the whole body of a person can be included in the training data together with ground-truth segmentation information that defines the current positions of the limbs and joints of the user's whole body (e.g., the positions of arms, torso, legs, head, shoulders, etc.). The machine learning model can be applied to a subset of the training data, such as one or more images in the training data, and an estimate or prediction of the ground-truth segmentation information that defines the current positions of the limbs and joints of the user's whole body (e.g., the positions of arms, torso, legs, head, shoulders, etc.) can be generated. The estimate or prediction can be compared with the corresponding ground-truth information to calculate the deviation. The deviation can then be used to update one or more parameters of the machine learning model. After updating one or more parameters, the machine learning model is applied to another subset of the training data, and these operations are repeated until the stopping criteria are reached.

[0088] Machine learning is a field of study that gives computers the ability to learn without being explicitly programmed. Machine learning explores the study and construction of algorithms (also referred to herein as tools) that can learn from existing data and make predictions about new data. Such machine learning tools operate by building models based on example training data in order to make data-driven predictions or decisions that are expressed as outputs or evaluations. Although examples are given for several machine learning tools, the principles given herein can be applied to other machine learning tools.

[0089] In some examples, different machine learning tools can be used. For example, job postings can be classified or scored using logistic regression (LR), naive Bayes, random forest (RF), neural network (NN), matrix factorization, and support vector machine (SVM) tools.

[0090] Two common problems in machine learning are classification problems and regression problems. Classification problems, also known as categorization problems, aim to classify items into one of several categorical values ​​(e.g., is this object an apple or an orange?). Regression algorithms aim to quantify some items (e.g., by providing values ​​as real numbers).

[0091] Machine learning algorithms use features to analyze data to generate estimates. Each of the features is a separate measurable property of the phenomenon being observed. The concept of a feature is related to the concept of explanatory variables used in statistical techniques such as linear regression. Selecting features that are informative, discriminative, and independent is important for the effective operation of MLPs in pattern recognition, classification, and regression. Features can be of different types, such as numeric features, strings, and graphs.

[0092] In one example, features may be of different types and may include one or more of content, concepts, attributes, historical data, and / or user data, just as examples.

[0093] Machine learning algorithms use training data to find correlations between identified features that influence an outcome or evaluation. In some examples, the training data includes labeled data, which is known data about one or more identified features and one or more outcomes (e.g., detecting communication patterns, detecting meaning of a message, generating a summary of a message, detecting action items in a message, detecting urgency in a message, detecting a relationship between a user and a sender, calculating a score attribute, calculating a message score, etc.).

[0094] Using the training data and the identified features, the machine learning tool is trained at the machine learning program training location. The machine learning tool evaluates the values ​​of the features associated with the training data. The result of the training is a trained machine learning program.

[0095] When performing an evaluation using a trained machine learning program, new data is provided as input to the trained machine learning program, and the trained machine learning program generates an evaluation as output.

[0096] Machine learning programs support two types of phases, namely a training phase and a prediction phase. In the training phase, supervised learning, unsupervised learning, or reinforcement learning can be used. For example, a machine learning program (1) receives features (e.g., as structured or labeled data in supervised learning) and / or (2) identifies features in the training data (e.g., unstructured or unlabeled data for unsupervised learning). In the prediction phase, the machine learning program uses the features to analyze the query data to generate a result or prediction, as an example of evaluation.

[0097] In the training phase, feature engineering is used to identify features and can include identifying informative, discriminative, and independent features for machine learning programs to operate effectively in pattern recognition, classification, and regression. In some examples, the training data includes labeled data, which is known data about pre-identified features and one or more outcomes. Each of the features can be a variable or attribute, such as a measurable property of a process, item, system, or phenomenon represented by a data set (e.g., training data).

[0098] During the training phase, the machine learning program uses the training data to find correlations between features that influence the predicted outcome or assessment.

[0099] The machine learning program is trained during a training phase of machine learning program training using the training data and the identified features. The machine learning program evaluates the values ​​of the features associated with the training data. The result of the training is a trained machine learning program (e.g., a trained or learned model).

[0100] In addition, the training phase may involve machine learning, where the training data is structured (e.g., labeled during preprocessing operations), and the trained machine learning program implements a relatively simple neural network that can perform, for example, classification and clustering operations. In other examples, the training phase may involve deep learning, where the training data is unstructured, and the trained machine learning program implements a deep neural network that can perform both feature extraction and classification / clustering operations.

[0101] The neural network generated during the training phase and implemented within the trained machine learning program may include a hierarchical (e.g., layered) organization of neurons. For example, neurons (or nodes) may be hierarchically arranged into multiple layers, including an input layer, an output layer, and multiple hidden layers. Each layer within the neural network may have one or more neurons, each of which is operable to calculate a small function (e.g., an activation function). For example, if the activation function generates a result that exceeds a certain threshold, the output can be passed from the neuron (e.g., a transmitting neuron) to a connected neuron (e.g., a receiving neuron) in a consecutive layer. The connection between neurons also has an associated weight that defines the influence on the input from the transmitting neuron to the receiving neuron.

[0102] In some examples, the neural network can also be one of a variety of different types of neural networks, including, by way of example only, a single-layer feedforward network, an artificial neural network (ANN), a recurrent neural network (RNN), a symmetrically connected neural network, and an unsupervised pre-trained network, a convolutional neural network (CNN), or a recurrent neural network.

[0103] During the prediction phase, the evaluation is performed using a trained machine learning program. The query data is provided as input to the trained machine learning program, and the trained machine learning program generates an evaluation as output in response to receiving the query data.

[0104] Image processing system 402 provides a user-based publishing platform that enables a user to select a geographic location on a map and upload content associated with the selected geographic location. The user can also specify under what circumstances a particular media overlay should be provided to other users. Image processing system 402 generates a media overlay that includes the uploaded content and associates the uploaded content with the selected geographic location.

[0105] The augmented creation system 408 supports the augmented reality developer platform and includes applications for content creators (e.g., artists and developers) to create and publish augmentations (e.g., augmented reality experiences) for client applications 104. The augmented creation system 408 provides content creators with a library of built-in features and tools, including, for example, custom shaders, tracking techniques, and templates.

[0106] In some examples, the enhancement creation system 408 provides a merchant-based publishing platform that enables merchants to select specific enhancements associated with a geographic location through a bidding process. For example, the enhancement creation system 408 associates the media overlay of the highest bidding merchant with the corresponding geographic location for a predefined amount of time.

[0107] The user management system 410 is operationally responsible for managing user data and profiles, and includes a social networking system 412 that maintains information about relationships between users of the volumetric content presentation system 100 .

[0108] The collection management system 414 is operationally responsible for managing collections or collections of media (e.g., collections of text, image video, and audio data). Collections of content (e.g., messages, including images, videos, text, and audio) can be organized into "event libraries" or "event stories." Such collections can be made available for a specified time period (e.g., for the duration of an event related to the content). For example, content related to a concert can be made available as a "story" for the duration of the concert. The collection management system 414 can also be responsible for publishing icons that provide notifications of specific collections to the user interface of the client application 104.

[0109] The collection management system 414 includes a curation interface that allows collection managers to manage and curate collections of specific content. For example, the curation interface enables an event organizer to curate a collection of content related to a specific event (e.g., to remove inappropriate content or redundant messages). In addition, the collection management system 414 uses machine vision (or image recognition technology) and content rules to automatically curate content collections. In some examples, compensation may be paid to users for including user-generated content in a collection. In such cases, the collection management system 414 operates to automatically pay such users for use of their content.

[0110] The client application 104 presents a graphical user interface (e.g., a login page or title screen) for the external resource. During, before, or after presenting the login page or title screen, the client application 104 determines whether the launched external resource has been previously authorized to access the user data of the client application 104. In response to determining that the launched external resource has been previously authorized to access the user data of the client application 104, the client application 104 presents another graphical user interface of the external resource, which includes the functions and features of the external resource. In response to determining that the launched external resource has not been previously authorized to access the user data of the client application 104, after a threshold time period (e.g., 3 seconds) of displaying the login page or title screen of the external resource, the client application 104 slides up (e.g., animating the menu to emerge from the bottom of the screen to the middle or other part of the screen) a menu for authorizing the external resource to access the user data. The menu identifies the type of user data that the external resource will be authorized to use. In response to receiving a user selection of the accept option, the client application 104 adds the external resource to the authorized external resource list and allows the external resource to access the user data from the client application 104. External resources are authorized by the client application 104 to access user data under the OAuth 2 framework.

[0111] The client application 104 controls the type of user data shared with the external resource based on the type of external resource that is authorized. For example, an external resource comprising a full-scale application is provided with access to a first type of user data (e.g., a two-dimensional avatar of a user with or without different avatar characteristics). As another example, an external resource comprising a small-scale version of an application (e.g., a web-based version of the application) is provided with access to a second type of user data (e.g., payment information, a two-dimensional avatar of the user, a three-dimensional avatar of the user, and an avatar with various avatar characteristics). Avatar characteristics include different ways to customize the look and feel of an avatar, such as different poses, facial features, clothing, etc.

[0112] The immersive experience creation system 416 is responsible for creating an immersive AR experience based on the existing volumetric content. In doing so, the immersive experience creation system 416 may segment the volumetric content using one or more known machine learning or artificial intelligence image processing techniques to obtain one or more volumetric content items. The immersive experience creation system 416 presents an interactive interface operable to receive a selection of one or more volumetric content items. In some embodiments, the interactive interface may present volumetric content containing volumetric content items. The volumetric content items may be outlined, highlighted, or otherwise visually distinguished from other elements to indicate that they are available for user selection. The immersive experience creation system 416 receives input indicating a selection of one or more volumetric content items from the interactive interface. The selection of one or more volumetric content items may be used to present one or more volumetric content items superimposed on a second real-world environment, thereby creating an immersive AR experience. More details on creating an AR experience and presenting one or more volumetric content items superimposed on a second real-world environment are discussed below.

[0113] Figure 5 is a diagram illustrating a data structure 500 that may be stored in database 116 of presentation server system 108 according to some examples. Although the contents of database 116 are illustrated as including a plurality of tables, it should be appreciated that data may be stored in other types of data structures (e.g., as an object-oriented database).

[0114] Database 116 includes message data stored in message table 504. For any particular message, the message data includes at least message sender data, message recipient (or receiver) data, and payload. Figure 5 Additional details regarding information that may be included in a message and included within the message data stored in message table 504 are described.

[0115] The entity table 506 stores entity data and is linked (e.g., used as a reference) to the entity graph 508 and the profile data 502. The entities whose records are maintained within the entity table 506 may include individuals, corporate entities, organizations, objects, places, events, etc. Regardless of the entity type, any entity for which the presence server system 108 stores data may be an identifiable entity. Each entity is provided with a unique identifier as well as an entity type identifier (not shown).

[0116] The entity graph 508 stores information about relationships and associations between entities. By way of example only, such relationships may be social, professional (e.g., working at a common company or organization), interest-based, or activity-based. Some relationships between entities may be one-way, such as an individual user subscribing to digital content from a business or publishing user (e.g., a newspaper or other digital media channel or brand). Other relationships may be two-way, such as a "friend" relationship between various users of the volumetric content presentation system 100.

[0117] Certain permissions and relationships may be attached to each relationship, and may also be attached to each direction of the relationship. For example, a bidirectional relationship (e.g., a friend relationship between various users) may include authorization for disclosure of digital content items between the various users, but may impose certain restrictions or filters on the disclosure of such digital content items (e.g., based on content characteristics, location data, or time of day data). Similarly, a subscription relationship between a personal user and a business user may impose varying degrees of restrictions on the disclosure of digital content from the business user to the personal user, and may greatly restrict or prevent the disclosure of digital content from the personal user to the business user. As an example of an entity, a particular user may record certain restrictions in a record for that entity within the entity table 506 (e.g., via privacy settings). Such privacy settings may apply to all types of relationships in the context of the volumetric content presentation system 100, or may selectively apply to only certain types of relationships.

[0118] The profile data 502 stores multiple types of profile data about a particular entity. The profile data 502 can be selectively used and presented to other users of the volumetric content presentation system 100 based on the privacy settings specified by the particular entity. In the case where the entity is a person, the profile data 502 includes, for example, the user name, phone number, address, settings (such as notification and privacy settings), and an avatar representation (or a collection of such avatar representations) selected by the user. The particular user can then selectively include one or more of these avatar representations within the content of messages communicated via the volumetric content presentation system 100, as well as on a map interface displayed to other users by the client application 104. The collection of avatar representations can include a "status avatar" that presents a graphical representation of a status or activity that the user can choose to convey at a particular time.

[0119] Where the entity is a group, the profile data 502 for the group may similarly include one or more avatar representations associated with the group, in addition to the group name, members, and various settings related to the group (eg, notifications).

[0120] Database 116 also stores enhancement data, such as overlays or filters, in enhancement table 510. Enhancement data is associated with and applied to videos (whose data is stored in video table 512) and images (whose data is stored in image table 514).

[0121] In some examples, filters are overlays that are displayed as overlays on an image or video during presentation to a recipient user. Filters can be of various types, including user-selected filters from a set of filters presented to a sending user by client application 104 when the sending user is composing a message. Other types of filters include geolocation filters (also referred to as geofilters), which can be presented to a sending user based on a geographic location. For example, a geolocation filter specific to a nearby or special location can be presented by client application 104 within a user interface based on geographic location information determined by a global positioning system (GPS) unit of user device 102.

[0122] Another type of filter is a data filter, which can be selectively presented to the sending user by the client application 104 based on other input or information collected during the message creation process by the user device 102. Examples of data filters include the current temperature at a particular location, the current speed at which the sending user is traveling, the battery life of the user device 102, or the current time.

[0123] Other augmented data that can be stored in the image table 514 include augmented reality content items (eg, corresponding to an application lens or augmented reality experience). The augmented reality content items can be real-time special effects and sounds that can be added to an image or video.

[0124] As described above, augmented data includes volumetric, augmented reality (AR), virtual reality (VR) and mixed reality (MR) content items, overlays, image transformations, images, and modifications that can be applied to image data (e.g., video or images). This includes real-time modifications that modify the user's field of view that the user can see through the lens of the user device 102 or modify the image when captured using the device sensor (e.g., one or more cameras) in the user device 102, and then display the image with the modification on the screen of the user device 102. This also includes modifications to stored content such as video clips in a collection or group that can be modified. For example, in a user device 102 that can access multiple volumetric content items, a user can use a single video clip with multiple volumetric content items to see how different volumetric content items will modify the stored clip. Similarly, real-time video capture can use modifications to show how the video image currently captured by the sensor of the user device 102 will modify the captured data. Such data can be simply displayed on the screen without being stored in the memory, or the content captured by the device sensor can be recorded and stored in the memory with or without modification (or both). In some systems, the preview feature can show how different augmented reality content items will be displayed in different windows of the display at the same time. For example, this can enable multiple windows with different pseudo-random animations to be viewed on the display at the same time.

[0125] Thus, data and various systems using augmented reality content items or other such transformation systems that modify content using such data may involve: detection of objects (e.g., faces, hands, bodies, cats, dogs, surfaces, objects, etc.); tracking of such objects as they leave, enter, and move around the field of view in a video frame; and modifying or transforming such objects as they are tracked. In various examples, different methods for implementing such transformations may be used. Some examples may involve generating a three-dimensional mesh model of one or more objects, and using transformations of the models and animated textures within a video to implement the transformations. In some examples, tracking of points on an object may be used to place an image or texture (which may be two-dimensional or three-dimensional) at the tracked location. In yet another example, neural network analysis of a video frame may be used to place an image, model, or texture in content (e.g., a frame of an image or video). Thus, augmented reality content items refer both to images, models, and textures used to create transformations in content, and to additional modeling and analysis information required to implement such transformations using object detection, tracking, and placement.

[0126] The story table 516 stores data about a collection of messages and related images, video or audio data that are compiled into a collection (e.g., a story or library). The creation of a particular collection can be initiated by a particular user (e.g., each user for whom a record is maintained in the entity table 506). A user can create a "personal story" in the form of a collection of content that has been created and sent / broadcasted by the user. To this end, the user interface of the client application 104 can include a user-selectable icon to enable the sending user to add specific content to his or her personal story.

[0127] A collection may also constitute a "live story" as a collection of content from multiple users, which is created manually, automatically, or using a combination of manual and automatic techniques. For example, a "live story" may constitute a curated stream of user-submitted content from various locations and events. Users whose client devices have location services enabled and who are at a common location event at a particular time may be presented with the option of contributing content to a particular live story, for example, via the user interface of the client application 104. Live stories may be identified to the user by the client application 104 based on his / her location. The end result is a "live story" told from a community perspective.

[0128] Another type of content collection is called a "location story" that enables users whose user devices 102 are located in a particular geographic location (e.g., on a college or university campus) to contribute to a particular collection. In some examples, contributions to location stories may require secondary authentication to verify that the end user belongs to a particular organization or other entity (e.g., is a student on a university campus).

[0129] As mentioned above, video table 512 stores video data that, in some examples, is associated with messages whose records are maintained within message table 504. Similarly, image table 514 stores image data associated with messages for which message data is stored in entity table 506. Entity table 506 may associate various enhancements from enhancement table 510 with various images and videos stored in image table 514 and video table 512.

[0130] 6A to 6H is a conceptual diagram illustrating presentation of example volumetric content according to some examples. Fig. 6AThe use of volumetric capture in the creation of volumetric content according to an example embodiment is shown. Volumetric content 622 is created based on video and / or images generated by a volumetric recording system 602. The video and / or images capture a first real-world three-dimensional environment 620 from multiple angles. The captured video and / or images are converted into volumetric content 622 using artificial intelligence methods. For some examples, the conversion of the captured video or captured photos into volumetric content 622 can be performed by one or more processors in the volumetric recording system 602 or by at least one of one or more processors in the application server 112, the database server 114, and the user device 102 using artificial intelligence methods. The resulting volumetric content 622 includes a volumetric representation of one or more elements of the first real-world three-dimensional environment 620.

[0131] In some examples, the volume recording system 602 may include a set of cameras arranged in a manner partially or completely surrounding the first real-world three-dimensional environment 620 to capture video or photos of the first real-world three-dimensional environment 620 and multiple elements from various different angles. In some examples, the volume recording system 602 may include a drone that hovers above and flies around the first real-world three-dimensional environment 620 to capture video or photos of the first real-world three-dimensional environment 620. In additional examples, the volume recording system 602 may include a network-connected device (i.e., a mobile phone) or one or more of the user devices 102 that have the ability to capture video or photos.

[0132] exist Fig. 6A In the example shown, a first real-world three-dimensional environment 620 includes a real-world person 608, a real-world basket 604, a real-world basketball 606, and a real-world chair 610. During volumetric capture, the real-world person 608 throws the real-world basketball 606 toward the real-world basket 604, while the real-world chair 610 is stationary. The volumetric recording system 602 can capture a video of the first real-world three-dimensional environment 620. In the captured video, the real-world person 608 throws the real-world basketball 606 toward the real-world basket 604, while the real-world chair 610 is only located in the first real-world three-dimensional environment 620. The volumetric content 622 generated based on the captured video includes volumetric representations of the real-world person 608, the real-world basket 604, the real-world basketball 606, and the real-world chair 610. That is, the volumetric content 622 includes volumetric content items corresponding to the volumetric representations of each of the real-world person 608, the real-world basket 604, the real-world basketball 606, and the real-world chair 610.

[0133] In other words, the volumetric content items included in the volumetric content may correspond to real-world counterparts and may be generated based on the real-world counterparts. These real-world counterparts are not necessarily from Fig. 6A , these real-world counterparts may also come from different volumetric captures performed at different times or locations.

[0134] In some examples, the volumetric content items may also include volumetric content items corresponding to virtual objects. A virtual object may be an imaginary, fictional, or computer-generated object that no longer exists in the real world. For example, a virtual object may be a dinosaur, a fire-breathing dragon, a zombie, or a wizard. Volumetric content items generated based on virtual objects may be stored in a library of volumetric content items (e.g., stored in database 116).

[0135] Reference Figure 6B , volumetric content 622 generated based on a first real-world three-dimensional environment 620 is shown. Volumetric content 622 includes an AR person 614, an AR hoop 616, an AR basketball 612, and an AR chair 618. The AR person is generated based on the real-world person 608, the AR hoop 616 is generated based on the real-world hoop 604, the AR basketball 612 is generated based on the real-world basketball 606, and the AR chair 618 is generated based on the real-world chair 610. Figure 6B In the example presentation shown in , when presented by user device 102 (e.g., user device 102) of user 124 (e.g., user 124), AR person 614 may appear to be performing a throwing motion because AR person 614 is generated based on one or more videos of real-world person 608 performing the throwing motion.

[0136] Reference Figure 6C , an example presentation 632 of volumetric content 622 by user device 102 is shown. The example presentation 632 includes a real-world chair 610 in a second real-world environment and an AR person 614 superimposed on the second real-world environment visible to user 124 via user device 102. User 124 and real-world person 608 may be different people or may be the same person. The second real-world environment may be the first real-world three-dimensional environment 620 or a different real-world environment. The second real-world environment is determined by the user's location and the content visible to the user at that location. For example, when the user is on the roof of a building, the second real-world environment may be the roof because the roof is visible to the user via user device 102; alternatively, when the skyline of the city is visible to user 124 looking away from the roof via user device 102, the second real-world environment may be the skyline of the city.

[0137] In this example, the second real-world environment is a bedroom. Figure 6CThe real world chair 624 may be Fig. 6A 610. User 124 can see the chair in the bedroom; at the same time, user device 102 renders AR person 614 superimposed on the chair as if AR person 614 were interacting with the chair. In some examples, AR person 614 may appear to be sitting in the chair, posing next to the chair, and performing various animated actions around the chair. As a result, the rendering of volumetric content 622 may include a person sitting in a chair in the bedroom. In some examples, rendering 632 may include rendering of AR person 614 performing a throwing action next to the chair in the bedroom. The throwing action corresponds to the throwing action performed by real-world person 608 during volumetric capture.

[0138] In some examples, presentation 632 is performed in response to identifying a real-world object in the second real-world environment. In some examples, user device 102 may use an artificial intelligence classification method to identify the real-world object in the second real-world environment as real-world chair 610. In response to identifying real-world chair 610, user device 102 may automatically cause presentation of AR person 614 superimposed on real-world chair 610 in volumetric content 622 of user 124.

[0139] Reference Fig.6D , an example presentation 634 of volumetric content 622 by a user device 102 is shown. In this example, the presentation 634 includes a real-world recycling bin 644 and two volumetric content items in a second real-world environment. The presentation 634 of the volumetric content 622 by the user device 102 includes presenting the first volumetric content item and the second volumetric content item superimposed on the second real-world environment visible to the user 124 via the user device 102, such that the two volumetric content items appear to interact with the real-world chair 624 in the second real-world environment.

[0140] In this example, the second real-world environment is an office, and the real-world recycling bin 644 is in the office. The first volumetric content item is the AR person 614, and the second volumetric content item is the AR basketball 612. In some examples, in providing presentation 634, the user device 102 may present the AR person 614 throwing the AR basketball 612 toward the real-world recycling bin 644 and presenting the AR basketball 612 entering the real-world recycling bin 644, so that the two volumetric content items appear to interact with the real-world recycling bin 644.

[0141] Reference Fig. 6E, an example presentation 636 of volumetric content 622 by user device 102 is shown. Presentation 636 of volumetric content 622 includes real-world person 626 and volumetric content items in a second real-world environment. In providing presentation 636, user device 102 presents the volumetric content items superimposed on the second real-world environment visible to user 124 via user device 102, making the volumetric content items appear to interact with real-world person 626 in the second real-world environment. As an example, presentation 636 can include presentation of AR basketball 612 being thrown by real-world person 626. Real-world person 626 can be the same person as real-world person 608 or can be a different person.

[0142] Reference Fig. 6F , an example presentation 638 of volumetric content 622 by user device 102 is shown. The presentation 638 includes a real-world person 626 in a second real-world environment and two volumetric content items. In providing the presentation 638, user device 102 may cause the presentation of the first volumetric content item and the second volumetric content item superimposed on the second real-world environment visible to user 124 via user device 102, such that the two volumetric content items appear to interact with the real-world person 626 in the second real-world environment.

[0143] In this example, the second real-world environment is a basketball court, and the real-world person 626 may be the real-world person 608 or may be a different person. The first volumetric content item is the AR basketball hoop 616, and the second volumetric content item is the AR virtual flaming basketball 628. In some examples, the user device 102 may make the two volumetric content items appear to be interacting with the real-world person 626 by presenting the real-world person 626 throwing the AR virtual flaming basketball 628 toward the AR basketball hoop 616. The process of both the real-world person 626 throwing the AR virtual flaming basketball 628 and the AR virtual flaming football 628 appearing to be passing through the AR basketball hoop 616 provides an appearance of interaction.

[0144] Reference Figure 6G , an example presentation 640 of volumetric content 622 by user device 102 is shown. Figure 6G In the example shown in , presentation 640 includes AR person 614 and AR basketball 612 superimposed on a second real-world environment.

[0145] In this example, the second real-world environment is a basketball court. Presentation 640 may include presenting AR person 614 throwing AR basketball 612 into the air.

[0146] Reference Figure 6H, an example presentation 642 of volumetric content 622 by user device 102 is shown. The presentation 642 includes AR person 614, AR chair 618, and additional volumetric content items. User device 102 may cause the presentation of AR person 614, AR chair 618, and additional volumetric content items superimposed on a second real-world environment visible to user 124 via user device 102, such that the two volumetric content items appear to interact with AR person 614 in the second real-world environment.

[0147] In this example, the second real world environment is a stage. The additional volumetric content item is an AR virtual chair 630. The AR virtual chair 630 may be an imaginary element that does not exist in the physical world. In providing presentation 642, the display device presents an AR person 614 posing next to the AR chair 618 and the AR virtual chair 630, thereby presenting the interaction between the volumetric content items.

[0148] Fig. 7A and 7B 700 is a flowchart illustrating the operation of the volumetric content presentation system 100 in performing a method 700 for presenting a volumetric item according to some examples. The method 700 may be implemented in computer-readable instructions executed by one or more processors, such that the operations of the method 700 may be performed in part or in whole by functional components of the volumetric content presentation system 100; therefore, the method 700 is described below by way of example with reference to the method 700. However, it should be understood that at least some of the operations of the method 700 may be deployed on various other hardware configurations other than the volumetric content presentation system 100.

[0149] At operation 702, the volumetric content presentation system 100 accesses a plurality of volumetric content items (eg, Figure 6B 6, AR basketball 612, AR person 614, AR hoop 616, and AR chair 618 are shown in FIG. 6 . Each volumetric content item from the plurality of volumetric content items includes a volumetric representation of an element of a first real-world three-dimensional environment. The volumetric content items may include, for example, a volumetric representation of a real-world person or a real-world object in the first real-world three-dimensional environment. The volumetric content (e.g., volumetric content 622) may be accessed from database 116.

[0150] At operation 704, the volumetric content rendering system 100 obtains volumetric content items from the volumetric content. For some embodiments, in order to obtain volumetric content items from the volumetric content, the volumetric content rendering system 100 may perform one or more image processing techniques such as segmentation to extract volumetric content items from the volumetric content. When performing such image processing, the volumetric content rendering system 100 may utilize various known artificial intelligence methods. For some embodiments, the volumetric content may be stored so that each volumetric content item may be accessed independently (e.g., the volumetric content item may be stored as a separate file) and used in these embodiments.

[0151] At operation 706, the volumetric content presentation system 100 causes the display device to present an interactive interface operable to receive a selection of a volumetric content item from the volumetric content for presentation as part of the new AR experience. For some embodiments, the interactive interface may be presented in conjunction with the presentation of the volumetric content, and in these embodiments, the interactive interface allows the user to select the volumetric content item directly from the virtual content. For some embodiments, the interactive interface includes a library of volumetric content items for selection (e.g., a set of pre-existing volumetric content items).

[0152] At operation 708, the volumetric content presentation system 100 receives input from the interactive interface indicating a selection of a volumetric content item. For some embodiments, the input may correspond to a selection of the volumetric content item directly from the presentation of the volumetric content. For some embodiments, the input may correspond to a selection of the volumetric content item from a library of volumetric content items.

[0153] At operation 710, the volumetric content presentation system 100 causes presentation of a volumetric content item superimposed on a second real-world environment in a field of view of a user of a display device. For some implementations, the presentation of the volumetric content is projected onto one or more optical elements (e.g., left optical element 208 and right optical element 210) of a display device (e.g., user device 102).

[0154] In a first example of presentation of a volumetric content item, the volumetric content item includes a volumetric representation of a person and the real-world element is an object (e.g., real-world chair 610). In this example, presenting the volumetric content item may include presenting a volumetric representation of a person that appears to interact with a real-world object.

[0155] In a second example, the volumetric content item includes a volumetric representation of a real-world object from a first real-world environment, and the real-world element from the second environment is a real-world person. In this example, presenting the volumetric content item may include presenting a volumetric representation of the real-world object that appears to interact with the real-world person in the second real-world environment.

[0156] In a third example, the volumetric content item is a first volumetric content item, and presenting the volumetric content item includes presenting a second volumetric content item corresponding to the virtual object. That is, the volumetric content presentation system 100 causes the display device to present the volumetric content item from the volumetric content accessed at operation 704 and the second volumetric content item (e.g., corresponding to the virtual object) superimposed on the second real-world environment.

[0157] In a fourth example, the volumetric content item is a first volumetric content item including a volumetric representation of a first real-world element from a first real-world environment, and presenting the volumetric content item includes presenting a second volumetric content item including a volumetric representation of a second real-world element from the first real-world environment. In this example, both the first and second volumetric content items are presented as an overlay on the second real-world environment.

[0158] In a fifth example, the volumetric content item is a first volumetric content item including a volumetric representation of a first real-world element from a first real-world environment, and presenting the volumetric content item includes presenting a second volumetric content item and a third volumetric content item corresponding to the virtual object, the second volumetric content item including a volumetric representation of a second real-world element from the first real-world environment. In this example, the first, second, and third volumetric content items are presented as overlays on the second real-world environment.

[0159] At operation 712, the volumetric content presentation system 100 generates new volumetric content based on presenting the volumetric content item superimposed on the second real-world environment. The new volumetric content includes the volumetric content item and one or more elements of the second real-world environment. In some cases, the new volumetric content includes a volumetric representation of the volumetric content item and one or more elements of the second real-world environment. At operation 714, the volumetric content presentation system 100 stores the new volumetric content for subsequent presentation.

[0160] like Figure 7B As shown, method 700 may further include operations 716, 718, and 720 in some embodiments. Consistent with these embodiments, operations 716, 718, and 720 may be performed as part of operation 710, where volumetric content presentation system 100 causes presentation of a volumetric content item superimposed on a second real-world environment in a user's field of view of a display device.

[0161] At operation 716, in causing the rendering, the volumetric content rendering system 100 identifies a real-world element from the second real-world environment, and at operation 718, the volumetric content rendering system 100 causes the display device to render the volumetric content item superimposed on the real-world element. For some implementations, in response to identifying the real-world element, the volumetric content rendering system 100 causes the display device to render the volumetric content item.

[0162] At operation 720, the volumetric content rendering system 100 causes the display device to render the volumetric content item in a manner that makes the volumetric content item appear to be interacting with the real-world element. The volumetric content rendering system 100 may utilize one or more known image processing techniques on image data generated by one or more cameras in communication with the display device to identify the real-world element. Various known artificial intelligence techniques may be used to perform image processing. For some embodiments, in response to identifying the real-world element, rendering the volumetric content item superimposed on the real-world element is automatically performed.

[0163] Figure 8 802 is a block diagram illustrating a software architecture 802 that may be installed on any one or more of the devices described herein. The software architecture 802 is supported by hardware, such as a machine 804, including a processor 806, a memory 808, and an I / O component 810. In this example, the software architecture 802 may be conceptualized as a stack of layers, each of which provides specific functionality. The software architecture 802 includes layers such as an operating system 812, a library 814, a framework 816, and an application 818. In operation, the application 818 enables an API call 820 through the software stack and receives a message 822 in response to the API call 820.

[0164] The operating system 812 manages hardware resources and provides public services. The operating system 812 includes, for example, a kernel 824, services 826, and drivers 828. The kernel 824 serves as an abstraction layer between the hardware and other software layers. For example, the kernel 824 provides memory management, processor management (e.g., scheduling), component management, network and security settings, and other functions. Services 826 can provide other public services to other software layers. Drivers 828 are responsible for controlling or interfacing with the underlying hardware. For example, drivers 828 may include display drivers, camera drivers, or Low-power drivers, Flash drivers, Serial communication drivers (e.g., USB drivers), Drivers, audio drivers, power management drivers, etc.

[0165] The library 814 provides a common low-level infrastructure used by the application 818. The library 814 may include a system library 830 (e.g., a C standard library) that provides functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. In addition, the library 814 may include an API library 832, such as a media library (e.g., a library that supports rendering and manipulation of various media formats such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), a graphics library (e.g., an OpenGL framework for rendering in two dimensions (2D) and three dimensions (3D) in graphical content on a display), a database library (e.g., SQLite that provides various relational database functions), a network library (e.g., WebKit that provides web browsing functions), etc. The library 814 may also include a variety of other libraries 834 to provide many other APIs to the application 818.

[0166] The framework 816 provides a common high-level infrastructure used by the applications 818. For example, the framework 816 provides various graphical user interface (GUI) functions, high-level resource management, and high-level positioning services. The framework 816 may provide a wide range of other APIs that can be used by the applications 818, some of which may be specific to a particular operating system or platform.

[0167] In an example, applications 818 may include a home application 836, a contact application 838, a browser application 840, a book reader application 842, a positioning application 844, a media application 846, a messaging application 848, a game application 850, and a variety of other applications such as third-party applications 852. Applications 818 are programs that execute functions defined in the program. Various programming languages, such as object-oriented programming languages ​​(e.g., Objective-C, Java, or C++) or procedural programming languages ​​(e.g., C or assembly language) may be used to create one or more of the applications 818 constructed in various ways. In a specific example, third-party applications 852 (e.g., provided by an entity other than the vendor of a particular platform using ANDROID TM or IOS TM Software Development Kit (SDK) can be used to develop applications on platforms such as IOS TM ANDROID TM , Mobile software running on the mobile operating system of the phone or another mobile operating system. In this example, the third-party application 852 can initiate an API call 820 provided by the operating system 812 to facilitate the functions described herein.

[0168] Fig. 9 It is a diagrammatic representation of a machine 900 in which instructions 902 (e.g., software, programs, applications, applet, app, or other executable code) for causing the machine 900 to perform any one or more of the methods discussed herein may be executed. For example, the instructions 902 may cause the machine 900 to perform any one or more of the methods described herein. The instructions 902 convert a general, unprogrammed machine 900 into a specific machine 900 that is programmed to perform the functions described and shown in the manner described. The machine 900 may operate as a standalone device or may be coupled (e.g., networked) to other machines for operation. In a networked deployment, the machine 900 may operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 900 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web device, a network router, a network switch, a network bridge, or any machine capable of sequentially or otherwise executing instructions 902 specifying actions to be taken by the machine 900. In addition, although a single machine 900 is shown, the term "machine" should also be taken to include a collection of machines that individually or jointly execute instructions 902 to perform any one or more of the methods discussed herein. For example, the machine 900 may include a user device 102 or any of a plurality of server devices forming part of a presentation server system 108. In some examples, the machine 900 may also include both a client and a server system, where certain operations of a particular method or algorithm are performed on the server side, and where certain operations of a particular method or algorithm are performed on the client side.

[0169] The machine 900 may include a processor 904, a memory 906, and an input / output I / O component 908, which may be configured to communicate with each other via a bus 910. In an example, the processor 904 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, a processor 912 and a processor 914 that execute instructions 902. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously. Although Fig. 9 Multiple processors 904 are shown, but the machine 900 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0170] The memory 906 includes a main memory 916, a static memory 918, and a storage unit 920, all of which are accessible to the processor 904 via the bus 910. The main memory 906, the static memory 918, and the storage unit 920 store instructions 902 that implement any one or more of the methods or functions described herein. The instructions 902 may also reside, completely or partially, within the main memory 916, within the static memory 918, within a machine-readable medium 922 within the storage unit 920, within at least one of the processors 904 (e.g., within a cache memory of a processor), or within any suitable combination thereof during execution of the instructions 902 by the machine 900.

[0171] The I / O components 908 may include various components that receive input, provide output, generate output, send information, exchange information, capture measurements, etc. The specific I / O components 908 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine will most likely not include such a touch input device. It should be understood that the I / O components 908 may include Fig. 9924 and 926. In various examples, the I / O components 908 may include a user output component 924 and a user input component 926. The user output component 924 may include a visual component (e.g., a display such as a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), an acoustic component (e.g., a speaker), a tactile component (e.g., a vibration motor, a resistance mechanism), other signal generators, etc. The user input component 926 may include an alphanumeric input component (e.g., a keyboard, a touch screen configured to receive alphanumeric input, an optical keyboard, or other alphanumeric input component), a point-based input component (e.g., a mouse, a touch pad, a trackball, a joystick, a motion sensor, or another pointing instrument), a tactile input component (e.g., a physical button, a touch screen or other tactile input component that provides the location and force of a touch or touch gesture), an audio input component (e.g., a microphone), etc.

[0172] In another example, the I / O component 908 may include a biometric component 928, a motion component 930, an environmental component 932, or a positioning component 934, as well as various other components. For example, the biometric component 928 includes components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component 930 includes an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, and a rotation sensor component (e.g., a gyroscope).

[0173] Environmental component 932 includes, for example, one or more cameras (with still image / photo and video capabilities), an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas detection sensor that detects concentrations of hazardous gases for safety or measures pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment.

[0174] With respect to cameras, user device 102 may have a camera system including, for example, a front-facing camera on a front surface of user device 102 and a rear-facing camera on a rear surface of user device 102. The front-facing camera may, for example, be used to capture still images and videos of a user of user device 102 (e.g., a "selfie"), which may then be enhanced using the enhancement data (e.g., filters) described above. The rear-facing camera may, for example, be used to capture still images and videos in a more traditional camera mode, where the images are similarly enhanced using the enhancement data. In addition to the front-facing camera and the rear-facing camera, user device 102 may also include a 360° camera for capturing 360° photos and videos.

[0175] Additionally, the camera system of user device 102 may include dual rear cameras (e.g., a main camera and a depth sensing camera), or even triple, quad, or penta rear camera configurations located on the front and rear sides of user device 102. For example, these multiple camera systems may include a wide-angle camera, an ultra-wide-angle camera, a telephoto camera, a macro camera, and a depth sensor.

[0176] The positioning component 934 includes a position sensor component (for example, a GPS receiver component), an altitude sensor component (for example, an altimeter or barometer that detects air pressure, and the altitude can be obtained based on the air pressure), an orientation sensor component (for example, a magnetometer), etc.

[0177] A variety of technologies may be used to achieve communication. The I / O components 908 also include a communication component 936 that is operable to couple the machine 900 to a network 938 or device 940 via a corresponding coupling or connection. For example, the communication component 936 may include a network interface component or another suitable device that interfaces with the network 938. In other examples, the communication component 936 may include a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, Components (e.g. Low energy consumption), Device 940 may be another machine or any of a variety of peripheral devices (eg, a peripheral device coupled via USB).

[0178] In addition, the communication component 936 can detect the identifier or include a component operable to detect the identifier. For example, the communication component 936 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional bar codes such as universal product code (UPC) bar codes, multi-dimensional bar codes such as quick response (QR) codes, Aztec codes, data matrix, data symbol (Dataglyph), MaxiCode, PDF417, Ultra Code, UCC RSS-2D bar codes, and other optical codes), or an acoustic detection component (e.g., a microphone for identifying tagged audio signals). In addition, various information can be obtained via the communication component 936, such as a location via Internet Protocol (IP) geolocation, ... Location determined by signal triangulation, location obtained via detection of NFC beacon signals that can indicate a specific location, etc.

[0179] Various memories (e.g., main memory 916, static memory 918, and memory of processor 904) and storage unit 920 may store one or more sets of instructions and data structures (e.g., software) implemented or used by any one or more of the methods or functions described herein. These instructions (e.g., instructions 902) when executed by processor 904 cause various operations to implement the disclosed examples.

[0180] The instructions 902 may be sent or received using a transmission medium via a network interface device (e.g., a network interface component included in the communication component 936) and using any of several well-known transmission protocols (e.g., the Hypertext Transfer Protocol (HTTP)) over the network 938. Similarly, the instructions 902 may be sent or received using a transmission medium via a coupling (e.g., a peer-to-peer coupling) to the device 940.

[0181] Glossary

[0182] "Carrier signal" refers to any intangible medium that can store, encode or carry instructions for execution by a machine and includes digital or analog communication signals or other intangible media that facilitates the transmission of such instructions. Instructions may be transmitted or received over a network using a transmission medium via a network interface device.

[0183] "Client Device" refers to any machine that interfaces with a communications network to obtain resources from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a portable digital assistant (PDA), a smart phone, a tablet computer, an ultrabook, a netbook, a laptop computer, a multiprocessor system, a microprocessor-based or programmable consumer electronics product, a game console, a set-top box, a display device, or any other communications device that a user may use to access a network.

[0184] "Communications Network" means one or more parts of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless local area network (WLAN), a wide area network (WAN), a wireless wide area network (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the Public Switched Telephone Network (PSTN), a Plain Old Telephone Service (POTS) network, a cellular telephone network, a wireless network, A network, another type of network, or a combination of two or more such networks. For example, the network or a portion of the network may include a wireless network or a cellular network, and the coupling may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other types of cellular or wireless couplings. In this example, the coupling may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimization (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rate GSM evolution (EDGE) technology, including the third generation partnership project (3GPP) of 3G, fourth generation wireless (4G) network, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world microwave access interoperability (WiMAX), long term evolution (LTE) standards, other data transmission technologies defined by various standard setting organizations, other long distance protocols, or other data transmission technologies.

[0185] "Component" refers to a device, physical entity or logic with boundaries, which is defined by function or subroutine calls, branch points, APIs or other technologies that provide partitioning or modularization for specific processing or control functions. Components can be combined with other components via their interfaces to perform machine processing. Components can be packaged functional hardware units designed for use with other components, and part of a program that generally performs specific functions of related functions. Components can constitute software components (e.g., codes implemented on machine-readable media) or hardware components. "Hardware components" are tangible units that can perform certain operations and can be configured or arranged in a certain physical manner. In various examples, one or more computer systems (e.g., independent computer systems, client computer systems, or server computer systems) or one or more hardware components (e.g., processors or processor groups) of a computer system can be configured by software (e.g., applications or application parts) to operate to perform certain operations described herein. Hardware components can also be implemented mechanically, electronically, or in any suitable combination thereof. For example, a hardware component may include a dedicated circuit or logic that is permanently configured to perform certain operations. A hardware component may be a dedicated processor, such as a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). The hardware component may also include a programmable logic or circuit system that is temporarily configured to perform certain operations by software. For example, the hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or a specific component of a machine) that is uniquely customized to perform the configured function, and is no longer a general-purpose processor. It should be understood that the decision to mechanically implement the hardware component in a dedicated and permanently configured circuit or in a temporarily configured (e.g., configured by software) circuit system can be driven for cost and time considerations. Therefore, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity, that is, an entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate or perform certain operations described herein in a certain manner. Considering the example that the hardware component is temporarily configured (e.g., programmed), it is not necessary to configure or instantiate each hardware component in the hardware component at any time. For example, in the case where the hardware component includes a general-purpose processor that becomes a special-purpose processor by software configuration, the general-purpose processor can be configured to be a different special-purpose processor (e.g., including different hardware components) at different times. Software accordingly configures one or more specific processors to, for example, constitute a specific hardware component at one time and to constitute a different hardware component at a different time. Hardware components can provide information to other hardware components and can receive information from other hardware components. Therefore, the described hardware components can be considered to be communicatively coupled.In the case of multiple hardware components being present at the same time, communication can be achieved by signal transmission (e.g., by appropriate circuits and buses) between two or more hardware components in the hardware components. In examples where multiple hardware components are configured or instantiated at different times, communication between such hardware components can be achieved, for example, by storing information in a memory structure accessible to multiple hardware components and retrieving information in the memory structure. For example, a hardware component can perform an operation and store the output of the operation in a memory device coupled to it in communication. Then, other hardware components can subsequently access the memory device to retrieve and process the stored output. The hardware component can also initiate communication with an input or output device, and can operate on resources (e.g., a collection of information). The various operations of the example methods described herein can be performed at least in part by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform related operations. Whether it is temporarily configured or permanently configured, such a processor can constitute a processor-implemented component that operates to perform one or more operations or functions described herein. As used herein, "processor-implemented components" refer to hardware components implemented using one or more processors. Similarly, the method described herein can be implemented at least in part by a processor, wherein one or more specific processors are examples of hardware. For example, at least some operations in the operation of the method can be performed by one or more processors or the parts implemented by the processor. In addition, one or more processors can also operate to support the execution of related operations in a "cloud computing" environment or as a "software as a service" (SaaS) operation. For example, at least some operations in the operation can be performed by a computer (as an example of a machine including a processor) group, wherein these operations can be accessed via a network (for example, the Internet) and via one or more appropriate interfaces (for example, API). The execution of some operations in the operation can be distributed between processors, not only resident in a single machine, but deployed across multiple machines. In some examples, a processor or a part implemented by a processor can be located in a single geographical location (for example, in a home environment, an office environment or a server cluster). In other examples, a processor or a part implemented by a processor can be distributed across multiple geographical locations.

[0186] "Computer-readable storage media" refers to both machine storage media and transmission media. Therefore, these terms include both storage devices / media and carrier / modulated data signals. The terms "machine-readable medium", "computer-readable medium" and "device-readable medium" refer to the same thing and can be used interchangeably in this disclosure.

[0187] "Ephemeral messages" are messages that are accessible for a limited duration. Ephemeral messages can be text, images, videos, etc. The access time for ephemeral messages can be set by the sender of the message. Alternatively, the access time can be a default setting or a setting specified by the recipient. Regardless of the setting technique, the message is ephemeral.

[0188] “Machine storage media” refers to a single or multiple storage devices and media (e.g., centralized or distributed databases and associated caches and servers) that store executable instructions, routines, and data. Thus, the term should be considered to include, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGA, and flash memory devices; magnetic disks, such as internal hard disks and removable magnetic disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms “machine storage media,” “device storage media,” and “computer storage media” mean the same thing and may be used interchangeably in this disclosure. The terms “machine storage media,” “computer storage media,” and “device storage media” expressly exclude carrier waves, modulated data signals, and other such media, at least some of which are encompassed by the term “signal media.”

[0189] “Non-transitory computer-readable storage medium” refers to tangible media capable of storing, encoding, or carrying instructions to be executed by a machine.

[0190] "Signal medium" refers to any intangible medium that can store, encode or carry instructions executed by a machine and includes digital or analog communication signals or other intangible media that facilitate the communication of software or data. The term "signal medium" should be deemed to include any form of modulated data signals, carrier waves, etc. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.

Claims

1. A method comprising: accessing, by at least one hardware processor, volumetric content comprising a plurality of volumetric content items, a volumetric content item from the plurality of volumetric content items comprising a volumetric representation of an element of a first real-world environment; obtaining, by the at least one hardware processor, the volumetric content item from the volumetric content; as well as Presentation is caused, by a display device, of the volumetric content item superimposed on a second real-world environment in a field of view of a user of the display device.

2. The method according to claim 1, further comprising: presenting an interactive interface operable to receive a selection of the volumetric content item from the volumetric content for presentation; as well as Input is received from the interactive interface indicating a selection of the volumetric content item, wherein the causing to present is performed in response to receiving the input.

3. The method according to claim 1, further comprising: generating new volumetric content based on rendering the volumetric content item superimposed on the second real-world environment; as well as The new volume content is stored for subsequent presentation.

4. The method according to claim 1, wherein: Said causing presentation of said volumetric content item superimposed on said second real-world environment comprises: identifying a real-world element in the second real-world environment; and The volumetric content item is caused to be presented superimposed on the real-world element.

5. The method according to claim 4, wherein: In response to identifying a real-world element in the second real-world environment, automatically causing presentation of the volumetric content item superimposed on the real-world element is performed.

6. The method according to claim 4, wherein: The causing presentation of the volumetric content item superimposed on the real-world element includes causing the display device to present the volumetric content item that appears to interact with the real-world element.

7. The method according to claim 4, wherein: The volumetric content item corresponds to a real-world object from the first real-world environment; and The real-world element from the second real-world environment corresponds to a real-world person.

8. The method according to claim 4, wherein: The volumetric content item corresponds to a real-world person from the first real-world environment; and The real-world elements from the second real-world environment correspond to real-world objects.

9. The method according to claim 4, wherein: The volumetric content item corresponds to a virtual object; and The real-world element from the second real-world environment corresponds to a real-world object or a real-world person.

10. The method of claim 1, wherein: The volumetric content item is a first volumetric content item; and The method further comprises obtaining a second volumetric content item from the volumetric content; as well as The causing presentation of the volumetric content item superimposed on the second real-world environment comprises causing presentation of the first volumetric content item and the second volumetric content item superimposed on the second real-world environment in a field of view of a user of the display device.

11. A system comprising: one or more hardware processors; as well as at least one memory storing instructions for causing the one or more hardware processors to perform the following operations, the operations comprising: accessing, by at least one hardware processor, volumetric content comprising a plurality of volumetric content items, a volumetric content item from the plurality of volumetric content items comprising a volumetric representation of an element of a first real-world environment; obtaining, by the at least one hardware processor, the volumetric content item from the volumetric content; and Presentation is caused, by a display device, of the volumetric content item superimposed on a second real-world environment in a field of view of a user of the display device.

12. The system according to claim 11, further comprising: presenting an interactive interface operable to receive a selection of the volumetric content item from the volumetric content for presentation; as well as Input is received from the interactive interface indicating a selection of the volumetric content item, wherein the causing to present is performed in response to receiving the input.

13. The system of claim 11, further comprising: generating new volumetric content based on rendering the volumetric content item superimposed on the second real-world environment; as well as The new volume content is stored for subsequent presentation.

14. The system according to claim 11, wherein: Said causing presentation of said volumetric content item superimposed on said second real-world environment comprises: identifying a real-world element in the second real-world environment; and The volumetric content item is caused to be presented superimposed on the real-world element.

15. The system of claim 14, wherein: In response to identifying a real-world element in the second real-world environment, automatically causing presentation of the volumetric content item superimposed on the real-world element is performed.

16. The system of claim 14, wherein: The causing presentation of the volumetric content item superimposed on the real-world element includes causing the display device to present the volumetric content item that appears to interact with the real-world element.

17. The system of claim 14, wherein: The volumetric content item corresponds to a real-world object from the first real-world environment; and The real-world element from the second real-world environment corresponds to a real-world person.

18. The system of claim 14, wherein: The volumetric content item corresponds to a real-world person from the first real-world environment; and The real-world elements from the second real-world environment correspond to real-world objects.

19. The system of claim 14, wherein: The volumetric content item corresponds to a virtual object; and The real-world element from the second real-world environment corresponds to a real-world object or a real-world person.

20. A machine-readable medium storing instructions which, when executed by a computer system, cause the computer system to perform the following operations, comprising: accessing volumetric content comprising a plurality of volumetric content items, a volumetric content item from the plurality of volumetric content items comprising a volumetric representation of an element of a first real-world environment; obtaining the volume content item from the volume content; as well as Presentation is caused, by a display device, of the volumetric content item superimposed on a second real-world environment in a field of view of a user of the display device.

Images