Distributed generation of virtual content

Abstract

Systems and techniques are described for establishing one or more virtual sessions between users. For example, a first device may send a call setup request for a virtual representation call of a virtual session to a second device, and may receive a call acceptance from the second device indicating acceptance of the call setup request. A first device may send first grid information for a first virtual representation of a first user of the first device and first grid animation parameters for the first virtual representation to a second device. The first device may receive, from the second device, second grid information for a second virtual representation of a second user of the second device and second grid animation parameters for the second virtual representation. The first device may generate a second virtual representation of the second user based on the second mesh information and the second mesh animation parameter.

Description

Technical Field

[0001] The present disclosure generally relates to virtual content for virtual environments or portions of virtual environments.For example, aspects of the present disclosure include systems and techniques for providing distributed generation of virtual content. Background Art

[0002] Extended reality (XR) (e.g., virtual reality, augmented reality, mixed reality) systems can provide users with virtual experiences by immersing them in a completely virtual environment (composed of virtual content) and / or can provide users with augmented or mixed reality experiences by combining the real world or physical environment with a virtual environment.

[0003] One example use case for providing virtual, augmented, or mixed reality XR content to a user is presenting a "metaverse" experience to the user. A metaverse is essentially a virtual universe that includes one or more three-dimensional (3D) virtual worlds. For example, a metaverse virtual environment may allow a user to virtually interact with other users (e.g., in a social setting, in a virtual meeting, etc.), to virtually purchase goods, services, property, or other items, to play computer games, and / or to experience other services.

[0004] In some cases, a user may be represented in a virtual environment (e.g., a metaverse virtual environment) as a virtual representation of the user, sometimes referred to as an avatar. In any virtual environment, it is important that the system be able to establish a call to a virtual representation with a high-quality avatar that represents the person in an effective manner. Summary of the invention

[0005] Systems and techniques for providing distributed generation of virtual content for a virtual environment (e.g., a metaverse virtual environment) are described. According to at least one example, a method for generating virtual content at a first device in a distributed system is provided. The method includes: receiving input information associated with the second device or at least one of a user of the second device from a second device associated with a virtual session; generating a virtual representation of the user of the second device based on the input information; generating a virtual scene from the perspective of a user of a third device associated with the virtual session, wherein the virtual scene includes the virtual representation of the user of the second device; and sending one or more frames depicting the virtual scene from the perspective of the user of the third device to the third device.

[0006] In another example, an apparatus for generating virtual content at a first device in a distributed system is provided, the apparatus comprising at least one memory and at least one processor (e.g., configured in a circuit) coupled to the at least one memory. The at least one processor is configured to: receive input information associated with the second device or at least one of a user of the second device from a second device associated with a virtual session; generate a virtual representation of the user of the second device based on the input information; generate a virtual scene from the perspective of a user of a third device associated with the virtual session, wherein the virtual scene includes the virtual representation of the user of the second device; and output one or more frames depicting the virtual scene from the perspective of the user of the third device for transmission to the third device.

[0007] In another example, a non-transitory computer-readable medium is provided having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to: receive input information associated with the second device or at least one of a user of the second device from a second device associated with a virtual session; generate a virtual representation of the user of the second device based on the input information; generate a virtual scene from the perspective of a user of a third device associated with the virtual session, wherein the virtual scene includes the virtual representation of the user of the second device; and output one or more frames depicting the virtual scene from the perspective of the user of the third device for transmission to the third device.

[0008] In another example, an apparatus for generating virtual content at a first device in a distributed system is provided. The apparatus includes: a component for receiving input information associated with the second device or at least one of a user of the second device from a second device associated with a virtual session; a component for generating a virtual representation of the user of the second device based on the input information; a component for generating a virtual scene from the perspective of a user of a third device associated with the virtual session, wherein the virtual scene includes the virtual representation of the user of the second device; and a component for sending one or more frames depicting the virtual scene from the perspective of the user of the third device to the third device.

[0009] As another example, a method for establishing one or more virtual sessions between users is provided. The method includes: sending a call establishment request for a virtual representation call of a virtual session to a second device by a first device; receiving a call acceptance indicating acceptance of the call establishment request from the second device at the first device; sending first grid information for a first virtual representation of a first user of the first device to the second device by the first device; sending first grid animation parameters for the first virtual representation of the first user of the first device to the second device by the first device; receiving second grid information for a second virtual representation of a second user of the second device from the second device at the first device; receiving second grid animation parameters for a second virtual representation of a second user of the second device from the second device at the first device; and generating a second virtual representation of a second user of the second device at the first device based on the second grid information and the second grid animation parameters.

[0010] In another example, a first device for establishing one or more virtual sessions between users is provided, the first device comprising at least one memory and at least one processor (e.g., configured in a circuit) coupled to the at least one memory. The at least one processor is configured to: send a call establishment request for a virtual representation call for a virtual session to a second device; receive a call acceptance indicating acceptance of the call establishment request from the second device; send first grid information for a first virtual representation of a first user of the first device to the second device; send first grid animation parameters for the first virtual representation of the first user of the first device to the second device; receive second grid information for a second virtual representation of a second user of the second device from the second device; receive second grid animation parameters for the second virtual representation of the second user of the second device from the second device; and generate a second virtual representation of the second user of the second device based on the second grid information and the second grid animation parameters.

[0011] In another example, a non-transitory computer-readable medium of a first device is provided, having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to: send a call establishment request for a virtual representation call for a virtual session to a second device; receive a call acceptance from the second device indicating acceptance of the call establishment request; send first grid information for a first virtual representation of a first user of the first device to the second device; send first grid animation parameters for the first virtual representation of the first user of the first device to the second device; receive second grid information for a second virtual representation of a second user of the second device from the second device; receive second grid animation parameters for the second virtual representation of the second user of the second device from the second device; and generate a second virtual representation of the second user of the second device based on the second grid information and the second grid animation parameters.

[0012] In another example, a first device for establishing one or more virtual sessions between users is provided. The device includes: a component for sending a call establishment request for a virtual representation call for a virtual session to a second device; a component for receiving a call acceptance indicating acceptance of the call establishment request from the second device; a component for sending first grid information for a first virtual representation of a first user of the first device to the second device; a component for sending first grid animation parameters for the first virtual representation of the first user of the first device to the second device; a component for receiving second grid information for a second virtual representation of a second user of the second device from the second device; a component for receiving second grid animation parameters for a second virtual representation of the second user of the second device from the second device; and a component for generating a second virtual representation of the second user of the second device based on the second grid information and the second grid animation parameters.

[0013] In another example, a method for establishing one or more virtual sessions between users is provided. The method includes: receiving, by a server device, a call establishment request for a virtual representation call of a virtual session from a first client device; sending, by the server device, the call establishment request to a second client device; receiving, by the server device, a call acceptance indicating acceptance of the call establishment request from the second client device; sending, by the server device, the call acceptance to the first client device; receiving, from the first client device based on the call acceptance, first grid information for a first virtual representation of a first user of the first client device; sending, by the server device, first grid animation parameters for the first virtual representation of the first user of the first client device; generating, by the server device, a first virtual representation of the first user of the first client device based on the first grid information and the first grid animation parameters; and sending, by the server device, the first virtual representation of the first user of the first client device to the second client device.

[0014] In another example, a server device for establishing one or more virtual sessions between users is provided, the server device comprising at least one memory and at least one processor (e.g., configured in a circuit) coupled to the at least one memory. The at least one processor is configured to: receive a call establishment request for a virtual representation call of a virtual session from a first client device; send the call establishment request to a second client device; receive a call acceptance indicating acceptance of the call establishment request from the second client device; transmit the call acceptance to the first client device; receive first grid information for a first virtual representation of a first user of the first client device from the first client device based on the call acceptance; transmit first grid animation parameters for the first virtual representation of the first user of the first client device; generate a first virtual representation of the first user of the first client device based on the first grid information and the first grid animation parameters; and transmit the first virtual representation of the first user of the first client device to the second client device.

[0015] In another example, a non-transitory computer-readable medium of a server device is provided, having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to: receive a call establishment request for a virtual representation call for a virtual session from a first client device; send the call establishment request to a second client device; receive a call acceptance from the second client device indicating acceptance of the call establishment request; transmit the call acceptance to the first client device; receive first grid information for a first virtual representation of a first user of the first client device from the first client device based on the call acceptance; transmit first grid animation parameters for the first virtual representation of the first user of the first client device; generate a first virtual representation of the first user of the first client device based on the first grid information and the first grid animation parameters; and transmit the first virtual representation of the first user of the first client device to the second client device.

[0016] In another example, a server device for establishing one or more virtual sessions between users is provided. The device includes: a component for receiving a call establishment request for a virtual representation call for a virtual session from a first client device; a component for sending the call establishment request to a second client device; a component for receiving a call acceptance indicating acceptance of the call establishment request from the second client device; a component for sending the call acceptance to the first client device; a component for receiving first grid information for a first virtual representation of a first user of the first client device from the first client device based on the call acceptance; a component for transmitting first grid animation parameters for the first virtual representation of the first user of the first client device; a component for generating a first virtual representation of the first user of the first client device based on the first grid information and the first grid animation parameters; and a component for sending the first virtual representation of the first user of the first client device to the second client device.

[0017] In another example, a method for establishing one or more virtual sessions between users is provided. The method includes: sending a call establishment request for a virtual representation call of a virtual session to a second device by a first device; receiving a call acceptance indicating acceptance of the call establishment request from the second device at the first device; sending input information associated with the first device or at least one of the users of the first device to a third device by the first device, the input information being used to generate a virtual representation of the user of the second device and a virtual scene from the perspective of the user of the second device; and receiving information of the virtual scene from the third device.

[0018] As another example, a device for establishing one or more virtual sessions between users is provided. The device includes at least one memory and at least one processor coupled to at least one memory having instructions stored thereon. When the instructions are executed by at least one processor, the at least one processor is caused to: send a call establishment request for a virtual representation call for a virtual session to a second device by a first device; receive a call acceptance indicating acceptance of the call establishment request from the second device at the first device; send input information associated with the first device or at least one of the users of the first device to a third device by the first device, the input information being used to generate a virtual representation of the user of the second device and a virtual scene from the perspective of the user of the second device; and receive information of the virtual scene from the third device.

[0019] In another example, a non-transitory computer-readable medium having instructions stored thereon is provided. The instructions, when executed by one or more processors, cause the one or more processors to: send a call establishment request for a virtual representation call of a virtual session to a second device by a first device; receive a call acceptance indicating acceptance of the call establishment request from the second device at the first device; send input information associated with the first device or at least one of the users of the first device to a third device by the first device, the input information being used to generate a virtual representation of the user of the second device and a virtual scene from the perspective of the user of the second device; and receive information of the virtual scene from the third device.

[0020] As another example, an apparatus for establishing one or more virtual sessions between users, the method comprising: sending, by a first device to a second device, a call establishment request for a virtual representation call of a virtual session; receiving, at the first device, a call acceptance from the second device indicating acceptance of the call establishment request; sending, by the first device to a third device, input information associated with the first device or at least one of the users of the first device, the input information being used to generate a virtual representation of the user of the second device and a virtual scene from the perspective of the user of the second device; and receiving information of the virtual scene from the third device.

[0021] In some aspects, one or more devices described herein are, are part of, and / or include an extended reality (XR) device or system (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a mobile device (e.g., a mobile phone or other mobile device), a wearable device, a wireless communication device, a camera, a personal computer, a laptop computer, a vehicle or a computing device or component of a vehicle, a server computer or server device (e.g., an edge or cloud-based server, a personal computer acting as a server device, a mobile device (such as a mobile phone) acting as a server device, an XR device acting as a server device, a vehicle acting as a server device, a network router, or other device acting as a server device), another device, or a combination thereof. In some aspects, the device includes a camera or multiple cameras for capturing one or more images. In some aspects, the device also includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the above-mentioned device may include one or more sensors (e.g., one or more inertial measurement units (IMUs), such as one or more gyroscopes, one or more gyroscopes, one or more accelerometers, any combination thereof, and / or other sensors).

[0022] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used alone to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0023] The foregoing and other features and aspects will become more fully apparent by reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Illustrative examples of the present application are described in detail below with reference to the following drawings:

[0025] Figure 1 is a schematic diagram illustrating an example of an extended reality (XR) system according to aspects of the present disclosure;

[0026] Figure 2 is a schematic diagram illustrating an example of a three-dimensional (3D) collaborative virtual environment according to aspects of the present disclosure;

[0027] Figure 3 is a schematic diagram illustrating an example of an XR system according to aspects of the present disclosure, in which two client devices exchange information for generating a virtual representation of a user and constituting a virtual scene;

[0028] Figure 4 is a schematic diagram illustrating another example of an XR system according to aspects of the present disclosure;

[0029] Figure 5 is a schematic diagram illustrating an example configuration of a client device according to aspects of the present disclosure;

[0030] Figure 6 is a schematic diagram illustrating an example of an XR system including an animation and scene rendering system in communication with a client device according to aspects of the present disclosure;

[0031] Figure 7 is a schematic diagram illustrating another example of an XR system including an animation and scene rendering system in communication with a client device according to aspects of the present disclosure;

[0032] Figure 8 is a schematic diagram showing an illustrative example of a user virtual representation system of an animation and scene rendering system according to aspects of the present disclosure;

[0033] Fig. 9 is a schematic diagram showing an illustrative example of a scene composition system of an animation and scene rendering system according to aspects of the present invention;

[0034] Fig.10is a schematic diagram illustrating another example of an XR system including an animation and scene rendering system in communication with a client device according to aspects of the present disclosure;

[0035] Fig.11 is a schematic diagram illustrating another example of an XR system including an animation and scene rendering system in communication with a client device according to aspects of the present disclosure;

[0036] Fig.12 is a flow chart illustrating an example of a process of generating virtual content at a device of a distributed system according to aspects of the present disclosure;

[0037] Fig.13 A schematic diagram showing an example of a user's grid, an example of a user's normal map, an example of a user's albedo map, an example of a user's specular map, and an example of personalization for a user according to aspects of the present invention;

[0038] Fig.14A is a schematic diagram showing an example of an original (non-retopologized) mesh according to aspects of the present disclosure;

[0039] Fig. 14B is a schematic diagram illustrating an example of a retopological mesh according to aspects of the present disclosure;

[0040] Fig.15 is a diagram illustrating an example of a technique for performing avatar animation according to aspects of the present disclosure;

[0041] Fig.16 is a schematic diagram illustrating an example of an avatar calling flow directly between client devices according to aspects of the present disclosure;

[0042] Fig.17 is a diagram illustrating an example of an XR system configured to perform aspects described herein;

[0043] Fig.18 is a schematic diagram showing an example of an avatar call flow for establishing an avatar call between client devices using a server device according to aspects of the present disclosure;

[0044] Fig.19 is a diagram illustrating another example of an XR system configured to perform aspects described herein according to aspects of the present disclosure;

[0045] Fig. 20 is a diagram illustrating another example of an XR system configured to perform aspects described herein according to aspects of the present disclosure;

[0046] Fig.21is a diagram illustrating another example of an XR system configured to perform aspects described herein according to aspects of the present disclosure;

[0047] Fig. 22 is a diagram illustrating an example of an example of an XR system configured to perform aspects described herein according to aspects of the present disclosure;

[0048] Fig.23 is a diagram illustrating an example of an example of an XR system configured to perform aspects described herein according to aspects of the present disclosure;

[0049] Fig.24 is a signaling diagram illustrating communications between various devices according to aspects of the present disclosure;

[0050] Fig.25 is an example of a device-to-device call flow for establishing an avatar call between a client device without an edge server and a decoder on a target device for view-dependent and / or view-independent textures in accordance with aspects of the present disclosure;

[0051] Fig.26 is a diagram illustrating an example of an example of an XR system configured to perform aspects described herein according to aspects of the present disclosure;

[0052] Fig. 27 is a diagram illustrating an example of an example of an XR system configured to perform aspects described herein according to aspects of the present disclosure;

[0053] Fig.28 is a signaling diagram illustrating communications between various devices according to aspects of the present disclosure;

[0054] Fig.29 is an example of a device-to-device call flow for establishing an avatar call between client devices without an edge server and all processing (e.g., decoding, avatar animation, etc.) for view-dependent and / or view-independent textures on a source or transmitting / sending client device in accordance with aspects of the present disclosure;

[0055] Fig.30 is a diagram illustrating an example of an example of an XR system configured to perform aspects described herein according to aspects of the present disclosure;

[0056] Fig.31 is a diagram illustrating an example of an example of an XR system configured to perform aspects described herein according to aspects of the present disclosure;

[0057] Fig.32 is a signaling diagram illustrating communications between various devices according to aspects of the present disclosure;

[0058] Fig.33is an example of a device-to-device call flow for establishing an avatar call between a client device having an edge server and a decoder on a server device for view-dependent and / or view-independent textures in accordance with aspects of the present disclosure;

[0059] Fig.34 is a diagram illustrating an example of an example of an XR system configured to perform aspects described herein according to aspects of the present disclosure;

[0060] Fig.35 is a diagram illustrating an example of an example of an XR system configured to perform aspects described herein according to aspects of the present disclosure;

[0061] Fig.36 is a signaling diagram illustrating communications between various devices according to aspects of the present disclosure;

[0062] Fig.37 is an example of a device-to-device call flow for establishing an avatar call between a client device with an edge server and all processing (e.g., decoding, avatar animation, etc.) for view-dependent and / or view-independent textures on a source or transmitting / sending client device in accordance with aspects of the present disclosure;

[0063] Fig.38 is a signaling diagram illustrating another example of communications between various devices according to aspects of the present disclosure;

[0064] Fig.39 is a signaling diagram illustrating another example of communications between various devices according to aspects of the present disclosure;

[0065] Fig.40 is a signaling diagram illustrating another example of communications between various devices according to aspects of the present disclosure;

[0066] Fig.41 is a block diagram illustrating an example of a deep learning network according to some examples;

[0067] Fig.42 is a block diagram illustrating an example of a convolutional neural network according to some examples;

[0068] Fig.43 is a diagram illustrating an example of a computing system according to aspects of the present disclosure;

[0069] Fig.44 is a flowchart illustrating an example of a process of generating virtual content in a distributed system according to aspects of the present disclosure; and

[0070] Fig.45 is a flow chart illustrating an example of generating virtual content in a distributed system according to aspects of the present disclosure. DETAILED DESCRIPTION

[0071] Certain aspects of the present disclosure are provided below. Some of these aspects can be applied independently, and some of them can be applied in combination, which is obvious to those skilled in the art. In the following description, for the purpose of explanation, specific details are set forth in order to provide a thorough understanding of various aspects of the application. However, it is obvious that various aspects can be practiced without these specific details. The drawings and description are not intended to be restrictive.

[0072] The following description provides only example aspects and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the following description of the example aspects will provide a feasible description for implementing the example aspects for those skilled in the art. It should be understood that various changes may be made to the functions and arrangements of the elements without departing from the spirit and scope of the present application as set forth in the appended claims.

[0073] As previously described, an extended reality (XR) system or device can provide an XR experience to a user by presenting virtual content to the user (e.g., for a fully immersive experience) and / or can combine a view of a real-world or physical environment with a display of a virtual environment (composed of virtual content). The real-world environment can include real-world objects (also called physical objects) such as people, vehicles, buildings, tables, chairs, and / or other real-world or physical objects. As used herein, the terms XR system and XR device are used interchangeably. Examples of XR systems or devices include head-mounted displays (HMDs), smart glasses (e.g., AR glasses, MR glasses, etc.), and the like.

[0074] XR systems may include virtual reality (VR) systems that facilitate interaction with VR environments, AR systems that facilitate interaction with augmented reality (AR) environments, mixed reality (MR) systems that facilitate interaction with MR environments, and / or other XR systems. For example, VR provides a complete immersive experience in a three-dimensional (3D) computer-generated VR environment or video that depicts a virtual version of a real-world environment. In some cases, VR content may include VR videos that can be captured and rendered at very high quality, potentially providing a truly immersive virtual reality experience. Virtual reality applications may include games, training, education, sports videos, online shopping, and the like. VR content may be rendered and displayed using a VR system or device that completely covers the user's eyes during the VR experience, such as a VR HMD or other VR head-mounted device.

[0075] AR is a technology that provides virtual or computer-generated content (referred to as AR content) on a user's view of a physical real-world scene or environment. AR content can include any virtual content, such as video, images, graphical content, location data (e.g., global positioning system (GPS) data or other location data), sound, any combination thereof, and / or other enhanced content. AR systems are designed to enhance (or expand) rather than replace a person's current perception of reality. For example, a user can see a real static or moving physical object through an AR device display, but the user's visual perception of the physical object can be enhanced or strengthened by a virtual image of the object (e.g., a real-world car is replaced by a virtual image of Delaney), by AR content added to a physical object (e.g., virtual wings added to an active animal), by AR content displayed relative to a physical object (e.g., information virtual content displayed near a sign on a building, a virtual coffee cup virtually anchored to a real-world table in one or more images (e.g., placed on top of it), etc.), and / or by displaying other types of AR content. Various types of AR systems can be used for games, entertainment, and / or other applications.

[0076] MR technology can combine aspects of VR and AR to provide users with an immersive experience. For example, in an MR environment, real-world and computer-generated objects can interact (e.g., real people can interact with virtual people as if the virtual people were real people).

[0077] The XR environment can be interactive in a way that appears real or physical. As a user experiencing an XR environment (e.g., an immersive VR environment) moves in the real world, the virtual content presented (e.g., images presented in a virtual environment in a VR experience) also changes, giving the user the perception that the user is moving within the XR environment. For example, the user can turn left or right, look up or down, and / or move forward or backward, thereby changing the user's viewpoint of the XR environment. The XR content presented to the user can change accordingly, making the user's experience in the XR environment as seamless as in the real world.

[0078] In some cases, the XR system can match the relative pose and movement of objects and devices in the physical world. For example, the XR system can use tracking information to calculate the relative pose of features of devices, objects, and / or the real-world environment in order to match the relative position and movement of devices, objects, and / or the real-world environment. In some examples, the XR system can use the pose and movement of one or more devices, objects, and / or the real-world environment to present content relative to the real-world environment in a convincing manner. Relative pose information can be used to match virtual content to the user's perceived motion and the spatiotemporal state of devices, objects, and the real-world environment. In some cases, the XR system can track parts of the user (e.g., the user's hands and / or fingertips) to allow the user to interact with virtual content items.

[0079] An XR system or device can facilitate interaction with different types of XR environments (e.g., a user can use an XR system or device to interact with an XR environment). An example of an XR environment is a metaverse virtual environment. A user can participate in one or more virtual sessions with other users by virtually interacting with other users (e.g., in a social environment, in a virtual meeting, etc.), virtually purchasing items (e.g., goods, services, property, etc.), virtually playing computer games, and / or experiencing other services in a metaverse virtual environment. In an illustrative example, a virtual session provided by an XR system may include a 3D collaborative virtual environment for a group of users. Users can interact with each other via virtual representations of users in the virtual environment. Users can experience the virtual environment visually, auditorily, tactilely, or otherwise while interacting with virtual representations of other users.

[0080] A virtual representation of a user can be used to represent the user in a virtual environment. The virtual representation of a user is also referred to as an avatar in this article. The avatar representing the user can imitate the user's appearance, movement, habits and / or other characteristics. The virtual representation or avatar can be generated / animated in real time based on the input captured from the user's device. The range of avatars can range from basic synthetic 3D representations to more realistic representations of the user. In some examples, the user may expect the avatar representing the person in the virtual environment to behave as a digital twin of the user. In any virtual environment, it is important for the XR system to effectively generate high-quality avatars (e.g., realistically representing the person's appearance, movement, etc.) in a low-latency manner. It may also be important for the XR system to render audio in an efficient manner to enhance the XR experience.

[0081] For example, in the example of a 3D collaborative virtual environment from above, the XR system of a user from a user group may display virtual representations (or avatars) of other users sitting at a virtual table or at a specific location in a virtual room. The virtual representations of the users and the background of the virtual environment should be displayed in a realistic manner (e.g., as if the users were sitting together in the real world). The heads, bodies, arms, and hands of the users may be animated as they move in the real world. Audio may need to be rendered spatially or may be rendered in mono. The delay in rendering and animating the virtual representations should be minimal in order to maintain a high-quality user experience.

[0082] The computational complexity of generating a virtual environment by an XR system can impose significant power and resource requirements, which can be a limiting factor in achieving an XR experience (e.g., reducing the ability of an XR device to effectively generate and animate virtual content in a low-latency manner). For example, when implementing an XR application, the computational complexity of rendering and animating a user's virtual representation and composing a virtual scene can impose significant power and resource requirements on the device. Such power and resource requirements are exacerbated by the recent trend to implement such technology in mobile and wearable devices (e.g., HMDs, XR glasses, etc.) and to make such devices smaller, lighter, and more comfortable (e.g., by reducing the heat emitted by the device) for users to wear for longer periods of time. In view of these factors, a user's XR device (e.g., HMD) may have difficulty rendering and animating virtual representations of other users, and composing scenes and generating target views of the virtual environment for display to the user of the XR device.

[0083] Described herein are systems, apparatuses, electronic devices, methods (also referred to as processes), and computer-readable media (collectively referred to herein as “systems and techniques”) for providing distributed generation of virtual content for a virtual environment (e.g., a metaverse virtual environment). In some aspects, an animation and scene rendering system may receive input information associated with one or more devices (e.g., at least one XR device) of a respective user participating in or associated with a virtual session (e.g., a 3D collaborative virtual meeting in a metaverse environment, a computer or virtual game, or other virtual session). For example, the input information received from a first device may include information representing a face of a first user of the first device (e.g., a code or feature representing facial appearance or other information), information representing a body of the first user of the first device (e.g., a code or feature representing body appearance, body posture, or other information), information representing one or more hands of the first user of the device (e.g., a code or feature representing hand appearance, hand posture, or other information), posture information of the first device (e.g., a posture in six degrees of freedom (6-DOF), referred to as a 6-DOF posture), audio associated with the environment in which the first device is located, and / or other information.

[0084] The animation and scene rendering system may process input information to generate and / or animate a corresponding virtual representation of each user of each device of one or more devices. For example, the animation and scene rendering system may process input information from a first device to generate and / or animate a virtual representation (or avatar) for a first user of the first device. Animation of a virtual representation refers to modifying the position, movement, behavior, or other features of a virtual representation, such as to match the corresponding position, movement, behavior, etc. of a corresponding user in the real world or physical space. In some aspects, in order to generate and / or animate a virtual representation of the first user's face (referred to as a facial representation), generate and / or animate a virtual representation of the first user's body (referred to as a body representation), and generate and / or animate a virtual representation of the first user's hair (referred to as a hair representation). In some cases, the animation and scene rendering system may combine or append a facial representation with a body representation to generate a combined virtual representation. The animation and scene rendering system may then add a hair representation to the combined virtual representation to generate a final virtual representation of the user for one or more content frames.

[0085] The animation and scene rendering system can then compose a virtual scene or environment for the virtual session, and generate a corresponding target view (e.g., a content frame, such as a virtual content or a combination of real world and virtual content) for each device from the corresponding perspective of each device or user in the virtual scene or environment. The composed virtual scene includes virtual representations of the users involved in the virtual session and any virtual background of the scene. In one example, using information received from the first device and information received from other devices in one or more devices associated with the virtual session, the animation and scene rendering system can compose a virtual scene (including virtual representations of users and background information) and generate a frame representing a view of the virtual scene from the perspective of a second user of a second device of one or more devices associated with the virtual session (representing the view of the second user in the real or physical world). In some aspects, a view of the virtual scene can be generated from the perspective of the second user based on the posture of the first device (corresponding to the posture of the first user) and the posture of any other user / device and also based on the posture of the second device (corresponding to the posture of the second user). For example, a view of the virtual scene can be generated from the perspective of the second user based on the relative difference between the posture of the second device and each corresponding posture of the first device and other devices.

[0086] In some cases, when composing a virtual scene, the animation and scene rendering system may perform re-lighting of each respective virtual representation of one or more users. For example, the animation and scene rendering system may determine the lighting in the virtual scene or in the real-world environment in which the first user is located. The animation and scene rendering system may modify the lighting of the virtual representation of the first user to account for the determined lighting so that the virtual representation of the first user appears as realistic as possible when viewed by other users in the virtual scene.

[0087] The animation and scene rendering system can send the frame (e.g., after encoding or compressing the frame) to the second device. The second device can display the frame (e.g., after decoding or decompressing the frame) so that the second user can view the virtual scene from the second user's perspective.

[0088] In some cases, an avatar may be represented using one or more meshes (e.g., including multiple vertices, edges, and / or faces in three-dimensional space) with corresponding materials. The materials may include normal textures, diffuse or albedo textures, specular textures, any combination thereof, and / or other materials or textures. Various materials or textures may need to be obtained from registration or offline reconstruction.

[0089] The goal of generating a virtual representation or avatar for a user is to generate a mesh with various materials (e.g., normal, albedo, specular, etc.) that represent the user. In some cases, the mesh must have a known topology. However, meshes from scanners (e.g., LightCage, 3DMD, etc.) may not satisfy such constraints. To address such issues, the mesh can be retopologized after scanning, which will parameterize the mesh. Therefore, the mesh for the avatar can be associated with a plurality of animation parameters that define how the avatar will be animated during the virtual session. In some aspects, the animation parameters may include encoding parameters (e.g., codes or features) from a neural network, some or all of which may be used to generate an animated avatar. In some cases, the animation parameters may include facial codes or features, codes or features representing facial blend shapes, codes or features representing hand joints, codes or features representing body joints, codes or features representing head poses, audio streams (or encoded representations of audio streams), any combination thereof, and / or other parameters.

[0090] In order to transfer data of an avatar mesh between devices for an interactive virtual experience between users, the parameters used to animate the mesh may require a large data transfer rate. For example, for each frame, the parameters may need to be sent from one device of a first user to a second device of a second user at a frame rate of 30-60 frames per second (FPS) so that the second device can animate the avatar at that frame rate. It is necessary to be able to establish a call to a virtual representation or avatar in an efficient manner.

[0091] In some aspects, systems and techniques for providing an effective communication framework for virtual representation calls of a virtual environment (e.g., a metaverse virtual environment) are described herein. In some aspects, the process for establishing an avatar call can be performed directly between user devices, or can be performed via a server. In order to establish a call, a first device of a first user can send (e.g., directly or via a server) mesh information for reception by a second device of a second user, the mesh information defining a virtual representation or avatar of the first user for participating in a virtual session (e.g., a 3D collaborative virtual meeting, a computer or virtual game, or other virtual session in a metaverse environment) between the first user and the second user (and in some cases one or more other users). The mesh information may include information defining a mesh (e.g., including vertices, edges, and / or faces of a mesh) and information defining a texture and / or material of a mesh (e.g., a normal map defining a normal texture of a mesh, an albedo map defining a diffuse or albedo texture of a mesh, a specular map defining a specular texture of a mesh, any combination thereof, and / or other materials or textures). The second device may also send (e.g., directly or via a server) mesh information defining a virtual representation or avatar of the second user for reception by the first device of the first user. In some cases, the mesh information may only need to be sent once at the beginning of the call, with only mesh animation parameters exchanged thereafter, as described below. In some aspects, the mesh information may be compressed.

[0092] Once a call is established between the first device and the second device, the first device and the second device may exchange (e.g., directly or via a server) mesh animation parameters during the call so that the respective devices may animate a virtual representation during the virtual session. The mesh animation parameters may include information representing a face of a first user of the first device (e.g., a code or feature representing a facial appearance or other information), information representing a body of the first user of the first device (e.g., a code or feature representing a body appearance, a body posture, or other information), information representing one or more hands of the first user of the first device (e.g., a code or feature representing a hand appearance, a hand posture, or other information), posture information of the first device (e.g., a posture in six degrees of freedom (6-DOF), referred to as a 6-DOF posture), audio associated with an environment in which the first device is located, and / or other information. For example, an animation and scene rendering system of a device of a second user participating in the virtual session may process the mesh information and animation parameters from the first device to generate and / or animate a virtual representation or avatar for the first user of the first device during the virtual session. In some aspects, the mesh animation parameters may be compressed.

[0093] By using such a call flow and sending code or features representing the animation of the user's virtual representation (or avatar) during the call, the transmission data rate required to provide the information required to animate the virtual representation is greatly reduced. In some aspects, the mesh information and / or mesh animation parameters can be compressed, which can further reduce the transmission data rate.

[0094] Other examples include flows such as: device-to-device flows without an edge server and decoder on the target or receiving client device (e.g., for view-dependent and view-independent textures), device-to-device flows without an edge server and all processing (e.g., decoding, avatar animation, etc.) on the source or transmitting / sending client device (e.g., for view-dependent and view-independent textures), device-to-device flows with an edge server and decoder on a server device such as an edge server or cloud server (e.g., for view-dependent and view-independent textures), and device-to-device flows with an edge server and all processing (e.g., decoding, avatar animation, etc.) on the source or transmitting / sending client device (e.g., for view-dependent and view-independent textures).

[0095] As discussed herein, such solutions may provide a photo-realistic framework for a virtual representation (e.g., avatar) of a user in a virtual environment (e.g., for use in a metaverse).

[0096] Various aspects of the application will be described with respect to the accompanying drawings.

[0097] Figure 1 An example of an extended reality system 100 is shown. As shown, the extended reality system 100 includes a device 105, a network 120, and a communication link 125. In some cases, the device 105 may be an extended reality (XR) device, which may generally implement aspects of extended reality, including virtual reality (VR), augmented reality (AR), mixed reality (MR), etc. A system including the device 105, the network 120, or other elements in the extended reality system 100 may be referred to as an extended reality system.

[0098] Device 105 may overlay virtual objects in view 130 with real-world objects. For example, view 130 may generally refer to visual input to user 110 via device 105, a display generated by device 105, a configuration of virtual objects generated by device 105, etc. For example, view 130-A may refer to visible real-world objects (also referred to as physical objects) and visible virtual objects that are overlaid on or coexist with real-world objects at some initial time. View 130-B may refer to visible real-world objects and visible virtual objects that are overlaid on or coexist with real-world objects at some later time. As discussed herein, the positional differences in real-world objects (e.g., and thus overlaid virtual objects) may result from the view 130-A shifting to view 130-B at 135 due to head movement 115. In another example, view 130-A may refer to a completely virtual environment or scene at an initial time, and view 130-B may refer to a virtual environment or scene at a later time.

[0099] In general, the device 105 may generate, display, project, etc., a virtual object and / or a virtual environment to be viewed by the user 110 (e.g., where a portion of the virtual object and / or the virtual environment may be displayed based on a head pose prediction of the user 110 according to the techniques described herein). In some examples, the device 105 may include a transparent surface (e.g., optical glass) such that a virtual object may be displayed on the transparent surface to overlay the virtual object on a real-world object viewed through the transparent surface. Additionally or alternatively, the device 105 may project the virtual object onto the real-world environment. In some cases, the device 105 may include a camera and may display a real-world object (e.g., as a frame or image captured by the camera) and a virtual object overlaid on the displayed real-world object. In various examples, the device 105 may include aspects of a virtual reality headset, smart glasses, a live feed camera, a GPU, one or more sensors (e.g., such as one or more IMUs, image sensors, microphones, etc.), one or more output devices (e.g., such as speakers, displays, smart glasses, etc.), etc.

[0100] In some cases, head movement 115 may include rotation of user 110's head, translational head movement, etc. Device 105 may update view 130 of user 110 based on head movement 115. For example, device 105 may display view 130-A of user 110 before head movement 115. In some cases, after head movement 115, device 105 may display view 130-B to user 110. When view 130-A shifts to view 130-B, the extended reality system (e.g., device 105) may render or update virtual objects and / or other portions of the virtual environment for display.

[0101] In some cases, extended reality system 100 may provide various types of virtual experiences, such as a three-dimensional (3D) collaborative virtual environment, to a group of users (eg, including user 110 ). Figure 2 2 is a diagram illustrating an example of a 3D collaborative virtual environment 200 in which various users interact with each other in a virtual session via virtual representations (or avatars) of the users in the virtual environment 200. The virtual representations include a virtual representation 202 of a first user, a virtual representation 204 of a second user, a virtual representation 206 of a third user, a virtual representation 208 of a fourth user, and a virtual representation 210 of a fifth user. Other background information of the virtual environment 200 is also shown, including a virtual calendar 212, a virtual web page 214, and a virtual video conferencing interface 216. Users can experience the virtual environment visually, audibly, tactilely, or otherwise from each user's perspective while interacting with the virtual representations of other users. For example, the virtual environment 200 is shown from the perspective of the first user (represented by virtual representation 202).

[0102] It is important for XR systems to efficiently generate high-quality virtual representations (or avatars) with low latency. It may also be important for XR systems to render audio in an efficient manner to enhance the XR experience. For example, in Figure 2 In an example of a 3D collaborative virtual environment 200 of FIG. 1 , a first user's XR system (e.g., XR system 100) displays virtual representations 204-210 of other users participating in a virtual session. The users' virtual representations 204-210 and the background of the virtual environment 200 should be displayed in a realistic manner (e.g., as if the users were meeting in a real-world environment), such as by animating the heads, bodies, arms, and hands of the other users' virtual representations 204-210 as the users move in the real world. Audio captured by the other users' XR systems may need to be rendered spatially, or may be rendered in mono for output to the first user's XR system. The latency in rendering and animating the virtual representations 204-210 should be minimal so that the first user's user experience is as if the user is interacting with other users in a real-world environment.

[0103] Extended reality systems typically involve an XR device (e.g., HMD, smart glasses such as AR glasses, etc.) rendering and animating a virtual representation of the user and composing a virtual scene. Figure 3is a diagram illustrating an example of an extended reality system 300 in which two client devices (client device 302 and client device 304) exchange information for generating (e.g., rendering and / or animating) a virtual representation of a user and composing a virtual scene (e.g., virtual environment 200) including the virtual representation of the user. For example, client device 302 may send gesture information to client device 304. Client device 304 may use the gesture information to generate a rendering of a virtual representation of the user of client device 302 from the perspective of client device 304.

[0104] However, the computational complexity of generating the virtual environment may introduce significant power and resource requirements on the client devices 302, 304, resulting in excessive power and resource requirements on the client devices when implementing XR applications. This complexity may limit the ability of the client devices to generate XR experiences in a high-quality and low-latency manner. These power and resource requirements are exacerbated because XR technologies are often implemented in mobile and wearable devices (e.g., HMDs, XR glasses, etc.), which are manufactured in smaller, lighter, and more comfortable (e.g., by reducing the heat emitted by the device) form factors, with the idea that users can wear the XR devices for extended periods of time. Given such issues, Figure 3 The XR system 300 shown in FIG. 3 may have difficulty providing a high-quality XR experience.

[0105] As previously mentioned, this document describes systems and techniques for providing distributed generation of virtual content for a virtual environment or scene (eg, a metaverse virtual environment). Figure 4 4 is a schematic diagram showing an example of a system 400 according to various aspects of the present disclosure. As shown, the system 400 includes a client device 405, an animation and scene rendering system 410, and a storage device 415. Although the system 400 shows two devices 405, a single animation and scene rendering system 410, a single storage device 415, and a single network 420, the present disclosure is applicable to any system architecture with one or more devices 405, animation and scene rendering systems 410, storage devices 415, and networks 420. In some cases, the storage device 415 can be part of the animation and scene rendering system 410. The device 405, the animation and scene rendering system 410, and the storage device 415 can communicate with each other via the network 420 using a communication link 425 and exchange information such as multimedia packets, multimedia data, multimedia control information, and posture prediction parameters for the generation of virtual content that supports XR. In some cases, a portion of the technology described herein for providing distributed generation of virtual content can be performed by one or more of the devices 405, and a portion of the technology can be performed by the animation and scene rendering system 410, or both.

[0106] Device 405 may be an XR device (e.g., a head mounted display (HMD), XR glasses such as virtual reality (VR) glasses, augmented reality (AR) glasses, etc.), a mobile device (e.g., a cellular phone, a smart phone, a personal digital assistant (PDA), etc.), a wireless communication device, a tablet computer, a laptop computer, and / or other devices that support various types of communication and functional features related to multimedia (e.g., transmitting, receiving, broadcasting, streaming, sinking, capturing, storing, and recording multimedia data). Device 405 may be additionally or alternatively referred to by those skilled in the art as a user equipment (UE), a user device, a smart phone, a Bluetooth device, a Wi-Fi device, a mobile station, a subscriber station, a mobile unit, a subscriber unit, a wireless unit, a remote unit, a mobile device, a wireless device, a wireless communication device, a remote device, an access terminal, a mobile terminal, a wireless terminal, a remote terminal, a handset, a user agent, a mobile client, a client, and / or some other suitable terminology. In some cases, device 405 is also capable of communicating directly with another device (e.g., using a peer-to-peer (P2P) or device-to-device (D2D) protocol, such as using a sidelink communication). For example, device 405 may be able to receive or send various information, such as instructions or commands (eg, multimedia-related information), from another device 405 .

[0107] Device 405 may include application 430 and multimedia manager 435. Although system 400 shows device 405 including both application 430 and multimedia manager 435, application 430 and multimedia manager 435 may be optional features of device 405. In some cases, application 430 may be a multimedia-based application that may receive (e.g., download, stream, broadcast) from animation and scene rendering system 410, storage 415, or another device 405, or send (e.g., upload) multimedia data to animation and scene rendering system 410, storage 415, or another device 405 via use of communication link 425.

[0108] The multimedia manager 435 may be part of a general purpose processor, a digital signal processor (DSP), an image signal processor (ISP), a central processing unit (CPU), a graphics processing unit (GPU), a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), discrete gate or transistor logic components, discrete hardware components, or any combination thereof, or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination thereof, and / or the like designed to perform the functions described in the present disclosure. For example, the multimedia manager 435 may process multimedia (e.g., image data, video data, audio data) from the local memory or storage device 415 of the device 405 and / or write multimedia data to the local memory or storage device 415 of the device 405.

[0109] The multimedia manager 435 may also be configured to provide multimedia enhancement, multimedia restoration, multimedia analysis, multimedia compression, multimedia streaming, and multimedia synthesis, among other functions. For example, the multimedia manager 435 may perform white balancing, cropping, scaling (e.g., multimedia compression), adjusting resolution, multimedia stitching, color processing, multimedia filtering, spatial multimedia filtering, artifact removal, frame rate adjustment, multimedia encoding, multimedia decoding, and multimedia filtering. By way of further example, according to the techniques described herein, the multimedia manager 435 may process multimedia data to support server-based gesture prediction for XR.

[0110] The animation and scene rendering system 410 may be a server of a device, such as a data server, a cloud server, a server associated with a multimedia subscription provider, a proxy server, a web server, an application server, a communication server, a home server, a mobile server, an edge or cloud-based server, a personal computer acting as a server device, a mobile device (such as a mobile phone) acting as a server device, an XR device acting as a server device, a network router, any combination thereof, or other server devices. In some cases, the animation and scene rendering system 410 may include a multimedia distribution platform 440. In some cases, the multimedia distribution platform 440 may be a device or system separate from the animation and scene rendering system 410. The multimedia distribution platform 440 may allow the device 405 to discover, browse, share, and download multimedia via the network 420 using the communication link 425, and thus provide digital distribution of multimedia from the multimedia distribution platform 440. Therefore, digital distribution may be in the form of delivering media content (such as audio, video, images) without using physical media but through an online delivery medium (such as the Internet). For example, the device 405 may upload or download multimedia-related applications for streaming, downloading, uploading, processing, enhancing, etc. multimedia (e.g., images, audio, video). The animation and scene rendering system 410 or the multimedia distribution platform 440 may also send various information to the device 405, such as instructions or commands (e.g., multimedia-related information) for downloading multimedia-related applications on the device 405.

[0111] The storage device 415 can store various information, such as instructions or commands (e.g., multimedia related information). For example, the storage device 415 can store multimedia 445, information from the device 405 (e.g., gesture information, representation information of a virtual representation of a user or an avatar, such as codes or features related to facial representation, body representation, hand representation, etc., and / or other information). The device 405 and / or the animation and scene rendering system 410 can use the communication link 425 to retrieve the stored data from the storage device 415 via the network 420 and / or send more data to the storage device 415. In some examples, the storage device 415 can be a memory device (e.g., read-only memory (ROM), random access memory (RAM), cache memory, buffer memory, etc.) that stores various information (such as instructions or commands (e.g., multimedia related information)), a relational database (e.g., a relational database management system (RDBMS) or a structured query language (SQL) database), a non-relational database, a network database, an object-oriented database, or other types of databases.

[0112] The network 420 may provide encryption, access authorization, tracking, Internet Protocol (IP) connections, and other access, calculation, modification, and / or functionality. Examples of the network 420 may include any combination of cloud networks, local area networks (LANs), wide area networks (WANs), virtual private networks (VPNs), wireless networks (e.g., using 802.11), cellular networks (using third generation (3G), fourth generation (4G), long term evolution (LTE), or new radio (NR) systems (e.g., fifth generation (5G)), etc.).

[0113] The communication link 425 shown in the system 400 may include an uplink transmission from the device 405 to the animation and scene rendering system 410 and the storage device 415, and / or a downlink transmission from the animation and scene rendering system 410 and the storage device 415 to the device 405. The wireless link 425 can send bidirectional communication and / or unidirectional communication. In some examples, the communication link 425 can be a wired connection or a wireless connection or both. For example, the communication link 425 may include one or more connections, including but not limited to Wi-Fi, Bluetooth, Bluetooth Low Energy (BLE), cellular, Z-WAVE, 802.11, peer-to-peer, LAN, wireless local area network (WLAN), Ethernet, FireWire, fiber optics, and / or other connection types associated with wireless communication systems.

[0114] In some aspects, a user of device 405 (referred to as a first user) can participate in a virtual session with one or more other users (including a second user of an attached device). In such an example, the animation and scene rendering system 410 can process information received from device 405 (e.g., directly received from device 405, received from storage device 415, etc.) to generate and / or animate a virtual representation (or avatar) of the first user. The animation and scene rendering system 410 can compose a virtual scene that includes a virtual representation of the user and, in some cases, background virtual information from the perspective of a second user of the attached device. The animation and scene rendering system 410 can send frames of the virtual scene (e.g., via network 120) to the attached device. Further details on these aspects are provided below.

[0115] Figure 5 5 is a diagram showing an example of a device 500. The device 500 may be implemented as a client device (eg, Figure 4The device 500 may be implemented as an animation and scene rendering system (e.g., animation and scene rendering system 410) or as a central processing unit (CPU) 510 having a CPU memory 515, a GPU 525 having a GPU memory 530, a display 545, a display buffer 535 storing data associated with rendering, a user interface unit 505, and a system memory 540. For example, the system memory 540 may store a GPU driver 520 (shown as included in the CPU 510, as described below) having a compiler, a GPU program, a natively compiled GPU program, etc. The user interface unit 505, the CPU 510, the GPU 525, the system memory 540, the display 545, and the extended reality manager 550 may communicate with each other (e.g., using a system bus).

[0116] Examples of CPU 510 include, but are not limited to, a digital signal processor (DSP), a general-purpose microprocessor, an application-specific integrated circuit (ASIC), a field-programmable logic array (FPGA), or other equivalent integrated or discrete logic circuits. Figure 5 510 and GPU 525 may be integrated into a single unit. The CPU 510 may execute one or more software applications. Examples of applications may include operating systems, word processors, web browsers, email applications, spreadsheets, video games, audio and / or video capture, playback or editing applications, or other such applications that initiate the generation of image data to be presented via display 545. As shown, the CPU 510 may include a CPU memory 515. For example, the CPU memory 515 may represent an on-chip storage device or memory used when executing machine or object code. The CPU memory 515 may include one or more volatile or non-volatile memories or storage devices, such as flash memory, magnetic data media, optical storage media, etc. The CPU 510 is able to read values ​​from or write values ​​to the CPU memory 515 faster than reading values ​​from or writing values ​​to the system memory 540, which may be accessed, for example, via a system bus.

[0117] GPU 525 can represent one or more special processors for performing graphics operations. For example, GPU 525 can be a special hardware unit with fixed functions and programmable components for rendering graphics and executing GPU applications. GPU 525 can also include DSP, general-purpose microprocessor, ASIC, FPGA or other equivalent integrated or discrete logic circuits. GPU 525 can be built with a highly parallel structure that provides more efficient processing of complex graphics-related operations than CPU 510. For example, GPU 525 can include multiple processing elements configured to operate on multiple vertices or pixels in a parallel manner. The highly parallel nature of GPU 525 can allow GPU 525 to generate graphic images (e.g., graphical user interfaces and two-dimensional or three-dimensional graphics scenes) for display 545 faster than CPU 510.

[0118] In some cases, GPU 525 may be integrated into the motherboard of device 500. In other examples, GPU 525 may be present on a graphics card or other device or component installed in a port in the motherboard of device 500, or may be otherwise incorporated into a peripheral device configured to interoperate with device 500. As shown, GPU 525 may include GPU memory 530. For example, GPU memory 530 may represent an on-chip storage device or memory used when executing machine or object code. GPU memory 530 may include one or more volatile or non-volatile memory or storage devices, such as flash memory, magnetic data media, optical storage media, etc. GPU 525 is able to read values ​​from GPU memory 530 or write values ​​to GPU memory 530 faster than reading values ​​from system memory 540 or writing values ​​to system memory 540, which may be accessed, for example, via a system bus. That is, GPU 525 can read data from and write data to GPU memory 530 without using a system bus to access off-chip memory. This operation may allow GPU 525 to operate in a more efficient manner by reducing the need for GPU 525 to read and write data via the system bus, which may be subject to heavy bus traffic.

[0119] The display 545 represents a unit capable of displaying video, images, text, or any other type of data for consumption by a viewer. In some cases, such as when the device 500 is implemented as an animation and scene rendering system, the device 500 may not include the display 545. The display 545 may include a liquid crystal display (LCD), a light emitting diode (LED) display, an organic LED (OLED), an active matrix OLED (AMOLED), etc. The display buffer 535 represents a memory or storage device dedicated to storing data for presenting images (such as computer-generated graphics, still images, video frames, etc. for the display 545). The display buffer 535 may represent a two-dimensional buffer including a plurality of storage locations. In some cases, the number of storage locations within the display buffer 535 may generally correspond to the number of pixels to be displayed on the display 545. For example, if the display 545 is configured to include 640×480 pixels, the display buffer 535 may include 640×480 storage locations that store pixel color and intensity information, such as red, green, and blue pixel values ​​or other color values. The display buffer 535 may store the final pixel value of each pixel processed by the GPU 525. The display 545 may retrieve the final pixel value from the display buffer 535 and display the final image based on the pixel value stored in the display buffer 535.

[0120] The user interface unit 505 represents a unit with which a user can interact with other units of the device 500 (such as the CPU 510) or otherwise interface to communicate with other units of the device 500 (such as the CPU 510). Examples of the user interface unit 505 include, but are not limited to, a trackball, a mouse, a keyboard, and other types of input devices. The user interface unit 505 may also be or include a touch screen, and the touch screen may be incorporated as part of the display 545.

[0121] The system memory 540 may include one or more computer-readable storage media. Examples of the system memory 540 include, but are not limited to, random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disk read-only memory (CD-ROM) or other optical disk storage, magnetic disk storage or other magnetic storage, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and can be accessed by a computer or processor. The system memory 540 can store program modules and / or instructions that can be accessed by the CPU 510 for execution. In addition, the system memory 540 can store user applications and application surface data associated with the applications. In some cases, the system memory 540 can store information used by other components of the device 500 and / or information generated by other components of the device 500. For example, the system memory 540 can act as a device memory for the GPU 525 and can store data to be operated by the GPU 555 and data generated by operations performed by the GPU 525.

[0122] In some examples, system memory 540 may include instructions that cause CPU 510 or GPU 525 to perform functions attributed to CPU 510 or GPU 525 in various aspects of the present disclosure. In some examples, system memory 540 may be considered a non-transitory storage medium. The term "non-transitory" should not be interpreted to mean that system memory 540 is non-removable. As an example, system memory 540 may be removed from device 500 and moved to another device. As another example, a system memory substantially similar to system memory 540 may be inserted into device 500. In some examples, non-transitory storage media may store data that may change over time (e.g., in RAM).

[0123] The system memory 540 may store the GPU driver 520 and the compiler, GPU program, and natively compiled GPU program. The GPU driver 520 may represent a computer program or executable code that provides an interface for accessing the GPU 525. The CPU 510 may execute the GPU driver 520 or a portion thereof to interface with the GPU 525, and for this reason, the GPU driver 520 may be executed in the CPU 510. Figure 5510. The GPU driver 520 is accessible by programs executed by the CPU 510 or other executable programs, including the GPU program stored in the system memory 540. Thus, when one of the software applications executing on the CPU 510 requires graphics processing, the CPU 510 can provide graphics commands and graphics data to the GPU 525 for rendering to the display 545 (e.g., via the GPU driver 520).

[0124] In some cases, the GPU program may include, for example, code written in a high-level (HL) programming language using an application programming interface (API). Examples of APIs include Open Graphics Library ("OpenGL"), DirectX, Render-Man, WebGL, or any other public or proprietary standard graphics API. Instructions may also conform to so-called heterogeneous computing libraries, such as Open Computing Language ("OpenCL"), DirectCompute, etc. Typically, an API includes a predetermined standardized command set executed by associated hardware. API commands allow a user to instruct the hardware components of the GPU 525 to execute commands without the user knowing the details of the hardware components. In order to process graphics rendering instructions, the CPU 510 may issue one or more rendering commands to the GPU 525 (e.g., through the GPU driver 520) to cause the GPU 525 to perform some or all of the rendering of graphics data. In some examples, the graphics data to be rendered may include a list of graphics primitives (e.g., points, lines, triangles, quadrilaterals, etc.).

[0125] The GPU program stored in the system memory 540 may call or otherwise include one or more functions provided by the GPU driver 520. The CPU 510 typically executes a program in which the GPU program is embedded, and when encountering the GPU program, passes the GPU program to the GPU driver 520. The CPU 510 executes the GPU driver 520 in this context to process the GPU program. That is, for example, the GPU driver 520 may process the GPU program by compiling the GPU program into an object or machine code that can be executed by the GPU 525. The object code may be referred to as a natively compiled GPU program. In some examples, a compiler associated with the GPU driver 520 may operate in real time or near real time to compile the GPU program during the execution of the program in which the GPU program is embedded. For example, a compiler typically represents a unit that simplifies the HL instructions defined in the HL programming language into low-level (LL) instructions of the LL programming language. After compilation, these LL instructions can be executed by a specific type of processor or other type of hardware, such as an FPGA, an ASIC, etc. (including but not limited to the CPU 510 and the GPU 525).

[0126] exist Figure 5 In the example of , when executing HL code including a GPU program, the compiler can receive the GPU program from CPU 510. That is, the software application executed by CPU 510 can call GPU driver 520 (e.g., via a graphics API) to issue one or more commands to GPU 525 for rendering one or more graphics primitives into a displayable graphics image. The compiler can compile the GPU program to generate a natively compiled GPU program that conforms to the LL programming language. The compiler can then output the natively compiled GPU program including LL instructions. In some examples, the LL instructions can be provided to GPU 525 in the form of a drawing primitive list (e.g., triangles, rectangles, etc.).

[0127] LL instructions (e.g., which may alternatively be referred to as primitive definitions) may include vertex specifications specifying one or more vertices associated with the primitives to be rendered. The vertex specifications may include the position coordinates of each vertex, and in some cases, include other attributes associated with the vertex, such as color coordinates, normal vectors, and texture coordinates. Primitive definitions may include primitive type information, scaling information, rotation information, and the like. Based on instructions issued by a software application (e.g., a program in which a GPU program is embedded), the GPU driver 520 may formulate one or more commands, one or more commands specifying one or more operations for the GPU 525 to perform in order to render the primitives. When the GPU 525 receives a command from the CPU 510, it may decode the command and configure one or more processing elements to perform the specified operation, and may output rendered data to the display buffer 535.

[0128] The GPU 525 may receive a locally compiled GPU program, and then in some examples, the GPU 525 renders one or more images and outputs the rendered images to the display buffer 535. For example, the GPU 525 may generate a plurality of primitives to be displayed at the display 545. Primitives may include one or more of lines (including curves, splines, etc.), points, circles, ellipses, polygons (e.g., triangles), or any other two-dimensional primitives. The term "primitive" may also refer to three-dimensional primitives, such as cubes, cylinders, spheres, cones, pyramids, tori, etc. In general, the term "primitive" refers to any basic geometric shape or element that can be rendered by the GPU 525 to be displayed as an image (or a frame in the context of video data) via the display 545. The GPU 525 may transform primitives and other attributes of primitives (e.g., defining color, texture, lighting, camera configuration, or other aspects) into a so-called "world space" by applying one or more model transformations (which may also be specified in the state data). Once transformed, the GPU 525 can apply a view transform to the active camera (which can also be specified in the state data defining the camera) to transform the coordinates of the primitives and lights into camera or eye space. The GPU 525 can also perform vertex shading to render the appearance of the primitives in view of any active lights. The GPU 525 can perform vertex shading in one or more of the above-mentioned model, world, or view space.

[0129] Once the primitives are shaded, the GPU 525 may perform projection to project the image into a regular observation volume. After transforming the model from the eye space to the regular observation volume, the GPU 525 may perform clipping to remove any primitives that do not reside at least partially within the regular observation volume. For example, the GPU 525 may remove any primitives that are not within the camera's frame. The GPU 525 may then map the coordinates of the primitives from the observation volume to the screen space, effectively reducing the three-dimensional coordinates of the primitives to the two-dimensional coordinates of the screen. Given the transformed and projected vertices that define the primitives with their associated shading data, the GPU 525 may then rasterize the primitives. In general, rasterization may refer to the task of taking an image described in a vector graphics format and converting it into a raster image (e.g., a pixelated image) to be output on a video display or stored in a bitmap file format.

[0130] The GPU 525 may include a dedicated fast bin buffer (e.g., a fast memory buffer such as GMEM that can be referenced by the GPU memory 530). As discussed herein, the rendering surface can be divided into bins. In some cases, the bin size is determined by the format (e.g., pixel color and depth information) and the rendering target resolution divided by the total amount of GMEM. The number of bins can vary based on the device 500 hardware, the target resolution size, and the target display format. The rendering pass can draw (e.g., render, write, etc.) pixels into GMEM (e.g., with high bandwidth matching the capabilities of the GPU). The GPU 525 can then parse the GMEM (e.g., write the blended pixel values ​​from the GMEM as a single layer burst to the display buffer 535 or the frame buffer in the system memory 540). This can be referred to as bin-based or tile-based rendering. When all bins are completed, the driver can swap buffers and start the binning process again for the next frame.

[0131] For example, the GPU 525 may implement a tile-based architecture that renders an image or render target by dividing the image into multiple parts (referred to as tiles or bins). The size of the bin may be determined based on the size of the GPU memory 530 (e.g., which may alternatively be referred to herein as GMEM or cache), the resolution of the display 545, the color or Z precision of the render target, etc. When implementing tile-based rendering, the GPU 525 may perform a binning pass and one or more rendering passes. For example, with respect to a binning pass, the GPU 525 may process the entire image and sort the rasterized primitives into bins.

[0132] Device 500 may use sensor data, sensor statistics, or other data from one or more sensors. Some examples of monitoring sensors may include an IMU, an eye tracker, a tremor sensor, a heart rate sensor, etc. In some cases, an IMU may be included in device 500 and may use some combination of an accelerometer, a gyroscope, or a magnetometer to measure and report specific force, angular rate, and sometimes orientation of the body.

[0133] As shown, device 500 may include an extended reality manager 550. Extended reality manager 550 may implement aspects of extended reality, augmented reality, virtual reality, etc. In some cases, such as when device 500 is implemented as a client device (e.g., Figure 4When the device 500 is configured to be in a state of being in a physical environment, the device 500 may be configured to be in a state of being in a physical environment, such as a physical environment, ... Figure 4 When the animation and scene rendering system 410 is used, the extended reality manager 550 can process information provided by the client device as input information to generate and / or animate a virtual representation for a user of the client device.

[0134] Figure 6 6 is a diagram illustrating another example of an XR system 600 including a client device 602, a client device 604, and a client device 606 in communication with an animation and scene rendering system 610. As shown, each of the client devices 602, 604, and 606 provides input information to the animation and scene rendering system 610. In some examples, the input information from the client device 602 may include information representing a face of a user of the client device 602 (e.g., a code or feature representing facial appearance or other information), information representing a body of the user of the client device 602 (e.g., a code or feature representing body appearance, body posture, or other information), information representing one or more hands of the user of the client device 602 (e.g., a code or feature representing hand appearance, hand posture, or other information), posture information of the client device 602 (e.g., a posture in six degrees of freedom (6-DOF), referred to as a 6-DOF posture), audio associated with an environment in which the client device 602 is located, any combination thereof, and / or other information. Similar information may be provided from the client device 604 and the client device 606.

[0135] The user virtual representation system 620 of the animation and scene rendering system 610 can process information (e.g., facial representation, body representation, posture information, etc.) provided from each of the client devices 602, 604, 606, and can generate and / or animate a virtual representation (or avatar) for each respective user of the client devices 602, 604, 606. As used herein, animation of a user's virtual representation (or avatar) refers to modifying the position, movement, behavior, or other characteristics of the virtual representation so that the animation matches the corresponding position, movement, behavior, etc. of the corresponding user in the real world environment or space. In some aspects, as described below with respect to Figure 8To describe in more detail, to generate and / or animate a virtual representation of a first user, the user virtual representation system 620 may generate and / or animate a virtual representation of a face of the user of the client device 602 (the facial representation), generate and / or animate a virtual representation of a body of the user of the client device 602 (the body representation), and generate and / or animate a virtual representation of the hair of the user of the client device 602 (the hair representation). In some cases, the user virtual representation system 620 may combine or append the facial representation with the body representation to generate a combined virtual representation. The user virtual representation system 620 may then add the hair representation to the combined virtual representation to generate a virtual representation for one or more content frames (in the example embodiment). Figure 6 The scene rendering system may send one or more frames to the client devices 602, 604, 606.

[0136] The scene synthesis system 622 of the animation and scene rendering system 610 can then synthesize a virtual scene or environment for the virtual session and generate a corresponding target view (e.g., a content frame, such as virtual content or a combination of real world and virtual content) for each of the client devices 602, 604, 606 from the corresponding perspective of each device or user in the virtual scene or environment. The virtual scene synthesized by the scene synthesis system 622 includes virtual representations of the users involved in the virtual session and any virtual background of the scene. In one example, using information received from other client devices 604, 606, the scene synthesis system 622 can synthesize the virtual scene (including the virtual representations of the users and the background information) and generate a frame representing a view of the virtual scene from the perspective of the user of the client device 602.

[0137] Figure 7 is a diagram illustrating an example of an XR system 700 configured to perform aspects described herein. The XR 700 system includes a user participating in a virtual session (in some cases with a Figure 7 The client device 702 and the client device 704 may include a first XR device (e.g., a virtual reality (VR) head mounted display (HMD), augmented reality (AR) or mixed reality (MR) glasses, etc.), and the client device 704 may include a second XR device.

[0138] The frames transmitted to the client device 704 include a view of the virtual scene from the perspective of the user of the client device 704 based on the relative difference between the pose of the client device 704 and each corresponding pose of the client device 702 and any other devices providing information to the animation and scene rendering system 710 (e.g., the user can view virtual representations of other users in the virtual scene as if the users were together in the physical space). Figure 7 In the example of , client device 702 can be considered a source (e.g., an information source for generating at least one frame for a virtual scene), and client device 704 can be considered a target (e.g., a target for receiving at least one frame generated for a virtual scene).

[0139] As described above, information sent by a client device to the animation and scene rendering system (e.g., sent from client device 702 to animation and scene rendering system 710) may include information representing a face of a user of client device 702 (e.g., a code or feature representing facial appearance, or other information), information representing a body of a user of client device 702 (e.g., a code or feature representing body appearance, a posture of the body, or other information), information representing one or more hands of a user of client device 702 (e.g., a code or feature representing hand appearance, a posture of the hand, or other information), posture information of client device 702 (e.g., a posture in six degrees of freedom (6-DOF), referred to as a 6-DOF posture), audio associated with the user of client device 702, the environment in which client device 702 is located, any combination thereof, and / or other information.

[0140] For example, computing device 702 may include face engine 709, gesture engine 712, body engine 714, hand engine 716, and audio decoder 718. In some aspects, computing device 702 may include a body engine configured to generate a virtual representation of a user's body. Figure 7 Other components or engines than those shown (e.g. Figure 5 One or more components on the device 500, Fig.43 4300, etc.). Computing device 704 is shown to include a video decoder 732, a re-projection engine 734, a display 736, and a future pose prediction engine 738. In some cases, each client device (e.g., Figure 6 Client device 702, client device 702, client device 602, Figure 6 Client device 604, Figure 6The client device 606 of the present invention may include a face engine, a pose engine, a body engine, a hand engine, an audio decoder (and in some cases an audio decoder or a combined audio encoder-decoder), a video decoder (and in some cases a video encoder or a combined video encoder-decoder), a reprojection engine, a display, and a future pose prediction engine. Figure 7 These engines or components are not shown with respect to client device 702 and client device 704 because Figure 7 In the example of FIG. 7 , client device 702 is a source device and client device 704 is a target device.

[0141] The facial engine 709 of the client device 702 may receive one or more input frames 715 from one or more cameras of the client device 702. For example, the input frames 715 received by the facial engine 709 may include frames (or images) captured by one or more cameras having a field of view of the mouth of the user of the client device 702, the left eye of the user, and the right eye of the user. Other images may also be processed by the facial engine 709. In some cases, the input frames 715 may be included in a sequence of frames (e.g., a video, a sequence of independent or still images, etc.). The facial engine 709 may generate and output a code (e.g., a feature vector or multiple feature vectors) representing the face of the user of the client device 702. The facial engine 709 may send the code representing the user's face to the animation and scene rendering system 710. In an illustrative example, the facial engine 709 may include one or more facial encoders, the one or more facial encoders including one or more machine learning systems (e.g., deep learning networks, such as deep neural networks) trained to represent the user's face with a code or feature vector. In some cases, the face engine 709 may include a separate encoder for each type of image processed by the face engine 709, such as a first encoder for frames or images of the mouth, a second encoder for frames or images of the right eye, and a third encoder for frames or images of the left eye. Training may include supervised learning (e.g., using labeled images and one or more loss functions, such as mean square error MSE), semi-supervised learning, unsupervised learning, etc.). For example, a deep neural network may generate and output a code (e.g., a feature vector or multiple feature vectors) representing the face of a user of the client device 702. The code may be a latent code (or bitstream) that can be decoded by a face decoder (not shown) of the animation and scene rendering system 710, which is trained to decode the code (or feature vector) representing the user's face in order to generate a virtual representation of the face (e.g., a face mesh). For example, the face decoder of the animation and scene rendering system 710 may decode the code received from the client device 702 to generate a virtual representation of the user's face.

[0142] The posture engine 712 can determine the posture (e.g., 6-DOF posture) of the client device 702 in the 3D environment (and therefore determine the head posture of the user of the client device 702). The 6-DOF posture can be an absolute posture. In some cases, the posture engine 712 may include a 6-DOF tracker, which can track three rotation data (e.g., including pitch, roll, and yaw) and three translation data (e.g., horizontal displacement, vertical displacement, and depth displacement relative to a reference point). The posture engine 712 (e.g., 6-DOF tracker) can receive sensor data as input from one or more sensors. In some cases, the posture engine 712 includes one or more sensors. In some examples, the one or more sensors may include one or more inertial measurement units (IMUs) (e.g., accelerometers, gyroscopes, etc.), and the sensor data may include IMU samples from one or more IMUs. The posture engine 712 can determine raw posture data based on the sensor data. The raw gesture data may include 6DOF data representing the gesture of the client device 702, such as three-dimensional rotation data (eg, including pitch, roll, and yaw) and three-dimensional translation data (eg, horizontal displacement, vertical displacement, and depth displacement relative to a reference point).

[0143] The body engine 714 may receive one or more input frames 715 from one or more cameras of the client device 702. The frames received by the body engine 714 and the camera used to capture the frames may be the same or different from the frames received by the face engine 709 and the camera used to capture those frames. For example, the (one or more) input frames received by the body engine 714 may include frames (or images) captured by one or more cameras having a field of view of the body of the user of the client device 702 (e.g., parts other than the face, such as the user's neck, shoulders, torso, lower body, feet, etc.). The body engine 714 may perform one or more techniques to output a representation of the body of the user of the client device 702. In one example, the body engine 714 may generate and output a 3D mesh representing the shape of the body (e.g., including multiple vertices, edges, and / or faces in 3D space). In another example, the body engine 714 may generate and output a code (e.g., a feature vector or multiple feature vectors) representing the body of the user of the client device 702. In one illustrative example, the body engine 714 may include one or more body encoders that include one or more machine learning systems (e.g., deep learning networks, such as deep neural networks) trained to represent the user's body with a code or feature vector. Training may include supervised learning (e.g., using labeled images and one or more loss functions, such as MSE), semi-supervised learning, unsupervised learning, etc.). For example, the deep neural network may generate and output a code (e.g., a feature vector or multiple feature vectors) representing the body of the user of the client device 702. The code may be a latent code (or bitstream) that can be decoded by a body decoder (not shown) of the animation and scene rendering system 710, which is trained to decode the code (or feature vector) representing the user's virtual body in order to generate a virtual representation of the body (e.g., a body mesh). For example, the body decoder of the animation and scene rendering system 710 may decode the code received from the client device 702 to generate a virtual representation of the user's body.

[0144] The hand engine 716 may receive one or more input frames 715 from one or more cameras of the client device 702. The frames received by the hand engine 716 and the camera(s) used to capture the frames may be the same or different than the frames received by the face engine 709 and / or the body engine 714 and the camera(s) used to capture those frames. For example, the input frames received by the hand engine 716 may include frames (or images) captured by one or more cameras having a field of view of the hands of the user of the client device 702. The hand engine 716 may perform one or more techniques to output a representation of one or more hands of the user of the client device 702. In one example, the hand engine 716 may generate and output a 3D mesh representing the shape of the hand (e.g., including multiple vertices, edges, and / or faces in 3D space). In another example, the hand engine 716 may output a code (e.g., a feature vector or multiple feature vectors) representing one or more hands. For example, the hand engine 716 may include one or more body encoders that include one or more machine learning systems (e.g., deep learning networks, such as deep neural networks) that are trained (e.g., using supervised learning, semi-supervised learning, unsupervised learning, etc.) to represent the user's hands with codes or feature vectors. The code may be a latent code (or bitstream) that can be decoded by a hand decoder (not shown) of the animation and scene rendering system 710, which is trained to decode the code (or feature vector) representing the user's virtual hand in order to generate a virtual representation of the body (e.g., a body mesh). For example, the hand decoder of the animation and scene rendering system 710 can decode the code received from the client device 702 to generate a virtual representation of the user's hand (or in some cases, one hand).

[0145] The audio decoder 718 (e.g., an audio decoder or a combined audio encoder-decoder) can receive audio data 717, such as audio obtained using one or more microphones of the client device 702. The audio decoder 718 can encode or compress the audio data and send the encoded audio to the animation and scene rendering system 710. The audio decoder 718 can perform any type of audio encoding to compress the audio data, such as a coding technique based on a modified discrete cosine transform (MDCT). The encoded audio can be decoded by an audio decoder (not shown) of the animation and scene rendering system 710, which is configured to perform the inverse process of the audio encoding process performed by the audio decoder 718 to obtain decoded (or decompressed) audio.

[0146] The user virtual representation system 720 of the animation and scene rendering system 710 can receive input information received from the client device 702 and use the input information to generate and / or animate a virtual representation (or avatar) of the user of the client device 702. In some aspects, the user virtual representation system 720 can also use the future predicted pose of the client device 704 to generate and / or animate the virtual representation of the user of the client device 702. In such aspects, the future pose prediction engine 738 of the client device 704 can predict the pose of the client device 704 (e.g., corresponding to the predicted head pose, body pose, hand pose, etc. of the user) based on the target pose (e.g., target pose 737) and send the predicted pose to the user virtual representation system 720. For example, the future pose prediction engine 738 can predict the future pose of the client device 704 at a future time (e.g., which can be the time of the prediction) based on the model (e.g., corresponding to the head position, head orientation, line of sight, etc. of the user of the client device 704). Figure 1 The future time T may correspond to a time when the client device 704 will output or display a target view frame (e.g., target view frame 757). As used herein, references to a pose of a client device (e.g., client device 704) and a head pose, body pose, etc., of a user of the client device may be used interchangeably.

[0147] Predicted poses can be useful when generating virtual representations, since in some cases virtual objects may appear delayed to the user when compared to the object's expected view to the user or compared to the real-world object the user is viewing (e.g., in AR, MR, or VR see-through scenarios). Figure 1 As an illustrative example, in the absence of head motion or pose prediction, updating of a virtual object in view 130-B from a previous view 130-A may be delayed until a head pose measurement is made so that the position, orientation, size, etc. of the virtual object can be updated accordingly. In some cases, the delay may be due to system latency (e.g., end-to-end system latency between the client device 704 and a system or device that renders virtual content (such as the animation and scene rendering system 710)), which may be caused by rendering, time warping, or both. In some cases, such delays may be referred to as round-trip latency or dynamic registration error. In some cases, the error may be large enough that a user of the client device 704 may perform a head motion (e.g., a head pose measurement may be made) before the pose measurement is ready for display. Figure 1 ). Therefore, it may be beneficial to predict the head movement 115 so that virtual objects associated with the view 130 -B can be determined and updated in real time based on the prediction (eg, pattern) in the head movement 115 .

[0148] As described above, user virtual representation system 720 can generate and / or animate a virtual representation of a user of client device 702 using input information received from client device 702 and future predicted pose information from future pose prediction engine 738. As described herein, animation of a user's virtual representation can include modifying the position, movement, behavior, or other characteristics of the virtual representation to match a corresponding position, movement, behavior, etc., of the user in the real world or physical space. Figure 8 An example of a user virtual representation system 720 is described. In some aspects, the user virtual representation system 720 can be implemented as a deep learning network, such as a neural network (e.g., a convolutional neural network (CNN), an autoencoder, or other type of neural network) trained using supervised learning, semi-supervised learning, unsupervised learning, etc. based on input training data (e.g., a code representing a user's face, a code representing a user's body, a code representing a user's hand, pose information such as 6-DOF pose information of a client device of the user, inverse kinematics information, etc.) to generate and / or animate a virtual representation of a user of a client device.

[0149] Figure 8 is a diagram showing an illustrative example of a user virtual representation system 720. A facial representation animation engine 842 of the user virtual representation system 720 may generate and / or animate a virtual representation (referred to as a facial representation) of a face (or entire head) of a first user of a client device 702. The facial representation may include a 3D mesh (including a plurality of vertices, edges, and / or faces in a 3D space) representing the shape of the face or head of the user of the client device 702 and texture information representing features of the face.

[0150] like Figure 8As shown, the facial representation animation engine 842 can obtain (e.g., receive, retrieve, etc.) a facial code (e.g., a feature vector or multiple feature vectors) output from the facial engine 709, a source pose of the client device 702 (e.g., a 6-DOF pose output by the pose engine 712), and a target pose (e.g., target pose 737) of the client device 704 (e.g., a 6-DOF pose output by the pose engine of the client device 704 and received by the animation and scene rendering system 710). The facial representation animation engine 842 can also obtain texture information (e.g., color and surface details of the face) and depth information (shown as enrolled texture and depth information 841). The texture information and depth information can be obtained during an enrollment phase, where a user uses one or more cameras of a client device (e.g., client devices 702, 704, etc.) to capture one or more images of the user. The images can be processed to determine text and depth information of the user's face (and in some cases the user's body and / or hair). Using the facial codes, the source and target poses, and the registered texture and depth information 841, the facial representation animation engine 842 can generate (or animate) the face of the user of the client device 702 in a pose from the perspective of the user of the client device 704 (e.g., based on the relative difference between the pose of the client device 704 and each corresponding pose of the client device 702 and any other device providing information to the animation and scene rendering system 710) and using facial features corresponding to the features of the user's face. For example, the facial representation animation engine 842 can calculate a mapping based on the facial codes, the source and target poses, and the registered texture and depth information 841. The mapping can be learned using offline training data (e.g., using supervised training, unsupervised training, semi-supervised training, etc.). The facial representation animation engine 842 can use the mapping to determine how the animation of the user's face should appear based on the input and registered data.

[0151] The face-body combiner engine 844 of the user virtual representation system 720 can obtain, generate, and / or animate a virtual representation of the body of a user of the client device 702 (referred to as a body representation). Figure 8As shown, the face-body combiner engine 844 can obtain the pose of one or more hands of the user of the client device 702 output by the hand engine 716, the source pose of the client device 702, and the target pose of the client device 704. In some aspects, the face-body combiner engine 844 can also obtain the pose of the body of the user of the client device 702 output by the body engine 714. In other aspects, the pose of the body can be estimated using inverse kinematics. For example, using the head pose (e.g., 6DOF head pose or device pose) from the pose engine 712 and the hand pose from the hand engine 716, the face-body combiner engine 844 can estimate the joint parameters of the user's body to manipulate the body, arms, legs, etc. The face-body combiner engine 844 can estimate the joint parameters using any suitable kinematic equations (e.g., inverse kinematics equations or systems of equations, such as the forward and backward inverse kinematics (FABRIK) heuristic method). Using the hand pose, source pose, and target pose, and the body pose or results of inverse kinematics, the face-body combiner engine 844 can determine the body pose of the user of the client device 702 from the perspective of the user of the client device 704 (e.g., based on the relative difference between the pose of the client device 704 and each corresponding pose of the client device 702 and any other device providing information to the animation and scene rendering system 710).

[0152] The face-body combiner engine 844 may combine or append the face representation with the body representation to generate a combined virtual representation. For example, as described above, the face representation may include a 3D mesh of the face or head of the user of the client device 702 and texture information representing the features of the face. The body representation may include a 3D mesh of the body of the user of the client device 702 and texture information representing the features of the body. The face-body combiner engine 844 may combine or merge the previously synthesized 3D mesh of the face representation (or the entire head) with the 3D mesh of the body representation (including one or more hands) to generate a combined virtual representation (e.g., a combined 3D mesh of the face / head and body). In some cases, the 3D mesh of the body representation may be a generic 3D mesh of a male or female (corresponding to whether the user is male or female). In one example, the vertices of the face / head 3D mesh and the body 3D mesh corresponding to the neck of the user are merged (e.g., connected together) to make a seamless transition between the face / head and body of the user. After combining or merging the 3D mesh of the face representation / head with the 3D mesh of the body representation, the two 3D meshes will be a joint 3D mesh. The face-body combiner engine 844 can then ensure that the skin color of the body mesh matches the skin diffuse color (e.g., albedo) and other materials of the face / head mesh. For example, the face-body combiner engine 844 can estimate the skin parameters of the head and can transfer the skin parameters to the skin of the body. In one illustrative example, the face-body combiner engine 844 can use a transfer map (e.g., provided by Autodesk TM Provides a diffuse map that represents the diffuse color of the face / head mesh to the body mesh.

[0153] The hair animation engine 846 of the user virtual representation system 720 can generate and / or animate a virtual representation of the hair of the user of the client device 702 (referred to as a hair representation). The hair representation may include a 3D mesh representing the shape of the hair (including multiple vertices, edges and / or faces in 3D space) and texture information representing the characteristics of the hair. The hair animation engine 846 can use any suitable technology to perform hair animation. In one example, the hair animation engine 846 can obtain or receive a reference 3D mesh based on a hairpin or hair strand that can be animated based on the movement of the head. In another example, the hair animation engine 846 can perform an image restoration technique by adding color to a 2D image output (output by the user virtual representation system 720) of the user's virtual representation (or avatar) based on the movement or posture of the head and / or body. An illustrative example of an algorithm that can be used by the hair animation engine 846 to generate and / or animate a virtual representation of the user's hair includes a multi-input conditional hair image generation adversarial network (GAN) (MichiGAN).

[0154] In some aspects, the facial representation animation engine 842 may also obtain the registered texture and depth information 841, which may include text and depth information of the hair of the user of the client device 702. The animation and scene rendering system 720 may then combine or add the hair representation to the combined virtual representation to generate an end-user virtual representation 847 of the user of the client device 702. For example, the animation and scene rendering system 720 may combine a combined 3D mesh (including a combination of a face mesh and a body mesh including one or more hands) with a 3D mesh of the hair to generate the end-user virtual representation 847.

[0155] return Figure 7 , the user virtual representation system 720 can output the virtual representation of the user of the client device 702 to the scene synthesis system 722 of the animation and scene rendering system 710. Background scene information 719 and other user virtual representations 724 of other users participating in the virtual session (if any) are also provided to the scene synthesis system 722. The background scene information 719 can include the lighting of the virtual scene, virtual objects in the scene (e.g., virtual buildings, virtual streets, virtual animals, etc.), and / or other details related to the virtual scene (e.g., sky, clouds, etc.). In some cases, such as in an AR, MR, or VR perspective setting (where a video frame of the real world environment is displayed to the user), the lighting information can include the lighting of the real world environment in which the client device 702 and / or the client device 704 (and / or other client devices participating in the virtual session) are located. In some cases, the future predicted posture from the future posture prediction engine 738 of the client device 704 can also be input to the scene synthesis system 722.

[0156] Using the virtual representation of the user of client device 702, background scene information 719, and virtual representations of other users (if any) (and in some cases future predicted poses), scene composition system 722 can synthesize a target view frame of the virtual scene using a view of the virtual scene from the perspective of the user of client device 704 based on the relative difference between the pose of client device 704 and each corresponding pose of client device 702 and any other client devices of the user participating in the virtual session. For example, the composed target view frame can include a mixture of the virtual representation of the user of client device 702, background scene information 719 (e.g., lighting, background objects, sky, etc.), and virtual representations of other users that may be involved in the virtual session. The poses of the virtual representations of any other users are also based on the pose of client device 704 (corresponding to the pose of the user of client device 704).

[0157] Fig. 9is a diagram illustrating an example of a scene synthesis system 722. As described above, the background scene information 719 may include lighting information of a virtual scene (or, in some cases, lighting information of a real-world environment). The lighting extraction engine 952 of the scene synthesis system 722 may obtain the background scene information 719 and extract or determine the lighting information of the virtual scene (or real-world scene) from the background scene information 719. For example, a high dynamic range (HDR) map of a scene (e.g., a high dynamic range image (HDRI) map including a 360-degree image) may be captured by rotating a camera around the scene in multiple directions (e.g., in all directions) from a single viewpoint in some cases. The HDR map or image maps the lighting characteristics of the scene by storing real-world values ​​of light and reflection at each location in the scene. The scene synthesis system 722 may then extract light from the HDR map or image (e.g., using one or more convolution kernels).

[0158] The lighting extraction engine 952 may output the extracted lighting information to the relighting engine 950 of the scene synthesis system 722. When generating a target view image of the virtual scene, the relighting engine 950 may use the lighting information extracted from the background scene information 719 to perform relighting of user virtual representations A 947, user virtual representations i 948, to user virtual representations N 949 (where there are a total of N user virtual representations, N being greater than or equal to 0) representing the users participating in the virtual session. User virtual representation A 947 may represent Figure 7 948 to N 949 may represent one or more other users participating in the virtual session. The relighting engine 950 may modify the lighting of virtual representation A 947, user virtual representation i 948, to user virtual representation N 949 so that the virtual representations appear as realistic as possible when viewed by the user of the client device 704. The results of the relighting performed by the relighting engine 950 include relighting user virtual representation A 951, relighting user virtual representation i 953, to relighting user virtual representation N 955.

[0159] In some examples, the relighting engine 950 may include one or more neural networks that are trained (e.g., using supervised training, unsupervised training, semi-supervised training, etc.) to relight the foreground (and in some cases the background) of the virtual scene using the extracted light information. For example, the relighting engine 950 may segment the foreground (e.g., the user) from one or more images of the scene, and may estimate the geometry and / or normal map of the user. The relighting engine 950 may use the normal map and lighting information from the lighting extraction engine 952 (e.g., an HDRI map or the user's current environment) to estimate an albedo map that includes albedo information for the user. The relighting engine 950 may then use the albedo information and the lighting information from the lighting extraction engine 952 (e.g., an HDRI map) to relight the scene. In some cases, the relighting engine 950 may use techniques other than machine learning.

[0160] The blending engine 956 of the scene composition system 722 may blend the re-lit virtual representations 951, 953, and 955 into the background scene represented by the background scene information 719 to generate the target view frame 757. In one illustrative example, the blending engine 956 may apply Poisson image blending (e.g., using image gradients) to blend the re-lit user virtual representations 951, 953, and 955 into the background scene. In another illustrative example, the blending engine 956 may include one or more neural networks that are trained (e.g., using supervised training, unsupervised training, semi-supervised training, etc.) to blend the re-lit user virtual representations 951, 953, and 955 into the background scene.

[0161] return Figure 7 , the spatial audio engine 726 can receive as input the encoded audio generated by the audio decoder 718, and in some cases, receive as input the future predicted gesture information from the future gesture prediction engine 738. The input is used to generate audio that is spatially oriented according to the gesture of the user of the client device 704. The lip sync engine 728 can synchronize the animation of the lips of the virtual representation of the user of the client device 702 depicted in the target view frame 757 with the spatial audio output by the spatial audio engine 726. The target view frame 757 (after being synchronized by the lip sync engine 728) can be output to the video encoder 730. The video encoder 730 can encode (or compress) the target view frame 757 using a video encoding technique (e.g., according to any suitable video codec, such as Advanced Video Coding (AVC), High Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), Moving Picture Experts Group (MPEG), etc.). The animation and scene rendering system 710 can then transmit the target view frame 757 to the client device 704 via the network.

[0162] The video decoder 732 of the client device 704 can decode the encoded target view frame 757 using an inverse of the video encoding technique performed by the video encoder 730 (e.g., according to a video codec such as AVC, HEVC, VVC, etc.). The reprojection engine 734 can perform reprojection to reproject the virtual content of the decoded target view frame according to the predicted pose determined by the future pose prediction engine 738. The reprojected target view frame can then be displayed on the display 736 so that a user of the client device 704 can view the virtual scene from the user's perspective.

[0163] Fig.10 Another example of an XR system 1000 configured to perform aspects described herein is shown. Fig.10 As shown, the XR system 1000 includes Figure 7 The present invention also includes some of the same components (with the same reference numerals) as XR system 700 of the present invention, and also includes additional or alternative components compared to XR system 700. Face engine 709, gesture engine 712, hand engine 716, audio decoder 718, spatial audio engine 726, scene synthesis system 722, lip sync engine 728, video encoder, video decoder, re-projection engine 734, display 736, and future gesture projection engine 738 are configured to perform the same as Figure 7 In some aspects, XR system 1000 may include a body engine configured to generate a virtual representation of the user's body. Similarly, face-body combiner engine 844 and hair animation engine 846 are configured to perform the same or similar operations as the same components of XR system 700. Figure 8 The same components may perform the same or similar operations.

[0164] Similar to Figure 7 The XR system 700, XR system 1000 system includes participating in a virtual session (in some cases with Fig.10 The client device 1002 and the client device 1004 may include a first XR device (e.g., a VR HMD, AR or MR glasses, etc.) and a second XR device (e.g., a VR HMD, AR or MR glasses, etc.). The animation and scene rendering system 1010 is shown to include an audio decoder 1025, which is configured to decode an audio bitstream received from the audio decoder 718.

[0165] Facial engine in Fig.10 715. The facial engine 709 of the client device 1002 may receive one or more input frames 715 from one or more cameras of the client device 1002. In some examples, the input frames 715 received by the facial engine 709 may include frames (or images) captured by one or more cameras having a field of view of the mouth of the user of the client device 1002, the user's left eye, and the user's right eye. The facial engine 709 may generate and output a code (e.g., a feature vector or multiple feature vectors) representing the face of the user of the client device 1002. The facial engine 709 may send the code representing the user's face to the animation and scene rendering system 1010. As described above, the facial engine 709 may include one or more machine learning systems (e.g., deep learning networks, such as deep neural networks) trained to represent the user's face with a code or feature vector. As described above, the facial engine 709 may include one or more machine learning systems (e.g., deep learning networks, such as deep neural networks) trained to represent the user's face with a code or feature vector. Fig.10 As shown by the “3x” symbol in , the facial engine 709 may include a separate encoder (e.g., a separate encoder neural network) for each type of image processed by the facial engine 709, such as a first encoder for frames or images of the mouth of a user of the client device 1002, a second encoder for frames or images of the right eye of the user of the client device 1002, and a third encoder for frames or images of the left eye of the user of the client device 1002.

[0166] The face decoder 1013 of the animation and scene rendering system 1010 is trained to decode the code (or feature vector) representing the user's face in order to generate a virtual representation of the face (e.g., a face mesh). The face decoder 1013 can receive the code (e.g., latent code or bitstream) from the face engine 709, and can decode the code received from the client device 1002 to generate a virtual representation of the user's face (e.g., a 3D mesh of the user's face). The virtual representation of the user's face can be output to the view-dependent texture synthesis engine 1021 of the animation and scene rendering system 1010, as described below. In some cases, the view-dependent texture synthesis engine 1021 can be part of the user virtual representation system 720 described previously.

[0167] The geometry encoder engine 1011 of the client device 1002 can generate a 3D model (e.g., a 3D deformable model or 3DMM) of the user's head or face based on one or more frames 715. In some aspects, the 3D model can include representations of facial expressions in frames from the one or more frames 715. In one illustrative example, the facial expression representation can be formed by a blend shape. The blend shape can semantically represent the movement of a muscle or part of a facial feature (e.g., opening / closing a jaw, raising / lowering an eyebrow, opening / closing an eye, etc.). In some cases, each blend shape can be represented by a blend shape coefficient paired with a corresponding blend shape vector. In some examples, the facial model can include a representation of the facial shape of the user in the frame. In some cases, the facial shape can be represented by a facial shape coefficient paired with a corresponding facial shape vector. In some embodiments, the geometry encoder engine 1011 (e.g., a machine learning model) can be trained (e.g., during a training process) to enforce a consistent facial shape (e.g., consistent facial shape coefficient) of the 3D facial model, regardless of the pose (e.g., pitch, yaw, and roll) associated with the 3D facial model. For example, when a 3D facial model is rendered into a 2D image or frame for display, a projection technique may be used to project the 3D facial model onto the 2D image or frame.

[0168] In some aspects, the blend shape coefficients can be further refined by additionally introducing relative poses between the client device 1002 and the client device 1004. For example, a neural network can be used to process the relative pose information to generate a view-dependent 3DMM geometry that approximates the true geometry. In another example, the facial geometry can be determined directly from the input frame 715, such as by estimating additional vertex residuals from the input frame 715 and producing more accurate facial expression details for texture synthesis.

[0169] The 3D model of the user's head (or an encoded representation of the 3D model, such as a latent representation or feature vector representing the 3D model) can be output from the geometry encoder engine 1011 and received by the pre-processing engine 1019 of the animation and scene rendering system 1010. As described above, the pre-processing engine 1019 can also receive the future pose predicted by the future pose prediction engine 738 of the client device 1004 and the pose (e.g., 6-DOF pose) of the client device 1002 provided by the pose engine 712. The pre-processing engine 1019 can process the 3D model of the user's head and the pose of the client devices 1002, 1004 to generate a face of the user of the client device 1004 with detailed facial expressions for a particular pose (e.g., from the perspective of the pose of the user of the client device 1004 relative to the pose of the user of the client device 1002). Pre-processing may be performed because, in some cases, the animation and scene rendering system 1010 is configured to modify a neutral facial expression of a user of the client device 1002 (e.g., obtained offline as a registered neutral face by capturing an image of the user's face or scanning) to animate a virtual representation (or avatar) of the user of the client device 1002 with different expressions at different viewpoints. To do this, the animation and scene rendering system 1010 may need to understand how to modify the neutral expression in order to animate the virtual representation of the user of the client device 1002 with the correct expression and pose. The pre-processing engine 1019 may encode the expression information provided by the geometry encoder engine 1011 (and, in some cases, the pose information provided by the pose engine 712 and / or the future pose information from the future pose prediction engine 738) so that the animation and scene rendering system 1010 (e.g., the view-dependent texture synthesis engine 1021 and / or the non-rigid alignment engine 1023) can understand the expression and pose of the virtual representation of the user of the client device 1002 to be synthesized.

[0170] In one illustrative example, the 3D model (which may be a 3DMM as described above) includes a position map defining the position of each point of the 3D model and normal information defining the normal of each point on the position map. The pre-processing engine 1019 may expand or convert the normal information and the position map into a coordinate UV space (which may be referred to as a UV face position map). In some cases, the UV face position map may provide and / or represent a 2D map of the face of the user of the client device 1004 captured in the input frame 715. For example, the UV face position map may be a 2D image that records and / or maps the 3D positions of points (e.g., pixels) in a UV space (e.g., a 2D texture coordinate system). U in UV space and V in UV space may represent axes of the UV face position map (e.g., axes of a 2D texture of a face). In one illustrative example, U in UV space may represent a first axis (e.g., a horizontal X axis) of the UV face position map, and V in UV space may represent a second axis (e.g., a vertical Y axis) of the UV face position map. In some examples, the UV position map can record, model, identify, represent and / or calculate the 3D shape, structure, contour, depth and / or other details of the face (and / or the facial region of the head). In some embodiments, a machine learning model (e.g., a neural network) can be used to generate the UV facial position map.

[0171] Thus, the UV facial position map can encode the expression of the face (and in some cases the pose of the head), thereby providing information that can be used by the animation and scene rendering system 1010 (e.g., the view-dependent texture synthesis engine 1021 and / or the non-rigid alignment engine 1023) to animate the facial representation for the user of the client device 1002. The pre-processing engine 1019 can output the pre-processed 3D model of the user's head to the view-dependent texture synthesis engine 1021 and the non-rigid alignment engine 1023. In some cases, the non-rigid alignment engine 1023 can be part of the user virtual representation system 720 described previously.

[0172] As described above, the view-dependent texture synthesis engine 1021 receives a virtual representation of the user's face (face representation) from the face decoder 1013 and receives pre-processed 3D model information from the pre-processing engine 1019. The view-dependent texture synthesis engine 1021 may combine information associated with the 3D model of the user's head (output from the pre-processing engine 1019) with the user's face representation (output from the face decoder 1013) to generate a complete head model having facial features of the user of the client device 1002. In some cases, the information associated with the 3D model of the user's head output from the pre-processing engine 1019 may include the UV face position map described previously. The view-dependent texture synthesis engine 1021 may also obtain the information associated with the 3D model of the user's head (output from the pre-processing engine 1019) as described above. Figure 8The enrolled texture and depth information 841 discussed. The view-dependent texture synthesis engine 1021 can add texture and depth from the enrolled texture and depth information 841 to a full head model of a user of the client device 1002. In some cases, the view-dependent texture synthesis engine 1021 can include a deep learning network, such as a neural network (e.g., a convolutional neural network (CNN), an autoencoder, or other type of neural network). For example, the view-dependent texture synthesis engine 1021 can be a neural network that is trained (e.g., using supervised learning, semi-supervised learning, unsupervised learning, etc.) to process the UV facial position map (and in some cases, facial information such as facial information output by the facial decoder 1013) to generate a full head model of the user.

[0173] The non-rigid alignment engine 1023 can move the 3D model of the head of the user of the client device 1002 in a translation direction to ensure that the geometry and texture are aligned or overlapped, which can reduce the gap between the geometry and texture. The face-body combiner engine 844 can then combine the body model ( Fig.10 ) and the facial representation output by the view-dependent texture synthesis engine 1021 to generate a combined virtual representation of the user of the client device 702. The hair animation engine 846 can generate a hair model that can be combined with the combined virtual representation of the user to generate a final user virtual representation (e.g., final user virtual representation 847) of the user of the client device 1002. The scene synthesis system 722 can then generate a target view frame 757, process the target view frame 757 as described above, and send the target view frame 757 (e.g., after being encoded by the video encoder 730) to the client device 1004.

[0174] As mentioned above Figure 7 As discussed, the video decoder 732 of the client device 1004 may decode the encoded target view frame 757 using an inverse technique of the video encoding technique performed by the video encoder 730 (e.g., according to a video codec such as AVC, HEVC, VVC, etc.). The reprojection engine 734 may perform reprojection to reproject the virtual content of the decoded target view frame according to the predicted pose determined by the future pose prediction engine 738. The reprojected target view frame may then be displayed on the display 736 so that a user of the client device 1004 may view the virtual scene from the user's perspective.

[0175] Fig.11 Another example of an XR system 1100 configured to perform aspects described herein is shown. Fig.11 As shown in FIG. 1 , the XR system 1100 includes Figure 7 XR System 700 and / or Fig.101000, and also includes additional or alternative components compared to XR system 700 and XR system 1000. Face engine 709, face decoder 1013, geometry encoder engine 1011, pose engine 712, pre-processing engine 1019, view-dependent texture synthesis engine 1021, non-rigid alignment engine 1023, face-body combiner engine 844, hair animation engine 846, audio decoder 718, and other similar components are configured to perform the same Figure 7 XR System 700 or Fig.10 The same components of XR system 1000 may operate the same or similarly.

[0176] XR system 1100 may be similar to XR systems 700 and 1000 in that client device 1102 and client device 1104 participate in a virtual session (in some cases, with Fig.11 The XR system 1100 may differ from the XR systems 700 and 1000 in that the client device 1104 may be local to the animation and scene rendering system 1110. For example, the client device 1104 may be communicatively connected to the animation and scene rendering system 1110 using a local connection such as a wired connection (e.g., a High Definition Multimedia Interface (HDMI) cable, etc.). In one illustrative example, the client device 1104 may be a television, a computer monitor, or other device other than an XR device. In some cases, the client device 1104 (e.g., a television, a computer monitor, etc.) may be configured to display 3D content.

[0177] The client device 1102 can send information via a network (e.g., a local network, an RTC network, or other wireless network) to the animation and scene rendering system 1110. The animation and scene rendering system 1110 can generate an animated virtual representation (or avatar) of the user of the client device 1102 for a virtual scene from the perspective of the user of the client device 1104, and can output the animated virtual representation to the client device 1104 via a local connection.

[0178] In some aspects, the animation and scene rendering system 1110 may include a mono audio engine 1126 instead of Figure 7 and Fig.10 In some cases, the animation and scene rendering system 1110 may include a spatial audio engine 726. In some cases, the animation and scene rendering system 1110 may include a spatial audio engine 726.

[0179] Fig.12An example of a process 1200 for generating virtual content at a first device of a distributed system is shown. According to aspects described herein, the first device may include an animation and scene rendering system (e.g., an edge or cloud-based server, a personal computer acting as a server device, a mobile device such as a mobile phone acting as a server device, an XR device acting as a server device, a network router, or other device acting as a server or other device). Illustrative examples of animation and scene rendering systems include Figure 7 Animation and scene rendering system 710, Fig.10 Animation and scene rendering system 1010 and Fig.11 Animation and scene rendering system 1110. In some examples, a component (eg, a chipset, a processor, a memory, any combination thereof, and / or other components) may perform one or more operations of process 1200.

[0180] In block 1202, a first device (or a component thereof) may receive input information associated with the second device or at least one of a user of the second device from a second device associated with a virtual session. In some aspects, the input information includes information representing a face of a user of the second device, information representing a body of a user of the second device, information representing one or more hands of a user of the second device, gesture information of the second device, audio associated with an environment in which the second device is located, any combination thereof, and / or other information. In some cases, the information representing the body of the user of the second device includes a gesture of the body. In some examples, the information representing one or more hands of a user of the second device includes a corresponding gesture of each of the one or more hands.

[0181] At block 1204, the first device (or a component thereof) may generate a virtual representation of a user of the second device based on the input information. In some aspects, to generate the virtual representation, the first device (or a component thereof) may use information representing a face of the user, posture information of the second device, and posture information of the third device to generate a virtual representation of the face of the user of the second device. The first device (or a component thereof) may also use the posture information of the second device and the posture information of the third device to generate a virtual representation of the body of the user of the second device. The first device (or a component thereof) may generate a virtual representation of the hair of the user of the second device. In some cases, the first device (or a component thereof) is configured to further use information representing the body of the user to generate a virtual representation of the body of the user of the second device. In some examples, the first device (or a component thereof) is configured to further use inverse kinematics to generate a virtual representation of the body of the user of the second device. In some cases, the first device (or a component thereof) is configured to further use information representing one or more hands of the user to generate a virtual representation of the body of the user of the second device. In some aspects, the first device (or a component thereof) may combine the virtual representation of the face with the virtual representation of the body to generate a combined virtual representation. The first device (or a component thereof) may add a virtual representation of the hair to the combined virtual representation.

[0182] At block 1206, the first device (or a component thereof) may generate a virtual scene from the perspective of a user of a third device associated with the virtual session. The virtual scene includes a virtual representation of a user of a second device. In some aspects, to generate the virtual scene, the first device (or a component thereof) may obtain a background representation of the virtual scene and adjust lighting of the virtual representation of the user of the second device based on the background representation of the virtual scene to generate a modified virtual representation of the user. To generate the virtual scene, the first device (or a component thereof) may combine the background representation of the virtual scene with the modified virtual representation of the user.

[0183] At block 1208 , the first device may send (or a component thereof may output for transmission) to the third device one or more frames depicting the virtual scene from the perspective of a user of the third device.

[0184] In some aspects, the first device (or its component) may generate a virtual representation of a user of the third device based on input information from the third device. The first device (or its component) may generate a virtual scene including a virtual representation of the user of the third device from the perspective of the user of the second device. The first device may also send (or its component may output for transmission) one or more frames depicting the virtual scene from the perspective of the user of the second device to the second device.

[0185] In some cases, a device or apparatus configured to perform the operations of process 1200 and / or other processes described herein may include a processor, a microprocessor, a microcomputer, or other components of a device configured to perform the steps of process 1200 and / or other processes. In some examples, such a device or apparatus may include one or more sensors configured to capture image data and / or other sensor measurements. In some examples, such a computing device or apparatus may include one or more sensors and / or cameras configured to capture one or more images or videos. In some cases, such a device or apparatus may include a display for displaying an image. In some examples, one or more sensors and / or cameras are separated from a device or apparatus, in which case the device or apparatus receives sensed data. Such a device or apparatus may also include a network interface configured to transmit data.

[0186] Components of a device or apparatus configured to perform one or more operations of process 1200 and / or other processes described herein may be implemented in circuits. For example, a component may include and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and / or other suitable electronic circuits), and / or may include and / or be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. The computing device may also include a display (as an example of an output device or in addition to an output device), a network interface configured to transmit and / or receive data, any combination thereof, and / or other components. The network interface may be configured to transmit and / or receive data based on an Internet Protocol (IP) or other types of data.

[0187] The process 1200 is illustrated as a logical flow diagram, the operations of which represent a sequence of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, an operation represents a computer-executable instruction stored on one or more computer-readable storage media, which, when executed by one or more processors, performs the operation. In general, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the process.

[0188] In addition, the processes described herein (e.g., process 1200 and / or other processes) can be performed under the control of one or more computer systems configured with executable instructions, and can be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed together on one or more processors, through hardware, or a combination thereof. As described above, the code can be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program including multiple instructions that can be executed by one or more processors. The computer-readable or machine-readable storage medium can be non-transitory.

[0189] As mentioned above, a virtual representation (e.g., an avatar) is an important component of a virtual environment. A virtual representation (or avatar) is a 3D representation of a user and allows the user to interact with a virtual scene. There are different ways to represent a virtual representation (e.g., an avatar) of a user and corresponding animation data. For example, an avatar can be purely synthetic, or it can be an accurate representation of the user (e.g., as in Figure 2 204, etc. shown in the 3D collaborative virtual environment 200 of FIG. 1 ). The virtual representations (or avatars) may need to be captured or reoriented in real time to reflect the user's actual movements, body postures, facial expressions, etc.

[0190] Various animation assets may be required to model and / or animate a virtual representation or avatar. For example, one or more meshes (e.g., including multiple vertices, edges, and / or faces in three-dimensional space) with corresponding materials may be used to represent an avatar of a user. Materials may include normal textures (e.g., represented by a normal map), diffuse or albedo textures, specular textures, any combination thereof, and / or other materials or textures. For example, an animated avatar is an animated mesh and corresponding assets (e.g., normal, albedo, specular, etc.) for each frame. Fig.13 1 is a diagram showing an example of a user's mesh 1301, an example of a user's normal map 1302, an example of a user's albedo map 1304, an example of a user's specular map 1306, and an example of a user's personalized parameters 1308. For example, personalized parameters 1308 may be neural network parameters (e.g., weights, biases, etc.) that may be pre-trained, fine-tuned, or set by the user. Various materials or textures may need to be obtained from registration or offline reconstruction.

[0191] In some aspects, assets may be view-dependent (eg, generated for a certain view pose) or may be view-independent (eg, generated for all possible view poses). In some cases, realism may be compromised for view-independent assets.

[0192] The goal of generating an avatar for a user is to generate a mesh for the user with various materials (e.g., normals, albedo, specularity, etc.). In some cases, the mesh must have a known topology. However, the mesh from a scanner (e.g., LightCage, 3DMD, etc.) may not satisfy such constraints. To solve such problems, the mesh can be retopologized after scanning, which will make the mesh parameterized. Fig.14A is a diagram showing an example of an original (non-retopologized) mesh 1402 . Fig. 14B 1404. Retopologizing a mesh for a virtual representation or avatar causes the mesh to become parameterized with a plurality of animation parameters that define how the avatar will animate during a virtual session.

[0193]

[0013] Animation of a virtual representation (eg, avatar) may be performed using a variety of techniques. Fig.15 1500 is a diagram illustrating an example of a technique for performing avatar animation. As shown, camera sensors of a head mounted display (HMD) are used to capture images of a user's face, including an eye camera for capturing images of the user's eyes, a face camera for capturing visible portions of the face (e.g., a mouth, chin, cheeks, a portion of a nose, etc.), and other sensors for capturing other sensor data (e.g., audio, etc.). A machine learning (ML) model (e.g., a neural network model) may process the images to generate a 3D mesh and texture of a 3D facial avatar. The mesh and texture may then be rendered by a rendering engine to generate a rendered image (e.g., view-dependent rendering, such as Fig.15 as shown).

[0194] Large data transfer rates may be required to transfer mesh information and animation parameters of the mesh between devices for avatar animation to facilitate an interactive virtual experience between users. For example, for each frame, the parameters may need to be sent from one device of a first user to a second device of a second user at a frame rate of 30-60 frames per second (FPS) so that the second device can render the avatar for each frame at that frame rate.

[0195] As previously described, systems and techniques are also described herein for providing an efficient communication framework for virtual representation invocation of a virtual environment (e.g., a metaverse virtual environment). In some aspects, the process for establishing an avatar invocation can be performed directly between client devices (also referred to as user devices), or can be performed via a server.

[0196] Fig.16is an example of an avatar call flow directly between client devices, including a first client device 1602 of a first user (shown as client device A, also referred to as user A) and a second client device 1604 of a second user (shown as client device B, also referred to as user B). In one illustrative example, the first client device 1602 is a first XR device (e.g., an HMD configured to display VR, AR, and / or other XR content), and the second client device 1604 is a second XR device (e.g., an HMD configured to display VR, AR, and / or other XR content). Although Fig.16 Two client devices are shown, but Fig.16 The call flow can be between a first client device 1602 and a plurality of other client devices. As shown, the first client device 1602 can establish an avatar call between the first client device 1602 and the second client device 1604 by sending a call establishment request 1606 to the second client device 1604 to establish the avatar call. The second client device 1604 can accept the avatar call, such as based on user input from the second user, the second user accepting the invitation to participate in a virtual session (e.g., a 3D collaborative virtual meeting, a computer or virtual game, or other virtual session in a metaverse environment) with the first user and in some cases with one or more other users. The second client device 1604 can transmit a call acceptance 1608 indicating the acceptance of the avatar call.

[0197] Upon accepting the avatar call, the first client device 1602 may provide (e.g., transmit, etc.) mesh information 1610 to the second client device 1604 (e.g., to a secure location on the second client device 1604). The mesh information defines an avatar (or other virtual representation) of the first user for use in participating in a virtual session (e.g., a 3D collaborative virtual meeting, a computer or virtual game, etc.) between the first user and the second user and, in some cases, one or more other users. In some cases, the mesh information may include information defining a mesh (or geometry) of the first user's avatar, such as vertices, edges, and / or faces of the mesh. The mesh information may also include assets, such as information defining a texture and / or material of the mesh. For example, an asset may include a normal map (e.g., a normal texture) defining the mesh. Fig.13 Normal map 1302 of the mesh), an albedo map defining the diffuse or albedo texture of the mesh (e.g., Fig.13 ), a specular map that defines the specular texture of the mesh (e.g., Fig.13Specular reflection map 1306), any combination thereof, and / or other materials or textures. In some cases, the asset can be obtained during a registration process, during which asset information (e.g., normal, albedo, specular reflection, etc.) can be obtained for the user (e.g., via one or more captured images).

[0198] The second client device 1604 may also provide (e.g., transmit, etc.) mesh information 1612 defining an avatar (or other virtual representation) of the second user to the first client device 1602 (e.g., to a secure location on the first client device 1602). Similar to the mesh information 1610, the mesh information 1612 of the second user may include information defining a mesh of the avatar and other assets associated with the avatar of the second user (e.g., a normal map, an albedo map, a specular map, etc.). In some aspects, the mesh information may be compressed.

[0199] In some aspects, the grid information 1610 and the grid information 1612 may be provided (e.g., sent, etc.) only once at the start of the call. After the first client device 1602 obtains the grid information 1612 of the second user and the second client device 1604 obtains the grid information 1610 of the first user, the first client device 1602 and the second client device 1604 may only need to exchange grid animation parameters 1614 during the avatar call. For example, once the call is established between the first client device 1602 and the second client device 1604, the first client device 1602 and the second client device 1604 may exchange grid animation parameters during the call so that the first client device 1602 can animate the second user's avatar during the virtual session, and the second client device 1604 can animate the first user's avatar during the virtual session.

[0200] The virtual session may be presented as a plurality of frames. For each of the plurality of frames constituting the virtual session, the mesh animation parameters may include information representing the face of the first user of the first client device 1602 (e.g., a code or feature representing facial appearance or other information, such as information about the face of the first user of the first client device 1602). Figure 7 , Fig.10 information describing the first user of the first client device 1602, etc.), information representing the body of the first user of the first client device 1602 (e.g., a code or feature representing the appearance of the body, a posture of the body, or other information, such as information about Figure 7 , Fig.10 information describing the first client device 1602, information representing one or more hands of the first user of the first client device 1602 (e.g., codes or features representing the appearance of the hands, hand gestures, or other information, such as information about Figure 7 , Fig.10information described in the description, etc.), posture information of the first client device 1602 (e.g., posture in six degrees of freedom (6-DOF), referred to as 6-DOF posture, such as about Figure 7 , Fig.10 , etc.), audio associated with the environment in which the first client device 1602 is located (such as about Figure 7 , Fig.10 etc.), and / or other information. In some aspects, mesh animation parameters can be compressed.

[0201] In some aspects, the animation and scene rendering system of the second client device 1604 (e.g., similar to Figure 7 Animation and scene rendering system 710, Fig.10 Animation and scene rendering system 1010, Fig.11 The animation and scene rendering system 1110 of the first client device 1602 can process the mesh information 1610 and the animation parameters from the first client device 1602 to generate and / or animate the avatar of the first user during the virtual session. Similarly, the animation and scene rendering system of the first client device 1602 can process the mesh information 1612 and the animation parameters from the second client device 1604 to generate and / or animate the avatar of the second user during the virtual session.

[0202] Once the virtual session is complete, the avatar call ends 1616. In some cases, the grid information may be deleted by the first client device 1602 and the second client device 1604 at the end of the avatar call.

[0203] Fig.17 is a diagram showing an apparatus configured to perform aspects described herein (such as with Fig.16 FIG. 17 is a diagram of an example of an XR system 1700 that illustrates an example of an avatar call flow associated with an operation. Fig.17 As shown, the XR system 1700 includes Figure 7 XR System 700 and Fig.10 Some of the same components (with the same reference numerals) as the XR system 1000 of FIG. 709, the facial encoder 709, the pose engine 712, the hand engine 716, the audio decoder 718, the user virtual representation system 720, the spatial audio engine 726, the lip sync engine 728, the reprojection engine 734, the display 736, the future pose projection engine 738, the geometry encoder engine 1011, and the audio decoder 1025 are configured to perform the same Figure 7 XR System 700 and / or Fig.10 The same or similar operations of the same components of XR system 1000 may also be included in the XR system 1700. Figure 7 , Fig.10 , Fig.11and / or Fig.12 The absence Fig.17 Components shown in, such as scene synthesis system 722, pre-processing engine 1019 and / or other components.

[0204] like Fig.17 As shown, the user virtual representation system 720 may use the registration data 1750 of the first user (user A) of the first client device 1602 to generate a virtual representation (eg, avatar) of the first user. The registration data 1750 of the first user may include Fig.16 Grid information 1610. Fig.16 As described, mesh information 1610 may include information defining a mesh of an avatar of a first user of first client device 1602 and other assets associated with the avatar (eg, a normal map, an albedo map, a specular map, etc.). Fig.16 The mesh animation parameters 1614 described may include Fig.17 The facial code of the facial encoder 709, from Fig.17 The face blend shapes from the geometry encoder engine 1011 (which may include a 3DMM head encoder in some cases), Fig.17 The hand joint code of the hand engine 716, from Fig.17 The head pose of the pose engine 712, and in some cases from Fig.17 The audio stream of the audio decoder 718.

[0205] Fig.18 is an example of an avatar call flow for establishing an avatar call between client devices using the server device 1805. Fig.18 The communication is shown between a first client device 1802 (shown as client device A, also referred to as user A) and a server device 1805. Although not shown, the server device 1805 also communicates with one or more other client devices participating in the virtual session with the first client device 1802. The one or more other client devices may include at least a second client device of a second user. In an illustrative example, the first client device 1802 is a first XR device (e.g., an HMD configured to display VR, AR, and / or other XR content), and the second client device is a second XR device (e.g., an HMD configured to display VR, AR, and / or other XR content). The server device 1805 may include a secure or trusted server or multiple secure or trusted servers.

[0206] As shown, the first client device 1802 can establish an avatar call between the first client device 1802 and one or more other client devices by sending a call establishment request 1806 to the server device 1805 to establish the avatar call. Fig.18 The call flow includes a first client device 1802 initiating a call, but any client device can send a call establishment request 1806 to a server device 1805 to establish an avatar call. The server device 1805 can send the call establishment request 1806 to one or more other client devices.

[0207] Any of the one or more other client devices (e.g., a second client device) may accept the avatar call. For example, the second client device may accept the avatar call based on user input from a second user accepting an invitation to participate in a virtual session (e.g., a 3D collaborative virtual meeting in a metaverse environment, a computer or virtual game, or other virtual session) with the first user and, in some cases, with one or more other users of the one or more other devices. Any of the one or more other devices that accept the avatar call may send a call acceptance to the server device 1805, and the server device 1805 may send a call acceptance 1808 indicating the avatar call acceptance to the first client device 1802.

[0208] Upon accepting the avatar call, the first client device 1802 may send mesh information 1810 to the server device 1805 (e.g., to a secure location on the server device 1805). The server device 1805 may provide (e.g., send, upload for access, etc.) the mesh information to one or more other client devices. The mesh information defines an avatar (or other virtual representation) of the first user for use in participating in a virtual session between the first user and one or more other users (e.g., a 3D collaborative virtual meeting, a computer or virtual game, etc.). In some cases, the mesh information may include information defining a mesh (or geometry) of the first user's avatar, such as vertices, edges, and / or faces of the mesh. The mesh information may also include assets, such as information defining a texture and / or material of the mesh. For example, an asset may include a normal map (e.g., a normal texture) defining the mesh. Fig.13 Normal map 1302 of the mesh), an albedo map defining the diffuse or albedo texture of the mesh (e.g., Fig.13 ), a specular map that defines the specular texture of the mesh (e.g., Fig.13 Specular reflection map 1306), any combination thereof, and / or other materials or textures. In some cases, the asset can be obtained during a registration process, during which asset information (e.g., normal, albedo, specular reflection, etc.) can be obtained for the user (e.g., via one or more captured images).

[0209] Each of the one or more client devices that receive the call establishment request 1806 for the virtual session can provide their respective mesh information to the server device 1805. For example, the second client device can send mesh information defining the avatar (or other virtual representation) of the second user to the server device 1805 (e.g., to a secure location on the server device 1805). In some cases, the server device 1805 can provide the corresponding mesh information of each client device to other client devices participating in the virtual session. In other cases, the server device 1805 can use the mesh information to render the scene of the virtual session.

[0210] In some aspects, the first client device 1802 (and in some cases, one or more other client devices) can register an account with the server device 1805. In such aspects, the first client device 1802 can provide the grid information 1810 once to the server device 1805. After establishing the account and providing the grid information 1810 to the server device 1805, the first user of the first client device 1802 can log into the account for authentication by the server device 1805 for future avatar invocations without having to re-provide (e.g., re-upload, re-send, etc.) the grid information 1810 to the server device 1805. In some cases, the first client device 1802 can provide any updates to the grid information 1810 to the server device 1810.

[0211] In some aspects, for a particular avatar call, the grid information 1810 (as well as the grid information from one or more other client devices) may be provided to the server device 1805 only once at the start of the avatar call. After the first client device 1802 and the one or more other devices provide their respective grid information to the server device 1805, the first client device 1802 and the one or more other devices may only need to provide the grid animation parameters 1814 to the server device 1805 during the avatar call. In some cases, once the call is established between the first client device 1802 and the second client device, the first client device 1802 and the second client device may exchange the grid animation parameters via the server device 1805 during the call, so that the first client device 1802 can animate the avatar of the second user during the virtual session, and the second client device can animate the avatar of the first user during the virtual session. In other cases, the server device 1805 may use the animation parameters (and grid information) of various client devices to present the avatars of the first client device 1802 and the one or more other client devices and the scene of the virtual session from the perspectives of the various client devices.

[0212] The virtual session may be presented as a plurality of frames. For each of the plurality of frames constituting the virtual session, the mesh animation parameters may include information representing the face of the first user of the first client device 1802 (e.g., a code or feature representing facial appearance or other information, such as information about the face of the first user of the first client device 1802). Figure 7 , Fig.10 information describing the first user of the first client device 1802, etc.), information representing the body of the first user of the first client device 1802 (e.g., a code or feature representing the appearance of the body, a posture of the body, or other information, such as information about Figure 7 , Fig.10 information describing the first client device 1802, information representing one or more hands of the first user of the first client device 1802 (e.g., codes or features representing the appearance of the hands, hand gestures, or other information, such as information about Figure 7 , Fig.10 information described in the description, etc.), posture information of the first client device 1802 (e.g., posture in six degrees of freedom (6-DOF), referred to as 6-DOF posture, such as about Figure 7 , Fig.10 , etc.), audio associated with the environment in which the first client device 1802 is located (such as about Figure 7 , Fig.10 etc.), and / or other information. In some aspects, mesh animation parameters can be compressed.

[0213] In some aspects, as described above, server device 1805 can use the animation parameters (and mesh information) of the various client devices to present the avatars and scenes of the virtual session of first client device 1802 and one or more other client devices from the perspectives of the various client devices. Fig.18 As shown, server device 1805 can provide (e.g., transmit, upload for access, etc.) a particular scene generated by server device 1805 to first client device 1802 and one or more other client devices via split rendering. For example, using mesh information 1810 and mesh animation parameters for first client device 1802, server device 1805 can generate a rendering of an avatar of a first user of first client device 1802 from the perspective of a second user of a second client device. In other aspects, each client device can present avatars of one or more other users of one or more other client devices. For example, in such aspects, the animation and scene rendering system of first client device 1802 (e.g., similar to Figure 7 Animation and scene rendering system 710, Fig.10 Animation and scene rendering system 1010, Fig.11The animation and scene rendering system 1110, etc.) can process mesh information 1812 and animation parameters from one or more other client devices to generate and / or animate avatars for one or more other users during a virtual session.

[0214] Once the virtual session is complete, the avatar call ends 1816. In some cases, the grid information may be deleted by the server device 1605 at the end of the avatar call.

[0215] Fig.19 is a diagram showing a system configured to perform aspects described herein (e.g., with Fig.18 FIG. 19 is a diagram of an example of an XR system 1900 that illustrates an example of an avatar call flow associated with an operation. Fig.19 As shown in FIG. 1 , the XR system 1900 includes Figure 7 XR System 700 and Fig.10 Some of the same components (with the same numbers) as the XR system 1000 of the present invention. The facial encoder 709, the pose engine 712, the hand engine 716, the audio decoder 718, the background scene information 719, the user virtual representation system 720, the scene synthesis system 722, the spatial audio engine 726, the lip sync engine 728, the video encoder 730, the video decoder 732, the re-projection engine 734, the display 736, the future pose projection engine 738, the geometry encoder engine 1011 and the audio decoder 1025 are configured to perform the same Figure 7 XR System 700 and / or Fig.10 The same or similar operations of the same components of XR system 1000 may also be included in the XR system 1900. Figure 7 , Fig.10 , Fig.11 and / or Fig.12 The absence Fig.19 Components shown in, such as pre-processing engine 1019 and / or other components.

[0216] like Fig.19 As shown, the user virtual representation system 720 may use the registration data 1950 of the first user (user A) of the first client device 1802 to generate a virtual representation (eg, avatar) of the first user. The registration data 1950 of the first user may include Fig.18 As previously described, the mesh information 1810 may include a mesh defining an avatar of a first user of the first client device 1802 and other assets associated with the avatar (e.g., a normal map, an albedo map, a specular reflection map, etc., such as Fig.13 1301, normal map 1302, albedo map 1304, specular map 1306 and / or personalized parameters 1308). Fig.18The mesh animation parameters described can include Fig.19 The facial code of the facial encoder 709, from Fig.19 The face blend shapes from the geometry encoder engine 1011 (which may include a 3DMM head encoder in some cases), Fig.19 The hand joint code of the hand engine 716, from Fig.19 The head pose of the pose engine 712, and in some cases from Fig.19 The audio stream of the audio decoder 718.

[0217] Fig. 20 is a diagram showing an apparatus configured to perform aspects described herein (such as with Fig.18 FIG. 2 is a diagram of an example of an XR system 2000 that includes operations associated with an avatar call flow (an avatar call flow associated with an avatar call flow). The XR system 2000 can be used when one or more view-dependent textures are present. For example, a view-dependent texture is an image that can only be viewed within certain poses, while a view-independent texture is an image that can be viewed regardless of a person's pose. For example, an image generated using a view-dependent texture may be valid (e.g., may be visible) within a certain range of poses (e.g., a certain number of degrees from a certain pose). If the person's pose deviates beyond the range, the image may have quality issues or no longer be visible. In some cases, a more accurate pose estimate may be used when generating a view-dependent texture because the view-dependent texture may only be valid within a certain range of poses. Although the view-dependent texture may be limited to a certain range of poses, the size of the view-dependent texture may be smaller than the view-independent texture, and this allows for potential memory, bandwidth, and / or computational savings.

[0218] like Fig. 20 As shown, the XR system 2000 includes Figure 7 XR System 700 and Fig.10 Some of the same components (with the same reference numerals) as the XR system 1000 of FIG. 709, the facial encoder 709, the pose engine 712, the hand engine 716, the audio decoder 718, the user virtual representation system 720, the spatial audio engine 726, the lip sync engine 728, the video encoder 730, the video decoder 732, the re-projection engine 734, the display 736, the future pose projection engine 738, the geometry encoder engine 1011, and the audio decoder 1025 are configured to perform the same Figure 7 XR System 700 and / or Fig.10As shown, XR system 2000 may use information 219 to generate one or more virtual representations (e.g., avatars) by user virtual representation system 720 and / or render a view of a scene from the user's perspective by user view rendering engine 2075, the view including background information (e.g., similar to Figure 7 Background scene information 719), and other users (including one or more other users) update animation parameters and registration data. Fig. 20 As shown, the user virtual representation system 720 may use the registration data 2050 of the first user (user A) to generate a virtual representation (eg, avatar) of the first user. The registration data 2050 of the first user may include Fig.18 The mesh information 1810 may include a mesh defining an avatar of a first user of the first client device 1802 and other assets associated with the avatar (e.g., a normal map, an albedo map, a specular reflection map, etc., for example). Fig.13 The XR system 2000 also includes a scene definition / synthesis system 2022, which can be used to synthesize and / or define the scene of the virtual environment or session. The XR system 2000 is shown as including a trusted edge server, which includes certain components. In some cases, the XR system 2000 may not include an edge server. In some examples, Figure 7 , Fig.10 , Fig.11 and / or Fig.17 and / or Fig.19 Any of the XR systems may include Fig. 20 The XR system 2000 may also include an edge server as shown in FIG. Figure 7 , Fig.10 and / or Fig.11 Not in Fig. 20 Components shown in, such as pre-processing engine 1019 and / or other components.

[0219] Fig.21 is a diagram showing a system configured to perform aspects described herein (e.g., with Fig.18 FIG. 21 is an example of an example of an XR system 2100 that illustrates operations associated with an avatar call flow. The XR system 2100 may be used when one or more view-independent textures are present. As described above, a view-independent texture is an image that can be viewed regardless of a person's posture. Fig.21 As shown, the XR system 2100 includes Figure 7 XR System 700 and Fig.10Some of the same components (with the same reference numerals) as the XR system 1000 of FIG. 709, the facial encoder 709, the gesture engine 712, the hand engine 716, the audio decoder 718, the user virtual representation system 720, the spatial audio engine 726, the lip sync engine 728, the video encoder 730, the video decoder 732, the reprojection engine 734, the display 736, the geometry encoder engine 1011, and the audio decoder 1025 are configured to perform the same Figure 7 XR System 700 and / or Fig.10 As shown, XR system 2100 may use information 2119 to generate one or more virtual representations (e.g., avatars) by user virtual representation system 720 and / or render a view of a scene from the user's perspective by user view rendering engine 2175, the view including background information (e.g., similar to Figure 7 's background scene information 719), and other users (including one or more other users) update animation parameters and registration data. The XR system 2100 also includes a scene definition / synthesis system 2122, which can be used to synthesize and / or define the scene of the virtual environment or session. The XR system 2100 is shown as including a trusted edge server that includes certain components, such as the scene definition / synthesis system 2122. In some cases, the trusted cloud server can be configured to establish and / or control / direct computing operations that can be performed by the trusted edge server. For example, the scene definition / synthesis system 2122 executed on the trusted cloud server can control / direct systems on the trusted edge server, such as the user virtual representation system 720 and / or the user view rendering system 2175. In some cases, the XR system 2100 may not include an edge server. In some examples, Figure 7 , Fig.10 , Fig.11 and / or Fig.17 and / or Fig.19 Any of the XR systems may include Fig.21 The XR system 2100 may also include an edge server from Figure 7 , Fig.10 , Fig.11 and / or Fig.12 The absence Fig.21 Components shown in, such as pre-processing engine 1019 and / or other components.

[0220] As described above, other example call flows can be based on a device-to-device flow without an edge server and decoder on a target or receiving client device (e.g., for view-dependent textures and view-independent textures), a device-to-device flow without an edge server and all processing (e.g., decoding, avatar animation, etc.) on a source or transmitting / sending client device (e.g., for view-dependent textures and view-independent textures), a device-to-device flow with an edge server and decoder on a server device (e.g., for view-dependent textures and view-independent textures), and a device-to-device flow with an edge server and all processing (e.g., decoding, avatar animation, etc.) on a source or transmitting / sending client device (e.g., for view-dependent textures and view-independent textures). As previously described, a view-dependent texture can be generated for a certain view pose, and a view-independent texture can be generated for all possible view poses.

[0221] Fig. 22 is a diagram illustrating an example of an example of an XR system 2200 configured to perform aspects described herein, such as operations associated with device-to-device streaming without an edge server and decoder on a target device (e.g., a target HMD) for view-dependent textures. Fig. 22 As shown, the XR system 2200 includes Figure 7 XR System 700, Fig.10 XR System 1000, Fig. 20 XR System 2000, Fig.21 Similar components (e.g., facial encoder 709, body engine 714, hand engine 716, future pose prediction engine 738, etc.) are configured to perform the same as those of XR system 2100 (with the same reference numerals). Figure 7 XR System 700, Fig.10 XR System 1000, Fig. 20 XR System 2000, Fig.21 As shown, the XR system 2200 may use information 2119 to generate one or more virtual representations (e.g., avatars) by the user virtual representation system 720 and / or render a view of the scene from the user's perspective by the user view rendering engine 2175, which includes background information (e.g., in some cases similar to Figure 7719), and other users (including one or more other users) update animation parameters and registration data. The user virtual representation system 720 can pass view-related textures to the user view rendering system 2175. Passing view-related textures can help reduce the amount of memory and / or bandwidth used to send textures. The XR system 2200 also includes a scene definition / synthesis system 2122, which can be used to synthesize and / or define the scene of a virtual environment or session. The XR system 2200 is shown as including a trusted cloud server, which includes certain components, such as the scene definition / synthesis system 2122. In some cases, the scene definition / synthesis system 2122 executed on the trusted cloud server can control / direct systems on a trusted edge server, such as the user virtual representation system 720 and / or the user view rendering system 2175. In some cases, the XR system 2200 may not include a cloud server. The XR system 2200 may also include Figure 7 , Fig.10 , Fig.11 and / or Fig.12 The absence Fig. 22 Components shown in, such as pre-processing engine 1019 and / or other components.

[0222] Fig.23 is a diagram illustrating an example of an example of an XR system 2300 configured to perform aspects described herein, such as operations associated with device-to-device streaming without an edge server and decoder on a target device (e.g., a target HMD) for view-independent textures. Fig.23 As shown, the XR system 2300 includes Figure 7 XR System 700, Fig.10 XR System 1000, Fig. 20 XR System 2000, Fig.21 Similar components (e.g., face encoder 709, body engine 714, hand engine 716, etc.) are configured to perform the same as those of XR system 2100 (with the same reference numerals). Figure 7 XR System 700, Fig.10 XR System 1000, Fig. 20 XR System 2000, Fig.21 As shown, the XR system 2300 may use information 2119 to generate one or more virtual representations (e.g., avatars) by the user virtual representation system 720 and / or render a view of the scene from the user's perspective by the user view rendering engine 2175, the view including background information (e.g., similar to the background information in some cases). Figure 7The XR system 2300 also includes a scene definition / synthesis system 2122, which can be used to synthesize and / or define the scene of the virtual environment or session. The XR system 2300 is shown as including a trusted cloud server, which includes certain components. In some cases, the XR system 2300 may not include a cloud server. The XR system 2300 may also include a Figure 7 , Fig.10 , Fig.11 and / or Fig.12 The absence Fig.23 Components shown in, such as pre-processing engine 1019 and / or other components.

[0223] Fig.24 2400 is a signaling diagram illustrating communications between a first user equipment (UE) (shown as UE1 2402), a proxy call setup and control function (P-CSCF), a service call setup and control function (S-CSCF), an interrogating call setup and control function (I-CSCF) (collectively referred to as P / S / I-CSCF 2404), a media resource function (MRF) 2406, a multimedia telephony application server (MMTel AS) 2408, and a second UE (shown as UE2 2410) to establish an XR session. The operations of the signaling diagram 2400 may be performed by Fig. 22 XR System 2200 and / or Fig.23 The XR system 2300 performs. In some cases, the operations of the signaling diagram 2400 may be performed in accordance with 3GPP technical specification group Service and System Aspects (SA) WG4 (SA4).

[0224] As shown, a call establishment process 2412 may be performed to establish a call between a first UE 2402 and a second UE 2410. After the call is established, a scene description retrieval process 2414 may be performed. In some cases, during the scene description retrieval process 2414, a scene definition / synthesis system (e.g., Figure 22 to Figure 23 The scene definition / composition system 2022, etc.) can retrieve information about the scene to be rendered in the XR session, such as user registration data (e.g., Figure 22 to Figure 23 User A registration data 2150, etc.) and background information (e.g., Figure 22 to Figure 23 In some cases, the scene definition / synthesis system may also set up / establish computing processes on the trusted edge server, such as a user virtual representation system (e.g., user virtual representation system 720) and a user view rendering system (e.g., Figure 22-23 The scene description update process 2416 may apply updates to the information about the scene as needed. During the augmented reality (AR) media and metadata exchange process 2418 between the first UE 2402 and the second UE 2410, the first UE 2402 sends 2420 (e.g., transmits) the stored avatar data (e.g., once sent data) to the second UE 2410. In some cases, the avatar data (e.g., the stored avatar data sent 2420 from the first UE to the second UE) may also be referred to as user registration data (e.g., such as Figure 22-Figure 23 The first UE sends 2422 (e.g., transmits) information based on facial expressions, gestures, etc. (e.g., animation parameters such as facial codes, facial blend shapes, hand joints, head gestures, and / or Fig.21 or Fig. 22 The second UE 2410 performs an avatar animation 2424 (e.g., using Fig.21 or Fig. 22 The second UE 2410 renders the user view 2426 (e.g., using Fig.21 or Fig. 22 User view rendering system 2175).

[0225] Fig.25 is an example of a device-to-device call flow for establishing an avatar call for view-dependent and / or view-independent textures between a client device without an edge server and a decoder on a target device (eg, a target HMD). Fig.25 The communication is shown between a first client device 2502 (shown as client device A, also referred to as user A, which may be an HMD, AR glasses, etc.) and a second client device 2505 (shown as client device B, also referred to as user B, which may be an HMD, AR glasses, etc.). In some cases, Fig.25 The communication of the calling process can be done by Fig. 22 XR System 2200 and / or Fig.23 The XR system 2300 performs

[0226] like Fig.25As shown, the first client device 2502 establishes the call (e.g., by sending a request message 2506 to the second client device 2505), and the second client device 2505 accepts the call (e.g., by sending a response 2508 or confirmation message to the first client device 2502). In some cases, the first client device 2502 can then send the user A's grid and assets 2510 (e.g., registered data) to a secure location on the second client device 2505. In other cases, the second client device 2505 can obtain the grid and / or assets from the memory or storage device (e.g., cache) of the second client device 2505, such as if the first client device 2502 and the second client device 2505 are often involved in the call. In some cases, the second client device 2505 can send the grid and / or assets 2512 (e.g., registered data) to a secure location on the first client device 2502. In other cases, the first client device 2502 can obtain the grid and / or assets from the memory or storage device (e.g., cache) of the first client device 2502. The call is then established 2514, and the first client device 2502 and the second client device 2505 exchange mesh animation parameters during the call. The call can then end 2516, in which case the mesh and assets can be deleted from a secure location on the first client device 2502 and the second client device 2505, or if the user is trusted, the mesh and assets can be maintained for further use.

[0227] Fig.26 is a diagram illustrating an example of an example of an XR system 2600 configured to perform aspects described herein, such as operations associated with device-to-device flows without an edge server and all processing (e.g., decoding, avatar animation, etc.) on a source or transmission / sending client device (shown as a source HMD and coprocessor) for view-dependent textures. Fig.26 As shown, the XR system 2600 includes Figure 7 XR System 700, Fig.10 XR System 1000, Fig. 20 XR System 2000, Fig.21 Similar components (e.g., facial encoder 709, body engine 714, hand engine 716, future pose prediction engine 738, etc.) are configured to perform the same as those of XR system 2100 (with the same reference numerals). Figure 7 XR System 700, Fig.10 XR System 1000, Fig. 20 XR System 2000, Fig.21The same or similar operations may be performed by the same components of the XR system 2100, etc. As shown, the user virtual representation system 720 of the transmitting / sending client device may use the registration data 2150 of the user of the transmitting / sending client device (shown as user A) to generate a virtual representation (e.g., an avatar) of user A. The encoder 2630 may encode the virtual representation of user A and send the encoded information to the user view rendering engine 2175 of the target or receiving client device (shown as a target HMD and a coprocessor). The user view rendering engine 2175 may use the encoded information and information 2619 (e.g., background information, in some cases similar to Figure 7 The XR system 2600 also includes a scene definition / composition system 2122, which can be used to synthesize and / or define the scene of the virtual environment or session. In some cases, the receiving client device (target HMD and coprocessor) may not include the video encoder 730, such as based on a transmitting / sending client device (source HMD and coprocessor) with an encoder 2630. The XR system 2600 is shown as including a trusted cloud server, which includes certain components. In some cases, the XR system 2600 may not include a cloud server. The XR system 2600 may also include a video encoder 730 from Figure 7 , Fig.10 , Fig.11 and / or Fig.12 The absence Fig.26 Components shown in, such as pre-processing engine 1019 and / or other components.

[0228] Fig. 27 is a diagram illustrating an example of an example of an XR system 2700 configured to perform aspects described herein, such as operations associated with device-to-device flows without an edge server and all processing (e.g., decoding, avatar animation, etc.) for view-independent textures on a source or transmitting / sending client device (shown as a source HMD and coprocessor). Fig. 27 As shown, the XR system 2700 includes Figure 7 XR System 700, Fig.10 XR System 1000, Fig. 20 XR System 2000, Fig.21 Similar components (e.g., face encoder 709, body engine 714, hand engine 716, etc.) are configured to perform the same as those of XR system 2100 (with the same reference numerals). Figure 7 XR System 700, Fig.10 XR System 1000, Fig. 20 XR System 2000, Fig.21 The same or similar operations may be performed by the same components of the XR system 2100, etc. As shown, the user virtual representation system 720 of the transmitting / sending client device may use the registration data 2150 of the user of the transmitting / sending client device (shown as user A) to generate a virtual representation (e.g., an avatar) of user A. The encoder 2730 may encode the virtual representation of user A and send the encoded information to the user view rendering engine 2175 of the target or receiving device (shown as a target HMD and a coprocessor). The user view rendering engine 2175 may use the encoded information and information 2719 (e.g., background information, in some cases similar to Figure 7 The XR system 2700 may also include a scene definition / composition system 2122 that can be used to synthesize and / or define the scene of a virtual environment or session. The XR system 2700 is shown as including a trusted cloud server that includes certain components. In some cases, the XR system 2700 may not include a cloud server. The XR system 2700 may also include a scene definition / composition system 2122 that can be used to synthesize and / or define the scene of a virtual environment or session. Figure 7 , Fig.10 , Fig.11 and / or Fig.12 The absence Fig. 27 Components shown in, such as pre-processing engine 1019 and / or other components.

[0229] Fig.28 2800 is a signaling diagram illustrating communications between a first UE (shown as UE1), a P-CSCF, an S-CSCF, an I-CSCF, an MRF, an MMTel AS, and a second UE (shown as UE2). The operations of the signaling diagram 2800 may be performed by Fig.26 XR System 2600 and / or Fig. 27 In some cases, the operations of signaling diagram 2800 may be performed in accordance with 3GPP SA4. As shown in the figure, at operation 13, during the AR media and metadata exchange between the first UE and the second UE, the first UE (e.g., using Fig.26 or Fig. 27 The user virtual representation system 720) accesses the stored avatar data (e.g., from Fig.26 or Fig. 27 The user A registration data 2150) and obtain information about the user's facial expressions, postures, etc. (e.g., animation parameters, such as facial codes from the facial encoder 709, hand joints from the hand engine 716, etc., such as Fig.26 or Fig. 27 At operation 14, the first UE performs an avatar animation (eg, using Fig.26 or Fig. 27 At operation 15, the first UE sends (e.g., transmits) a mesh of a user of the first UE to the second UE. At operation 16, the second UE renders the user view (e.g., using Fig.26 or Fig. 27 User view rendering system 2175).

[0230] Fig.29 is an example of a device-to-device call flow for establishing an avatar call between client devices without an edge server and all processing (e.g., decoding, avatar animation, etc.) to the source or transmission / sending client device for view-dependent and / or view-independent textures. Fig.29 The communication is shown between a first client device 2902 (shown as client device A, also referred to as user A, which may be an HMD, AR glasses, etc.) and one or more other client devices 2905 (which may be a single other client device or multiple other client devices). In some cases, Fig.29 The communication of the calling process can be done by Fig.26 XR System 2600 and / or Fig. 27 The XR system 2700 performs

[0231] like Fig.29 As shown, the first client device 2902 establishes a call (e.g., by sending a request message 2906 to one or more other client devices 2905) and at least one client device (or a user of at least one client device) from the one or more other client devices 2905 accepts the call (e.g., by sending a response 2908 or confirmation message to the first client device 2902). The call is then established 2910, and each client device (including the first client device 2902 and each of the one or more other client devices 2905 that accepted the call) can send a three-dimensional (3D) video or 3D video update of a virtual representation (or avatar) of the corresponding user to the other user. The call can then end 2912.

[0232] Fig.30 is a diagram illustrating an example of an example of an XR system 3000 configured to perform aspects described herein, such as with a server device (e.g., Fig.30 Operations for view-dependent textures associated with device-to-device streaming of edge servers and decoders on a trusted edge server and / or a trusted cloud server. Fig.30 As shown, the XR system 3000 includes Figure 7XR System 700, Fig.10 XR System 1000, Fig. 20 XR System 2000, Fig.21 Similar components (e.g., facial encoder 709, geometry encoder engine 1011, hand engine 716, future pose prediction engine 738, etc.) are configured to perform the same as those of XR system 2100. Figure 7 XR System 700, Fig.10 XR System 1000, Fig. 20 XR System 2000, Fig.21 The same or similar operations of the same components of the XR system 2100 and the like. As shown, the user virtual representation system 720 of the trusted edge server can use the registration data 2150 of the user (shown as user A) of the transmission / sending client device to generate a virtual representation (e.g., avatar) of user A (with view-dependent textures). The registration data 2150 of the first user may include Fig.18 The mesh information 1810 may include a mesh defining an avatar of a first user of the first client device 1802 and other assets associated with the avatar (e.g., a normal map, an albedo map, a specular reflection map, etc., for example). Fig.13 The user view rendering engine 2175 on the trusted edge server may use the view-related texture information and information 2119 (e.g., background information, in some cases similar to Figure 7 The XR system 3000 also includes a scene definition / synthesis system 2122 (which may be on a cloud server) that may be used to synthesize and / or define the scene of a virtual environment or session. The XR system 3000 is shown as including a trusted cloud server that includes certain components. In some cases, the XR system 3000 may not include a cloud server. The XR system 3000 may also include a scene definition / synthesis system 2122 from a cloud server. Figure 7 , Fig.10 , Fig.11 and / or Fig.12 The absence Fig.30 Components shown in, such as pre-processing engine 1019 and / or other components.

[0233] Fig.31 is a diagram illustrating an example of an example of an XR system 3100 configured to perform aspects described herein, such as with a server device (e.g., Fig.31 Operations for view-independent textures associated with device-to-device streaming of edge servers and decoders on a trusted edge server and / or a trusted cloud server. Fig.31 As shown, the XR system 3100 includes Figure 7 XR System 700, Fig.10 XR System 1000, Fig. 20 XR System 2000, Fig.21 Similar components (e.g., face encoder 709, geometry encoder engine 1011, hand engine 716, etc.) are configured to perform the same as those of XR system 2100. Figure 7 XR System 700, Fig.10 XR System 1000, Fig. 20 XR System 2000, Fig.21 The same or similar operations of the same components of the XR system 2100 and the like. As shown, the user virtual representation system 720 of the trusted edge server can use the registration data 2150 of the user (shown as user A) of the transmission / sending client device to generate a virtual representation (e.g., avatar) of user A (with view-independent textures). The user view rendering engine 2175 on the trusted edge server can use the view-independent texture information and information 2119 (e.g., background information, in some cases similar to Figure 7 The XR system 3100 also includes a scene definition / synthesis system 2122 (which may be on a cloud server) that may be used to synthesize and / or define the scene of a virtual environment or session. The XR system 3100 is shown as including a trusted cloud server that includes certain components. In some cases, the XR system 3100 may not include a cloud server. The XR system 3100 may also include a scene definition / synthesis system 2122 from a cloud server. Figure 7 , Fig.10 , Fig.11 and / or Fig.12 The absence Fig.31 Components shown in, such as pre-processing engine 1019 and / or other components.

[0234] Fig.32 3200 is a signaling diagram illustrating communications between a first UE (shown as UE1 3202), a P-CSCF, an S-CSCF, an I-CSCF (collectively referred to as P / S / I-CSCF 3204), an MRF 3206, an MMTel AS 3208, and a second UE (shown as UE2 3910). The operations of the signaling diagram 3200 may be performed by Fig.30 XR System 3000 and / or Fig.31 In some cases, the operations of signaling diagram 3200 may be performed in accordance with 3GPP SA4. In some cases, call establishment process 2412, scene description retrieval process 2414, and scene description update process 2416 may be based on Fig.30 and Fig.31 The like numbered components are substantially similar to those described above with respect to Fig.24 As shown, during the AR media and metadata exchange between the first UE 3202, the MRF 3206, and the second UE 3210, the first UE 3202 sends a message to the enhanced MRF 3206 (e.g., which may be Fig.30 and / or Fig.31 The first UE 3202 sends 3222 (e.g., transmits) information based on facial expressions, gestures, etc. (e.g., animation parameters such as facial codes, facial blend shapes, hand joints, head poses, and / or Fig.30 or Fig.31 MRF 3206 performs avatar animation 3224 (e.g., using Fig.30 or Fig.31 The MRF 3206 renders the user view 3226 (e.g., using Fig.30 or Fig.31 The MRF 3206 sends 3228 the rendered user view to the second UE 3210.

[0235] Fig.33 is used to connect to a server device (e.g. Fig.30 and / or Fig.31 An example of a device-to-device call flow for establishing an avatar call between an edge server and a decoder client device on a trusted edge server and / or a trusted cloud server for view-dependent and / or view-independent textures. Fig.33 The communication is shown between a first client device 3302 (shown as client device A, also referred to as user A, which may be an HMD, AR glasses, etc.) and one or more other client devices 3305 (which may be a single other client device or multiple other client devices). In some cases, Fig.33 The communication of the calling process can be done by Fig.30 XR System 3000 and / or Fig.31 The XR system 3100 performs

[0236] like Fig.33 As shown, the first client device 3302 establishes a call (e.g., by sending a request message 3306 to one or more other client devices 3305), and at least one client device (or a user of at least one client device) from the one or more other client devices 3305 accepts the call (e.g., by sending a response 3308 or confirmation message to the first client device 3302). In some cases, the first client device 3302 and each of the one or more other client devices 3305 that accept the call can send (e.g., transmit) the corresponding mesh, asset (e.g., albedo map, normal map, specular reflection map, etc.), machine learning parameters 3310 (e.g., neural network weights) (which can be part of the registration data) to a trusted server (e.g., Fig.30 and / or Fig.31 In other cases, if the corresponding client device has an account with the trusted server (or a service provider using the trusted server), the client device can obtain the grid, assets, etc. from the memory or storage (e.g., cache) of the corresponding client device. In this case, the corresponding client device can be authenticated by the trusted server so as to allow the client device to obtain the data.

[0237] A call 3312 is then established, and each client device (including the first client device 3302 and each of the one or more other client devices 3305 receiving the call) can send mesh animation parameters to the trusted server during the call. The trusted server can then send a specific scene 3314 generated by the trusted server to each client device (e.g., using Fig.30 or Fig.31 user virtual representation system 720 and / or user view rendering system 2175).

[0238] The call may then end 3316, in which case the grid and assets may be deleted from the trusted server (eg, a secure location or storage on the trusted server) or the client devices may log in from their respective accounts.

[0239] Fig.34 is a diagram illustrating an example of an example of an XR system 3400 configured to perform aspects described herein, such as operations associated with device-to-device streaming utilizing an edge server and all processing (e.g., decoding, avatar animation, etc.) on a source or transmission / sending client device (shown as a source HMD and coprocessor) for view-dependent textures. Fig.34 As shown, the XR system 3400 includes Figure 7 XR System 700, Fig.10 XR System 1000, Fig. 20 XR System 2000, Fig.21 Similar components (e.g., facial encoder 709, geometry encoder engine 1011, hand engine 716, future pose prediction engine 738, etc.) are configured to perform the same as those of XR system 2100. Figure 7 XR System 700, Fig.10 XR System 1000, Fig. 20 XR System 2000, Fig.21 The same or similar operations may be performed by the same components of the XR system 2100 and the like. As shown, the user virtual representation system 720 of the transmission / sending client device may use the registration data 2150 of the user of the transmission / sending client device (shown as user A) to generate a virtual representation (e.g., an avatar) of user A. The encoder 3430 may encode the virtual representation of user A and send the encoded information to the user view rendering engine 2175 on the edge server. The user view rendering engine 2175 may use the encoded information and information 3419 (e.g., background information, in some cases similar to Figure 7 The XR system 3400 also includes a scene definition / composition system 2122, which can be used to synthesize and / or define the scene of the virtual environment or session. In some cases, the receiving client device (target HMD and coprocessor) may not include the video encoder 730, such as based on a transmitting / sending client device (source HMD and coprocessor) with an encoder 3430. The XR system 3400 is shown as including a trusted cloud server, which includes certain components. In some cases, the XR system 3400 may not include a cloud server. The XR system 3400 may also include a video encoder 730 from Figure 7 , Fig.10 , Fig.11 and / or Fig.12 The absence Fig.34 Components shown in, such as pre-processing engine 1019 and / or other components.

[0240] Fig.35 is a diagram illustrating an example of an example of an XR system 3500 configured to perform aspects described herein, such as operations associated with device-to-device streaming utilizing an edge server and all processing (e.g., decoding, avatar animation, etc.) on a source or transmission / sending client device (shown as a source HMD and coprocessor) for view-independent textures. Fig.35As shown, the XR system 3500 includes Figure 7 XR System 700, Fig.10 XR System 1000, Fig. 20 XR System 2000, Fig.21 Similar components (e.g., face encoder 709, geometry encoder engine 1011, hand engine 716, etc.) are configured to perform the same as those of XR system 2100. Figure 7 XR System 700, Fig.10 XR System 1000, Fig. 20 XR System 2000, Fig.21 The same or similar operations may be performed by the same components of the XR system 2100 and the like. As shown, the user virtual representation system 720 of the transmission / sending client device may use the registration data 2150 of the user of the transmission / sending client device (shown as user A) to generate a virtual representation (e.g., an avatar) of user A. The encoder 3530 may encode the virtual representation of user A and send the encoded information to the user view rendering engine 2175 on the edge server. The user view rendering engine 2175 may use the encoded information and information 3519 (e.g., background information, in some cases similar to Figure 7 The XR system 3500 may also include a scene definition / composition system 2122 that may be used to synthesize and / or define the scene of a virtual environment or session. The XR system 3500 is shown as including a trusted cloud server that includes certain components. In some cases, the XR system 3500 may not include a cloud server. The XR system 3500 may also include a scene definition / composition system 2122 that may be used to synthesize and / or define the scene of a virtual environment or session. Figure 7 , Fig.10 , Fig.11 and / or Fig.12 The absence Fig.35 Components shown in, such as pre-processing engine 1019 and / or other components.

[0241] Fig.36 3600 is a signaling diagram illustrating communications between a first UE (shown as UE1), a P-CSCF, an S-CSCF, an I-CSCF, an MRF, an MMTel AS, and a second UE (shown as UE2). The operations of the signaling diagram 3600 may be performed by Fig.34 XR System 3400 and / or Fig.35In some cases, the operations of signaling diagram 3600 may be performed according to 3GPP SA4. As shown in the figure, at operation 13, during the AR media and metadata exchange between the first UE, the MRF, and the second UE, the first UE (e.g., using Fig.34 or Fig.35 The user virtual representation system 720) accesses the stored avatar data (e.g., from Fig.34 or Fig.35 The user A registration data 2150) and obtain information about the user's facial expressions, postures, etc. (e.g., animation parameters, such as facial codes from the facial encoder 709, hand joints from the hand engine 716, etc., such as Fig.34 or Fig.35 At operation 14, the first UE performs an avatar animation (eg, using Fig.34 or Fig.35 At operation 15, the first UE sends a request to the MRF (for example, which may be Fig.30 and / or Fig.31 The MRF maps the user view (eg, using Fig.34 or Fig.35 In operation 17, the MRF sends the rendered user view to the second UE.

[0242] Fig.37 is an example of a device-to-device call flow for establishing an avatar call between a client device with an edge server and all processing (e.g., decoding, avatar animation, etc.) for view-dependent and / or view-independent textures on the source or transmitting / sending client device. Fig.37 The communication is shown between a first client device 3702 (shown as client device A, also referred to as user A, which may be an HMD, AR glasses, etc.) and one or more other client devices 3705 (which may be a single other client device or multiple other client devices). In some cases, Fig.37 The communication of the calling process can be done by Fig.34 XR System 3400 and / or Fig.35 The XR System 3500 performs

[0243] like Fig.37As shown, a first client device 3702 establishes a call (e.g., by sending a request message 3706 to one or more other client devices 3705), and at least one client device (or a user of at least one client device) from the one or more other client devices 3705 accepts the call (e.g., by sending a response 3708 or confirmation message to the first client device 3702). The call 3710 is then established, and each client device (including the first client device 3702 and each of the one or more other client devices 3705 that accepted the call) can send a three-dimensional (3D) video or 3D video update of a virtual representation (or avatar) of the corresponding user to the trusted server during the call. The trusted server can then send a specific scene 3712 generated by the trusted server to each client device (e.g., using Fig.30 or Fig.31 user virtual representation system 720 and / or user view rendering system 2175).

[0244] The call may then end 3714, in which case the grid and assets may be deleted from the trusted server (eg, a secure location or storage on the trusted server), or the client devices may be logged out of their respective accounts.

[0245] Fig.38 3800 is a signaling diagram illustrating another example of communication between a first UE (shown as UE1), a P-CSCF, an S-CSCF, an I-CSCF, an MRF, an MMTel AS, and a second UE (shown as UE2) in an XR system including an edge server and a target device (e.g., a target HMD) having a decoder. In some cases, the operations of the signaling diagram 3800 may be performed according to 3GPP SA4. As shown in the figure, at operation 13, at the first UE, the MRF (e.g., which may be Fig.30 , Fig.31 During an AR media and metadata exchange between a first UE and a second UE (e.g., an edge server as part of the other figures described herein), the first UE may send (e.g., transmit) stored avatar data (e.g., from the MRF) to the MRF. Fig.34 or Fig.35At operation 14, the first UE may send (e.g., transmit) information about the user's facial expressions, postures, etc. to the MRF (e.g., animation parameters, such as facial codes from the facial encoder 709, hand joints from the hand engine 716, etc.). At operation 15, the MRF performs avatar animation (e.g., using the user virtual representation system 720). At operation 16, the MRF sends (e.g., transmits) a mesh of the user of the first UE to the second UE. At operation 17, UE2 renders a user view (e.g., using the user view rendering system 2175). In some cases, UE2 may perform the operations described above with respect to UE1 to allow UE1 to render the user view.

[0246] Fig.39 3900 is a signaling diagram illustrating communications between a first UE (shown as UE1 3902), a P-CSCF, an S-CSCF, an I-CSCF (collectively referred to as P / S / I-CSCF 3904), an MRF 3906, an MMTel AS 3908, and a second UE (shown as UE2 3910). The operations of the signaling diagram 3900 may be performed by Fig.30 XR System 3000 and / or Fig.31 In some cases, the operations of signaling diagram 3900 may be performed in accordance with 3GPP SA4. In some cases, call establishment process 2412, scene description retrieval process 2414, and scene description update process 2416 may be based on Fig.30 and Fig.31 The like numbered components are substantially similar to those described above with respect to Fig.24 As shown, during the AR media and metadata exchange 3918 between the first UE 3902, the MRF 3906, and the second UE 3910, the first UE 3902 sends a message to the enhanced MRF 3906 (e.g., which may be Fig.30 and / or Fig.31 The first UE 3902 sends 3924 (e.g., transmits) information based on facial expressions, gestures, etc. (e.g., animation parameters such as facial codes, facial blend shapes, hand joints, head poses, and / or Fig.30 or Fig.31 MRF 3906 performs avatar animation 3926 (e.g., using Fig.30 or Fig.31 The MRF 3906 renders the user view 4028 (e.g., using Fig.30 or Fig.31 The MRF 3906 sends 3930 the rendered user view to the second UE 3910. In some cases, the UE2 3910 may perform the operations described above with respect to the UE1 3902 to allow the UE1 3902 to receive the rendered user view from the MRF 3906.

[0247] Fig.40 4000 is a signaling diagram illustrating communications between a first UE (shown as UE1 4002), a P-CSCF, an S-CSCF, an I-CSCF (collectively referred to as P / S / I-CSCF 4004), an MRF 4006, an MMTel AS 4008, and a second UE (shown as UE2 4010). The operations of the signaling diagram 4000 may be performed by Fig.30 XR System 3000 and / or Fig.31 In some cases, the operations of signaling diagram 4000 may be performed in accordance with 3GPP SA4. In some cases, call establishment process 2412, scene description retrieval process 2414, and scene description update process 2416 may be based on Fig.30 and Fig.31 The like numbered components are substantially similar to those described above with respect to Fig.24 As shown, during the AR media and metadata exchange 4018 between the first UE 4002, the MRF 4006, and the second UE 4010, the first UE 4002 may generate or otherwise obtain 4020 (e.g., from a server) a 3D model of the avatar. The first UE 4002 then sends the augmented MRF 4006 (e.g., which may be Fig.30 and / or Fig.31 The first UE 4002 sends 4024 (e.g., transmits) information (e.g., animation parameters such as facial codes, facial blend shapes, hand joints, head poses, and / or other parameters) based on facial expressions, poses, etc. to the MRF 4006. Fig.30 or Fig.31 MRF 4006 performs avatar animation 4026 (e.g., using Fig.30 or Fig.31 The MRF 4006 renders the user view 4028 (e.g., using Fig.30 or Fig.31The MRF 4006 sends 4030 the rendered user view to the second UE 4010. In some cases, the UE2 4010 may perform the operations described above with respect to the UE1 4002 to allow the UE1 4002 to receive the rendered user view from the MRF 4006.

[0248] The avatar invocation flow described in this article (e.g., Fig.16 Device-to-device calling process, Fig.18 A server-based call flow, etc.) can provide a reduced transmission data rate when sending information used to animate a user's virtual representation. For example, a device-to-device call flow can include two parts, including sending mesh information (e.g., Fig.16 Grid information 1610, Fig.18 1810, etc.) and sending animation parameters (e.g. Fig.16 Animation parameters 1614, Fig.18 The server-based call flow may include three parts, including an initialization phase, a first conversion phase, and a second conversion phase, in which display images (e.g., two display images per frame) are sent to each client device in the server-based call flow. Using the call flow described herein, the data transfer rate of each of these phases can be kept to a minimum.

[0249] For example, during the initialization phase, information defining the grid and assets may be sent only once per avatar call and per user (or once per user for multiple avatar calls when the user has an account on the server, as described in the example above). Fig.18 In one illustrative example, during an initialization phase of invocation of an avatar of a user of a client device, the client device may transmit (e.g., to Fig.16 The second client device 1604 in the device-to-device call process, to Fig.18 The server device 1805 in the server-based call flow of the server device 1805, etc.) includes mesh information of a virtual representation (e.g., an avatar) of a user, including at least one mesh, a normal map, an albedo (or diffuse) map, a specular map, and in some cases at least a portion of a neural network (e.g., weights and / or other parameters of a decoder neural network for decoding mesh animation parameters (such as one or more codes or feature vectors)). The mesh information for the virtual representation of the user can be sent once at the beginning of the avatar call. In an illustrative example, Table 1 below provides an example of the size of the mesh information data:

[0250]

[0251] Table 1

[0252] As described above, the mesh information data may also be compressed using any compression technique, which may further reduce the size of the data.

[0253] As another example, in the first session phase, for each frame of the virtual session, the user's mesh animation parameters may be sent by the client device at the frame rate of the virtual session (such as at 30 FPS, 60 FPS, or other frame rates) (e.g., to the client device). Fig.16 The client device 1604 in the device-to-device call process sends Fig.18 In one illustrative example, the parameter updates may vary between 30 FPS and 60 FPS (wherein the updated mesh animation parameters may be transmitted at a rate varying between 30 FPS and 60 FPS). As described herein, the mesh animation parameters for each user may include facial codes, facial blend shapes, hand joint codes, head pose information, and an audio stream. In one illustrative example, Table 2 below provides an example of the size of the mesh animation parameters. The data in Table 2 is an approximation for a single user at a rate of 60 FPS, assuming that facial codes, facial blend shapes, and hand joints are 2 bytes / entry, and head pose is 4 bytes / entry.

[0254]

[0255] Table 2

[0256] As mentioned above, the data for the mesh animation parameters may also be compressed using any compression technique, which may further reduce the size of the data.

[0257] As another example, in the second dialog phase (for a server-based call flow), a server device (eg, server device 1805) may send a request to a server-based call flow (eg, Fig.18 The server device may send display images (e.g., two display images per frame, including one display image for each eye) to each client device in the call flow of the virtual session. The server device may send images at the frame rate of the virtual session, such as at 60fps. For example, assuming a display size of 4K (corresponding to a resolution of 3840×2160), the device server may send a total of 2×(3840×2160) streams at 60FPS for each user. In another example, the server device may stream 2 images at a resolution of 1024×768, and the client device may upscale the images to 4K resolution.

[0258]

[00136] As described herein, various aspects may be implemented using a deep network, such as a neural network or multiple neural networks. Fig.39is an illustrative example of a deep learning neural network 4100 that can be used by a 3D model training system. An input layer 4120 includes input data. In one illustrative example, the input layer 4120 may include data representing pixels of an input video frame. The neural network 4100 includes a plurality of hidden layers 4122a, 4122b to 4122n. The hidden layers 4122a, 4122b to 4122n include "n" hidden layers, where "n" is an integer greater than or equal to 1. The number of hidden layers can be made to include as many layers as required for a given application. The neural network 4100 also includes an output layer 4124 that provides outputs resulting from processing performed by the hidden layers 4122a, 4122b to 4122n. In one illustrative example, the output layer 4124 may provide classification for objects in the input video frame. The classification may include identifying a class of an object type (e.g., a person, dog, cat, or other object).

[0259] Neural network 4100 is a multi-layer neural network of interconnected nodes. Each node can represent a piece of information. Information associated with a node is shared between different layers, and each layer retains the information as it is processed. In some cases, neural network 4100 may include a feedforward network, in which case there is no feedback connection in which the output of the network is fed back into itself. In some cases, neural network 4100 may include a recursive neural network, which may have loops that allow information to be carried across nodes when reading in input.

[0260] Information can be exchanged between nodes by node-to-node interconnection between each layer. The nodes of input layer 4120 can activate a group of nodes in the first hidden layer 4122a. For example, as shown in the figure, each input node of input layer 4120 is connected to each node of the first hidden layer 4122a. The nodes of hidden layers 4122a, 4122b to 4122n can transform the information of each input node by applying an activation function to the information. Then, the information derived from the transformation can be transferred to the node of the next hidden layer 4122b and the node of the next hidden layer 4122b can be activated, and the node of the next hidden layer 4122b can perform its own specified function. Example functions include convolution, upsampling, data conversion and / or any other suitable function. Then, the output of hidden layer 4122b can activate the node of the next hidden layer, and so on. The output of the last hidden layer 4122n can activate one or more nodes of output layer 4124, and output is provided at the one or more nodes. In some cases, although a node in neural network 4100 (eg, node 4126) is shown as having multiple output lines, the node has a single output and all lines shown as outputting from the node represent the same output value.

[0261] In some cases, each node or interconnection between nodes can have a weight, which is a set of parameters derived from the training of neural network 4100. Once neural network 4100 is trained, it can be referred to as a trained neural network, which can be used to classify one or more objects. For example, an interconnection between nodes can represent a piece of information learned about the interconnected nodes. The interconnections can have tunable digital weights that can be tuned (e.g., based on a training data set), allowing neural network 4100 to adapt to the input and be able to learn as more and more data is processed.

[0262] The neural network 4100 is pre-trained to process features from the data in the input layer 4122 using different hidden layers 4122a, 4122b, through 4122n to provide an output through the output layer 4124. In an example where the neural network 4100 is used to recognize objects in images, the neural network 4100 may be trained using training data that includes both images and labels. For example, training images may be input into the network, where each training image has a label indicating the class of one or more objects in each image (essentially, indicating to the network what the objects are and what features they have). In one illustrative example, the training images may include an image of the number 2, in which case the label for the image may be [0 0 1 0 0 0 0 00 0].

[0263] In some cases, the neural network 4100 can use a training process called back propagation to adjust the weights of the nodes. Back propagation can include a forward pass, a loss function, a backward pass, and a weight update. The forward pass, loss function, backward pass, and parameter update are performed for one training iteration. This process can be repeated for a certain number of iterations for each set of training images until the neural network 4100 is trained well enough so that the weights of the layers are accurately tuned.

[0264] For the example of recognizing an object in an image, a forward pass may include passing a training image through the neural network 4100. Prior to training the neural network 4100, the weights are initially randomized. An image may include, for example, an array of numbers representing pixels of the image. Each number in the array may include a value from 0 to 255 describing the intensity of the pixel at that position in the array. In one example, the array may include a 28×28×3 array of numbers having 28 rows and 28 columns of pixels and 3 color components (such as red, green, and blue, or brightness and two chrominance components, etc.).

[0265] For the first training iteration of the neural network 4100, the output will likely include values ​​that do not favor any particular class due to the weights being randomly chosen at initialization. For example, if the output is a vector with probabilities that the object includes different classes, the probability values ​​for each different class may be equal or at least very similar (e.g., for ten possible classes, each class may have a probability value of 0.1). Using the initial weights, the neural network 4100 cannot determine low-level features and therefore cannot accurately determine what the classification of the object is likely to be. A loss function may be used to analyze the error in the output. Any suitable loss function definition may be used. An example of a loss function includes the mean squared error (MSE). The MSE is defined as , which is calculated by multiplying one half by the sum of the actual answer minus the square of the predicted (output) answer. The loss can be set equal to The value of .

[0266] For the first training image, the loss (or error) will be high because the actual value will be very different from the predicted output. The goal of training is to minimize the amount of loss so that the predicted output is the same as the training label. The neural network 4100 can perform a backward pass by determining which inputs (weights) contribute the most to the network's loss, and the weights can be adjusted so that the loss is reduced and ultimately minimized.

[0267] The derivative of the loss with respect to the weight (expressed as dL / dW, where W is the weight at a particular layer) can be calculated to determine the weight that contributes most to the network loss. After calculating the derivative, a weight update can be performed by updating all the weights of the filter. For example, the weights can be updated so that they change in the opposite direction of the gradient. The weight update can be expressed as ,in represents the weight, represents the initial weight, and Represents the learning rate. The learning rate can be set to any suitable value, where a high learning rate includes larger weight updates and a lower value indicates smaller weight updates.

[0268] Neural network 4100 may include any suitable deep network. One example includes a convolutional neural network (CNN), which includes an input layer and an output layer with multiple hidden layers between the input layer and the output layer. The hidden layers of the CNN include a series of convolution, nonlinear, pooling (for downsampling), and fully connected layers. Neural network 4100 may include any other deep network other than CNN, such as an autoencoder, a deep belief network (DBN), a recurrent neural network (RNN), etc.

[0269] Fig.42is an illustrative example of a convolutional neural network 4200 (CNN 4200). An input layer 4220 of the CNN 4200 includes data representing an image. For example, the data may include an array of numbers representing pixels of the image, with each number in the array comprising a value from 0 to 255 describing the intensity of the pixel at that position in the array. Using the previous example from above, the array may include a 28×28×3 array of numbers with 28 rows and 28 columns of pixels and 3 color components (e.g., red, green, and blue, or luma and two chroma components, etc.). The image may pass through a convolutional hidden layer 4222a, an optional non-linear activation layer, a pooling hidden layer 4222b, and a fully connected hidden layer 4222c to obtain an output at an output layer 4224. Although Fig.42 Only one of each hidden layer is shown in , but one of ordinary skill will appreciate that multiple convolutional hidden layers, nonlinear layers, pooling hidden layers, and / or fully connected layers may be included in CNN 4200. As previously described, the output may indicate a single category of an object, or may include a probability of a category that best describes an object in the image.

[0270] The first layer of CNN 4200 is a convolutional hidden layer 4222a. Convolutional hidden layer 4222a analyzes the image data of input layer 4220. Each node of convolutional hidden layer 4222a is connected to a region of nodes (pixels) of the input image called a receptive field. Convolutional hidden layer 4222a can be thought of as one or more filters (each filter corresponding to a different activation or feature map), where each convolution iteration of the filter is a node or neuron of convolutional hidden layer 4222a. For example, the region of the input image covered by the filter at each convolution iteration will be the receptive field of the filter. In an illustrative example, if the input image includes a 28×28 array, and each filter (and corresponding receptive field) is a 5×5 array, there will be 24×24 nodes in convolutional hidden layer 4222a. Each connection between a node and the receptive field of that node learns a weight, and in some cases learns an overall bias, so that each node learns to analyze its specific local receptive field in the input image. Each node of hidden layer 4222a will have the same weights and biases (referred to as shared weights and shared biases). For example, a filter has an array of weights (numbers) and the same depth as the input. For the video frame example, the filter would have a depth of 3 (according to the three color components of the input image). An illustrative example size of the filter array is 5×5×3, corresponding to the size of the node's receptive field.

[0271] The convolutional nature of the convolutional hidden layer 4222a is due to the fact that each node of the convolutional layer is applied to its corresponding receptive field. For example, the filter of the convolutional hidden layer 4222a can start at the upper left corner of the input image array and can be convolved around the input image. As described above, each convolution iteration of the filter can be considered as a node or neuron of the convolutional hidden layer 4222a. At each convolution iteration, the value of the filter is multiplied by the corresponding number of original pixel values ​​of the image (e.g., a 5×5 filter array multiplied by a 5×5 array of input pixel values ​​at the upper left corner of the input image array). The multiplications from each convolution iteration can be added together to obtain the sum for that iteration or node. Next, the process continues at the next position in the input image based on the receptive field of the next node in the convolutional hidden layer 4222a.

[0272] For example, the filter can be moved by the stride amount to the next receptive field. The stride amount can be set to 1 or other suitable amount. For example, if the stride amount is set to 1, the filter will move 1 pixel to the right at each convolution iteration. Processing the filter at each unique position of the input volume produces a number representing the filter result at that position, resulting in a sum value being determined for each node of the convolution hidden layer 4222a.

[0273] The mapping from the input layer to the convolutional hidden layer 4222a is called an activation map (or feature map). The activation map includes the value of each node that represents the filtering result at each location of the input volume. The activation map may include an array that includes the various summed values ​​produced by each iteration of the filter over the input volume. For example, if a 5×5 filter is applied to each pixel of a 28×28 input image (with a stride of 1), the activation map will include a 24×24 array. The convolutional hidden layer 4222a may include several activation maps in order to identify multiple features in the image. Fig.42 The example shown in includes three activation maps. Using the three activation maps, the convolutional hidden layer 4222a can detect three different kinds of features, each of which is detectable across the entire image.

[0274] In some examples, a nonlinear hidden layer can be applied after the convolutional hidden layer 4222a. Nonlinear layers can be used to introduce nonlinearity into a system that already computes linear operations. An illustrative example of a nonlinear layer is a rectified linear unit (ReLU) layer. The ReLU layer can apply the function f(x) = max(0, x) to all values ​​in the input volume, which changes all negative activations to 0. Thus, the ReLU can increase the nonlinear properties of the CNN 4200 without affecting the receptive field of the convolutional hidden layer 4222a.

[0275] The pooling hidden layer 4222b can be applied after the convolutional hidden layer 4222a (and after the non-linear hidden layer when used). The pooling hidden layer 4222b is used to simplify the information in the output from the convolutional hidden layer 4222a. For example, the pooling hidden layer 4222b can take each activation map output from the convolutional hidden layer 4222a and generate a compressed activation map (or feature map) using a pooling function. Max pooling is an example of a function performed by a pooling hidden layer. The pooling hidden layer 4222a can use other forms of pooling functions, such as average pooling, L2 norm pooling, or other suitable pooling functions. The pooling function (e.g., a maximum pooling filter, an L2 norm filter, or other suitable pooling filter) is applied to each activation map included in the convolutional hidden layer 4222a. Fig.42 In the example shown, three pooling filters are used for the three activation maps in the convolutional hidden layer 4222a.

[0276] In some examples, max pooling can be used by applying a max pooling filter (e.g., having a size of 2×2) with a stride (e.g., equal to the dimension of the filter, such as stride 2) to the activation map output from the convolutional hidden layer 4222a. The output from the max pooling filter includes the maximum number in each sub-region of the filter convolution. Using a 2×2 filter as an example, each unit in the pooling layer can summarize a region of 2×2 nodes in the previous layer (each node is a value in the activation map). For example, four values ​​(nodes) in the activation map will be analyzed by a 2×2 max pooling filter at each iteration of the filter, where the maximum of the four values ​​is output as the “max” value. If such a max pooling filter is applied to an activation filter with a dimension of 24×24 nodes from the convolutional hidden layer 4222a, the output from the pooling hidden layer 4222b will be an array of 12×12 nodes.

[0277] In some examples, an L2-norm pooling filter may also be used. The L2-norm pooling filter involves computing the square root of the sum of the squares of the values ​​in a 2×2 region (or other suitable region) of the activation map (rather than computing the maximum value as done in max pooling), and using the computed value as the output.

[0278] Intuitively, a pooling function (e.g., max pooling, L2 norm pooling, or other pooling functions) determines whether a given feature is found anywhere in an image region and discards the exact location information. This can be done without affecting the results of feature detection because once a feature is found, the exact location of the feature is not as important as its approximate location relative to other features. Max pooling (and other pooling methods) provides the benefit of pooling far fewer features, thereby reducing the number of parameters required in later layers of the CNN 4200.

[0279] The last layer of connections in the network is a fully connected layer that connects each node from the pooling hidden layer 4222b to each output node in the output layer 4224. Using the above example, the input layer includes 28×28 nodes that encode the pixel intensities of the input image, the convolutional hidden layer 4222a includes 3×24×24 hidden feature nodes based on applying a 5×5 local receptive field (for the filter) to the three activation maps, and the pooling layer 4222b includes a layer of 3×12×12 hidden feature nodes based on applying a max pooling filter to a 2×2 region on each of the three feature maps. Extending this example, the output layer 4224 can include ten output nodes. In such an example, each node of the 3×12×12 pooling hidden layer 4222b is connected to each node of the output layer 4224.

[0280] The fully connected layer 4222c can obtain the output of the previous pooling layer 4222b (which should represent an activation map of high-level features) and determine the features that are most relevant to a particular category. For example, the fully connected layer 4222c layer can determine the high-level features that are most strongly associated with a particular category and can include weights (nodes) for the high-level features. The product between the weights of the fully connected layer 4222c and the pooling hidden layer 4222b can be calculated to obtain the probabilities of different classes. For example, if the CNN 4200 is used to predict that an object in a video frame is a person, a high value will be present in the activation map, which represents high-level features of a person (e.g., there are two legs, a face is present at the top of the object, two eyes are present at the upper left and upper right of the face, a nose is present in the middle of the face, a mouth is present at the bottom of the face, and / or other features common to people).

[0281] In some examples, the output from output layer 4224 may include an M-dimensional vector (in the previous example, M=10), where M may include the number of classes that the program must choose from when classifying an object in an image. Other example outputs may also be provided. Each number in the N-dimensional vector may represent the probability that an object belongs to a certain category. In an illustrative example, if the 10-dimensional output vector representing ten different categories of objects is [0 0 0.05 0.8 0 0.15 0 0 0 0], the vector indicates that there is a 5% probability that the image is an object of the third category (e.g., a dog), an 80% probability that the image is an object of the fourth category (e.g., a person), and a 15% probability that the image is an object of the sixth category (e.g., a kangaroo). The probability of a class can be considered as the confidence level that the object is part of that class.

[0282] Fig.43 is a diagram illustrating an example of a system for implementing certain aspects of the present technology. In particular, Fig.43An example of a computing system 4300 is shown, which may be any computing device, for example, constituting an internal computing system, a remote computing system, a camera, or any component thereof, wherein the components of the system communicate with each other using connection 4305. Connection 4305 may be a physical connection using a bus, or a direct connection into processor 4310, such as in a chipset architecture. Connection 4305 may also be a virtual connection, a networked connection, or a logical connection.

[0283] In some aspects, computing system 4300 is a distributed system in which the functionality described in the present disclosure may be distributed across a data center, multiple data centers, a peer-to-peer network, etc. In some aspects, one or more of the described system components represent a number of such components, each component performing some or all of the functionality for which the component is described. In some aspects, a component may be a physical or virtual device.

[0284] The example system 4300 includes at least one processing unit (CPU or processor) 4310 and connections 4305 that couple various system components including system memory 4315, such as read only memory (ROM) 4320 and random access memory (RAM) 4325, to the processor 4310. The computing system 4300 may include a cache 4311 of high-speed memory directly connected to the processor 4310, in close proximity to the processor 4310, or integrated as part of the processor 4310.

[0285] Processor 4310 may include any general purpose processor and hardware services or software services, such as services 4332, 4334, and 4336 stored in storage device 4330, configured to control processor 4310 as well as special purpose processors where the software instructions are incorporated into the actual processor design. Processor 4310 may be essentially a completely independent computing system including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0286] To enable user interaction, computing system 4300 includes input device 4345, which can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, etc. Computing system 4300 can also include output device 4335, which can be one or more of a plurality of output mechanisms. In some cases, a multimodal system can enable a user to provide multiple types of input / output to communicate with computing system 4300. Computing system 4300 can include communication interface 4340, which can generally govern and manage user input and system output.

[0287] The communication interface may use a wired and / or wireless transceiver to perform or facilitate receiving and / or transmitting wired or wireless communications, including utilizing an audio jack / plug, a microphone jack / plug, a universal serial bus (USB) port / plug, an Apple® Lightning® port / plug, an Ethernet port / plug, a fiber optic port / plug, a proprietary wired port / plug, BLUETOOTH® wireless signal transmission, BLUETOOTH® low energy (BLE) wireless signal transmission, IBEACON® wireless signal transmission, radio frequency identification (RFID) wireless signal transmission, near field communication (NFC) wireless signal transmission, dedicated short range communication (SRC) wireless signal transmission, or wireless communication with a wireless controller. DSRC) wireless signal transmission, 802.11 Wi-Fi wireless signal transmission, WLAN signal transmission, visible light communication (VLC), world interoperability for microwave access (WiMAX), infrared (IR) communication wireless signal transmission, public switched telephone network (PSTN) integrated services digital signal transmission network (ISDN) signal transmission, 3G / 4G / 5G / Long Term Evolution (LTE) cellular data network wireless signal transmission, ad-hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or some combination thereof.

[0288] The communication interface 4340 may also include one or more GNSS receivers or transceivers for determining the location of the computing system 4300 based on one or more signals received from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the Global Positioning System (GPS) based in the United States, the Global Navigation Satellite System (GLONASS) based in Russia, the BeiDou Navigation Satellite System (BDS) based in China, and the Galileo GNSS based in Europe. There is no limitation to operating on any particular hardware arrangement, and thus the basic features here can be easily replaced with improved hardware or firmware arrangements when such arrangements are developed.

[0289] The storage device 4330 may be a non-volatile and / or non-transitory and / or computer-readable memory of a device and may be a hard disk or other type of computer-readable medium that can store data accessible by a computer, such as a magnetic cassette, a flash memory card, a solid-state memory device, a digital versatile disk, a magnetic cassette, a floppy disk, a flexible disk, a hard disk, a magnetic tape, a magnetic stripe / stripe, any other magnetic storage medium, a flash memory, a memristor memory, any other solid-state memory, a compact disc read-only memory (CD-ROM) disc, a rewritable compact disc (CD) disc, a digital video disc (DVD) disc, a Blu-ray disc (BDD) disc, a holographic disc, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a Memory Card, or a memory stick. Stick® card, smart card chip, Europay, Mastercard and Visa (EMV) chip, Subscriber Identity Module (SIM) card, Mini / Micro / Nano / Pico SIM card, another integrated circuit (IC) programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash EPROM (FLASHEPROM), cache memory (L1 / L2 / L3 / L4 / L5 / L#), resistive random access memory (RRAM / ReRAM), phase change memory (PCM), spin transfer torque RAM (STT-RAM), another memory chip or box, and / or a combination thereof.

[0290] Storage device 4330 may include software services, servers, services, etc., which, when the code defining such software is executed by processor 4310, causes the system to perform a function. In some aspects, a hardware service that performs a particular function may include a software component stored in a computer-readable medium in combination with the necessary hardware components (e.g., processor 4310, connection 4305, output device 4335, etc.) to perform that function. The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. Computer-readable media may include non-transitory media in which data may be stored and which does not include carrier waves and / or transient electronic signals propagated wirelessly or via a wired connection.

[0291] The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing or carrying instructions and / or data. Computer-readable media may include non-transitory media in which data may be stored and does not include carrier waves and / or transient electronic signals that are propagated wirelessly or via a wired connection. Examples of non-transitory media may include, but are not limited to, disks or tapes, optical storage media (such as compact disks (CDs) or digital versatile disks (DVDs)), flash memory, memory, or memory devices. Computer-readable media may have code and / or machine-executable instructions stored thereon, which may represent any combination of programs, functions, subroutines, programs, routines, subroutines, modules, software packages, categories, or instructions, data structures, or program statements. A code segment may be coupled to another code segment or hardware circuit by passing and / or receiving information, data, independent variables, parameters, or memory contents. Information, independent variables, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0292] In some aspects, computer readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer readable storage media explicitly excludes media such as energy, carrier signals, electromagnetic waves, and signals themselves.

[0293] Fig.44 4400 is a flowchart illustrating a process 4400 for generating virtual content in a distributed system according to aspects of the present disclosure. The process 4400 may be performed by a computing device (or apparatus) or a component (eg, a chipset, a codec, etc.) of a computing device (eg, a processor ... Figure 5 CPU 510 and / or GPU 525 and / or Fig.43 The computing device may be an animation and scene rendering system (e.g., an edge or cloud-based server, a personal computer acting as a server device, a mobile device such as a mobile phone acting as a server device, an XR device acting as a server device, a network router, or other device acting as a server or other device). The operations of process 4400 may be implemented as a single processor (e.g., Figure 5 CPU 510 and / or GPU 525, and / or Fig.43 A software component that is executed and runs on the processor 4312).

[0294] At block 4402, the computing device (or a component thereof) may receive a request from a second device (eg, a mobile device such as a Figure 1 Equipment 105, Figure 3 Client devices 302, 304, Figure 4 Client device 405, device 500, Figure 6 Client devices 602, 604 and / or 606, Figure 7 Client devices 702, 704, Fig.43 The computing system 4300 of the second device or the like receives input information associated with the second device or at least one of the users of the second device. In some cases, the input information includes at least one of the following: information representing the face of the user of the second device (e.g., via a face engine), information representing the body of the user of the second device (e.g., via a body engine), information representing one or more hands of the user of the second device (e.g., via a hand engine), posture information of the second device (e.g., via a posture engine), or audio associated with the environment in which the second device is located (e.g., via an audio decoder). In some cases, the information representing the body of the user of the second device includes a posture of the body, and wherein the information representing the one or more hands of the user of the second device includes a corresponding posture of each of the one or more hands.

[0295] At block 4404, the computing device (or a component thereof) may generate a virtual representation of a user of the second device based on the input information (eg, via Figure 7 In some cases, the computing device (or its components) may generate a virtual representation of the face of the user of the second device using information representing the face of the user, the posture information of the second device, and the posture information of the third device; generate a virtual representation of the body of the user of the second device using the posture information of the second device and the posture information of the third device; and generate a virtual representation of the hair of the user of the second device. In some cases, the virtual representation of the body of the user of the second device is further generated using information representing the body of the user. In some cases, the virtual representation of the body of the user of the second device is further generated using inverse kinematics. In some cases, the virtual representation of the body of the user of the second device is further generated using information representing one or more hands of the user. In some cases, the computing device (or its components) may combine the virtual representation of the face with the virtual representation of the body to generate a combined virtual representation; and add the virtual representation of the hair to the combined virtual representation. In some cases, the computing device (or its components) may generate a virtual representation of the user of the third device based on input information from the third device; generate a virtual scene including the virtual representation of the user of the third device from the perspective of the user of the second device; and send one or more frames depicting the virtual scene from the perspective of the user of the second device to the second device.

[0296] At block 4406, the computing device (or a component thereof) may generate a virtual scene from the perspective of a user of a third device associated with the virtual session (eg, via Figure 7 The scene synthesis system 722 as Fig.39 In some cases, the computing device (or a component thereof) may obtain a background representation of the virtual scene; adjust lighting of the virtual representation of the user of the second device based on the background representation of the virtual scene to generate a modified virtual representation of the user; and combine the background representation of the virtual scene with the modified virtual representation of the user.

[0297] At block 4408 , the computing device (or a component thereof) may output one or more frames depicting the virtual scene from the perspective of a user of the third device for transmission to the third device.

[0298] Fig.45 4500 is a flowchart illustrating a process 4500 for generating virtual content in a distributed system according to aspects of the present disclosure. The process 4500 may be performed by a computing device (or apparatus) or a component of a computing device (eg, a chipset, a codec, etc.), such as Figure 5 CPU 510 and / or GPU 525 and / or Fig.43 The computing device may be a mobile device (e.g., a mobile phone, Figure 1 Equipment 105, Figure 3 Client devices 302, 304, Figure 4 Client device 405, device 500, Figure 6 Client devices 602, 604 and / or 606, Figure 7 Client devices 702, 704, Fig.43 The operations of process 4500 may be implemented as a computer system 4300 on one or more processors (e.g., Figure 5 CPU 510 and / or GPU 525, and / or Fig.43 A software component that is executed and runs on the processor 4312).

[0299] At block 4502, a computing device (or a component thereof) may send a message to a second device (eg, Fig.40 UE 4010, Figure 1 Equipment 105, Figure 3 Client devices 302, 304, Figure 4 Client device 405, device 500, Figure 6 Client devices 602, 604 and / or 606, Figure 7 Client devices 702, 704, Fig.43In some cases, the computing device (or a component thereof) may send a virtual representation of a user of the first device to the third device.

[0300] At block 4504 , the computing device (or a component thereof) may receive a call acceptance from the second device indicating acceptance of the call establishment request (eg, as part of the call establishment process 2412 ).

[0301] In box 4506, the computing device (or its components) may transmit input information associated with the first device or at least one of the users of the first device to a third device (e.g., an animation and scene rendering system such as an edge or cloud-based server, a personal computer acting as a server device, a mobile device such as a mobile phone acting as a server device, an XR device acting as a server device, a network router, or other device acting as a server or other device), the input information is used to generate a virtual representation of the user of the second device and a virtual scene from the perspective of the user of the second device. In some cases, the computing device (or its components) may send two or more images of the user of the first device to the third device for generating the virtual representation of the user of the first device. In some cases, the computing device (or its components) may send the virtual representation of the user of the first device to the third device. In some cases, the computing device (or its components) may obtain the virtual representation of the user of the first device through the first device. In some cases, obtaining the virtual representation of the user of the first device includes generating the virtual representation of the user of the first device by the first device.

[0302] At block 4508, the computing device (or a component thereof) may receive information of a virtual scene from a third device. In some cases, the computing device (or a component thereof) may receive a virtual representation of a user of the second device from the third device; and render the virtual scene from the perspective of a user of the first device, the virtual scene including the virtual representation of the user of the second device. In some cases, the information of the virtual scene includes one or more frames depicting the virtual scene from the perspective of the user of the first device, the virtual scene including the virtual representation of the user of the second device.

[0303] Specific details are provided in the above description to provide a thorough understanding of the aspects and examples provided herein. However, it will be appreciated by those skilled in the art that these aspects can be practiced without these specific details. For clarity, in some cases, the present technology may be presented as including separate functional blocks, which include devices, device components, steps or routines in the method embodied in software or a combination of hardware and software. In addition to those components shown in the accompanying drawings and / or described in this article, additional components may be used. For example, circuits, systems, networks, processes and other components may be shown as components in block diagram form, so as not to obscure these aspects with unnecessary details. In other examples, known circuits, processes, algorithms, structures and techniques may be shown without unnecessary details, so as to avoid obscuring these aspects.

[0304] Various aspects may be described above as a process or method depicted as a flowchart, flow diagram, data flow diagram, structure diagram, or block diagram. Although a flowchart may describe an operation as a sequential process, many operations may be performed in parallel or simultaneously. In addition, the order of the operations may be rearranged. A process terminates when its operations are completed, but may have additional steps not included in the figure. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, its termination may correspond to the function returning to a calling function or a main function.

[0305] The processes and methods according to the above examples can be implemented using computer executable instructions stored in or otherwise available from computer readable media. Such instructions may include, for example, instructions and data that enable or otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a specific function or group of functions. Parts of the computer resources used may be accessed over a network. Computer executable instructions may be, for example, binary files, intermediate format instructions such as assembly language, firmware, source code. Examples of computer readable media that can be used to store instructions, information used, and / or information created during the methods according to the described examples include disks or optical disks, flash memory, USB devices provided with non-volatile memory, networked storage devices, etc.

[0306] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing the necessary tasks may be stored in a computer-readable or machine-readable medium. (One or more) processors may perform the necessary tasks. Typical examples of form factors include laptop computers, smart phones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functions described herein may also be embodied in peripheral devices or add-on cards. As another example, such functions may also be implemented between different chips or different processes executed in a single device on a circuit board.

[0307] The instructions, the media for carrying such instructions, the computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functions described in this disclosure.

[0308] In the foregoing description, various aspects of the present application are described with reference to their specific aspects, but those skilled in the art will recognize that the present application is not limited thereto. Therefore, although the illustrative aspects of the present application have been described in detail herein, it should be understood that the inventive concept can be implemented and adopted differently in other ways, and the appended claims are intended to be interpreted as including such variations, except for being limited by the prior art. The various features and aspects of the above-mentioned application can be used individually or in combination. In addition, without departing from the broader spirit and scope of this specification, various aspects can be utilized in any number of environments and applications outside the environment and application described herein. Therefore, the description and the accompanying drawings are considered to be illustrative rather than restrictive. For the purpose of illustration, the method is described in a specific order. It should be understood that, in alternative aspects, the method can be performed in an order different from the described order.

[0309] One of ordinary skill will understand that the less than ("<") and greater than (">") symbols or terms used herein may be replaced by less than or equal to ("≤") and greater than or equal to ("≥") symbols, respectively, without departing from the scope of the present specification.

[0310] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform those operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform those operations, or any combination thereof.

[0311] The phrase “coupled to” means that any component is physically connected directly or indirectly to another component, and / or any component is in direct or indirect communication with another component (eg, connected to another component via a wired or wireless connection, and / or other suitable communication interface).

[0312] Claim language or other language in the present disclosure that recites "at least one of" a group and / or "one or more" a group indicates that one member of the group or multiple members of the group (in any combination) satisfies the claim. For example, claim language that recites "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language that recites "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one" in a set and / or "one or more" in a set does not limit the set to the items listed in the set. For example, claim language that recites "at least one of A and B" or "at least one of A or B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0313] Claim language or other language reciting "at least one processor is configured to," "at least one processor is configured to," "the processor is configured to," etc. indicates that one processor or multiple processors (in any combination) can perform the associated operations. For example, claim language reciting "at least one processor is configured to: X, Y, and Z" means that a single processor can be used to perform operations X, Y, and Z; or that each of multiple processors is tasked with some subset of operations X, Y, and Z, so that multiple processors together perform X, Y, and Z; or that a group of multiple processors work together to perform operations X, Y, and Z. In another example, claim language reciting "at least one processor is configured to: X, Y, and Z" can mean that any single processor can perform only at least a subset of operations X, Y, and Z.

[0314] The various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the examples disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been generally described above with respect to their functionality. Whether such functions are implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. The technician may implement the described functions in different ways for each specific application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0315] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as a general-purpose computer, a wireless communication device handset, or an integrated circuit device with multiple uses (including applications in wireless communication device handsets and other devices). Any features described as modules or components may be implemented together in an integrated logic device, or separately as discrete but interoperable logic devices. If implemented in software, the technique may be implemented at least in part by a computer-readable data storage medium including a program code, the program code including instructions for executing one or more of the methods, algorithms, and / or operations described above when executed. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may include a memory or data storage medium, such as a random access memory (RAM) (e.g., synchronous dynamic random access memory (SDRAM)), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic or optical data storage medium, and the like. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.

[0316] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. This processor may be configured to perform any of the techniques described in the present invention. A general-purpose processor may be a microprocessor; but in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, a combination of one or more microprocessors and a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or implementation suitable for the device in the technology described herein.

[0317] Illustrative aspects of the disclosure include:

[0318] Aspect 1. A method for generating virtual content at a first device in a distributed system, the method comprising: receiving input information associated with at least one of the second device or a user of the second device from a second device associated with a virtual session; generating a virtual representation of the user of the second device based on the input information; generating a virtual scene from a perspective of a user of a third device associated with the virtual session, wherein the virtual scene includes the virtual representation of the user of the second device; and sending one or more frames depicting the virtual scene from the perspective of the user of the third device to the third device.

[0319] Aspect 2. The method according to aspect 1, wherein the input information includes at least one of information representing a face of a user of the second device, information representing a body of the user of the second device, information representing one or more hands of the user of the second device, gesture information of the second device, or audio associated with an environment in which the second device is located.

[0320] Aspect 3. The method according to aspect 2, wherein the information representing the body of the user of the second device comprises a posture of the body, and wherein the information representing the one or more hands of the user of the second device comprises a corresponding posture of each of the one or more hands.

[0321] Aspect 4. The method according to any one of Aspects 2 or 3, wherein generating the virtual representation comprises: generating a virtual representation of the face of the user of the second device using information representing the face of the user, the posture information of the second device, and the posture information of the third device; generating a virtual representation of the body of the user of the second device using the posture information of the second device and the posture information of the third device; and generating a virtual representation of the hair of the user of the second device.

[0322] Aspect 5. The method according to aspect 4, wherein the information representing the user's body is further used to generate a virtual representation of the body of the user of the second device.

[0323] Aspect 6. The method according to any one of aspects 4 or 5, wherein the virtual representation of the body of the user of the second device is further generated using inverse kinematics.

[0324] Aspect 7. The method according to any one of aspects 4 to 6, wherein information representing one or more hands of the user is further used to generate a virtual representation of the body of the user of the second device.

[0325] Aspect 8. The method according to any one of aspects 4 to 7, further comprising: combining the virtual representation of the face with the virtual representation of the body to generate a combined virtual representation; and adding the virtual representation of the hair to the combined virtual representation.

[0326] Aspect 9. The method of any one of aspects 1 to 8, wherein generating the virtual scene comprises: obtaining a background representation of the virtual scene; adjusting lighting of the virtual representation of the user of the second device based on the background representation of the virtual scene to generate a modified virtual representation of the user; and combining the background representation of the virtual scene with the modified virtual representation of the user.

[0327] Aspect 10. The method according to any one of aspects 1 to 9 further comprises: generating a virtual representation of a user of the third device based on input information from the third device; generating a virtual scene including the virtual representation of the user of the third device from the perspective of the user of the second device; and sending one or more frames depicting the virtual scene from the perspective of the user of the second device to the second device.

[0328] Aspect 11. An apparatus associated with a first device for generating virtual content in a distributed system, comprising: at least one memory; and at least one processor, coupled to the at least one memory and configured to: receive input information associated with at least one of the second device or a user of the second device from a second device associated with a virtual session; generate a virtual representation of the user of the second device based on the input information; generate a virtual scene from the perspective of a user of a third device associated with the virtual session, wherein the virtual scene includes the virtual representation of the user of the second device; and output one or more frames depicting the virtual scene from the perspective of the user of the third device for transmission to the third device.

[0329] Aspect 12. The apparatus according to aspect 11, wherein the input information includes at least one of information representing a face of a user of the second device, information representing a body of the user of the second device, information representing one or more hands of the user of the second device, gesture information of the second device, or audio associated with an environment in which the second device is located.

[0330] Aspect 13. The apparatus of aspect 12, wherein the information representing the body of the user of the second device comprises a posture of the body, and wherein the information representing the one or more hands of the user of the second device comprises a corresponding posture of each of the one or more hands.

[0331] Aspect 14. The apparatus according to any one of Aspects 12 or 13, wherein the at least one processor is configured to: generate a virtual representation of the face of the user of the second device using information representing the face of the user, the posture information of the second device, and the posture information of the third device; generate a virtual representation of the body of the user of the second device using the posture information of the second device and the posture information of the third device; and generate a virtual representation of the hair of the user of the second device.

[0332] Aspect 15. The apparatus of aspect 14, wherein the information representing the user's body is further used to generate a virtual representation of the body of the user of the second device.

[0333] Aspect 16. The apparatus according to any one of aspects 14 or 15, wherein the virtual representation of the body of the user of the second device is further generated using inverse kinematics.

[0334] Aspect 17. The apparatus according to any one of aspects 14 to 16, wherein information representing one or more hands of the user is further used to generate a virtual representation of the body of the user of the second device.

[0335] Aspect 18. The apparatus according to any one of aspects 14 to 17, wherein the at least one processor is configured to: combine the virtual representation of the face with the virtual representation of the body to generate a combined virtual representation; and add the virtual representation of the hair to the combined virtual representation.

[0336] Aspect 19. The apparatus of any of Aspects 11 to 18, wherein the at least one processor is configured to: obtain a background representation of the virtual scene; adjust lighting of the virtual representation of the user of the second device based on the background representation of the virtual scene to generate a modified virtual representation of the user; and combine the background representation of the virtual scene with the modified virtual representation of the user.

[0337] Aspect 20. The apparatus of any one of Aspects 11 to 19, wherein the at least one processor is configured to: generate a virtual representation of a user of the third device based on input information from the third device; generate a virtual scene including the virtual representation of the user of the third device from the perspective of the user of the second device; and output one or more frames depicting the virtual scene from the perspective of the user of the second device for transmission to the second device.

[0338] Aspect 21. The apparatus according to any one of aspects 11 to 20, wherein the apparatus is implemented as a first device and further comprises at least one transceiver configured to: receive input information from a second device; and send one or more frames to a third device.

[0339] Aspect 22. A non-transitory computer readable medium having stored thereon instructions which, when executed by one or more processors, cause the one or more processors to perform operations according to any one of Aspects 1 to 21.

[0340] Aspect 23. An apparatus for generating virtual content in a distributed system, the apparatus comprising one or more components for performing the operations according to any one of Aspects 1 to 21.

[0341] Aspect 31. A method for establishing one or more virtual sessions between users, the method comprising: sending, by a first device to a second device, a call establishment request for a virtual representation call of a virtual session; receiving, at the first device, a call acceptance from the second device indicating acceptance of the call establishment request; sending, by the first device to the second device, first grid information for a first virtual representation of a first user of the first device; sending, by the first device to the second device, first grid animation parameters for the first virtual representation of the first user of the first device; receiving, at the first device, from the second device, second grid information for a second virtual representation of a second user of the second device; receiving, at the first device, from the second device, second grid animation parameters for the second virtual representation of the second user of the second device; and generating, at the first device, a second virtual representation of the second user of the second device based on the second grid information and the second grid animation parameters.

[0342] Aspect 32. The method of aspect 31, wherein the first grid information is sent once for the virtual representation call for the virtual session, and wherein the second grid information is received once for the virtual representation call for the virtual session.

[0343] Aspect 33. The method of any of Aspects 31 or 32, wherein the first mesh animation parameters are sent and the second mesh animation parameters are received for each frame of the virtual session at a frame rate associated with the virtual session.

[0344] Aspect 34. The method according to any one of Aspects 31 to 33, wherein the second mesh animation parameter comprises at least one of information representing a face of a second user of the second device, information representing a body of the second user of the second device, information representing one or more hands of the second user of the second device, gesture information of the second device, or audio associated with an environment in which the second device is located.

[0345] Aspect 35. The method of aspect 34, wherein the information representing the body of the second user of the second device comprises a posture of the body, and wherein the information representing the one or more hands of the second user of the second device comprises a corresponding posture of each of the one or more hands.

[0346] Aspect 36. The method according to any one of Aspects 34 or 35, wherein generating the second virtual representation comprises: generating a virtual representation of the face of the second user of the second device using information representing the face of the second user of the second device and the posture information of the second device; generating a virtual representation of the body of the second user of the second device using the posture information of the second device; and generating a virtual representation of the hair of the second user of the second device.

[0347] Aspect 37. The method of aspect 36, wherein the information representing the body of the second user is further used to generate a virtual representation of the body of the second user of the second device.

[0348] Aspect 38. The method according to any one of aspects 36 or 37, wherein the virtual representation of the body of the second user of the second device is further generated using inverse kinematics.

[0349] Aspect 39. The method according to any one of aspects 36 to 38, wherein the information representing the one or more hands of the second user is further used to generate a virtual representation of the body of the second user of the second device.

[0350] Aspect 40. The method according to any one of aspects 36 to 39, further comprising: combining the virtual representation of the face with the virtual representation of the body to generate a combined virtual representation; and adding the virtual representation of the hair to the combined virtual representation.

[0351] Aspect 41. A method for establishing one or more virtual sessions between users, the method comprising: receiving, by a server device, a call establishment request for a virtual representation call of a virtual session from a first client device; sending, by the server device, the call establishment request to a second client device; receiving, by the server device, a call acceptance indicating acceptance of the call establishment request from the second client device; sending, by the server device, the call acceptance to the first client device; receiving, from the first client device, first grid information for a first virtual representation of a first user of the first client device based on the call acceptance; sending, by the server device, first grid animation parameters for the first virtual representation of the first user of the first client device; generating, by the server device, a first virtual representation of the first user of the first client device based on the first grid information and the first grid animation parameters; and sending, by the server device, the first virtual representation of the first user of the first client device to the second client device.

[0352] Aspect 42. The method according to Aspect 41, wherein the first grid information is received once for the virtual representation call for the virtual session.

[0353] Aspect 43. The method of any of Aspects 41 or 42, wherein the first mesh animation parameters are received for each frame of the virtual session at a frame rate associated with the virtual session.

[0354] Aspect 44. The method according to any one of Aspects 41 to 43, wherein the first mesh animation parameter comprises at least one of: information representing a face of a first user of the first client device, information representing a body of the first user of the first client device, information representing one or more hands of the first user of the first client device, gesture information of the first client device, or audio associated with an environment in which the first client device is located.

[0355] Aspect 45. The method of aspect 44, wherein the information representing the body of the first user of the first client device comprises a posture of the body, and wherein the information representing the one or more hands of the first user of the first client device comprises a corresponding posture of each of the one or more hands.

[0356] Aspect 46. An apparatus comprising at least one memory and at least one processor, the at least one processor being coupled to the at least one memory and configured to perform the operations according to any one of Aspects 31 to 40.

[0357] Aspect 47. An apparatus comprising at least one memory and at least one processor, the at least one processor being coupled to the at least one memory and configured to perform the operations according to any one of Aspects 41 to 45.

[0358] Aspect 48. A non-transitory computer readable medium having stored thereon instructions which, when executed by one or more processors, cause the one or more processors to perform operations according to any one of Aspects 31 to 40.

[0359] Aspect 49. A non-transitory computer readable medium having stored thereon instructions which, when executed by one or more processors, cause the one or more processors to perform operations according to any one of Aspects 31 to 55.

[0360] Aspect 50. An apparatus for generating virtual content in a distributed system, the apparatus comprising one or more components for performing the operations according to any one of Aspects 31 to 40.

[0361] Aspect 51. An apparatus for generating virtual content in a distributed system, the apparatus comprising one or more components for performing the operations according to any one of Aspects 41 to 45.

[0362] Aspect 61. A method for establishing one or more virtual sessions between users, the method comprising: sending, by a first device to a second device, a call establishment request for a virtual representation call for a virtual session; receiving, at the first device, a call acceptance from the second device indicating acceptance of the call establishment request; sending, by the first device to a third device, input information associated with at least one of the first device or a user of the first device, the input information being used to generate a virtual representation of a user of the second device and a virtual scene from the perspective of the user of the second device; and receiving information of the virtual scene from the third device.

[0363] Aspect 62. The method of Aspect 61, further comprising sending, by the first device, a virtual representation of a user of the first device to a third device.

[0364] Aspect 63. The method according to Aspect 62 further comprises: receiving a virtual representation of a user of the second device from a third device; and rendering a virtual scene from the perspective of a user of the first device, the virtual scene including the virtual representation of the user of the second device.

[0365] Aspect 64. The method according to any one of Aspects 61 or 62, wherein the information of the virtual scene comprises one or more frames depicting the virtual scene from the perspective of a user of the first device, the virtual scene comprising a virtual representation of a user of the second device.

[0366] Aspect 65. The method of Aspect 64, further comprising sending, by the first device to a third device, two or more images of a user of the first device for generating a virtual representation of the user of the first device.

[0367] Aspect 66. The method according to any of Aspects 64 or 65, further comprising sending the virtual representation of the user of the first device to a third device.

[0368] Aspect 67. The method of Aspect 66, further comprising obtaining, by the first device, a virtual representation of a user of the first device.

[0369] Aspect 68. The method of Aspect 67, wherein obtaining the virtual representation of the user of the first device comprises generating, by the first device, the virtual representation of the user of the first device.

[0370] Aspect 69. The method according to any one of Aspects 61 to 68, wherein the third device comprises a server hosting the media resource function.

[0371] Aspect 70. An apparatus comprising at least one memory and at least one processor, the at least one processor being coupled to the at least one memory and configured to perform the operations according to any one of Aspects 61 to 69.

[0372] Aspect 71. A non-transitory computer readable medium having stored thereon instructions which, when executed by one or more processors, cause the one or more processors to perform operations according to any one of aspects 61 to 69.

[0373] Aspect 72. An apparatus for generating virtual content in a distributed system, the apparatus comprising one or more components for performing the operations according to any one of Aspects 61 to 69.

Claims

1. A method for generating virtual content at a first device in a distributed system, the method comprising: receiving, from a second device associated with the virtual session, input information associated with at least one of the second device or a user of the second device; generating a virtual representation of the user of the second device based on the input information; generating a virtual scene from the perspective of a user of a third device associated with the virtual session, wherein the virtual scene includes the virtual representation of the user of the second device; as well as One or more frames depicting the virtual scene from the perspective of the user of the third device are sent to the third device.

2. The method according to claim 1, wherein: The input information includes at least one of information representing a face of the user of the second device, information representing a body of the user of the second device, information representing one or more hands of the user of the second device, posture information of the second device, or audio associated with an environment in which the second device is located.

3. The method according to claim 2, wherein: The information representing the body of the user of the second device includes a posture of the body, and wherein the information representing the one or more hands of the user of the second device includes a corresponding posture of each of the one or more hands.

4. The method according to claim 2, wherein: Generating the virtual representation comprises: generating a virtual representation of the face of the user of the second device using the information representing the face of the user, the posture information of the second device, and the posture information of the third device; generating a virtual representation of the body of the user of the second device using the gesture information of the second device and the gesture information of the third device; and A virtual representation of the hair of the user of the second device is generated.

5. The method according to claim 4, wherein: The virtual representation of the body of the user of the second device is also generated using the information representing the body of the user.

6. The method according to claim 4, wherein: The virtual representation of the body of the user of the second device is also generated using inverse kinematics.

7. The method according to claim 4, wherein: The virtual representation of the body of the user of the second device is also generated using the information representing the one or more hands of the user.

8. The method according to claim 4, further comprising: combining the virtual representation of the face with the virtual representation of the body to generate a combined virtual representation; as well as The virtual representation of the hair is added to the combined virtual representation.

9. The method according to claim 1, wherein: Generating the virtual scene includes: obtaining a background representation of the virtual scene; adjusting lighting of the virtual representation of the user of the second device based on the background representation of the virtual scene to generate a modified virtual representation of the user; and The background representation of the virtual scene is combined with the modified virtual representation of the user.

10. The method according to claim 1, further comprising: generating a virtual representation of the user of the third device based on the input information from the third device; generating a virtual scene including the virtual representation of the user of the third device from the perspective of the user of the second device; as well as One or more frames depicting the virtual scene from the perspective of the user of the second device are sent to the second device.

11. An apparatus associated with a first device for generating virtual content in a distributed system, comprising: at least one memory; as well as at least one processor coupled to at least one memory and configured to: receiving, from a second device associated with the virtual session, input information associated with at least one of the second device or a user of the second device; generating a virtual representation of the user of the second device based on the input information; generating a virtual scene from the perspective of a user of a third device associated with the virtual session, wherein the virtual scene includes the virtual representation of the user of the second device; as well as One or more frames depicting the virtual scene from the perspective of the user of the third device are output for transmission to the third device.

12. The device according to claim 11, wherein: The input information includes at least one of information representing a face of the user of the second device, information representing a body of the user of the second device, information representing one or more hands of the user of the second device, posture information of the second device, or audio associated with an environment in which the second device is located.

13. The device according to claim 12, wherein: The information representing the body of the user of the second device includes a posture of the body, and wherein the information representing the one or more hands of the user of the second device includes a corresponding posture of each of the one or more hands.

14. The device according to claim 12, wherein: The at least one processor is configured to: generating a virtual representation of the face of the user of the second device using the information representing the face of the user, the posture information of the second device, and the posture information of the third device; generating a virtual representation of a body of the user of the second device using the posture information of the second device and the posture information of the third device; as well as A virtual representation of the hair of the user of the second device is generated.

15. The device according to claim 14, wherein: The virtual representation of the body of the user of the second device is also generated using the information representing the body of the user.

16. The device according to claim 14, wherein: The virtual representation of the body of the user of the second device is also generated using inverse kinematics.

17. The device according to claim 14, wherein: The virtual representation of the body of the user of the second device is also generated using the information representing the one or more hands of the user.

18. The device according to claim 14, wherein: The at least one processor is configured to: combining the virtual representation of the face with the virtual representation of the body to generate a combined virtual representation; and The virtual representation of the hair is added to the combined virtual representation.

19. The device according to claim 11, wherein: The at least one processor is configured to: obtaining a background representation of the virtual scene; adjusting lighting of the virtual representation of the user of the second device based on the background representation of the virtual scene to generate a modified virtual representation of the user; as well as The background representation of the virtual scene is combined with the modified virtual representation of the user.

20. The device according to claim 11, wherein The at least one processor is configured to: generating a virtual representation of the user of the third device based on the input information from the third device; generating a virtual scene including the virtual representation of the user of the third device from the perspective of the user of the second device; as well as One or more frames depicting the virtual scene from the perspective of the user of the second device are output for transmission to the second device.

21. A method for establishing one or more virtual sessions between users, the method comprising: The first device sends a call establishment request for calling a virtual representation of a virtual session to the second device; receiving, at the first device, from the second device, a call acceptance indicating acceptance of the call establishment request; Sending, by the first device, input information associated with the first device or at least one of a user of the first device to a third device, the input information being used to generate a virtual representation of a user of the second device and a virtual scene from the perspective of the user of the second device; as well as The information of the virtual scene is received from the third device.

22. The method of claim 21, further comprising sending, by the first device, the virtual representation of the user of the first device to the third device.

23. The method according to claim 22, further comprising: receiving, from the third device, a virtual representation of the user of the second device; as well as The virtual scene is rendered from the perspective of the user of the first device, the virtual scene including the virtual representation of the user of the second device.

24. The method according to claim 21, wherein: The information of the virtual scene includes one or more frames depicting the virtual scene from the perspective of the user of the first device, the virtual scene including a virtual representation of the user of the second device.

25. The method of claim 24, further comprising sending, by the first device to the third device, two or more images of the user of the first device for generating a virtual representation of the user of the first device.

26. The method of claim 24, further comprising sending a virtual representation of the user of the first device to the third device.

27. The method of claim 26, further comprising obtaining, by the first device, a virtual representation of the user of the first device.

28. The method according to claim 27, wherein: Obtaining the virtual representation of the user of the first device includes generating, by the first device, the virtual representation of the user of the first device.

29. The method according to claim 21, wherein: The third device includes a server hosting a media resource function.

Images