Signaling Scene Descriptions for Multimedia Conferencing.

Abstract

Various embodiments include systems and methods for providing an immersive three-dimensional group session. Various embodiments include methods and devices for signaling a description of a scene with media components potentially coming from different parties. In various aspects, the scene graph may be signaled through the Session Description Protocol (SDP) during a Session Initiation Protocol (SIP) session setup. In various embodiments, the scene graph may include respective graphical output nodes assigned to be controlled by each of multiple participant computing devices in the immersive three-dimensional group session.

Description

Claiming priority

[0001] Related Applications

[0001] This application claims the benefit of priority to U.S. Provisional Application No. 62 / 990,895, entitled "Signaling of Scene Description For Multimedia Conferencing," filed March 17, 2020, the entire contents of which are incorporated herein by reference for all purposes. [Technical field]

[0002] The present invention relates to a system and method for providing an immersive three-dimensional group session. [Background technology]

[0003]

[0002] Long Term Evolution (LTE), Fifth Generation (5G) New Radio (NR), and other recently developed communication technologies enable wireless devices to communicate information at data rates that are orders of magnitude greater (e.g., in terms of gigabits per second) than were available just a few years ago.

[0004] Today's communication networks are also more secure, resistant to multipath fading, allow lower network traffic latency, and provide better communication efficiency (e.g., in terms of bits per second per unit of bandwidth used, etc.). These and other recent improvements have facilitated the emergence of the Internet of Things (IOT), large-scale machine-to-machine (M2M) communication systems, autonomous vehicles, and other technologies that rely on consistent and secure communications.

[0005]

[0004] In recent years, augmented reality software applications that combine real-world images from a user's physical environment with computer-generated imagery or virtual objects (VOs) have grown in popularity and use. Augmented reality software applications may add graphics, sounds, and / or haptic feedback to the natural world around the user of the application. Images, video streams, and information about people and / or objects may be presented to the user superimposed on the visual world as an augmented scene on a wearable electronic display or head-mounted device (e.g., smart glasses, augmented reality glasses, etc.). Summary of the Invention

[0006]

[0005] Various aspects include systems and methods for providing an immersive three-dimensional group session. Various embodiments include methods and devices for signaling a description of a scene with media components potentially coming from different parties. In various aspects, the scene graph may be signaled through a Session Description Protocol (SDP) during a Session Initiation Protocol (SIP) session setup. In various aspects, the scene graph may include respective graphical output nodes assigned to be controlled by each of a plurality of participant computing devices in the immersive three-dimensional group session. Various aspects may be implemented by a processor of a wireless device that is one of a plurality of participant computing devices operating in the immersive three-dimensional group session. Various aspects may include receiving a scene graph for the immersive three-dimensional group session, where the scene graph includes at least an own graphical output node assigned to be controlled by the wireless device and respective other graphical output nodes assigned to be controlled by each of the others of the plurality of participant computing devices; controlling components of the own graphical output node relative to a three-dimensional space of the immersive three-dimensional group session; sending components of the own graphical output node in a first media stream to the others of the plurality of participant computing devices; receiving components of the other graphical output node in a media stream from each of the others of the plurality of participant computing devices; and rendering the immersive three-dimensional group session on a display of the wireless device based at least in part on the components of the own graphical output node and the components of the other graphical output node.

[0007]

[0006] Some aspects may further include receiving a scene graph update including an indication of a new participant computing device for the immersive three-dimensional group session and an indication of a new graphical output node assigned to be controlled by the new participant computing device, receiving components of the new graphical output node in the second media stream from the new participant computing device, and rendering the immersive three-dimensional group session on a display of the wireless device based at least in part on the components of the own graphical output node, the components of the other graphical output node, and the components of the new graphical output node.

[0008]

[0007] Some aspects may further include receiving a session description protocol (SDP) for the immersive three-dimensional group session indicating an address of a data channel over which the scene graph is to be shared, where receiving the scene graph includes downloading the scene graph over the data channel.

[0009]

[0008] Some aspects may further include sending an offer to send or receive a scene graph to another of the multiple participant computing devices as part of a Session Initiation Protocol (SIP) setup for the immersive three-dimensional group session.

[0010]

[0009] Some aspects may further include sending an offer to send or receive a scene graph to another of the multiple participant computing devices, where the offer indicates a unique graphical output node.

[0011] In some aspects, the immersive three-dimensional group session is a Web Real-Time Communications (WebRTC) session.

[0012]

[0011] In some aspects, controlling components of the unique graphical output node relative to the three-dimensional space of the immersive three-dimensional group session may include controlling components of the unique graphical output node based at least in part on a determined position of the wireless device relative to the three-dimensional space of the immersive three-dimensional group session.

[0013]

[0012] In some aspects, controlling components of the unique graphical output node based at least in part on a determined position of the wireless device relative to the three dimensional space of the immersive three dimensional group session may include controlling components of the unique graphical output node based at least in part on a determined position of the wireless device relative to the three dimensional space of the immersive three dimensional group session and a determined orientation of the wireless device relative to the three dimensional space of the immersive three dimensional group session.

[0014]

[0013] A further aspect may include a wireless device having a processor configured to perform one or more operations of any of the methods summarized above. A further aspect may include a non-transitory processor-readable storage medium having stored thereon processor-executable instructions configured to cause a processor of the wireless device to perform the operations of any of the methods summarized above. A further aspect includes a wireless device having means for performing the functions of any of the methods summarized above. A further aspect includes a system-on-chip for use in a wireless device including a processor configured to perform one or more operations of any of the methods summarized above. A further aspect includes a system in a package including two system-on-chips for use in a wireless device including a processor configured to perform one or more operations of any of the methods summarized above.

[0015]

[0014] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate exemplary embodiments of the claims and, together with the general description given above and the detailed description given below, serve to explain the features of the claims. [Brief description of the drawings]

[0016] [Figure 1A] FIG. 1 is a system block diagram illustrating an exemplary communication system suitable for implementing various embodiments. [Figure 1B]

[0016] FIG. 1 illustrates a head-mounted device (eg, augmented reality glasses) in which various embodiments may be implemented. [Diagram 2]

[0017] 1 is a component block diagram illustrating an exemplary computing and wireless modem system suitable for implementing various embodiments. [Diagram 3]

[0018] FIG. 2 illustrates an example of a software architecture including radio protocol stacks for user and control planes in wireless communications, in accordance with various embodiments. [Figure 4]

[0019] 1 is a process flow diagram illustrating a method for supporting an immersive experience in a teleconference or telepresence session, according to various embodiments. [Diagram 5]

[0020] 1 is a process flow diagram illustrating a method for supporting an immersive experience in a teleconference or telepresence session, according to various embodiments. [Figure 6]

[0021] A diagram showing the structure of a scene graph document in glTF2.0. [Figure 7]

[0022] A diagram showing the structure of a scene graph. [Figure 8]

[0023] 1 is a call flow diagram illustrating operations for supporting an immersive experience in a teleconference or telepresence session, according to various embodiments. [Figure 9]

[0024] 1 is a process flow diagram illustrating a method for providing an immersive three-dimensional group session, according to various embodiments. [Figure 10]

[0025] 1 is a process flow diagram illustrating a method for providing an immersive three-dimensional group session, according to various embodiments. [Figure 11]

[0026] FIG. 1 is a component block diagram of an exemplary server suitable for implementing various embodiments. [Figure 12]

[0027] 1 is a component block diagram of a wireless device suitable for implementing various embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017]

[0028] Various embodiments will now be described in detail with reference to the accompanying drawings. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts. References made to specific examples and implementations are for illustrative purposes only and are not intended to limit the scope of the claims.

[0018]

[0029] Various embodiments may enable an immersive three-dimensional group session for multiple participant computing devices where a scene graph may include respective graphical output nodes assigned to be controlled by each of the multiple participant computing devices in the immersive three-dimensional group session. Various embodiments may enable multiple participant computing devices to share constituent media streams of their respective assigned graphical output nodes with each other in the immersive three-dimensional group session. By assigning each participant computing device control of its own respective graphical output node in the immersive three-dimensional group session and sharing constituent media streams of the graphical output nodes among the participant computing devices, various embodiments may support rendering an immersive three-dimensional group session with a shared three-dimensional space in which each participant computing device controls its own respective three-dimensional object in the shared three-dimensional space.

[0019]

[0030] The term "wireless device" is used herein to refer to any one or all of wireless router devices, wireless appliances, cellular telephones, smartphones, portable computing devices, personal or mobile multimedia players, laptop computers, tablet computers, smartbooks, ultrabooks, palmtop computers, wireless email receivers, multimedia Internet-enabled cellular telephones, medical devices and equipment, biometric sensors / devices, wearable devices including smart watches, smart clothing, smart glasses, smart wristbands, smart jewelry (e.g., smart rings, smart bracelets, etc.), entertainment devices (e.g., wireless gaming controllers, music and video players, satellite radios, etc.), wireless network-enabled Internet of Things (IoT) devices including smart meters / sensors, industrial manufacturing equipment, large and small machinery and appliances for home or business use, wireless communication elements in autonomous and semi-autonomous vehicles, wireless devices fixed or embedded in various mobile platforms, global positioning system devices, and similar electronic devices that include memory, wireless communication components, and programmable processors.

[0020]

[0031] Various embodiments may incorporate any of the Institute of Electrical and Electronics Engineers (IEEE) 16.11 standards, or IEEE 802.15.4 protocols (e.g., Thread, ZigBee, and Z-Wave), 6LoWPAN, Bluetooth Low Energy (BLE), LTE Machine Type Communications (LTE MTC), Narrowband LTE (NB-LTE), Cellular IoT (CIoT), Narrowband IoT (NB-IoT), BT Smart, Wi-Fi (e.g., Wi-Fi NAN, etc.), LTE-U, LTE-Direct, MuLTEfire, as well as relatively extended range wide area physical layer interfaces (PHYs) such as Random Phase Multiple Access (RPMA), Ultra Narrow Band (UNB), Low Power Long Range (LoRa), Low Power Long Range Wide Area Network (LoRaWAN), Weightless, or systems utilizing 3G, 4G or 5G, Cellular V2X, or further implementations or technologies thereof, including but not limited to cellular V2X, ... The wireless communications technology may be implemented in a device capable of transmitting and receiving RF signals in accordance with any of the wireless communications standards, including any of the 802.11 standards, the Bluetooth standards (e.g., Bluetooth 4, Bluetooth 5, etc.), Code Division Multiple Access (CDMA), Frequency Division Multiple Access (FDMA), Time Division Multiple Access (TDMA), Global System for Mobile Communications (GSM), GSM / General Packet Radio Service (GPRS), Enhanced Data GSM Environment (EDGE), Terrestrial Telecommunications (TETRA), Wideband-CDMA (W-CDMA), Evolution Data Optimized (EV-DO), 1xEV-DO, EV-DO Rev A, EV-DO Rev B, High Speed ​​Packet Access (HSPA), High Speed ​​Downlink Packet Access (HSDPA), High Speed ​​Uplink Packet Access (HSUPA), Evolved High Speed ​​Packet Access (HSPA+), Long Term Evolution (LTE), AMPS, or any other known signals.

[0021]

[0032] The term "system on chip" (SOC) is used herein to refer to a single integrated circuit (IC) chip that contains multiple resources and / or processors integrated on a single substrate. A single SOC may contain circuits for digital, analog, mixed signal, and radio frequency functions. A single SOC may also include any number of general purpose and / or special purpose processors (digital signal processors, modem processors, video processors, etc.), memory blocks (e.g., ROM, RAM, Flash, etc.), and resources (e.g., timers, voltage regulators, oscillators, etc.). A SOC may also include software for controlling the integrated resources and processors, as well as for controlling peripheral devices.

[0022]

[0033] The term "system in package" (SIP) may be used herein to refer to a single module or package containing multiple resources, computing units, cores and / or processors on two or more IC chips, substrates, or SOCs. For example, a SIP may include a single substrate on which multiple IC chips or semiconductor dies are stacked in a vertical configuration. Similarly, a SIP may include one or more multi-chip modules (MCMs) on which multiple ICs or semiconductor dies are packaged in a unified substrate. A SIP may also include multiple independent SOCs packaged in close proximity coupled to each other via high-speed communication circuits, such as on a single motherboard or in a single wireless device. The proximity of the SOCs facilitates high-speed communication and sharing of memory and resources.

[0023]

[0034] Various embodiments are described herein using the term "server" to refer to any computing device capable of functioning as a server, such as a master exchange server, web server, mail server, document server, content server, or any other type of server. A server may be a dedicated computing device or a computing device that includes a server module (e.g., running an application that can cause the computing device to operate as a server). The server module (e.g., server application) may be a full-featured server module or a simplified or secondary server module (e.g., a simplified or secondary server application) configured to provide synchronization services among dynamic databases on the receiver device. A simplified or secondary server may be a scaled-down version of server-type functionality that may be implemented on the receiver device, thereby enabling it to function as an Internet server (e.g., a corporate email server) only to the extent necessary to provide the functionality described herein.

[0024]

[0035] The phrase "head-mounted device" and its acronym (HMD) are used herein to refer to an electronic display system that is wearable and presents at least some computer-generated imagery to a user. An HMD may present only computer-generated imagery or a combination of computer-generated imagery and real-world images from the user's physical environment (i.e., what the user would see without glasses). An HMD may allow a user to observe the generated imagery in the context of a real-world scene. Non-limiting examples of head-mounted devices may include or be included in helmets, glasses, virtual reality glasses, augmented reality glasses, electronic goggles, and other similar technologies / devices. A head-mounted device may include various hardware elements, such as a processor, memory, a display, one or more cameras (e.g., a world-view camera, a gaze-view camera, etc.), and a wireless interface for connecting with the Internet, a network, or another computing device. In some embodiments, the head-mounted device processor may be configured to implement or execute an augmented reality software application.

[0025]

[0036] In some embodiments, the head-mounted device may be an accessory for and / or receive information from a wireless device (e.g., a desktop, laptop, smartphone, tablet computer, etc.), with all or part of the processing being performed on the processor of the wireless device. Thus, in various embodiments, the head-mounted device may be configured to perform all processing locally on the processor in the head-mounted device, to offload all of the primary processing to a processor in another computing device (e.g., a laptop that is in the same space as the head-mounted device, etc.), or to split the primary processing operations between the processor in the head-mounted device and a processor in the other computing device. In some embodiments, the processor in the other computing device may be a server in the "cloud" with which the processor in the head-mounted device or in the associated wireless device communicates via a network connection (e.g., a cellular network connection to the Internet).

[0026]

[0037] Telepresence services are becoming more capable, which allows the composition of several objects into a single immersive environment that conference participants can navigate and interact with more freely. The current Session Description Protocol (SDP) is limited in terms of support for describing rich composition and does not provide tools to support immersive telepresence environments. One particular type of telepresence or teleconferencing implementation is the immersive three-dimensional group session. In an immersive three-dimensional group session, each participant computing device in the session may render a three-dimensional graphical display of the session on its respective display such that each participant user in the immersive three-dimensional group session is presented with a virtual reality (VR) view of the three-dimensional space of the immersive three-dimensional group session. Three-dimensional objects, such as avatars, characters, etc., representing participants in the immersive three-dimensional group session may be observed by each participant and appear to move within the three-dimensional space of the immersive three-dimensional group session.

[0027]

[0038] Immersive Teleconferencing and Telepresence for Remote Terminals (ITT4RT) support is a standard being developed to facilitate immersive multi-computing device virtual reality (VR) video conferencing, such as immersive three-dimensional group sessions. ITT4RT use cases include compositing captured VR video, for example from a conference room, and other content, such as two-dimensional (2D) video slides. Work item descriptions related to ITT4RT indicate that ITT4RT is working towards enabling scenarios with two-way audio and one-way immersive video, such as a remote single user wearing an HMD and participating in a meeting would send audio and, optionally, 2D video (e.g., of the user's own presentation, screen sharing and / or capture).

[0028]

[0039] The complexity of a VR scene presents challenges for traditional Session Description Protocol (SDP) signaling, which can become quite unwieldy very quickly, such as when many remote users share their own content and compose them into the scene. SDP is simply not designed to carry scene description information. Various embodiments provide a solution to such a problem by providing an embodiment solution based on scene description to support compositing and overlay. Various embodiments provide a method and device for signaling a description of a scene with media components potentially coming from different parties. In various embodiments, the scene description may be signaled through SDP during a Session Initiation Protocol (SIP) session setup. In various embodiments, the scene description may be linked to other media streams in the session, for example, to leverage them as textures for overlays in an immersive conference scene, etc.

[0029]

[0040] A scene graph is a directed acyclic graph, usually just a plain tree structure, that represents an object-based hierarchy of the scene's geometry. Leaf nodes of the graph represent geometric primitives such as polygons. Each node in the graph holds pointers to its children. A child node may be, among other things, a group of other nodes, a geometric element, a transformation matrix, etc. Spatial transformations are attached to the nodes of the graph and are represented by transformation matrices. This structure of the scene graph has the advantage of reduced processing complexity, such as while traversing the graph for rendering. An exemplary operation simplified by the graph representation is a culling operation in which a branch of the graph is dropped from processing if the space of a parent node is not visible or is deemed not relevant to the rendering of the current view frustum (called level of detail culling). A scene graph may include various types of nodes, such as visual output nodes, audio source nodes, graphical output nodes, shared content nodes, etc. As a particular example, a graphical output node may define a three-dimensional object to be output in a three-dimensional space defined by the scene graph.

[0030]

[0041] Graphics Library (GL) Transmission Format (TF) (glTF) 2.0 (glTF2.0) is a new standard developed by Khronos to enable physically based rendering. glTF2.0 provides a compact and low-level representation of a scene graph. glTF2.0 provides a flat hierarchy of scene graph representations to simplify processing. glTF2.0 scene graphs are represented in JavaScript® Object Notation (JSON) to simplify integration in web environments. The glTF2.0 specification is designed to eliminate redundancies in the representation and provide efficient indexing of different objects in the scene graph. The Moving Picture Experts Group (MPEG) is working on extending glTF2.0 to add support for real-time media, scene updates, and other features.

[0031]

[0042] The scene graph, in various embodiments, may enable compositing a scene (also referred to as a space, e.g., a three-dimensional space) for an immersive presentation, such as an immersive three-dimensional group session. In some embodiments, the compositing may be performed in a call server, such as a multimedia resource function (MRF), a multipoint communication unit (MCU), a telepresence application server, etc. Alternatively, in some embodiments, a designated computing device participating in the conference may be responsible for creating the initial scene graph and sharing the scene graph with all other parties in the call (e.g., an immersive three-dimensional group session). This computing device may be the one that creates the main VR content, such as a computing device in a conference room with VR capture. In some embodiments, each computing device participating in the conference (e.g., an immersive three-dimensional group session) may contribute one or more nodes to the scene graph. In some embodiments, each node may identify or be assigned its associated transformation (e.g., in the form of a matrix, or individual translation and rotation operations) to properly position that node in the scene (or space), e.g., to properly position the node in three-dimensional space.

[0032]

[0043] In some embodiments, each computing device participating in a conference call (e.g., an immersive three-dimensional group session) may make an offer to send and receive a scene graph through session-level attributes. In some embodiments, the offer may indicate one or more unique graphical output nodes owned by the computing device sending the offer. As an example, each computing device participating in a conference call (e.g., an immersive three-dimensional group session) may make an offer to send and receive a scene graph through the following session-level attributes given in augmented Backus-Naur Form (ABNF) syntax: Session-Description=“a=scene-description:”SP mime-type[SP uri][SP sent-nodes]CRLF mime-type=“mime-type:”byte-string sent-nodes=“nodes-owned=1*(byte-string“;”) uri=“websocket-uri:”URI .

[0044] In some embodiments, the Uniform Resource Indicator (URI) parameter may be a WebSocket URI for a data channel over which the scene graph will be shared and updated. Alternatively, an application media session may be used with a protocol identifier, such as the following protocol identifier TCP / WSS / SD (Transmission Control Protocol / Websocket Secure / Session Description).

[0033]

[0045] In some embodiments, the WebSocket URI may be provided in accordance with the syntax and offer / answer negotiation defined in Internet Engineering Task Force (IETF) Request for Comments (RFC) 8124. An example schema for such a WebSocket URI provisioning may be as follows: m=application 50000 TCP / WSS / SD * a=setup:passive a=connection:new a=websocket-uri:wss: / / mrf.operator.com / call / 21323asd23 a=mime-type:model / gltf+json a=nodes-owned:node12,node13,node14

[0046] In various embodiments, a scene graph references media streams from a conference session that are used as components of nodes in a scene (e.g., three-dimensional space). One example is a video stream of a conference participant that is to be displayed in a rectangular region in a 3D scene (also called three-dimensional (3D) space). For example, the following URI format may be used:

[0034] url=”rtp: / / ”fqdn_or_ip“ / ”call_id“ / ”ssrc“ / ”mid

[0047] In this URI format, "fqdn_or_ip" represents the domain name or Internet Protocol (IP) address of the MRF or SIP proxy that manages the call (e.g., the immersive 3D group session). If no MRF or SIP proxy manages the call (e.g., the immersive 3D group session), "fqdn_or_ip" may represent the domain name or IP address of the SIP address of the host of the call (e.g., the immersive 3D group session). "call_id" provides a unique identifier for the current call or conference (e.g., the current immersive 3D group session). "ssrc" represents the synchronization source of the owner / sending participant of the media stream. Finally, "mid" represents the media session identifier provided in the SDP. Other forms of addressing may be defined, for example, as a Uniform Resource Name (URN).

[0035]

[0048] In some embodiments, when using Web Real-Time Communications (WebRTC), the session setup protocol may be left to the application. Some implementations rely on SIP over WebSockets for this purpose. However, other protocols may be used to set up and describe the call (e.g., an immersive 3D group session).

[0036]

[0049] Some embodiments may use a scene graph as an entry point into a conference call (e.g., an immersive 3D group session). In such an embodiment, all participants share a scene graph document that sets up the 3D scene (or 3D space) at the beginning of the call (e.g., an immersive 3D group session). The scene graph will define graph nodes for each participant and identify the components for which each participant needs to provide media streams. This may be done by a central call server, such as an MRF, to which all conference participants connect. The MRF may be configured to update the scene graph during the call (e.g., an immersive 3D group session), such as to add new nodes or remove nodes of newly joining or leaving call participants.

[0037]

[0050] In some embodiments, a participant computing device may use a link to a web page provided by the call server to join a WebRTC conference. The call server may provide the participant computing device with that web page along with a scene graph file that sets up an initial / default configuration in 3D space of call participants and materials (e.g., each participant will be assigned a visual node, an audio source node, and potentially also nodes for graphics and other shared content). Each participant computing device may add or modify the nodes it owns in the scene graph. Media streams that provide components for the nodes in the scene graph may be streamed using WebRTC. These streams may be exchanged directly or through a server such as a media proxy server.

[0038]

[0051] In some embodiments, computing devices participating in an ITT4RT conference may establish direct peer-to-peer WebSocket channels with each other, or connections may be offered to all parties by the MRF. In such embodiments, the WebSocket channels may use a text frame format. In a scene (e.g., in a three-dimensional space), node names may be unique and may be declared in the SDP to ensure that there are no naming conflicts in nodes provided by different computing devices in a call. In such embodiments, nodes in the scene graph may reference external media streams, such as other media streams declared in the SDP. In such embodiments, the receiver may mask nodes from some computing devices in the rendering process, such as based on user input.

[0039]

[0052] In some embodiments, the MRF may be the owner of the master scene graph by default, which is the computing device that sets the coordinate system and on which all other nodes are composited. In some embodiments, the MRF may also be the computing device that defines the primary camera in the scene (or space). In some embodiments, in the absence of a centralized MRF, the computing devices in the call may select one computing device to provide the primary scene graph, for example, by selecting the computing device that provides the VR content or the organizer of the call. In various embodiments, an overlay may be a 2D or 3D object placed in the scene (or space). In some embodiments, the geometry of the overlay and its texture may be defined by the nodes that correspond to that overlay object. A simple example is a set of slides that is played in a rectangular area shown in a VR scene (or VR space). In this example, the geometry may be a rectangle and the texture may come from a video media stream. The rectangle may be placed in the scene (or space). In a viewport-dependent overlay, the position of the rectangle may be locked to the camera orientation.

[0040]

[0053] 1A illustrates an example of a communication system 100 suitable for implementing various embodiments. The communication system 100 may be a 5G NR network or any other suitable network, such as an LTE network.

[0041]

[0054] The communications system 100 may include a heterogeneous network architecture including a core network 140 and various wireless devices (also referred to as user equipment (UE) computing devices) (illustrated in FIG. 1 as wireless devices 120a-120e). The communications system 100 may also include a number of base stations (illustrated as BS 110a, BS 110b, BS 110c, and BS 110d) and other network entities. A base station is an entity that communicates with the wireless devices (wireless devices or UE computing devices) and may also be referred to as a Node B, Node B, LTE evolved Node B (eNB), access point (AP), radio head, transmit receive point (TRP), new radio base station (NR BS), 5G Node B (NB), next generation Node B (gNB), etc. Each base station may provide communication coverage for a particular geographic area. In 3GPP, the term "cell" can refer to the coverage area of ​​a base station, the coverage area of ​​a base station subsystem serving this coverage area, or a combination thereof, depending on the context in which the term is used.

[0042]

[0055] The base stations 110a-110d may provide communication coverage for a macro cell, a pico cell, a femto cell, another type of cell, or a combination thereof. A macro cell may cover a relatively large geographic area (e.g., a few kilometers in radius) and may allow unrestricted access by wireless devices with service subscriptions. A pico cell may cover a relatively small geographic area and may allow unrestricted access by wireless devices with service subscriptions. A femto cell may cover a relatively small geographic area (e.g., a home) and may allow restricted access by wireless devices having an association with the femto cell (e.g., wireless devices in a Closed Subscriber Group (CSG)). A base station for a macro cell may be referred to as a macro BS. A base station for a pico cell may be referred to as a pico BS. A base station for a femto cell may be referred to as a femto BS or a home BS. In the example shown in FIG. 1A, base station 110a may be a macro BS for macro cell 102a, base station 110b may be a pico BS for pico cell 102b, and base station 110c may be a femto BS for femto cell 102c. Base stations 110a-110d may support one or more (e.g., three) cells. The terms “eNB,” “base station,” “NR BS,” “gNB,” “TRP,” “AP,” “Node B,” “5G NB,” and “cell” may be used interchangeably herein.

[0043]

[0056] In some examples, the cells may not be fixed and the geographic areas of the cells may move according to the location of the mobile base station. In some examples, the base stations 110a-110d may be interconnected to each other and to one or more other base stations or network nodes (not shown) in the communication system 100 through various types of backhaul interfaces, such as direct physical connections, virtual networks, or combinations thereof, using any suitable transport network.

[0044]

[0057] The base stations 110a-110d may communicate with the core network 140 via wired or wireless communications links 126. The wireless devices 120a-120e (UE computing devices) may communicate with the base stations 110a-110d via wireless communications links 122.

[0045]

[0058] The wired communications link 126 may use various wired networks (e.g., Ethernet, TV cable, telephone, fiber optics, and other forms of physical network connections) that may use one or more wired communications protocols, such as Ethernet, Point-to-Point Protocol, High Level Data Link Control (HDLC), Advanced Data Communications Control Protocol (ADCCP), and Transmission Control Protocol / Internet Protocol (TCP / IP).

[0046]

[0059] The communication system 100 may also include a relay station (e.g., relay BS 110d). A relay station is an entity that can receive a transmission of data from an upstream station (e.g., a base station or a wireless device) and send a transmission of data to a downstream station (e.g., a wireless device or a base station). A relay station may also be a wireless device that can relay a transmission for another wireless device. In the example shown in FIG. 1, relay station 110d may communicate with base station 110a and wireless device 120d to facilitate communication between macro base station 110a and wireless device 120d. A relay station may also be referred to as a relay base station, relay base station, relay, etc.

[0047]

[0060] The communications system 100 may be a heterogeneous network including different types of base stations, e.g., macro base stations, pico base stations, femto base stations, relay base stations, etc. These different types of base stations may have different transmit power levels, different coverage areas, and different susceptibility to interference in the communications system 100. For example, macro base stations may have high transmit power levels (e.g., 5-40 watts), while pico base stations, femto base stations, and relay base stations may have lower transmit power levels (e.g., 0.1-2 watts).

[0048]

[0061] A network controller 130 may couple to a set of base stations and provide coordination and control for these base stations. The network controller 130 may communicate with the base stations via a backhaul. The base stations may also communicate with each other directly or indirectly, e.g., via wireless or wireline backhaul.

[0049]

[0062] Wireless devices (UE computing devices) 120a, 120b, 120c may be dispersed throughout the communication system 100, and each wireless device may be fixed or mobile. A wireless device may also be referred to as an access terminal, UE, terminal, mobile station, subscriber unit, station, etc.

[0050]

[0063] The macro base station 110a may communicate with the communication network 140 via wired or wireless communication link 126. The wireless devices 120a, 120b, 120c may communicate with the base stations 110a-110d via wireless communication link 122. The core network 140 may be connected to other devices, such as a call server 150 (e.g., a multimedia resource function (MRF), a multipoint communication unit (MCU), a telepresence application server, etc.). In this manner, via a connection to the core network 140, the call server 150 may make available telepresence services, such as immersive teleconferencing and telepresence for remote terminals (ITT4RT) services, to the wireless devices 120a, 120b, 120c, 120d (e.g., from the core network 140 via link 126 and from the base stations 110a-110d via link 122). Although shown as outside the core network 140, the call server 150 may be part of the core network 140 itself.

[0051]

[0064] The wireless communication links 122, 124 may include multiple carrier signals, frequencies, or frequency bands, each of which may include multiple logical channels. The wireless communication links 122 and 124 may utilize one or more radio access technologies (RATs). Examples of RATs that may be used in wireless wide area network (WWAN) wireless communication links 122, 124 in the distributed communication system 100 include 3GPP LTE, 3G, 4G, 5G (e.g., NR), GSM, Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Worldwide Interoperability for Microwave Access (WiMAX), Time Division Multiple Access (TDMA), and other mobile telephony technology cellular RATs. Examples of RATs that may be used in the wireless local area network (WLAN) wireless communication links 122, 124 in the local communication system 100 include medium-range wireless protocols such as Wi-Fi, LTE-U, LTE-Direct, LAA, MuLTEfire, as well as relatively short-range RATs such as ZigBee, Bluetooth, and Bluetooth Low Energy (LE).

[0052]

[0065] Some wireless networks (e.g., LTE) utilize orthogonal frequency division multiplexing (OFDM) on the downlink and single-carrier frequency division multiplexing (SC-FDM) on the uplink. OFDM and SC-FDM partition the system bandwidth into multiple (K) orthogonal subcarriers, also commonly referred to as tones, bins, etc. Each subcarrier may be modulated with data. In general, modulation symbols are sent in the frequency domain for OFDM and in the time domain for SC-FDM. The spacing between adjacent subcarriers may be fixed, and the total number of subcarriers (K) may be dependent on the system bandwidth. For example, the spacing of the subcarriers may be 15 kHz, and the minimum resource allocation (called a "resource block") may be 12 subcarriers (or 180 kHz). Thus, the nominal fast file transfer (FFT) size may be equal to 128, 256, 512, 1024, or 2048 for system bandwidths of 1.25, 2.5, 5, 10, or 20 megahertz (MHz), respectively. The system bandwidth may also be partitioned into subbands. For example, a subband may cover 1.08 MHz (i.e., 6 resource blocks), and there may be 1, 2, 4, 8, or 16 subbands for 1.25, 2.5, 5, 10, or 20 MHz system bandwidth, respectively.

[0053]

[0066] Although the description of some embodiments may use terminology and examples related to LTE technology, various embodiments may be applicable to other wireless communication systems, such as New Radio (NR) or 5G networks. NR may include support for half-duplex operation using time division duplexing (TDD) utilizing OFDM with cyclic prefix (CP) on the uplink (UL) and downlink (DL). A single component carrier bandwidth of 100 MHz may be supported. An NR resource block may span 12 subcarriers with a subcarrier bandwidth of 75 kHz over a 0.1 ms duration. Each radio frame may consist of 50 subframes with a length of 10 ms. Thus, each subframe may have a length of 0.2 ms. Each subframe may indicate a link direction (i.e., DL or UL) for data transmission, and the link direction for each subframe may be dynamically switched. Each subframe may include DL / UL data as well as DL / UL control data. Beamforming may be supported, and the beam direction may be dynamically configured. Multiple-input multiple-output (MIMO) transmission with precoding may also be supported. MIMO configurations in DL may support up to eight transmit antennas with multi-layer DL transmission of up to eight streams and up to two streams per wireless device. Multi-layer transmission with up to two streams per wireless device may be supported. Aggregation of multiple cells may be supported with up to eight serving cells. Alternatively, NR may support different air interfaces other than an OFDM-based air interface.

[0054]

[0067] Some wireless devices may be considered machine type communication (MTC) wireless devices or evolved or extended machine type communication (eMTC) wireless devices. MTC and eMTC wireless devices include, for example, a robot, a drone, a remote device, a sensor, a meter, a monitor, a location tag, etc., that may communicate with a base station, another device (e.g., a remote device), or some other entity. A wireless node may provide, for example, connectivity for or to a network (e.g., a wide area network, such as the Internet or a cellular network) via a wired or wireless communication link. Some wireless devices may be considered Internet of Things (IoT) devices or may be implemented as NB-IoT (narrowband Internet of Things) devices. The wireless devices 120a-e may be included within a housing that stores components of the wireless device, such as a processor component, a memory component, similar components, or a combination thereof.

[0055]

[0068] In general, any number of communication systems and any number of wireless networks may be deployed in a given geographic area. Each communication system and wireless network may support a particular radio access technology (RAT) and may operate on one or more frequencies. The RAT may also be referred to as a radio technology, an air interface, etc. The frequencies may also be referred to as a carrier, a frequency channel, etc. Each frequency may support a single RAT in a given geographic area to avoid interference between communication systems of different RATs. In some cases, NR or 5G RAT networks may be deployed.

[0056]

[0069] In some implementations, two or more wireless devices 120a-e (e.g., shown as wireless device 120a and wireless device 120e) may communicate directly (e.g., without using base station 110a-110d as an intermediary to communicate with each other) using one or more sidelink channels 124. For example, wireless devices 120a-e may communicate using peer-to-peer (P2P) communications, device-to-device (D2D) communications, vehicle-to-everything (V2X) protocols (which may include vehicle-to-vehicle (V2V) protocols, vehicle-to-infrastructure (V2I) protocols, or similar protocols), mesh networks, or similar networks, or combinations thereof. In this case, wireless devices 120a-e may perform scheduling operations, resource selection operations, as well as other operations described elsewhere herein as being performed by base station 110a.

[0057]

[0070] FIG. 1B illustrates a head-mounted device 172 that may be configured according to various embodiments. With reference to FIGS. 1A and 1B, in the example illustrated in FIG. 1A, the head-mounted device 172 may be a particular implementation of a user equipment computing device (e.g., UE 120c, 120d, 120e). The head-mounted device 172 includes a frame 152, two optical lenses 154, an outward-facing world view image sensor / camera 158, an inward-facing gaze view sensor / camera 160, a sensor array 162, a memory 164, and a processor 156 communicatively coupled to a communication circuit 166. In various embodiments, the communication circuit 166 may support one or more RATs for supporting communication between various devices as described in the system 100 with reference to FIG. 1A. In some embodiments, the head-mounted device 172 may include capacitance touch sensing circuitry along an arm 180 of the frame or in a nose bridge 182 of the head-mounted device 172. In some embodiments, head mounted device 172 may also include sensors for monitoring physical conditions (e.g., location, movement, acceleration, orientation, altitude, etc.). The sensors may include any or all of a gyroscope, accelerometer, magnetometer, magnetic compass, altimeter, odometer, and pressure sensor. The sensors may also include various biosensors (e.g., heart rate monitor, body temperature sensor, carbon sensor, oxygen sensor, etc.) for collecting information related to the environment and / or user condition. Sensors may also be external to head mounted device 172 and paired or grouped to head mounted device 172 via a wired or wireless connection (e.g., Bluetooth, etc.).

[0058]

[0071] In some embodiments, the processor 156 may also be communicatively coupled to an image rendering device 168 (e.g., an image projector) embedded in the arm portion 180 of the frame 152 and configured to project an image onto the optical lens 154. In some embodiments, the image rendering device 168 may include a light emitting diode (LED) module, a light tunnel, a homogenizing lens, an optical display, a fold mirror, or other components well known in the art projectors or head mounted displays. In some embodiments (e.g., embodiments in which the image rendering device 168 is not included or used), the optical lens 154 may be or include a see-through or partially see-through electronic display. In some embodiments, the optical lens 154 includes an image generating element, such as a see-through organic light emitting diode (OLED) display element or a liquid crystal on silicon (LCOS) display element. In some embodiments, the optical lens 154 may include separate left and right eye display elements. In some embodiments, the optical lens 154 may include or act as a light guide for delivering light from the display element to the wearer's eyes.

[0059]

[0072] The outward-facing or world-view image sensor / camera 158 may be configured to capture real-world images from the user's physical environment and send corresponding image data to the processor 156. The processor 156 may combine the real-world images with computer-generated figures or virtual objects (VOs) to generate an augmented scene (or space) and render the augmented scene (or space) on the electronic display or optical lens 154 of the head-mounted device 172.

[0060]

[0073] The inward looking or line of sight view sensor / camera 160 may be configured to collect image data from the user's eye or facial structures surrounding the user's eye.

[0061]

[0074] The various embodiments may be implemented on a number of single-processor and multi-processor computer systems, including systems on a chip (SOC) or systems in a package (SIP). Figure 2 shows an exemplary computing system or SIP 200 architecture that may be used in a wireless device (UE computing device) implementing the various embodiments.

[0062]

[0075] 1A, 1B, and 2, the illustrated exemplary SIP 200 includes two SOCs 202, 204, a clock 206, a voltage regulator 208, and one or more wireless transceivers 266 configured to send and receive wireless communications via antennas (not shown) to and from network wireless devices, such as base station 110a and / or other wireless devices (e.g., wireless devices 120a-e). In some embodiments, the first SOC 202 operates as a central processing unit (CPU) of the wireless device that executes instructions of a software application program by performing arithmetic, logic, control, and input / output (I / O) operations specified by the instructions. In some embodiments, the second SOC 204 may operate as a dedicated processing unit. For example, the second SOC 204 may operate as a dedicated 5G processing unit responsible for managing high volume, high speed (e.g., 5 Gbps, etc.), and / or very high frequency short wavelength (e.g., 28 GHz mmWave spectrum, etc.) communications. In some embodiments, the wireless transceivers 266 may be wireless transceivers configured to support peer-to-peer (P2P) communications, device-to-device (D2D) communications, vehicle-to-anything (V2X) protocols (which may include vehicle-to-vehicle (V2V) protocols, vehicle-to-infrastructure (V2I) protocols, or similar protocols), Bluetooth communications, Wi-Fi communications, etc. In some embodiments, the wireless transceivers 266 may each be connected to the first SOC 202 and / or the second SOC 204 may be connected to each of the one or more wireless transceivers 266 by various (also referred to as interconnects, buses, etc.) physical connections 267, such as a Peripheral Component Interconnect Express (PCIe) connection, a Universal Serial Bus (USB) connection, a High Speed ​​Chip-to-Chip (HSIC) connection, an Ethernet connection, etc.

[0063]

[0076] The first SOC 202 may include a digital signal processor (DSP) 210, a modem processor 212, a graphics processor 214, an application processor 216, one or more coprocessors 218 (e.g., vector coprocessors) connected to one or more of the processors, memory 220, custom circuitry 222, system components and resources 224, an interconnect / bus module 226, one or more temperature sensors 230, a thermal management unit 232, and a thermal power envelope (TPE) component 234. The second SOC 204 may include a 5G modem processor 252, a power management unit 254, an interconnect / bus module 264, multiple mmWave transceivers 256, memory 258, and various additional processors 260, such as application processors, packet processors, etc.

[0064]

[0077] Each processor 210, 212, 214, 216, 218, 252, 260 may include one or more cores, and each processor / core may perform operations independent of the other processors / cores. For example, the first SOC 202 may include a processor that executes a first type of operating system (e.g., FreeBSD, LINUX, OS X, etc.) and a processor that executes a second type of operating system (e.g., MICROSOFT WINDOWS 10). In addition, any or all of the processors 210, 212, 214, 216, 218, 252, 260 may be included as part of a processor cluster architecture (e.g., a synchronous processor cluster architecture, an asynchronous or heterogeneous processor cluster architecture, etc.).

[0065]

[0078] The first SOC 202 and the second SOC 204 may include various system components, resources, and custom circuits for managing sensor data, analog-to-digital conversion, wireless data transmission, and for performing other specialized operations such as decoding data packets and processing encoded audio and video signals for rendering in a web browser. For example, the system components and resources 224 of the first SOC 202 may include power amplifiers, voltage regulators, oscillators, phase-locked loops, peripheral bridges, data controllers, memory controllers, system controllers, access ports, timers, and other similar components used to support processors and software clients running on the wireless device. The system components and resources 224 and / or custom circuits 222 may also include circuits for interfacing with peripheral devices such as cameras, electronic displays, wireless communication devices, external memory chips, etc.

[0066]

[0079] The first SOC 202 and the second SOC 204 may communicate via an interconnect / bus module 250. The various processors 210, 212, 214, 216, 218 may be interconnected to one or more memory elements 220, system components and resources 224, custom circuitry 222, and thermal management unit 232 via an interconnect / bus module 226. Similarly, the processor 252 may be interconnected to a power management unit 254, an mmWave transceiver 256, memory 258, and various additional processors 260 via an interconnect / bus module 264. The interconnect / bus modules 226, 250, 264 may include an array of reconfigurable logic gates and / or implement a bus architecture (e.g., CoreConnect, AMBA, etc.). Communication may be provided by an advanced interconnect, such as a high-performance network-on-chip (NoC).

[0067]

[0080] The first SOC 202 and / or the second SOC 204 may further include input / output modules (not shown) for communicating with resources external to the SOC, such as a clock 206 and a voltage regulator 208. Resources external to the SOC (e.g., clock 206, voltage regulator 208) may be shared by two or more of the internal SOC processors / cores.

[0068]

[0081] In addition to the exemplary SIP 200 described above, various embodiments may be implemented in a wide variety of computing systems, which may include a single processor, multiple processors, multi-core processors, or any combination thereof.

[0069]

[0082] FIG. 3 illustrates an example of a software architecture 300 including radio protocol stacks for a user plane and a control plane in wireless communications between a base station 350 (e.g., base station 110a) and a wireless device (UE computing device) 320 (e.g., wireless devices 120a-120e, 172, 200). With reference to FIGS. 1A-3, the wireless device 320 may implement the software architecture 300 to communicate with a base station 350 of a communication system (e.g., 100). In various embodiments, layers in the software architecture 300 may form logical connections with corresponding layers in the software of the base station 350. The software architecture 300 may be distributed among one or more processors (e.g., processors 212, 214, 216, 218, 252, 260). Although illustrated with respect to one over-the-air protocol stack, in a multi-SIM (Subscriber Identity Module) wireless device, software architecture 300 may include multiple protocol stacks, each of which may be associated with a different SIM (e.g., two protocol stacks, each associated with two SIMs, in a dual-SIM wireless communication device). Although described below with respect to an LTE communication layer, software architecture 300 may support any of a variety of standards and protocols for wireless communication and / or may include additional protocol stacks supporting any of a variety of standards and protocols wireless communication.

[0070]

[0083] The software architecture 300 may include a non-access stratum (NAS) 302 and an access stratum (AS) 304. The NAS 302 may include functions and protocols to support packet filtering, security management, mobility control, session management, and traffic and signaling between the SIM(s) of the wireless device (e.g., SIM(s) 204) and its core network 140. The AS 304 may include functions and protocols to support communication between the SIM(s) (e.g., SIM(s) 204) and supported access network entities (e.g., base stations). In particular, the AS 304 may include at least three layers (Layer 1, Layer 2, and Layer 3), each of which may include various sublayers.

[0071]

[0084] In the user and control planes, Layer 1 (L1) of the AS 304 may be a physical layer (PHY) 306 that may oversee functions enabling transmission and / or reception over the air interface. Examples of such physical layer 306 functions may include cyclic redundancy check (CRC) attachment, coding blocks, scrambling and descrambling, modulation and demodulation, signal measurement, MIMO, etc. The physical layer may include various logical channels, including a physical downlink control channel (PDCCH) and a physical downlink shared channel (PDSCH).

[0072]

[0085] In the user and control planes, Layer 2 (L2) of the AS 304 may be responsible for the link between the wireless device 320 and the base station 350 on the physical layer 306. In various embodiments, Layer 2 may include a Media Access Control (MAC) sublayer 308, a Radio Link Control (RLC) sublayer 310, and a Packet Data Convergence Protocol (PDCP) 312 sublayer, each of which forms a logical connection that terminates at the base station 350.

[0073]

[0086] In the control plane, Layer 3 (L3) of the AS 304 may include a Radio Resource Control (RRC) sublayer 3. Although not shown, the software architecture 300 may include additional Layer 3 sublayers, as well as various upper layers above Layer 3. In various embodiments, the RRC sublayer 313 may provide functions including broadcasting system information, paging, and establishing and releasing RRC signaling connections between the wireless device 320 and the base station 350.

[0074]

[0087] In various embodiments, the PDCP sublayer 312 may provide uplink functions including multiplexing between different radio bearers and logical channels, sequence number addition, handover data processing, integrity protection, ciphering, and header compression. In the downlink, the PDCP sublayer 312 may provide functions including in-order delivery of data packets, duplicate data packet detection, integrity verification, decryption, and header recovery.

[0075]

[0088] In the uplink, the RLC sublayer 310 may provide segmentation and concatenation of upper layer data packets, retransmission of lost data packets, and automatic repeat request (ARQ), while in the downlink, the RLC sublayer 310 functions may include reordering of data packets to compensate for out-of-order reception, reassembly of upper layer data packets, and ARQ.

[0076]

[0089] In the uplink, the MAC sublayer 308 may provide functions including multiplexing between logical and transport channels, random access procedures, logical channel priorities, and Hybrid ARQ (HARQ) operations. In the downlink, MAC layer functions may include channel mapping within a cell, demultiplexing, discontinuous reception (DRX), and HARQ operations.

[0077]

[0090] While the software architecture 300 may provide functionality for transmitting data over a physical medium, the software architecture 300 may further include at least one host layer 314 to provide data transfer services to various applications in the wireless device 320. In some embodiments, the application-specific functionality provided by the at least one host layer 314 may provide an interface between the software architecture and the general-purpose processor 206.

[0078]

[0091] In other embodiments, software architecture 300 may include one or more upper logical layers (e.g., transport, session, presentation, application, etc.) that provide host layer functionality. For example, in some embodiments, software architecture 300 may include a network layer (e.g., IP layer) where a logical connection terminates at a packet data network (PDN) gateway (PGW). In some embodiments, software architecture 300 may include an application layer where a logical connection terminates at another device (e.g., an end user device, a server, etc.). In some embodiments, software architecture 300 may further include a hardware interface 316 between physical layer 306 and communications hardware (e.g., one or more radio frequency (RF) transceivers) in AS 304.

[0079]

[0092] 4 illustrates a process flow diagram of an example method 400 for supporting an immersive experience in a teleconference or telepresence session, according to various embodiments. With reference to FIGS. 1A-4, method 400 may be implemented by a processor (such as 156, 212, 216, 252, or 260) of a wireless device (such as wireless devices 120a-120e, 172, 200, 320). In various embodiments, the operations of method 400 may be performed by a processor of a wireless device that is one of multiple participant computing devices in a teleconference or telepresence session, such as an immersive three-dimensional group session.

[0080]

[0093] At block 402, the process performs operations to indicate an offer to send and / or receive a scene graph as part of a Session Initiation Protocol (SIP) setup for the session. In some embodiments, the offer may indicate a graphical output node owned by the wireless device. In some embodiments, the session may be a WebRTC session.

[0081]

[0094] At block 404, the processor may perform operations to receive a Session Description Protocol (SDP) for the session indicating an address of a data channel over which a scene graph for the session will be shared. In some embodiments, the scene graph may define one or more nodes assigned to each computing device participating in the session. In various embodiments, the nodes may reference other media streams from other computing devices participating in the session, and the other media streams may be overlaid in the session. In various embodiments, the one or more nodes assigned to each computing device participating in the session may include one or more visual nodes, audio source nodes, graphics nodes, or shared content nodes.

[0082]

[0095] At block 406, the processor may perform operations to download the scene graph over a data channel.

[0083]

[0096] At block 408, the processor may perform operations to receive and render the session according to the scene graph for rendering on an image rendering device (e.g., 168). Receiving and rendering the session may include receiving a streaming service of the session according to the scene graph and rendering the session on a display. In some embodiments, the display of the session may be rendered on an HMD (e.g., 172), on a video conference room, on a volumetric display, or on any other image and sound rendering device, and receiving and rendering may include outputting the session to a user via the image and sound rendering device.

[0084]

[0097] At block 410, the processor may perform operations to add a node assigned to the wireless device to the scene graph or modify a node of one or more nodes assigned to the wireless device in the scene graph.

[0085]

[0098] FIG. 5 illustrates a process flow diagram of an exemplary method 500 for supporting an immersive experience in a teleconference or telepresence session, according to various embodiments. With reference to FIGS. 1A-5, the method 500 may be implemented by a processor (such as 156, 212, 216, 252, or 260) of a wireless device (such as wireless devices 120a-120e, 172, 200, 320) and / or a call server (such as call server 150). In various embodiments, the operations of the method 500 may be performed by a processor of a host computing device hosting the teleconference or telepresence session. In some embodiments, the host computing device may be a separate call server, such as an MRF, MCU, teleconference application server, etc. In some embodiments, the host computing device may be a wireless device that is one of multiple computing devices participating in a teleconference or telepresence session, such as an immersive three-dimensional group session. In various embodiments, the operations of the method 500 may be performed together with the operations of the method 400.

[0086]

[0099] At block 502, a processor of the host computing device may perform operations to receive an indication of an offer to send and / or receive a scene graph as part of a Session Initiation Protocol (SIP) setup for the session from a computing device participating in the session. In some embodiments, the session may be a WebRTC session. In some embodiments, the received indication may indicate a graphical output node owned by the wireless device sending the indication of the offer.

[0087]

[0100] At block 504, a processor of the host computing device may perform operations to generate a Session Description Protocol (SDP) for the session that indicates the address of a data channel over which a scene graph for the session will be shared. In some embodiments, the scene graph may define one or more nodes assigned to each computing device participating in the session.

[0088]

[0101] At block 506, a processor of the host computing device may perform operations to send an SDP to the computing devices participating in the session.

[0089]

[0102] At block 508, a processor of the host computing device may perform operations to send the session and the scene graph to the computing devices participating in the session. As an example, the host computing device may perform operations to stream the session and the scene graph to the computing devices participating in the session. In various embodiments, the computing devices participating in the session may be an HMD (e.g., 172), a TV in a conference room, a volumetric display, or other image and sound rendering devices, and sending the session and / or the scene graph may include sending the session and / or the scene graph so that the computing devices participating in the session can output the session to a user on the image and sound rendering device.

[0090]

[0103] FIG. 6 illustrates the configuration of a scene graph document 600 in glTF2.0 suitable for use in various embodiments. Referring to FIGS. 1A-6, in various embodiments, the scene graph document 600 may be part of a scene graph. The scene graph document 600 may include JSON elements that define a node hierarchy, material descriptions, lighting information, camera information, etc. The scene graph document 600 may include binary files (BIN) that define geometry information such as vertices and indexes, animation information such as key frames, skin information such as inverse bind matrices, etc. The scene graph document 600 may include a GL Shading Library (glSL) file that defines shader information. The scene graph document 600 may include various other types of files, such as Portable Network Graphics (PNG) files, Joint Photographic Experts Group (JPEG) files, etc., that define other information for the scene graph, such as textures.

[0091]

[0104] FIG. 7 illustrates a structure of a scene graph suitable for use in various embodiments. With reference to FIGS. 1A-7, in various embodiments, a scene graph may include multiple nodes. Each node may include child nodes that describe various components of the node, such as a camera view, mesh information, lighting information, etc. The scene graph may define a hierarchical relationship between attributes for rendering the mesh of the node, such as accessories, skins, buffer views, buffer information, materials, techniques, programs, shaders, textures, images, and samplers, as shown in FIG. 7.

[0092]

[0105] FIG. 8 is a call flow diagram illustrating operations for supporting immersive experiences in a teleconference or telepresence session according to various embodiments. With reference to FIGS. 1A-8, operations between the call participants (e.g., call participants #1 and #2) and the call server may include, in operation 1), the participants (e.g., call participants #1 and #2) using a provided link to a web page to join a WebRTC conference. In operation 2), the call server provides the participants (e.g., call participants #1 and #2) with the web page along with a scene graph file that sets up an initial / default configuration of the call participants (e.g., call participants #1 and #2) and materials in 3D space. For example, each participant will be assigned a visual node, an audio source node, and potentially also nodes for graphics and other shared content. In operation 3), each participant (e.g., call participants #1 and #2) may add or modify the nodes it owns in the scene graph. In operation 4), media streams providing components for the nodes in the scene graph may be streamed using WebRTC. These streams can be exchanged directly or through a server, for example a media proxy server.

[0093]

[0106] FIG. 9 is a process flow diagram illustrating a method 900 for providing an immersive three-dimensional group session, according to various embodiments. With reference to FIGS. 1A-9, method 900 may be implemented by a processor (such as 156, 212, 216, 252, or 260) of a wireless device (such as wireless devices 120a-120e, 172, 200, 320). In various embodiments, the operations of method 900 may be performed by a processor of a wireless device that is one of multiple participant computing devices in a teleconference or telepresence session, such as an immersive three-dimensional group session. In various embodiments, the operations of method 900 may be performed in conjunction with any one or more of the operations of methods 400 (FIG. 4) and / or 500 (FIG. 5).

[0094]

[0107] At block 902, the processor may perform operations including receiving a scene graph for an immersive three-dimensional group session, where the scene graph comprises at least a unique graphical output node assigned to be controlled by the wireless device and respective other graphical output nodes assigned to be controlled by each of the others of the plurality of participant computing devices. For example, the scene graph may be a scene graph such as that shown in FIG.

[0095]

[0108] The portion of the scene graph received in block 902 may be assigned to each participant computing device in the immersive three-dimensional group session, such as one or more nodes for each respective participant computing device. By parsing the scene graph, the processor of the wireless device may determine the nodes to be controlled by the wireless device. Assigning a graphical output node for each participant computing device may enable each participant computing device to control at least one graphical output node. The graphical output node may include components that define a three-dimensional object to be output in the three-dimensional space defined by the scene graph. For example, the three-dimensional object may include an avatar, character, or other representation, and the components of the graphical output node may define how the three-dimensional object is to be rendered in the three-dimensional space of the immersive three-dimensional group session. In this manner, by controlling the components of its assigned respective graphical output node, the processor of the wireless device may control how other participant computing devices view one or more three-dimensional objects associated with the wireless device, such as an avatar, character, or other representation selected by a user of the wireless device in the immersive three-dimensional group session. Similarly, by controlling the components of their respective assigned graphical output nodes, the other participant computing devices may control how users of the wireless devices view their respective associated three-dimensional object or objects, such as avatars, characters, or other representations, during the immersive three-dimensional group session.

[0096]

[0109] At block 904, the processor may perform operations including controlling components of the unique graphical output node relative to the three-dimensional space of the immersive three-dimensional group session. In some embodiments, the user may adjust the unique graphical node, such as moving the placement of the unique graphical node in the scene, independent of the position of the wireless device. For example, the user may select a location in the scene for displaying the user's three-dimensional object (e.g., avatar, character, etc.) in the scene. In some embodiments, controlling components of the unique graphical output node may include controlling components of the unique graphical output node based at least in part on a determined position of the wireless device relative to the three-dimensional space of the immersive three-dimensional group session. In some embodiments, a position of the wireless device may be determined in the three-dimensional space, such as a position relative to a center point, grid coordinates, or other reference for the three-dimensional space. Because some immersive three-dimensional group sessions may support movement of participants within the three-dimensional space for the immersive three-dimensional group session, determining the position of the wireless device may support rendering objects for the wireless device's graphical output node in the correct relative position. As a particular example, the lighting and / or camera components of the wireless device's assigned unique graphical output node may be controlled to reflect the lighting and / or camera components of the wireless device's current position in three-dimensional space for an immersive three-dimensional group session.

[0097]

[0110] In addition to controlling components of the unique graphical output node based on position at block 904, components of the unique graphical output node may be controlled based on other metrics, such as the orientation of the wireless device. For example, positional movements and orientation changes relative to a common reference point and common reference plane indicated by an accelerometer of the wireless device may be used to determine the position and orientation of the wireless device relative to the three-dimensional space of the immersive three-dimensional group session. As a particular example, the lighting and / or camera components of the assigned unique graphical output node of the wireless device may be controlled to reflect the lighting and / or camera components of the current position and current orientation of the wireless device in the three-dimensional space for the immersive three-dimensional group session. Controlling components of the unique graphical output node based on the current position and / or current orientation may be useful in implementations where the wireless device is a head-mounted device to allow movements of the user of the head-mounted device to be visually imparted to other participants in the immersive three-dimensional group session.

[0098]

[0111] At block 906, the processor may perform operations including sending components of the unique graphical output node in the first media stream to others of the plurality of participant computing devices. Those components may be sent in the media stream directly to the other participant computing devices and / or via a call server (e.g., 150), such as an MRF, MCU, teleconferencing application server, etc. In this manner, the processor of the wireless device may provide the media stream for its unique respective graphical output node to the other participant computing devices, thereby controlling how the other participant computing devices render that graphical output node.

[0099]

[0112] At block 908, the processor may perform operations including receiving components of other graphical output nodes in the media stream from each of the other ones of the plurality of participant computing devices. Those components may be received in the media stream directly from each of the other participant computing devices and / or via a call server (e.g., 150), such as an MRF, MCU, teleconferencing application server, etc. In this manner, the processor of the wireless device may receive media streams of graphical output nodes assigned to the other participant computing devices.

[0100]

[0113] At block 910, the processor may perform operations including rendering the immersive three-dimensional group session on a display of the wireless device based at least in part on components of its own graphical output node and components of the other graphical output nodes. For example, components of the media stream of the wireless device and components of the media streams of the other participant computing devices may be overlaid with components of the other media streams to generate a displayed output of the three-dimensional space of the immersive three-dimensional group session.

[0101]

[0114] The processor may perform the operations of blocks 904, 906, 908, and 910 continually during the immersive three-dimensional group session to render the immersive three-dimensional group session.

[0102]

[0115] FIG. 10 is a process flow diagram illustrating a method 1000 for providing an immersive three-dimensional group session, according to various embodiments. With reference to FIGS. 1A-10, the method 1000 may be implemented by a processor (such as 156, 212, 216, 252, or 260) of a wireless device (such as wireless devices 120a-120e, 172, 200, 320). In various embodiments, the operations of the method 1000 may be performed by a processor of a wireless device that is one of a plurality of participant computing devices in a teleconference or telepresence session, such as an immersive three-dimensional group session. In various embodiments, the operations of the method 1000 may be performed together with any one or more of the operations of the method 400 (FIG. 4), the method 500 (FIG. 5), and / or the method 900 (FIG. 9). As a particular example, the operations of the method 1000 may be performed as part of the operations of block 910 of the method 900 to render an immersive three-dimensional group session.

[0103]

[0116] At block 1002, the processor may perform operations including receiving a scene graph update including an indication of a new participant computing device for the immersive three-dimensional group session and an indication of a new graphical output node assigned to be controlled by the new participant computing device. The scene graph update may be sent by the host computing device in response to a new participant joining the immersive three-dimensional group session. In some embodiments, the scene graph update may be received directly from another participant computing device and / or via a call server (e.g., 150), such as an MRF, MCU, teleconferencing application server, etc.

[0104]

[0117] At block 1004, the processor may perform operations including receiving components of the new graphical output node in the second media stream from the new participant computing device. Those components may be received in the media stream directly from the new participant computing device and / or via a call server (e.g., 150), such as an MRF, MCU, teleconferencing application server, etc. In this manner, the processor of the wireless device may receive the media stream of the graphical output node assigned to the newly added participant computing device.

[0105]

[0118] At block 1006, the processor may perform operations including rendering the immersive three-dimensional group session on a display of the wireless device based at least in part on the components of the own graphical output node, the components of the other graphical output nodes, and the components of the new graphical output node. For example, the components of the media stream of the wireless device and the components of the media streams of the other participant computing devices, including the second media stream of the newly added participant computing device, may be overlaid with the components of the other media streams to generate a displayed output of the three-dimensional space of the immersive three-dimensional group session.

[0106]

[0119] Various embodiments may be implemented on various wireless network devices, an example of which is shown in FIG. 11 in the form of a wireless network computing device 1100 functioning as a network element of a communication network, such as a call server (e.g., call server 150). Such a network computing device may include at least the components shown in FIG. 11. With reference to FIGS. 1A-11, the network computing device 1100 may typically include a processor 1101 coupled to a volatile memory 1102 and a large capacity non-volatile memory, such as a disk drive 1103. The network computing device 1100 may also include a peripheral memory access device, such as a floppy disk drive, compact disk (CD), or digital video disk (DVD) drive 1106, coupled to the processor 1101. The network computing device 1100 may also include a network access port 1104 (or interface) coupled to the processor 1101 for establishing a data connection with a network, such as the Internet and / or a local area network coupled to other system computers and servers. The network computing device 1100 may include one or more antennas 1107 for sending and receiving electromagnetic radiation that may be connected to a wireless communications link. The network computing device 1100 may include additional access ports, such as USB, Firewire, Thunderbolt, etc., for coupling to peripherals, external memory, or other devices.

[0107]

[0120] Various embodiments may be implemented on various wireless devices (e.g., wireless devices 120a-120e, 172, 200, 320), an example of which is shown in FIG. 12 in the form of a smartphone 1200. With reference to FIGS. 1A-12, the smartphone 1200 may include a first SOC 202 (e.g., a SOC-CPU) coupled to a second SOC 204 (e.g., a 5G-enabled SOC). The first SOC 202 and the second SOC 204 may be coupled to internal memory 1206, 1216, a display 1212, and to a speaker 1214. Additionally, the smartphone 1200 may include an antenna 1204 for sending and receiving electromagnetic radiation that may be connected to a wireless data link, and / or a cellular telephone transceiver 266 coupled to one or more processors in the first SOC 202 and / or the second SOC 204. The smartphone 1200 also typically includes menu selection buttons or rocker switches 1220 for receiving user input.

[0108]

[0121] The exemplary smartphone 1200 also includes a sound encoding / decoding (CODEC) circuit 1210 that digitizes sound received from the microphone into data packets suitable for wireless transmission and decodes the received sound data packets to generate analog signals that are provided to a speaker to generate sound. One or more of the processors in the first SOC 202 and second SOC 204, the wireless transceiver 266, and the CODEC 1210 may also include digital signal processor (DSP) circuitry (not separately shown).

[0109]

[0122] The processors of the wireless network computing device 1100 and the smartphone 1200 may be any programmable microprocessor, microcomputer, or one or more multiple processor chips that may be configured by software instructions (applications) to perform various functions, including those of the various embodiments described below. In some wireless devices, multiple processors may be provided, such as one processor in the SOC 204 dedicated to wireless communication functions, one processor in the SOC 202 dedicated to running other applications, etc. Typically, software applications may be stored in memory 1206, 1216 before they are accessed and loaded into the processor. The processors may include sufficient internal memory to store application software instructions.

[0110]

[0123] Terms such as "component," "module," and "system" as used in this application are intended to include computer-related entities, such as, but not limited to, hardware, firmware, a combination of hardware and software, software, or software in execution, configured to perform certain operations or functions. For example, a component may be, but is not limited to, a process running on a processor, a processor, an object, an executable, a thread of execution, a program, and / or a computer. By way of example, both an application running on a wireless device and the wireless device may be referred to as a component. One or more components may reside within a process and / or thread of execution, and a component may be localized on one processor or core and / or distributed among two or more processors or cores. Furthermore, these components may execute from various non-transitory computer-readable media having various instructions and / or data structures stored thereon. Components may communicate via local and / or remote process, function or procedure calls, electronic signals, data packets, memory read / writes, and other known network, computer, processor, and / or process related communication methods.

[0111]

[0124] A number of different cellular and mobile communication services and standards may become available or are contemplated in the future, all of which may implement and benefit from the various embodiments. Such services and standards include, for example, 3rd Generation Partnership Project (3GPP), Long Term Evolution (LTE) systems, third generation wireless mobile communication technologies (3G), fourth generation wireless mobile communication technologies (4G), fifth generation wireless mobile communication technologies (5G), Global System for Mobile Communications (GSM), Universal Mobile Telecommunications System (UMTS), 3GSM, General Packet Radio Service (GPRS), and Code Division Multiple Access (CDMA) systems (e.g., cdmaOne, Examples of technologies include CDMA1020®, Enhanced Data Rates for GSM Evolution (EDGE), Advanced Mobile Phone System (AMPS), Digital AMPS (IS-136 / TDMA), Evolution Data Optimized (EV-DO), Digital Enhanced Cordless Telecommunications (DECT), Worldwide Interoperability for Microwave Access (WiMAX), Wireless Local Area Networks (WLAN), Wi-Fi Protected Access I and II (WPA, WPA2), and Integrated Digital Enhanced Network (iDEN). Each of these technologies involves, for example, the transmission and reception of voice, data, signaling, and / or content messages. It should be understood that any reference to terminology and / or technical details relating to a particular telecommunications standard or technology is for illustrative purposes only and is not intended to limit the scope of the claims to a particular communication system or technology unless expressly stated in the claim language.

[0112]

[0125] The various embodiments shown and described are provided merely as examples to illustrate various features of the claims. However, features shown and described with respect to any given embodiment are not necessarily limited to the associated embodiment and may be used or combined with other embodiments shown and described. Furthermore, the claims are not intended to be limited by any one exemplary embodiment. For example, one or more of the operations of methods 400, 500, 900, and / or 1000 may be replaced by or combined with one or more operations of methods 400, 500, 900, and / or 1000.

[0113]

[0126] The above method descriptions and process flow diagrams are provided only as illustrative examples and are not intended to require or imply that the operations of the various embodiments must be performed in the order presented. As will be appreciated by one of ordinary skill in the art, the order of operations in the above embodiments may be performed in any order. Words such as "then," "then," and "next" are not intended to limit the order of operations, but rather, these words are used to guide the reader through the method descriptions. Additionally, references to claim elements in the singular using, for example, the articles "a," "an," or "the" should not be construed as limiting the element to the singular.

[0114]

[0127] The various exemplary logic blocks, modules, components, circuits, and algorithmic operations described with respect to the embodiments disclosed herein may be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability of hardware and software, the various exemplary components, blocks, modules, circuits, and operations have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in various ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the claims.

[0115]

[0128] The hardware used to implement the various example logic, logic blocks, modules, and circuits described in connection with the embodiments disclosed herein may be implemented or performed using general purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of receiver smart objects, e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Alternatively, some operations or methods may be performed by circuitry that is specific to a given function.

[0116]

[0129] In one or more embodiments, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or codes on a non-transitory computer-readable storage medium or a non-transitory processor-readable storage medium. The operations of the methods or algorithms disclosed herein may be implemented in a processor-executable software module or processor-executable instructions that may reside on a non-transitory computer-readable storage medium or a non-transitory processor-readable storage medium. A non-transitory computer-readable storage medium or a non-transitory processor-readable storage medium may be any storage medium that can be accessed by a computer or processor. By way of example and not limitation, such a non-transitory computer-readable storage medium or a non-transitory processor-readable storage medium may include RAM, ROM, EEPROM, FLASH memory, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage smart objects, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Disk and disc as used herein include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc, where disks typically reproduce data magnetically and discs reproduce data optically with lasers. Combinations of the above are also included within the scope of non-transitory computer-readable media and non-transitory processor-readable media. Furthermore, operations of a method or algorithm may reside as one or any combination or set of code and / or instructions on a non-transitory processor-readable storage medium and / or a non-transitory computer-readable storage medium, which may be incorporated into a computer program product.

[0117]

[0130] The preceding description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the claims. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the scope of the claims. Thus, the present disclosure is not intended to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the following claims and the principles and novel features disclosed herein. The invention as described in the claims of the original application is set forth below. [C1] A method implemented by a processor of a wireless device that is one of a plurality of participant computing devices operating in an immersive three-dimensional group session, comprising: receiving a scene graph for an immersive three-dimensional group session, wherein the scene graph comprises at least a unique graphical output node assigned to be controlled by the wireless device and respective other graphical output nodes assigned to be controlled by each of others of the plurality of participant computing devices; controlling components of said unique graphical output node relative to the three-dimensional space of said immersive three-dimensional group session; sending the components of the unique graphical output node in a first media stream to the others of the plurality of participant computing devices; receiving, in a media stream from each of the other ones of the plurality of participant computing devices, components of the other graphical output node; rendering the immersive three-dimensional group session on a display of the wireless device based at least in part on the components of the unique graphical output node and the components of the other graphical output nodes; A method comprising: [C2] receiving a scene graph update including an indication of a new participant computing device for the immersive three-dimensional group session and an indication of a new graphical output node assigned to be controlled by the new participant computing device; receiving, in a second media stream from the new participant computing device, components of the new graphical output node; rendering the immersive three-dimensional group session on the display of the wireless device based at least in part on the components of the original graphical output node, the components of the other graphical output nodes, and the components of the new graphical output node; The method of claim C1, further comprising: [C3] receiving a Session Description Protocol (SDP) for the immersive three-dimensional group session indicating an address of a data channel over which the scene graph will be shared; Further equipped with The method of C1, wherein receiving the scene graph comprises downloading the scene graph over the data channel. [C4] sending an offer to send or receive the scene graph to the others of the plurality of participant computing devices as part of a Session Initiation Protocol (SIP) setup for the immersive three-dimensional group session; The method of claim C1, further comprising: [C5] sending an offer to the other one of the plurality of participant computing devices to send or receive the scene graph, wherein the offer indicates the unique graphical output node. The method according to C1. [C6] The method of C1, wherein the immersive three-dimensional group session is a Web Real-Time Communications (WebRTC) session. [C7] The method of C1, wherein controlling components of the unique graphical output node with respect to the three-dimensional space of the immersive three-dimensional group session comprises controlling components of the unique graphical output node based at least in part on a determined position of the wireless device with respect to the three-dimensional space of the immersive three-dimensional group session. [C8] The method of C7, wherein controlling components of the unique graphical output node based at least in part on the determined position of the wireless device relative to the three dimensional space of the immersive three dimensional group session comprises controlling components of the unique graphical output node based at least in part on the determined position of the wireless device relative to the three dimensional space of the immersive three dimensional group session and a determined orientation of the wireless device relative to the three dimensional space of the immersive three dimensional group session. [C9] Display and a processor coupled to the display; 11. A wireless device comprising: receiving a scene graph for an immersive three-dimensional group session, wherein the scene graph comprises at least a unique graphical output node assigned to be controlled by the wireless device and respective other graphical output nodes assigned to be controlled by each of a plurality of participant computing devices; controlling components of said unique graphical output node relative to the three-dimensional space of said immersive three-dimensional group session; sending the components of the unique graphical output node in a first media stream to others of the plurality of participant computing devices; and receiving the components of the other graphical output nodes in a media stream from each of the others of the plurality of participant computing devices. rendering the immersive three-dimensional group session on the display of the wireless device based at least in part on the components of the unique graphical output node and the components of the other graphical output nodes; 4. A wireless device comprising: [C10] The processor, receiving a scene graph update including an indication of a new participant computing device for the immersive three-dimensional group session and an indication of a new graphical output node assigned to be controlled by the new participant computing device; receiving, in a second media stream from the new participant computing device, components of the new graphical output node; rendering the immersive three-dimensional group session on the display of the wireless device based at least in part on the components of the original graphical output node, the components of the other graphical output nodes, and the components of the new graphical output node; 9. The wireless device of claim 8, further configured with processor-executable instructions for: [C11] The processor, receiving a Session Description Protocol (SDP) for the immersive three-dimensional group session indicating an address of a data channel over which the scene graph will be shared; further comprising processor-executable instructions for: The wireless device of C9, wherein the processor is further configured with processor-executable instructions for receiving the scene graph by downloading the scene graph over the data channel. [C12] The processor, sending an offer to send or receive the scene graph to the others of the plurality of participant computing devices as part of a Session Initiation Protocol (SIP) setup for the immersive three-dimensional group session; 9. The wireless device of claim 8, further configured with processor-executable instructions for: [C13] The processor, and further comprising processor-executable instructions for sending an offer to send or receive the scene graph to the other of the plurality of participant computing devices, wherein the offer indicates the unique graphical output node. 2. The wireless device according to claim 1 , [C14] The wireless device of C9, wherein the immersive three-dimensional group session is a Web Real-Time Communications (WebRTC) session. [C15] The method of C9, wherein the processor is further configured with processor-executable instructions for controlling components of the unique graphical output node relative to the three-dimensional space of the immersive three-dimensional group session by controlling components of the unique graphical output node based at least in part on a determined position of the wireless device relative to the three-dimensional space of the immersive three-dimensional group session. [C16] The wireless device of C15, further configured with processor-executable instructions for controlling components of the unique graphical output node based at least in part on the determined position of the wireless device with respect to the three-dimensional space of the immersive three-dimensional group session and a determined orientation of the wireless device with respect to the three-dimensional space of the immersive three-dimensional group session, the processor further configured with processor-executable instructions for controlling components of the unique graphical output node based at least in part on the determined position of the wireless device with respect to the three-dimensional space of the immersive three-dimensional group session and a determined orientation of the wireless device with respect to the three-dimensional space of the immersive three-dimensional group session. [C17] A non-transitory processor-readable medium having stored thereon processor-executable instructions, the processor-executable instructions causing a processor of a wireless device to: receiving a scene graph for an immersive three-dimensional group session, wherein the scene graph comprises at least a unique graphical output node assigned to be controlled by the wireless device and respective other graphical output nodes assigned to be controlled by each of a plurality of participant computing devices; controlling components of said unique graphical output node relative to the three-dimensional space of said immersive three-dimensional group session; sending the components of the unique graphical output node in a first media stream to others of the plurality of participant computing devices; and receiving the components of the other graphical output nodes in a media stream from each of the others of the plurality of participant computing devices. rendering the immersive three-dimensional group session on a display of the wireless device based at least in part on the components of the unique graphical output node and the components of the other graphical output nodes; 16. A non-transitory processor-readable medium configured to cause a processor to perform operations comprising: [C18] The stored processor-executable instructions may be configured to cause a processor of a wireless device to: receiving a scene graph update including an indication of a new participant computing device for the immersive three-dimensional group session and an indication of a new graphical output node assigned to be controlled by the new participant computing device; receiving, in a second media stream from the new participant computing device, components of the new graphical output node; rendering the immersive three-dimensional group session on the display of the wireless device based at least in part on the components of the original graphical output node, the components of the other graphical output nodes, and the components of the new graphical output node; 17. The non-transitory processor-readable medium of claim 17, configured to cause a processor to perform operations further comprising: [C19] The stored processor-executable instructions may be configured to cause a processor of a wireless device to: receiving a Session Description Protocol (SDP) for the immersive three-dimensional group session indicating an address of a data channel over which the scene graph will be shared; and configured to cause the device to perform operations further comprising: The processor-executable instructions are configured to cause a processor of a wireless device to perform an operation such that receiving the scene graph comprises downloading the scene graph over the data channel. 3. A non-transitory processor-readable medium as set forth in claim 2, wherein said processor-readable medium is a [C20] The stored processor-executable instructions cause a processor of a wireless device to: sending an offer to send or receive the scene graph to the others of the plurality of participant computing devices as part of a Session Initiation Protocol (SIP) setup for the immersive three-dimensional group session; 17. The non-transitory processor-readable medium of claim 17, configured to perform operations further comprising: [C21] The stored processor-executable instructions cause a processor of a wireless device to: sending an offer to send or receive the scene graph to the other of the plurality of participant computing devices, where the offer indicates the unique graphical output node; 3. A non-transitory processor-readable medium as set forth in claim 2, wherein said processor-readable medium is a [C22] The non-transitory processor-readable medium of C17, wherein the immersive three-dimensional group session is a Web Real-Time Communications (WebRTC) session. [C23] The non-transitory processor-readable medium of C17, wherein the stored processor-executable instructions are configured to cause a processor of a wireless device to perform operations such that controlling components of the unique graphical output node with respect to the three-dimensional space of the immersive three-dimensional group session comprises controlling components of the unique graphical output node based at least in part on a determined position of the wireless device with respect to the three-dimensional space of the immersive three-dimensional group session. [C24] The non-transitory processor-readable medium of C17, wherein the stored processor-executable instructions are configured to cause a processor of a wireless device to perform operations such as controlling components of the unique graphical output node based at least in part on the determined position of the wireless device relative to the three dimensional space of the immersive three dimensional group session, and controlling components of the unique graphical output node based at least in part on the determined position of the wireless device relative to the three dimensional space of the immersive three dimensional group session and a determined orientation of the wireless device relative to the three dimensional space of the immersive three dimensional group session. [C25] A wireless device, comprising: means for receiving a scene graph for an immersive three-dimensional group session, wherein said scene graph comprises at least a unique graphical output node assigned to be controlled by said wireless device and respective other graphical output nodes assigned to be controlled by each of a plurality of participant computing devices; means for controlling components of said unique graphical output node relative to the three-dimensional space of said immersive three-dimensional group session; means for sending the components of the unique graphical output node in a first media stream to others of the plurality of participant computing devices; means for receiving, in a media stream from each of the other ones of the plurality of participant computing devices, components of the other graphical output node; means for rendering the immersive three-dimensional group session on a display of the wireless device based at least in part on the components of the unique graphical output node and the components of the other graphical output nodes; A wireless device comprising: [C26] means for receiving a scene graph update including an indication of a new participant computing device for the immersive three-dimensional group session and an indication of a new graphical output node assigned to be controlled by the new participant computing device; means for receiving, in a second media stream from the new participant computing device, components of the new graphical output node; means for rendering the immersive three-dimensional group session on the display of the wireless device based at least in part on the components of the original graphical output node, the components of the other graphical output nodes, and the components of the new graphical output node; The wireless device of C25, further comprising: [C27] further comprising means for receiving a Session Description Protocol (SDP) for the immersive three-dimensional group session indicating an address of a data channel over which the scene graph is to be shared; The wireless device of C25, wherein the means for receiving the scene graph comprises means for downloading the scene graph over the data channel. [C28] means for sending an offer to send or receive the scene graph to the others of the plurality of participant computing devices as part of a Session Initiation Protocol (SIP) setup for the immersive three-dimensional group session; The wireless device of C25, further comprising: [C29] The method further comprises: means for sending an offer to send or receive the scene graph to the other one of the plurality of participant computing devices, wherein the offer indicates the unique graphical output node. 2. The wireless device according to claim 1 , [C30] The wireless device of C25, wherein the immersive three-dimensional group session is a Web Real-Time Communications (WebRTC) session.

Claims

1. 1. A method implemented by a processor of a wireless device that is one of a plurality of participant computing devices operating in an immersive three-dimensional group session, comprising: receiving a scene graph for an immersive three-dimensional group session, wherein said scene graph comprises at least a unique graphical output node assigned to be controlled by said wireless device and respective other graphical output nodes assigned to be controlled by each of others of said plurality of participant computing devices, said scene graph being received through a session description protocol during a session initiation protocol; controlling components of said unique graphical output node relative to the three-dimensional space of said immersive three-dimensional group session; sending the components of the unique graphical output node in a first media stream to the others of the plurality of participant computing devices; receiving, in a media stream from each of the other ones of the plurality of participant computing devices, components of the other graphical output node; rendering the immersive three-dimensional group session on a display of the wireless device based at least in part on the components of the unique graphical output node and the components of the other graphical output nodes; A method comprising:

2. receiving a scene graph update including an indication of a new participant computing device for the immersive three-dimensional group session and an indication of a new graphical output node assigned to be controlled by the new participant computing device; receiving, in a second media stream from the new participant computing device, components of the new graphical output node; rendering the immersive three-dimensional group session on the display of the wireless device based at least in part on the components of the original graphical output node, the components of the other graphical output nodes, and the components of the new graphical output node; The method of claim 1 further comprising:

3. receiving a Session Description Protocol (SDP) for the immersive three-dimensional group session indicating an address of a data channel over which the scene graph will be shared; Further equipped with The method of claim 1 , wherein receiving the scene graph comprises downloading the scene graph over the data channel.

4. sending an offer to send or receive the scene graph to the others of the plurality of participant computing devices as part of a Session Initiation Protocol (SIP) setup for the immersive three-dimensional group session; The method of claim 1 further comprising:

5. sending an offer to the other one of the plurality of participant computing devices to send or receive the scene graph, wherein the offer indicates the unique graphical output node. The method of claim 1.

6. The method of claim 1 , wherein the immersive three-dimensional group session is a Web Real-Time Communication (WebRTC) session.

7. Controlling components of the unique graphical output node with respect to the three dimensional space of the immersive three dimensional group session comprises controlling components of the unique graphical output node based at least in part on a determined position of the wireless device with respect to the three dimensional space of the immersive three dimensional group session, and optionally 2. The method of claim 1 , wherein controlling components of the unique graphical output node based at least in part on the determined position of the wireless device relative to the three-dimensional space of the immersive three-dimensional group session comprises controlling components of the unique graphical output node based at least in part on the determined position of the wireless device relative to the three-dimensional space of the immersive three-dimensional group session and a determined orientation of the wireless device relative to the three-dimensional space of the immersive three-dimensional group session.

8. A display and a processor coupled to the display; 11. A wireless device comprising: receiving a scene graph for an immersive three-dimensional group session, wherein said scene graph comprises at least a unique graphical output node assigned to be controlled by said wireless device and respective other graphical output nodes assigned to be controlled by each of a plurality of participant computing devices, said scene graph being received through a session description protocol during a session initiation protocol; controlling components of said unique graphical output node relative to the three-dimensional space of said immersive three-dimensional group session; sending the components of the unique graphical output node in a first media stream to others of the plurality of participant computing devices; receiving, in a media stream from each of the other ones of the plurality of participant computing devices, components of the other graphical output node; rendering the immersive three-dimensional group session on the display of the wireless device based at least in part on the components of the unique graphical output node and the components of the other graphical output nodes; 4. A wireless device comprising:

9. The processor, receiving a scene graph update including an indication of a new participant computing device for the immersive three-dimensional group session and an indication of a new graphical output node assigned to be controlled by the new participant computing device; receiving, in a second media stream from the new participant computing device, components of the new graphical output node; rendering the immersive three-dimensional group session on the display of the wireless device based at least in part on the components of the original graphical output node, the components of the other graphical output nodes, and the components of the new graphical output node; 10. The wireless device of claim 8, further comprising processor-executable instructions for:

10. The processor, receiving a Session Description Protocol (SDP) for the immersive three-dimensional group session indicating an address of a data channel over which the scene graph will be shared; further comprising processor-executable instructions for: The wireless device of claim 8 , wherein the processor is further configured with processor-executable instructions for receiving the scene graph by downloading the scene graph over the data channel.

11. The processor, sending an offer to send or receive the scene graph to the others of the plurality of participant computing devices as part of a Session Initiation Protocol (SIP) setup for the immersive three-dimensional group session; 10. The wireless device of claim 8, further comprising processor-executable instructions for:

12. The processor, sending an offer to the other of the plurality of participant computing devices to send or receive the scene graph, wherein the offer indicates the unique graphical output node; 9. The wireless device of claim 8.

13. The wireless device of claim 8 , wherein the immersive three-dimensional group session is a Web Real-Time Communications (WebRTC) session.

14. the processor is further configured with processor-executable instructions for controlling components of the unique graphical output node relative to the three-dimensional space of the immersive three-dimensional group session by controlling components of the unique graphical output node based at least in part on a determined position of the wireless device relative to the three-dimensional space of the immersive three-dimensional group session, and optionally 10. The wireless device of claim 8, wherein the processor is further configured with processor-executable instructions for controlling components of the unique graphical output node based at least in part on the determined position of the wireless device with respect to the three-dimensional space of the immersive three-dimensional group session by controlling components of the unique graphical output node based at least in part on the determined position of the wireless device with respect to the three-dimensional space of the immersive three-dimensional group session and a determined orientation of the wireless device with respect to the three-dimensional space of the immersive three-dimensional group session.

15. A non-transitory processor-readable medium having stored thereon processor-executable instructions, the processor-executable instructions causing a processor of a wireless device to: receiving a scene graph for an immersive three-dimensional group session, wherein said scene graph comprises at least a unique graphical output node assigned to be controlled by said wireless device and respective other graphical output nodes assigned to be controlled by each of a plurality of participant computing devices, said scene graph being received through a session description protocol during a session initiation protocol; controlling components of said unique graphical output node relative to the three-dimensional space of said immersive three-dimensional group session; sending the components of the unique graphical output node in a first media stream to others of the plurality of participant computing devices; receiving, in a media stream from each of the other ones of the plurality of participant computing devices, components of the other graphical output node; rendering the immersive three-dimensional group session on a display of the wireless device based at least in part on the components of the unique graphical output node and the components of the other graphical output nodes; 16. A non-transitory processor-readable medium configured to cause a processor to perform operations comprising:

Images