Signaling of scene descriptions for multimedia conferences

Abstract

To provide systems and methods for rendering an immersive three-dimensional group session.SOLUTION: A scene graph is signaled through a session description protocol (SDP) during a session initiation protocol (SIP) session setup. The scene graph includes respective graphical output nodes assigned to be controlled by a plurality of participant computing devices in an immersive three-dimensional group session.SELECTED DRAWING: Figure 4

Description

Claim of Priority

[0001] Related Applications

[0001] This application claims the benefit of priority of U.S. Provisional Application No. 62 / 990,895, filed on March 17, 2020, entitled "Signaling of Scene Description For Multimedia Conferencing", the entire content of which is incorporated herein by reference for all purposes.

Technical Field

[0002] Relates to systems and methods for providing an immersive three-dimensional group session.

Background Art

[0003]

[0002] Long Term Evolution (LTE (registered trademark)), 5th Generation (5G) New Radio (NR), and other recently developed communication technologies enable wireless devices to communicate information at data rates that are orders of magnitude greater than what was available just a few years ago (e.g., regarding gigabits per second, etc.).

[0004]

[0003] Today's communication networks are also more secure, resistant to multipath fading, enable lower network traffic latency, and provide better communication efficiency (e.g., regarding bits per second per unit of bandwidth used, etc.). These and other recent improvements have facilitated the emergence of technologies that rely on the Internet of Things (IoT), large-scale machine-to-machine (M2M) communication systems, autonomous vehicles, and other technologies that rely on consistent and secure communication.

[0005]

[0004] In recent years, augmented reality software applications that combine images of the real world from a user's physical environment with computer-generated images or virtual objects (VOs) have grown in popularity and use. Augmented reality software applications can add graphics, sound, and / or haptic feedback to the natural world around the user of the application. Images, video streams, and information about people and / or objects can be superimposed on the visual world as augmented scenes on a wearable electronic display or head-mounted device (e.g., smart glasses, augmented reality glasses, etc.) and presented to the user. [Overview of the project]

[0006]

[0005] Various embodiments include systems and methods for providing an immersive three-dimensional group session. Various embodiments may potentially include methods and devices for signaling a scene description with media components coming from different parties. In various embodiments, the scene graph may be signaled through a Session Description Protocol (SDP) during Session Initiation Protocol (SIP) session setup. In various embodiments, the scene graph may include each graphical output node assigned to be controlled by each of a plurality of participant computing devices in an immersive three-dimensional group session. Various embodiments may be implemented by a processor in a wireless device which is one of the plurality of participant computing devices operating in an immersive three-dimensional group session. Various embodiments may include receiving a scene graph for an immersive 3D group session, wherein the scene graph includes at least one own graphical output node assigned to be controlled by a wireless device and each of the other graphical output nodes assigned to be controlled by each of the other of a plurality of participant computing devices; controlling the components of the own graphical output node to the 3D space of the immersive 3D group session; sending the components of the own graphical output node in a first media stream to the other of the plurality of participant computing devices; receiving the components of the other graphical output nodes in media streams from each of the other of the plurality of participant computing devices; and rendering the immersive 3D group session on the display of the wireless device, at least in part, based on the components of the own graphical output node and the components of the other graphical output nodes.

[0007]

[0006] Some embodiments may further include receiving a scene graph update, which includes instructions for a new participant computing device for an immersive three-dimensional group session, and instructions for a new graphical output node assigned to be controlled by the new participant computing device; receiving components of a new graphical output node in a second media stream from the new participant computing device; and rendering the immersive three-dimensional group session on the display of a wireless device, at least in part, based on its own graphical output node components, other graphical output node components, and the new graphical output node components.

[0008]

[0007] Some embodiments may further include receiving a Session Description Protocol (SDP) for an immersive 3D group session that indicates the address of a data channel in which the scene graph will be shared, wherein receiving the scene graph includes downloading the scene graph via the data channel.

[0009]

[0008] Some embodiments may further include sending an offer to send or receive a scene graph to or from another of the multiple participant computing devices as part of setting up a Session Initiation Protocol (SIP) for an immersive 3D group session.

[0010]

[0009] Some embodiments may further include sending an offer to send or receive a scene graph to or from another of a plurality of participant computing devices, wherein the offer points to its own graphical output node.

[0011]

[0010] In some embodiments, the immersive 3D group session is a Web Real-Time Communication (WebRTC) session.

[0012]

[0011] In some embodiments, controlling the components of a unique graphical output node with respect to the three-dimensional space of an immersive three-dimensional group session may include controlling the components of a unique graphical output node based at least partially on the determined position of a wireless device with respect to the three-dimensional space of an immersive three-dimensional group session.

[0013]

[0012] In some embodiments, controlling the components of a custom graphical output node based at least partially on the determined position of the wireless device relative to the three-dimensional space of an immersive three-dimensional group session may include controlling the components of a custom graphical output node based at least partially on the determined position of the wireless device relative to the three-dimensional space of an immersive three-dimensional group session and the determined orientation of the wireless device relative to the three-dimensional space of an immersive three-dimensional group session.

[0014]

[0013] Further embodiments may include a wireless device having a processor configured to perform one or more operations of any of the methods summarized above. Further embodiments may include a non-temporary processor-readable storage medium storing processor-executable instructions configured to cause the processor of the wireless device to perform any of the operations of any of the methods summarized above. Further embodiments include a wireless device having means for performing any of the functions of any of the methods summarized above. Further embodiments include a system-on-a-chip for use in a wireless device including a processor configured to perform one or more operations of any of the methods summarized above. Further embodiments include a system in a package including two system-on-a-chips for use in a wireless device including processors configured to perform one or more operations of any of the methods summarized above.

[0015]

[0014] The accompanying drawings incorporated herein and forming part thereof illustrate exemplary embodiments of the claims and, together with the general description given above and the detailed description given below, serve to illustrate the features of the claims. [Brief explanation of the drawing]

[0016] [Figure 1A]

[0015] A system block diagram showing an exemplary communication system suitable for implementing various embodiments. [Figure 1B]

[0016] A diagram showing a head-mounted device (for example, augmented reality glasses) that can implement various embodiments. [Figure 2]

[0017] A block diagram of components illustrating an exemplary computing and wireless modem system suitable for implementing various embodiments. [Figure 3]

[0018] A diagram illustrating an example of a software architecture including a radio protocol stack for the user plane and control plane in wireless communication, according to various embodiments. [Figure 4]

[0019] A process flow diagram illustrating methods for supporting immersive experiences in teleconference or telepresence sessions through various embodiments. [Figure 5]

[0020] A process flow diagram illustrating methods for supporting immersive experiences in teleconference or telepresence sessions through various embodiments. [Figure 6]

[0021] A diagram showing the structure of a scene graph document in glTF 2.0. [Figure 7]

[0022] A diagram illustrating the structure of a scene graph. [Figure 8]

[0023] A call flow diagram illustrating the operation of supporting immersive experiences in teleconference or telepresence sessions through various embodiments. [Figure 9]

[0024] A process flow diagram showing a method for providing an immersive 3D group session according to various embodiments. [Figure 10]

[0025] A process flow diagram showing a method for providing an immersive 3D group session according to various embodiments. [Figure 11]

[0026] An exemplary server component block diagram suitable for implementing various embodiments. [Figure 12]

[0027] A wireless device component block diagram suitable for implementing various embodiments.

Best Mode for Carrying Out the Invention

Paragraph 0017

[0028] Various embodiments are described in detail while referring to the accompanying drawings. Whenever possible, the same reference numbers are used throughout the drawings to refer to the same or similar parts. References to specific examples and implementations are for purposes of illustration and are not intended to limit the claims.

Paragraph 0018

[0029] Various embodiments may enable an immersive 3D group session for multiple participant computing devices, wherein the scene graph may include a graphical output node assigned to each of the multiple participant computing devices to be controlled by each of them in the immersive 3D group session. Various embodiments may enable the multiple participant computing devices to share the media streams of the components of their respective assigned graphical output nodes with one another in the immersive 3D group session. By assigning each participant computing device control of its own respective graphical output node in the immersive 3D group session and sharing the media streams of the components of the graphical output nodes among the participant computing devices, various embodiments may support rendering an immersive 3D group session with a shared 3D space, in which each participant computing device controls its own respective 3D objects in the shared 3D space.

[0019]

[0030] The term “wireless device” is used herein to mean any one or all of the following: wireless router devices, wireless appliances, cellular phones, smartphones, portable computing devices, personal or mobile multimedia players, laptop computers, tablet computers, smartbooks, ultrabooks, palmtop computers, wireless email receivers, multimedia internet-enabled cellular phones, medical devices and equipment, biosensors / devices, wearable devices including smartwatches, smart clothing, smart glasses, smart wristbands, smart jewelry (e.g., smart rings, smart bracelets, etc.), entertainment devices (e.g., wireless gaming controllers, music and video players, satellite radios, etc.), wireless network-enabled Internet of Things (IoT) devices including smart meters / sensors, industrial manufacturing equipment, large and small machinery and equipment for home or business use, wireless communication elements in autonomous and semi-autonomous vehicles, wireless devices fixed to or embedded in various mobile platforms, global positioning system devices, and similar electronic devices including memory and wireless communication components and programmable processors.

[0020]

[0031] Various embodiments include systems utilizing any of the Institute of Electrical and Electronics Engineers (IEEE) 16.11 standards, or IEEE 802.15.4 protocols (e.g., Thread, ZigBee®, and Z-Wave), 6LoWPAN, Bluetooth® Low Energy (BLE), LTE Machine Type Communication (LTE MTC), Narrowband LTE (NB-LTE), Cellular IoT (CIoT), Narrowband IoT (NB-IoT), BT Smart, Wi-Fi® (e.g., Wi-Fi NAN), LTE-U, LTE-Direct, MuLTEfire, and a relatively extended range of wide-area physical layer interfaces (PHYs) such as Random Phase Multiple Access (RPMA), Ultra Narrowband (UNB), Low Power Long Range (LoRa), Low Power Long Range Wide Area Network (LoRaWAN), Weightless, or 3G, 4G, or 5G, cellular V2X, or further implementations thereof, or technologies. This can be implemented in devices capable of transmitting and receiving RF signals in accordance with any of the following wireless communication standards, including any of the 802.11 standards, Bluetooth standards (e.g., Bluetooth 4, Bluetooth 5, etc.), Code Division Multiple Access (CDMA), Frequency Division Multiple Access (FDMA), Time Division Multiple Access (TDMA), Global System for Mobile Communications (GSM®), GSM / General-Purpose Packet Radio Service (GPRS), Enhanced Data GSM Environment (EDGE), Terrestrial Infrastructure Radio (TETRA), Wideband CDMA (W-CDMA®), Evolution Data Optimized (EV-DO), 1xEV-DO, EV-DO Rev A, EV-DO Rev B, High-Speed ​​Packet Access (HSPA), High-Speed ​​Downlink Packet Access (HSDPA), High-Speed ​​Uplink Packet Access (HSUPA), Advanced High-Speed ​​Packet Access (HSPA+), Long-Term Evolution (LTE), AMPS, or any other known signals.

[0021]

[0032] The term “system on a chip” (SOC) is used herein to refer to a single integrated circuit (IC) chip that includes multiple resources and / or processors integrated on a single substrate. A single SOC may include circuitry for digital, analog, mixed-signal, and radio frequency functions. A single SOC may also include any number of general-purpose and / or dedicated processors (such as digital signal processors, modem processors, and video processors), memory blocks (e.g., ROM, RAM, and flash), and resources (e.g., timers, voltage regulators, and oscillators). A SOC may also include software for controlling the integrated resources and processors, as well as for controlling peripheral devices.

[0022]

[0033] The term “System in Package” (SIP) may be used herein to refer to a single module or package containing multiple resources, computing units, cores, and / or processors on two or more IC chips, substrates, or SOCs. For example, a SIP may include a single substrate on which multiple IC chips or semiconductor dies are stacked in a vertical configuration. Similarly, a SIP may include one or more multi-chip modules (MCMs) on which multiple ICs or semiconductor dies are packaged on an integrated substrate. A SIP may also include multiple independent SOCs coupled to each other via high-speed communication circuits and packaged in very close proximity, such as on a single motherboard or in a single wireless device. The proximity of the SOCs facilitates high-speed communication and the sharing of memory and resources.

[0023]

[0034] Various embodiments are described herein using the term “server” to refer to any computing device capable of functioning as a server, such as a master exchange server, web server, mail server, document server, content server, or any other type of server. A server may be a dedicated computing device or a computing device that includes a server module (for example, running an application that can make the computing device act as a server). The server module (for example, a server application) may be a full-function server module or a simplified or secondary server module (for example, a simplified or secondary server application) configured to provide synchronization services between dynamic databases on a receiver device. A simplified or secondary server is a miniaturized version of a server type of function that can be implemented on a receiver device, thereby enabling it to function as an internet server (for example, a corporate email server) only to the extent necessary to provide the functions described herein.

[0024]

[0035] The phrase “head-mounted device” and its acronym (HMD) are used herein to refer to an electronic display system that is wearable and presents a user with at least some computer-generated images. An HMD may present only computer-generated images, or a combination of computer-generated images and real-world images from the user’s physical environment (i.e., what the user would see without glasses). An HMD may enable the user to observe the generated images in the context of a real-world scene. Non-limiting examples of head-mounted devices include, or may be included in, helmets, glasses, virtual reality glasses, augmented reality glasses, electronic goggles, and other similar technologies / devices. A head-mounted device may include a variety of hardware elements, such as a processor, memory, a display, one or more cameras (e.g., a world-view camera, a gaze-view camera, etc.), and a wireless interface for connecting to the internet, a network, or another computing device. In some embodiments, the head-mounted device processor may be configured to implement or run an augmented reality software application.

[0025]

[0036] In some embodiments, the head-mounted device is an accessory for a wireless device (e.g., a desktop, laptop, smartphone, tablet computer, etc.) and / or can receive information from the wireless device, and all or part of the processing is performed on the processor of that wireless device. Thus, in various embodiments, the head-mounted device may be configured to perform all processing locally on the processor in the head-mounted device, offload all of the major processing to a processor in another computing device (e.g., a laptop located in the same space as the head-mounted device), or split the major processing operations between the processor in the head-mounted device and the processor in the other computing device. In some embodiments, the processor in the other computing device may be a server in the “cloud” with which the processor in the head-mounted device or the associated wireless device communicates via a network connection (e.g., a cellular network connection to the internet).

[0026]

[0037] Telepresence services are becoming more sophisticated, enabling the synthesis of several objects into a single immersive environment where meeting participants can navigate and interact more freely. Current Session Description Protocols (SDPs) are limited in their support for describing rich synthesis and do not provide the tools to support immersive telepresence environments. One particular type of telepresence or teleconferencing implementation is an immersive 3D group session. In an immersive 3D group session, each participant computing device in the session may render a 3D graphical display of the session on its respective display so that each participant user in the immersive 3D group session is presented with a virtual reality (VR) view of the 3D space of the immersive 3D group session. 3D objects, such as avatars and characters representing participants in the immersive 3D group session, may be observed by each participant and appear to move within the 3D space of the immersive 3D group session.

[0027]

[0038] Support for Immersive Teleconferencing and Telepresence for Remote Terminals (ITT4RT) is a standard being developed to facilitate immersive multi-computing device virtual reality (VR) video conferencing, such as immersive 3D group sessions. ITT4RT use cases include, for example, the compositing of captured VR video from a conference room with other content, such as 2D video slides. The ITT4RT work item description indicates that ITT4RT is working towards enabling scenarios involving two-way audio and one-way immersive video, where, for example, a single remote user wearing an HMD and participating in a meeting would send audio and, optionally, 2D video (e.g., the user's own presentation, screen sharing, and / or capture).

[0028]

[0039] The complexity of VR scenes presents challenges for conventional Session Description Protocol (SDP) signaling, which can quickly become unmanageable, for example, when many remote users share their own content and composite it into the scene. SDP is not simply designed to carry scene description information. Various embodiments offer solutions to such problems by providing scene description-based implementation solutions to support compositing and overlay. Various embodiments potentially provide methods and devices for signaling scene descriptions with media components coming from different parties. In various embodiments, scene descriptions may be signaled through SDP during Session Initiation Protocol (SIP) session setup. In various embodiments, scene descriptions may be linked to other media streams in the session, for example, to be used as textures for overlays in an immersive meeting scene.

[0029]

[0040] A scene graph is a directed acyclic graph, typically a simple tree structure, that represents an object-based hierarchy of the scene's geometry. The leaf nodes of the graph represent geometric primitives such as polygons. Each node in the graph holds a pointer to its children. Child nodes can be, in particular, groups of other nodes, geometric elements, transformation matrices, etc. Spatial transformations are attached to the nodes of the graph and represented by transformation matrices. This structure of the scene graph has the advantage of reduced processing complexity, such as when traversing the graph for rendering. An exemplary behavior simplified by the graph representation is the sorting behavior in which a branch of the graph is dropped from processing if the parent node's space is not visible or is considered irrelevant to the rendering of the current view frustum (called the level of detail sorting). A scene graph can contain various types of nodes, such as visual output nodes, audio source nodes, graphical output nodes, and shared content nodes. As a specific example, a graphical output node might define a three-dimensional object to be output into the three-dimensional space defined by the scene graph.

[0030]

[0041] The Graphics Library (GL) Transmission Format (TF) (glTF) 2.0 (glTF2.0) is a new standard developed by Khronos to enable physically based rendering. glTF2.0 provides a compact and low-level representation of scene graphs. glTF2.0 provides a flat hierarchy of scene graph representations to simplify processing. The glTF2.0 scene graph is represented in JavaScript® Object Notation (JSON) to facilitate integration in web environments. The glTF2.0 specification is designed to eliminate redundancy in representations and provide efficient indexing of different objects in the scene graph. The Moving Picture Expert Group (MPEG) is working on extensions to glTF2.0 to add support for real-time media, scene updates, and other features.

[0031]

[0042] A scene graph can, in various embodiments, enable the synthesis of scenes (also called spaces, such as three-dimensional space) for immersive presentations, such as immersive three-dimensional group sessions. In some embodiments, synthesis may be performed in a call server, such as a multimedia resource function (MRF), a multipoint communication unit (MCU), or a telepresence application server. Alternatively, in some embodiments, a designated computing device participating in a meeting may be responsible for creating the initial scene graph and sharing the scene graph with all other parties in the call (e.g., an immersive three-dimensional group session). This computing device may be the one creating the main VR content, such as a computing device in a conference room using VR capture. In some embodiments, each computing device participating in the meeting (e.g., an immersive three-dimensional group session) may contribute one or more nodes to the scene graph. In some embodiments, each node may identify or be assigned its associated transformations (e.g., in the form of matrices or individual transformations and rotations) in order to properly position the node in the scene (or space), for example, in three-dimensional space.

[0032]

[0043] In some embodiments, each computing device participating in a conference call (e.g., an immersive 3D group session) may make offers to send and receive scene graphs through session-level attributes. In some embodiments, the offer may point to one or more unique graphical output nodes owned by the computing device sending the offer. As an example, each computing device participating in a conference call (e.g., an immersive 3D group session) may make offers to send and receive scene graphs through the following session-level attributes given in augmented Backus-Naur Form (ABNF) syntax: Session-Description=“a=scene-description:”SP mime-type[SP uri][SP sent-nodes]CRLFmime-type=“mime-type:”byte-stringsent-nodes=“nodes-owned=1*(byte-string“;”)uri=“websocket-uri:”URI .

[0044] In some embodiments, the Uniform Resource Indicator (URI) parameter may be a WebSocket URI for the data channel through which the scene graph will be shared and updated. Alternatively, an application media session may be used with a protocol identifier, such as TCP / WSS / SD (Transmission Control Protocol / Websocket Secure / Session Description).

[0033]

[0045] In some embodiments, the WebSocket URI may be provided according to the syntax and offer / answer negotiation defined in Internet Engineering Task Force (IETF) Request for Comment (RFC) 8124. An exemplary schema for such WebSocket URI provisioning may be as follows: m=application 50000 TCP / WSS / SD *a=setup:passive a=connection:newa=websocket-uri:wss: / / mrf.operator.com / call / 21323asd23a=mime-type:model / gltf+jsona=nodes-owned:node12,node13,node14

[0046] In various embodiments, the scene graph refers to media streams from a meeting session used as constituent elements of nodes in a scene (e.g., three-dimensional space). One example is a video stream of meeting participants that will be displayed within a rectangular area in a 3D scene (also called three-dimensional (3D) space). For example, the following URI format may be used:

[0034] url=”rtp: / / ”fqdn_or_ip“ / ”call_id“ / ”ssrc“ / ”mid

[0047] In this URI format, "fqdn_or_ip" represents the domain name or Internet Protocol (IP) address of the MRF or SIP proxy managing the call (e.g., an immersive 3D group session). If the MRF or SIP proxy does not manage the call (e.g., an immersive 3D group session), "fqdn_or_ip" may represent the domain name or IP address of the SIP address of the host of the call (e.g., an immersive 3D group session). "call_id" provides a unique identifier for the current call or conference (e.g., the current immersive 3D group session). "ssrc" represents the synchronization source of the media stream owner / sending participant. Finally, "mid" represents the media session identifier provided in the SDP. Other forms of addressing can be defined, for example, as a Uniform Resource Name (URN).

[0035]

[0048] In some embodiments, when using Web Real-Time Communication (WebRTC), the session setup protocol may be left to the application. Some implementations rely on SIP over WebSocket for this purpose. However, other protocols may be used to set up and describe calls (e.g., immersive 3D group sessions).

[0036]

[0049] In some embodiments, a scene graph may be used as an entry point to a conference call (e.g., an immersive 3D group session). In such embodiments, all participants share a scene graph document that sets up a 3D scene (or 3D space) at the start of the call (e.g., an immersive 3D group session). The scene graph will define graph nodes for each participant and identify the components for which each participant needs to provide a media stream. This may be done by a central call server, such as an MRF, to which all conference participants connect. The MRF may be configured to update the scene graph during the call (e.g., an immersive 3D group session), for example, to add new nodes for newly joining or leaving call participants, or to remove nodes.

[0037]

[0050] In some embodiments, participant computing devices may use a link to a webpage provided by the call server to join a WebRTC conference. The call server may provide the participant computing devices with that webpage along with a scene graph file that sets up the initial / default configuration of the call participants and materials in 3D space (for example, each participant will be assigned a visual node, an audio source node, and potentially nodes for graphics and other shared content). Each participant computing device may add or modify the nodes it owns in the scene graph. Media streams that provide the building blocks for the nodes in the scene graph may be streamed using WebRTC. These streams may be exchanged directly or through a server such as a media proxy server.

[0038]

[0051] In some embodiments, computing devices participating in an ITT4RT conference may establish direct peer-to-peer WebSocket channels with each other, or connections may be offered to all parties by the MRF. In such embodiments, the WebSocket channel may use a text frame format. In a scene (for example, in 3D space), node names may be unique and may be declared in the SDP to ensure there are no naming conflicts in nodes provided by different computing devices during a call. In such embodiments, nodes in the scene graph may refer to external media streams, such as other media streams declared in the SDP. In such embodiments, the receiver may mask nodes from some computing devices during the rendering process, such as based on user input.

[0039]

[0052] In some embodiments, the MRF may, by default, be the owner of the master scene graph, which is a computing device that sets the coordinate system and from which all other nodes are composited. In some embodiments, the MRF may also be a computing device that defines the primary camera in the scene (or space). In some embodiments, if there is no centralized MRF, the computing device in a call may select one computing device to provide the primary scene graph, for example, by selecting the computing device that provides the VR content or the organizer of the call. In various embodiments, an overlay may be a 2D or 3D object placed in the scene (or space). In some embodiments, the geometry of an overlay and its texture may be defined by a node corresponding to that overlay object. A simple example is a set of slides played in a rectangular area shown in a VR scene (or VR space). In this example, the geometry may be a rectangle, and the texture may come from a video media stream. The rectangle may be placed in the scene (or space). In a viewport-dependent overlay, the position of the rectangle may be locked in the camera direction.

[0040]

[0053] Figure 1A shows an example of a communication system 100 suitable for implementing various embodiments. The communication system 100 may be a 5G NR network or any other suitable network such as an LTE network.

[0041]

[0054] The communication system 100 may include a heterogeneous network architecture that includes a core network 140 and various wireless devices (also called user equipment (UE) computing devices) (shown as wireless devices 120a-120e in Figure 1). The communication system 100 may also include several base stations (shown as BS110a, BS110b, BS110c, and BS110d) and other network entities. A base station is an entity that communicates with wireless devices (wireless devices or UE computing devices) and may also be called a node B, LTE evolved node B (eNB), access point (AP), radio head, transmit / receive point (TRP), new radio base station (NR BS), 5G node B (NB), next-generation node B (gNB), etc. Each base station may provide communication coverage to a specific geographic area. In 3GPP®, the term “cell” can refer to the coverage area of ​​a base station, the coverage area of ​​a base station subsystem that serves that coverage area, or a combination thereof, depending on the context in which the term is used.

[0042]

[0055] Base stations 110a-110d may provide communication coverage to macrocells, picocells, femtocells, other types of cells, or combinations thereof. Macrocells may cover relatively large geographical areas (e.g., a radius of several kilometers) and may enable unrestricted access by wireless devices subscribed to the service. Picocells may cover relatively small geographical areas and may enable unrestricted access by wireless devices subscribed to the service. Femtocells may cover relatively small geographical areas (e.g., a home) and may enable limited access by wireless devices associated with a femtocell (e.g., wireless devices in a limited subscriber group (CSG)). Base stations for macrocells are sometimes called macro BS. Base stations for picocells are sometimes called pico BS. Base stations for femtocells are sometimes called femto BS or home BS. In the example shown in Figure 1A, base station 110a may be a macro BS for macrocell 102a, base station 110b may be a pico BS for picocell 102b, and base station 110c may be a femto BS for femtocell 102c. Base stations 110a to 110d may support one or more (e.g., three) cells. The terms “eNB”, “base station”, “NR BS”, “gNB”, “TRP”, “AP”, “Node B”, “5G NB”, and “cell” may be used interchangeably herein.

[0043]

[0056] In some examples, cells may not be fixed, and the geographical area of ​​a cell may move according to the location of the mobile base station. In some examples, base stations 110a-110d may be interconnected with each other and with one or more other base stations or network nodes (not shown) in the communication system 100 through various types of backhaul interfaces, such as direct physical connections, virtual networks, or a combination thereof, using any suitable transport network.

[0044]

[0057] Base stations 110a to 110d can communicate with the core network 140 via wired or wireless communication link 126. Wireless devices 120a to 120e (UE computing devices) can communicate with base stations 110a to 110d via wireless communication link 122.

[0045]

[0058] The wired communication link 126 may use a variety of wired networks (e.g., Ethernet, TV cable, telephone, fiber optic, and other forms of physical network connections) that may use one or more wired communication protocols, such as Ethernet®, Point-to-Point Protocol, High-Level Data Link Control (HDLC), Advanced Data Communication Control Protocol (ADCCP), and Transmission Control Protocol / Internet Protocol (TCP / IP).

[0046]

[0059] The communication system 100 may also include a relay station (e.g., relay BS110d). A relay station is an entity that can receive data transmissions from an upstream station (e.g., a base station or wireless device) and send data transmissions to a downstream station (e.g., a wireless device or base station). A relay station may also be a wireless device that can relay transmissions for other wireless devices. In the example shown in Figure 1, relay station 110d may communicate with base station 110a and wireless device 120d to facilitate communication between macro base station 110a and wireless device 120d. A relay station may also be called a relay base station, relay station, or relay.

[0047]

[0060] The communication system 100 may be a heterogeneous network including different types of base stations, such as macro base stations, pico base stations, femto base stations, and relay base stations. These different types of base stations may have different transmission power levels, different coverage areas, and different impacts on interference within the communication system 100. For example, macro base stations may have high transmission power levels (e.g., 5 to 40 watts), while pico base stations, femto base stations, and relay base stations may have lower transmission power levels (e.g., 0.1 to 2 watts).

[0048]

[0061] The network controller 130 may be coupled to a set of base stations and may coordinate and control these base stations. The network controller 130 may communicate with base stations via backhaul. Base stations may also communicate with each other directly or indirectly, for example, via wireless or wireline backhaul.

[0049]

[0062] Wireless devices (UE computing devices) 120a, 120b, and 120c may be distributed throughout the entire communication system 100, and each wireless device may be fixed or mobile. Wireless devices may also be called access terminals, UEs, terminals, mobile stations, subscriber units, stations, etc.

[0050]

[0063] Macro base station 110a can communicate with communication network 140 via wired or wireless communication link 126. Wireless devices 120a, 120b, and 120c can communicate with base stations 110a to 110d via wireless communication link 122. Core network 140 can be connected to other devices, such as a call server 150 (e.g., a multimedia resource function (MRF), a multipoint communication unit (MCU), a telepresence application server, etc.). In this way, via its connection to core network 140, call server 150 can make telepresence services, such as immersive teleconferencing and telepresence (ITT4RT) services for remote terminals, available to wireless devices 120a, 120b, 120c, and 120d (e.g., from core network 140 via link 126 and from base stations 110a to 110d via link 122). Although shown as being outside the core network 140, the call server 150 may be part of the core network 140 itself.

[0051]

[0064] Wireless communication links 122 and 124 may include multiple carrier signals, frequencies, or frequency bands, each of which may include multiple logical channels. Wireless communication links 122 and 124 may utilize one or more radio access technologies (RATs). Examples of RATs that may be used in wireless wide area network (WWAN) wireless communication links 122 and 124 within the distributed communication system 100 include 3GPP LTE, 3G, 4G, 5G (e.g., NR), GSM, Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA®), Worldwide Interoperability for Microwave Access (WiMAX®), Time Division Multiple Access (TDMA), and other mobile telephone communication technology cellular RATs. Examples of RATs that may be used in wireless communication links 122 and 124 of the wireless local area network (WLAN) within the local communication system 100 include medium-range wireless protocols such as Wi-Fi, LTE-U, LTE-Direct, LAA, and MuLTEfire, as well as relatively short-range RATs such as ZigBee, Bluetooth, and Bluetooth Low Energy (LE).

[0052]

[0065] Some wireless networks (e.g., LTE) utilize orthogonal frequency division multiplexing (OFDM) on the downlink and single-carrier frequency division multiplexing (SC-FDM) on the uplink. OFDM and SC-FDM divide the system bandwidth into several (K) orthogonal subcarriers, commonly called tones or bins. Each subcarrier can be modulated with data. Generally, the modulation symbols are transmitted in the frequency domain in OFDM and in the time domain in SC-FDM. The spacing between adjacent subcarriers can be fixed, and the total number of subcarriers (K) can depend on the system bandwidth. For example, the subcarrier spacing can be 15 kHz, and the minimum resource allocation (called a "resource block") can be 12 subcarriers (or 180 kHz). Therefore, the nominal fast file transfer (FFT) sizes can be equal to 128, 256, 512, 1024, or 2048 for system bandwidths of 1.25, 2.5, 5, 10, or 20 megahertz (MHz), respectively. The system bandwidth can also be divided into subbands. For example, a subband may cover 1.08 MHz (i.e., 6 resource blocks), and there may be 1, 2, 4, 8, or 16 subbands for system bandwidths of 1.25, 2.5, 5, 10, or 20 MHz, respectively.

[0053]

[0066] While some embodiments may use terminology and examples related to LTE technology, various embodiments may be applicable to other wireless communication systems, such as New Radio (NR) or 5G networks. NR may include support for half-duplex operation using Time Division Duplex (TDD) and utilizing OFDM with cyclic prefixes (CP) on the uplink (UL) and downlink (DL). A single component carrier bandwidth of 100 MHz may be supported. An NR resource block may span 12 subcarriers with a subcarrier bandwidth of 75 kHz over a duration of 0.1 ms. Each radio frame may consist of 50 subframes, each with a length of 10 ms. Thus, each subframe may have a length of 0.2 ms. Each subframe may indicate the link direction for data transmission (i.e., DL or UL), and the link direction for each subframe may be dynamically switched. Each subframe may contain DL / UL data and DL / UL control data. Beamforming may be supported, and the beam direction may be dynamically configured. Multiple-input multiple-output (MIMO) transmission using precoding may also be supported. MIMO configurations in DL may support up to eight transmitting antennas with multi-layer DL transmission of up to eight streams and up to two streams per wireless device. Multi-layer transmission using up to two streams per wireless device may be supported. Aggregation of multiple cells may be supported with up to eight serving cells. Alternatively, NR may support different air interfaces other than OFDM-based air interfaces.

[0054]

[0067] Some wireless devices may be considered machine-type communications (MTC) wireless devices or advanced or enhanced machine-type communications (eMTC) wireless devices. MTC and eMTC wireless devices include, for example, robots, drones, remote devices, sensors, meters, monitors, and location tags that can communicate with base stations, other devices (e.g., remote devices), or any other entities. Wireless nodes may provide connectivity to or for a network (e.g., a wide area network such as the Internet or a cellular network) via wired or wireless communication links. Some wireless devices may be considered Internet of Things (IoT) devices or may be implemented as NB-IoT (Narrowband Internet of Things) devices. Wireless devices 120a-e may be contained within a housing that houses the components of the wireless device, such as processor components, memory components, similar components, or combinations thereof.

[0055]

[0068] Generally, any number of communication systems and wireless networks can be deployed within a given geographical area. Each communication system and wireless network may support a specific radio access technology (RAT) and may operate on one or more frequencies. RATs are sometimes called radio technologies or air interfaces. Frequencies are sometimes called carriers or frequency channels. Each frequency may support a single RAT within a given geographical area to avoid interference between communication systems of different RATs. In some cases, NR or 5G RAT networks may be deployed.

[0056]

[0069] In some implementations, two or more wireless devices 120a-e (indicated, for example, as wireless device 120a and wireless device 120e) may communicate directly using one or more sidelink channels 124 (for example, without using base stations 110a-110d as intermediaries for communication with each other). For example, wireless devices 120a-e may communicate using peer-to-peer (P2P) communication, device-to-device (D2D) communication, vehicle-to-anything (V2X) protocols (which may include vehicle-to-vehicle (V2V) protocols, vehicle-to-infrastructure (V2I) protocols, or similar protocols), mesh networks, or similar networks, or a combination thereof. In this case, wireless devices 120a-e may perform scheduling operations, resource selection operations, and other operations described elsewhere in this specification as being performed by base station 110a.

[0057]

[0070] Figure 1B shows a head-mounted device 172 that can be configured according to various embodiments. With respect to Figures 1A and 1B, in the example shown in Figure 1A, the head-mounted device 172 may be a specific implementation of a user-device computing device (e.g., UE120c, 120d, 120e). The head-mounted device 172 includes a frame 152, two optical lenses 154, an outward-facing world view image sensor / camera 158, an inward-facing gaze view sensor / camera 160, a sensor array 162, memory 164, and a processor 156 communicatively coupled to a communication circuit 166. In various embodiments, the communication circuit 166 may support one or more RATs to support communication between various devices, as described in system 100 with respect to Figure 1A. In some embodiments, the head-mounted device 172 may include a capacitance touch sensing circuit along the arm 180 of the frame or in the nose bridge 182 of the head-mounted device 172. In some embodiments, the head-mounted device 172 may also include sensors for monitoring physical conditions (e.g., location, motion, acceleration, orientation, altitude, etc.). The sensors may include any or all of the following: gyroscopes, accelerometers, magnetometers, magnetic compasses, altimeters, odometers, and pressure sensors. The sensors may also include various biosensors (e.g., heart rate monitors, body temperature sensors, carbon sensors, oxygen sensors, etc.) for collecting information relating to the environment and / or user state. The sensors may also be external to the head-mounted device 172 and may be paired or grouped with the head-mounted device 172 via wired or wireless connectivity (e.g., Bluetooth, etc.).

[0058]

[0071] In some embodiments, the processor 156 may also be communicatively coupled to an image rendering device 168 (e.g., an image projector), which may be embedded in an arm portion 180 of the frame 152 and configured to project an image onto the optical lens 154. In some embodiments, the image rendering device 168 may include a light-emitting diode (LED) module, an optical tunnel, a homogenizing lens, an optical display, a fold mirror, or other well-known components of a projector or head-mounted display. In some embodiments (e.g., embodiments that do not include or do not use the image rendering device 168), the optical lens 154 may be or include a see-through or partially see-through electronic display. In some embodiments, the optical lens 154 includes an image-generating element, such as a see-through organic light-emitting diode (OLED) display element or a liquid crystal on silicon (LCOS) display element. In some embodiments, the optical lens 154 may include separate left and right eye display elements. In some embodiments, the optical lens 154 may include or act as an optical guide for delivering light from the display element to the wearer's eyes.

[0059]

[0072] An outward-facing or world-view image sensor / camera 158 may be configured to capture real-world images from the user's physical environment and send the corresponding image data to a processor 156. The processor 156 may combine the real-world images with computer-generated images or virtual objects (VOs) to generate an augmented scene (or space) and render the augmented scene (or space) on the electronic display or optical lens 154 of the head-mounted device 172.

[0060]

[0073] The inward-facing or gaze-view sensor / camera 160 may be configured to collect image data from the user's eyes or the facial structure around the user's eyes.

[0061]

[0074] Various embodiments can be implemented on several single-processor and multi-processor computer systems, including system-on-chip (SOC) or system-in-package (SIP). Figure 2 shows an exemplary computing system or SIP200 architecture that may be used in wireless devices (UE computing devices) implementing various embodiments.

[0062]

[0075] Referring to Figures 1A, 1B, and 2, the illustrated exemplary SIP200 includes two SOCs 202 and 204, a clock 206, a voltage regulator 208, and one or more wireless transceivers 266 configured to send and receive wireless communications via antennas (not shown) to and from network wireless devices such as base stations 110a and / or other wireless devices (e.g., wireless devices 120a-e). In some embodiments, the first SOC 202 acts as the central processing unit (CPU) of the wireless device, executing instructions for a software application program by performing arithmetic, logical, control, and input / output (I / O) operations specified by the instructions. In some embodiments, the second SOC 204 may act as a dedicated processing unit. For example, the second SOC 204 may act as a dedicated 5G processing unit responsible for managing high-volume, high-speed (e.g., 5 Gbps, etc.) and / or ultra-high-frequency short-wavelength (e.g., 28 GHz mmWave spectrum, etc.) communications. In some embodiments, the wireless transceiver 266 may be configured to support peer-to-peer (P2P) communication, device-to-device (D2D) communication, vehicle-to-anything (V2X) protocols (which may include vehicle-to-vehicle (V2V) protocols, vehicle-to-infrastructure (V2I) protocols, or similar protocols), Bluetooth communication, Wi-Fi communication, and the like. In some embodiments, each wireless transceiver 266 may be connected to a first SOC 202, and / or a second SOC 204 may be connected to each of one or more wireless transceivers 266 by various physical connections 267 (also called interconnects, buses, etc.), such as Peripheral Components Interconnect Express (PCIe) connections, Universal Serial Bus (USB) connections, High-Speed ​​Chip-to-Chip (HSIC) connections, Ethernet connections, etc.

[0063]

[0076] The first SOC 202 may include a digital signal processor (DSP) 210, a modem processor 212, a graphics processor 214, an application processor 216, one or more coprocessors 218 (e.g., vector coprocessors) connected to one or more of the processors, memory 220, custom circuitry 222, system components and resources 224, an interconnect / bus module 226, one or more temperature sensors 230, a thermal management unit 232, and a thermal power envelope (TPE) component 234. The second SOC 204 may include a 5G modem processor 252, a power management unit 254, an interconnect / bus module 264, multiple mmWave transceivers 256, memory 258, and various additional processors 260 such as an application processor and a packet processor.

[0064]

[0077] Each processor 210, 212, 214, 216, 218, 252, 260 may include one or more cores, and each processor / core may operate independently of other processors / cores. For example, the first SOC202 may include processors running a first type of operating system (e.g., FreeBSD, LINUX®, OS X, etc.) and processors running a second type of operating system (e.g., MICROSOFT WINDOWS® 10). In addition, any or all of processors 210, 212, 214, 216, 218, 252, 260 may be included as part of a processor cluster architecture (e.g., a synchronous processor cluster architecture, an asynchronous or heterogeneous processor cluster architecture, etc.).

[0065]

[0078] The first SOC202 and the second SOC204 may include various system components, resources, and custom circuits for managing sensor data, analog-to-digital conversion, wireless data transmission, and for performing other specialized operations such as decoding data packets and processing encoded audio and video signals for rendering in a web browser. For example, the system components and resources 224 of the first SOC202 may include power amplifiers, voltage regulators, oscillators, phase-locked loops, peripheral bridges, data controllers, memory controllers, system controllers, access ports, timers, and other similar components used to support the processor and software client running on the wireless device. The system components and resources 224 and / or custom circuits 222 may also include circuits for interfaced with peripheral devices such as cameras, electronic displays, wireless communication devices, and external memory chips.

[0066]

[0079] The first SOC 202 and the second SOC 204 may communicate via the interconnect / bus module 250. Various processors 210, 212, 214, 216, and 218 may be interconnected via the interconnect / bus module 226 to one or more memory elements 220, system components and resources 224, custom circuits 222, and a thermal management unit 232. Similarly, processor 252 may be interconnected via the interconnect / bus module 264 to a power management unit 254, a mmWave transceiver 256, memory 258, and various additional processors 260. The interconnect / bus modules 226, 250, and 264 may include arrays of reconfigurable logic gates and / or implement a bus architecture (e.g., CoreConnect, AMBA, etc.). Communication may be provided by advanced interconnects such as high-performance network-on-chip (NoC).

[0067]

[0080] The first SOC202 and / or the second SOC204 may further include input / output modules (not shown) for communicating with resources outside the SOC, such as a clock 206 and a voltage regulator 208. These external resources (e.g., clock 206, voltage regulator 208) may be shared by two or more of the internal SOC processors / cores.

[0068]

[0081] In addition to the exemplary SIP200 described above, various embodiments can be implemented in a wide variety of computing systems, which may include a single processor, multiple processors, a multi-core processor, or any combination thereof.

[0069]

[0082] Figure 3 shows an example of a software architecture 300 that includes a radio protocol stack for the user plane and control plane in wireless communication between a base station 350 (e.g., base station 110a) and wireless devices (UE computing devices) 320 (e.g., wireless devices 120a-120e, 172, 200). Referring to Figures 1A-3, a wireless device 320 may implement the software architecture 300 to communicate with a base station 350 of a communication system (e.g., 100). In various embodiments, layers in the software architecture 300 may form logical connections with corresponding layers in the software of the base station 350. The software architecture 300 may be distributed among one or more processors (e.g., processors 212, 214, 216, 218, 252, 260). While the description is given for one radio protocol stack, in a multi-SIM (subscriber identification module) wireless device, the software architecture 300 may include multiple protocol stacks, each of which may be associated with a different SIM (for example, in a dual-SIM wireless communication device, there may be two protocol stacks, each associated with two SIMs). Although the LTE communication layer is described below, the software architecture 300 may support any of the various standards and protocols for wireless communication and / or may include additional protocol stacks that support any of the various standards and protocols for wireless communication.

[0070]

[0083] The software architecture 300 may include a non-access layer (NAS) 302 and an access layer (AS) 304. The NAS 302 may include functions and protocols for supporting packet filtering, security management, mobility control, session management, and traffic and signaling between one or more SIMs (e.g., one or more SIMs 204) of a wireless device and its core network 140. The AS 304 may include functions and protocols for supporting communication between one or more SIMs (e.g., one or more SIMs 204) and entities of a supported access network (e.g., base stations). In particular, the AS 304 may include at least three layers (Layer 1, Layer 2, and Layer 3), each of which may include various sublayers.

[0071]

[0084] In the user plane and control plane, the AS304's Layer 1 (L1) may be the Physical Layer (PHY) 306, which can oversee functions that enable transmission and / or reception over the air interface. Examples of such Physical Layer 306 functions may include cyclic redundancy check (CRC) attachments, coding blocks, scrambling and descrambling, modulation and demodulation, signal measurement, MIMO, etc. The Physical Layer may include various logical channels, including Physical Downlink Control Channels (PDCCHs) and Physical Downlink Shared Channels (PDSCHs).

[0072]

[0085] In the user plane and control plane, Layer 2 (L2) of AS304 may be responsible for the link between the wireless device 320 on physical layer 306 and the base station 350. In various embodiments, Layer 2 may include a Media Access Control (MAC) sublayer 308, a Radio Link Control (RLC) sublayer 310, and a Packet Data Convergence Protocol (PDCP) 312 sublayer, each of which forms a logical connection that terminates at the base station 350.

[0073]

[0086] In the control plane, Layer 3 (L3) of AS304 may include a Radio Resource Control (RRC) sublayer 3. Although not shown, the software architecture 300 may include additional Layer 3 sublayers, as well as various higher layers above Layer 3. In various embodiments, the RRC sublayer 313 may provide functions including broadcasting system information, paging, and establishing and releasing RRC signaling connections between wireless devices 320 and base stations 350.

[0074]

[0087] In various embodiments, the PDCP sublayer 312 may provide uplink functions including multiplexing between different radio bearers and logical channels, sequence numbering, handover data processing, integrity protection, encryption, and header compression. On the downlink, the PDCP sublayer 312 may provide functions including sequential distribution of data packets, duplicate data packet detection, integrity verification, decryption, and header restoration.

[0075]

[0088] On the uplink, the RLC sublayer 310 may provide segmentation and concatenation of upper-layer data packets, retransmission of lost data packets, and automatic retransmission requests (ARQ). On the downlink, the RLC sublayer 310 functionality may include reordering data packets to compensate for out-of-order reception, reassembly of upper-layer data packets, and ARQ.

[0076]

[0089] In the uplink, MAC sublayer 308 may provide functions including multiplexing between logical and transport channels, random access procedures, logical channel prioritization, and hybrid ARQ (HARQ) operation. In the downlink, MAC layer functions may include intra-cell channel mapping, multiplexing / decoupling, intermittent reception (DRX), and HARQ operation.

[0077]

[0090] While the software architecture 300 may provide functionality for transmitting data through a physical medium, the software architecture 300 may further include at least one host layer 314 to provide data transfer services to various applications within the wireless device 320. In some embodiments, application-specific functionality provided by at least one host layer 314 may provide an interface between the software architecture and the general-purpose processor 206.

[0078]

[0091] In other embodiments, the software architecture 300 may include one or more higher logical layers that provide host layer functionality (e.g., transport, session, presentation, application, etc.). For example, in some embodiments, the software architecture 300 may include a network layer (e.g., an IP layer) where the logical connection terminates at a packet data network (PDN) gateway (PGW). In some embodiments, the software architecture 300 may include an application layer where the logical connection terminates at another device (e.g., an end-user device, server, etc.). In some embodiments, the software architecture 300 may further include a hardware interface 316 between the physical layer 306 and communication hardware (e.g., one or more radio frequency (RF) transceivers) in the AS 304.

[0079]

[0092] Figure 4 shows a process flow diagram of an exemplary method 400 for supporting an immersive experience in a teleconference or telepresence session in various embodiments. Referring to Figures 1A to 4, method 400 can be implemented by a processor (such as 156, 212, 216, 252, or 260) in a wireless device (such as wireless devices 120a to 120e, 172, 200, 320). In various embodiments, the operation of method 400 can be carried out by a processor in a wireless device that is one of multiple participant computing devices in a teleconference or telepresence session, such as an immersive three-dimensional group session.

[0080]

[0093] In block 402, the process performs actions to direct an offer to send and / or receive a scene graph as part of the Session Initiation Protocol (SIP) setup for the session. In some embodiments, the offer may direct a graphical output node owned by a wireless device. In some embodiments, the session may be a WebRTC session.

[0081]

[0094] In block 404, the processor may perform an action to receive a Session Description Protocol (SDP) for the session that indicates the address of a data channel through which the scene graph for the session will be shared. In some embodiments, the scene graph may define one or more nodes assigned to each computing device participating in the session. In various embodiments, the nodes may reference other media streams from other computing devices participating in the session, and these other media streams may be overlaid in the session. In various embodiments, the one or more nodes assigned to each computing device participating in the session may include one or more visual nodes, audio source nodes, graphics nodes, or shared content nodes.

[0082]

[0095] In block 406, the processor may perform an action to download the scene graph via the data channel.

[0083]

[0096] In block 408, the processor may perform operations to receive and render a session according to a scene graph for rendering on an image rendering device (e.g., 168). Receiving and rendering a session may include receiving a streaming service of the session according to the scene graph and rendering the session on a display. In some embodiments, the display of the session may be rendered on an HMD (e.g., 172), on a video conference room, on a volume display, or on any other image and sound rendering device, and receiving and rendering may include outputting the session to the user via the image and sound rendering device.

[0084]

[0097] In block 410, the processor may perform actions to add a node assigned to a wireless device to the scene graph, or to modify one or more nodes in the scene graph that are assigned to a wireless device.

[0085]

[0098] Figure 5 shows a process flow diagram of an exemplary method 500 for supporting an immersive experience in a teleconference or telepresence session, according to various embodiments. Referring to Figures 1A to 5, method 500 may be implemented by a processor (such as 156, 212, 216, 252, or 260) of a wireless device (such as wireless devices 120a to 120e, 172, 200, 320) and / or a call server (such as call server 150). In various embodiments, the operation of method 500 may be carried out by a processor of a host computing device hosting the teleconference or telepresence session. In some embodiments, the host computing device may be a separate call server, such as an MRF, MCU, or teleconference application server. In some embodiments, the host computing device may be a wireless device that is one of several computing devices participating in a teleconference or telepresence session, such as an immersive 3D group session. In various embodiments, the operation of method 500 may be carried out in conjunction with the operation of method 400.

[0086]

[0099] In block 502, the host computing device's processor may perform actions to receive instructions for an offer from a computing device participating in the session to send and / or receive a scene graph as part of the Session Initiation Protocol (SIP) setup for the session. In some embodiments, the session may be a WebRTC session. In some embodiments, the received instructions may point to a graphical output node owned by the wireless device sending the offer instructions.

[0087]

[0100] In block 504, the host computing device's processor may perform an action to generate a session description protocol (SDP) for the session that points to the addresses of data channels through which a scene graph for the session will be shared. In some embodiments, the scene graph may define one or more nodes assigned to each computing device participating in the session.

[0088]

[0101] In block 506, the processor of the host computing device may perform an action to send an SDP to the computing devices participating in the session.

[0089]

[0102] In block 508, the host computing device's processor may perform an operation to send the session and scene graph to the computing devices participating in the session. As an example, the host computing device may perform an operation to stream the session and scene graph to the computing devices participating in the session. In various embodiments, the computing devices participating in the session may be HMDs (e.g., 172), TVs in a conference room, volume displays, or other image and sound rendering devices, and sending the session and / or scene graph may include sending the session and / or scene graph so that the computing devices participating in the session can output the session to the user on the image and sound rendering device.

[0090]

[0103] Figure 6 shows the configuration of a scene graph document 600 in glTF2.0 suitable for use in various embodiments. Referring to Figures 1A to 6, in various embodiments, the scene graph document 600 may be part of the scene graph. The scene graph document 600 may include JSON elements that define node hierarchy, material descriptions, lighting information, camera information, etc. The scene graph document 600 may include binary files (BINs) that define geometry information such as vertices and indices, animation information such as keyframes, skin information such as inverse bind matrices, etc. The scene graph document 600 may include GL shading library (glSL) files that define shader information. The scene graph document 600 may include various other types of files that define other information for the scene graph, such as textures, including Portable Network Graphics (PNG) files and Joint Photographic Expert Group (JPEG) files.

[0091]

[0104] Figure 7 shows a suitable scene graph structure for use in various embodiments. Referring to Figures 1A to 7, in various embodiments, the scene graph may contain multiple nodes. Each node may contain child nodes that describe various components of the node, such as camera views, mesh information, and lighting information. The scene graph may define hierarchical relationships between attributes for rendering the node's mesh, such as accessories, skins, buffer views, buffer information, materials, techniques, programs, shaders, textures, images, and samplers, as shown in Figure 7.

[0092]

[0105] Figure 8 is a call flow diagram illustrating the operation to support an immersive experience in a teleconference or telepresence session in various embodiments. Referring to Figures 1A to 8, the operation between the call participants (e.g., call participants #1 and #2) and the call server may include, in operation 1), the participants (e.g., call participants #1 and #2) using a provided link to a web page to join the WebRTC meeting. In operation 2), the call server provides the participants (e.g., call participants #1 and #2) with its web page along with a scene graph file that sets up the initial / default configuration in 3D space with the call participants (e.g., call participants #1 and #2) and materials. For example, each participant will be assigned a visual node, an audio source node, and potentially nodes for graphics and other shared content. In operation 3), each participant (e.g., call participants #1 and #2) may add or modify the nodes they own in the scene graph. In operation 4), a media stream providing the components for the nodes in the scene graph may be streamed using WebRTC. These streams can be exchanged directly or through a server, such as a media proxy server.

[0093]

[0106] Figure 9 is a process flow diagram illustrating Method 900 for providing an immersive 3D group session in various embodiments. Referring to Figures 1A to 9, Method 900 can be implemented by a processor (such as 156, 212, 216, 252, or 260) of a wireless device (such as wireless devices 120a to 120e, 172, 200, 320). In various embodiments, the operation of Method 900 can be carried out by a processor of a wireless device that is one of several participant computing devices in a teleconference or telepresence session, such as an immersive 3D group session. In various embodiments, the operation of Method 900 can be carried out in conjunction with one or more of the operations of Method 400 (Figure 4) and / or 500 (Figure 5).

[0094]

[0107] In block 902, the processor may perform an operation that includes receiving a scene graph for an immersive three-dimensional group session, where the scene graph comprises at least its own graphical output node assigned to be controlled by a wireless device, and each of the other graphical output nodes assigned to be controlled by each of the other of a plurality of participant computing devices. For example, the scene graph may be one as shown in Figure 8.

[0095]

[0108] The portion of the scene graph received in block 902 may be assigned to each participant computing device in an immersive 3D group session, such as one or more nodes for each participant computing device. By parsing the scene graph, the wireless device's processor may determine which nodes should be controlled by the wireless device. Assigning a graphical output node to each participant computing device may allow each participant computing device to control at least one graphical output node. A graphical output node may contain components that define a 3D object to be output into the 3D space defined by the scene graph. For example, a 3D object may include an avatar, character, or other representation, and the components of the graphical output node may define how the 3D object will be rendered in the 3D space of the immersive 3D group session. In this way, by controlling the components of each of its assigned graphical output nodes, the wireless device's processor may control how other participant computing devices observe one or more 3D objects associated with the wireless device, such as an avatar, character, or other representation selected by the wireless device's user during the immersive 3D group session. Similarly, by controlling the components of their respective assigned graphical output nodes, other participant computing devices may control how users of wireless devices observe their respective associated one or more three-dimensional objects, such as avatars, characters, or other representations, during immersive three-dimensional group sessions.

[0096]

[0109] In block 904, the processor may perform actions including controlling the components of a custom graphical output node relative to the three-dimensional space of an immersive three-dimensional group session. In some embodiments, a user may adjust a custom graphical node, such as moving the placement of the custom graphical node in the scene, independently of the position of the wireless device. For example, a user may select a position in the scene for displaying their three-dimensional object (e.g., avatar, character, etc.) in the scene. In some embodiments, controlling the components of a custom graphical output node may include controlling the components of the custom graphical output node based at least in part on the determined position of the wireless device relative to the three-dimensional space of the immersive three-dimensional group session. In some embodiments, the position of the wireless device may be determined in three-dimensional space, such as a center point for three-dimensional space, grid coordinates, or a position relative to another reference. Since some immersive three-dimensional group sessions may support the movement of participants in three-dimensional space for the immersive three-dimensional group session, determining the position of the wireless device may support rendering objects for the wireless device's graphical output node at the correct relative position. As a specific example, the lighting and / or camera components of a wireless device's assigned unique graphical output node may be controlled to reflect the lighting and / or camera components of the wireless device's current position in three-dimensional space for an immersive three-dimensional group session.

[0097]

[0110] In block 904, in addition to controlling the components of a custom graphical output node based on position, the components of a custom graphical output node may also be controlled based on other metrics, such as the orientation of the wireless device. For example, the movement of position and changes in orientation relative to a common reference point and common reference plane, as indicated by the wireless device's accelerometer, may be used to determine the position and orientation of the wireless device in three-dimensional space for an immersive three-dimensional group session. As a specific example, the lighting and / or camera components of a custom graphical output node assigned to a wireless device may be controlled to reflect the lighting and / or camera components of the wireless device's current position and current orientation in three-dimensional space for an immersive three-dimensional group session. Controlling the components of a custom graphical output node based on current position and / or current orientation may be useful in an implementation where the wireless device is a head-mounted device that allows the movement of the head-mounted device user to be visually communicated to other participants during an immersive three-dimensional group session.

[0098]

[0111] In block 906, the processor may perform operations including sending components of its own graphical output node in a first media stream to other participants computing devices. These components may be sent in the media stream directly to the other participants computing devices and / or via a call server (e.g., 150), such as an MRF, MCU, or teleconferencing application server. In this way, the processor of a wireless device can provide the other participants computing devices with a media stream for its own respective graphical output node, thereby controlling how the other participants computing devices render its graphical output node.

[0099]

[0112] In block 908, the processor may perform operations including receiving components of other graphical output nodes in a media stream from each of the other participant computing devices. These components may be received in the media stream directly from each of the other participant computing devices and / or via a call server (e.g., 150), such as an MRF, MCU, or teleconference application server. In this way, the processor of the wireless device may receive media streams of graphical output nodes assigned to other participant computing devices.

[0100]

[0113] In block 910, the processor may perform operations including rendering an immersive 3D group session on the display of a wireless device, at least in part, based on components of its own graphical output node and components of other graphical output nodes. For example, components of the media stream of the wireless device and components of the media stream of other participant computing devices may be overlaid together with components of the other media stream to generate a displayed output of the 3D space of the immersive 3D group session.

[0101]

[0114] The processor may continuously perform the operations of blocks 904, 906, 908, and 910 during an immersive 3D group session in order to render the immersive 3D group session.

[0102]

[0115] Figure 10 is a process flow diagram illustrating Method 1000 for providing an immersive 3D group session in various embodiments. Referring to Figures 1A to 10, Method 1000 may be implemented by a processor (such as 156, 212, 216, 252, or 260) of a wireless device (such as wireless devices 120a to 120e, 172, 200, 320). In various embodiments, the operation of Method 1000 may be carried out by a processor of a wireless device that is one of several participant computing devices in a teleconference or telepresence session, such as an immersive 3D group session. In various embodiments, the operation of Method 1000 may be carried out in conjunction with one or more of the operations of Method 400 (Figure 4), Method 500 (Figure 5), and / or Method 900 (Figure 9). As a specific example, the operation of Method 1000 may be carried out as part of the operation of block 910 of Method 900 to render an immersive 3D group session.

[0103]

[0116] In block 1002, the processor may perform operations including receiving a scene graph update, which includes instructions for a new participant computing device for an immersive 3D group session and instructions for a new graphical output node assigned to be controlled by the new participant computing device. The scene graph update may be sent by the host computing device in response to a new participant joining the immersive 3D group session. In some embodiments, the scene graph update may be received directly from another participant computing device and / or via a call server (e.g., 150), such as an MRF, MCU, or teleconferencing application server.

[0104]

[0117] In block 1004, the processor may perform an operation that includes receiving components of a new graphical output node in a second media stream from a new participant computing device. These components may be received in the media stream directly from the new participant computing device and / or via a call server (e.g., 150), such as an MRF, MCU, or teleconferencing application server. In this way, the processor of the wireless device may receive the media stream of a graphical output node assigned to a newly added participant computing device.

[0105]

[0118] In block 1006, the processor may perform operations including rendering an immersive 3D group session on the display of a wireless device, at least in part, based on the components of its own graphical output node, the components of other graphical output nodes, and the components of a new graphical output node. For example, the components of the media stream of the wireless device and the components of the media stream of other participant computing devices, including a second media stream of a newly added participant computing device, may be overlaid together with the components of other media streams to generate a displayed output of the 3D space of the immersive 3D group session.

[0106]

[0119] Various embodiments may be implemented on various wireless network devices, one example of which is shown in Figure 11 in the form of a wireless network computing device 1100 that functions as a network element of a communication network, such as a call server (e.g., call server 150). Such a network computing device may include at least the components shown in Figure 11. Referring to Figures 1A to 11, the network computing device 1100 may typically include a processor 1101 coupled to volatile memory 1102 and large-capacity non-volatile memory such as a disk drive 1103. The network computing device 1100 may also include peripheral memory access devices such as a floppy disk drive, compact disk (CD), or digital video disc (DVD) drive 1106 coupled to the processor 1101. The network computing device 1100 may also include a network access port 1104 (or interface) coupled to the processor 1101 for establishing data connectivity to networks such as the Internet and / or a local area network coupled to other system computers and servers. The network computing device 1100 may include one or more antennas 1107 for sending and receiving electromagnetic radiation that may be connected to a wireless communication link. The network computing device 1100 may include additional access ports, such as USB, Firewire, or Thunderbolt, for connecting to peripherals, external memory, or other devices.

[0107]

[0120] Various embodiments may be implemented on various wireless devices (e.g., wireless devices 120a-120e, 172, 200, 320), one example of which is shown in Figure 12 in the form of a smartphone 1200. Referring to Figures 1A-12, the smartphone 1200 may include a first SOC 202 (e.g., an SOC-CPU) coupled to a second SOC 204 (e.g., a 5G-enabled SOC). The first SOC 202 and the second SOC 204 may be coupled to internal memory 1206, 1216, a display 1212, and a speaker 1214. Furthermore, the smartphone 1200 may include an antenna 1204 for sending and receiving electromagnetic radiation which may be connected to a wireless data link, and / or a cellular telephone transceiver 266 coupled to one or more processors in the first SOC 202 and / or the second SOC 204. The smartphone 1200 also typically includes a menu selection button or rocker switch 1220 for receiving user input.

[0108]

[0121] A typical smartphone 1200 also includes an audio coding / decoding (codec) circuit 1210 that digitizes sound received from the microphone into data packets suitable for wireless transmission, and decodes the received sound data packets to generate an analog signal provided to the speaker for sound generation. Furthermore, one or more processors among the first SOC 202 and the second SOC 204, the wireless transceiver 266, and the codec 1210 may include a digital signal processor (DSP) circuit (not shown separately).

[0109]

[0122] The processors of the wireless network computing device 1100 and the smartphone 1200 may be any programmable microprocessor, microcomputer, or one or more multiple processor chips that can be configured by software instructions (applications) to perform a variety of functions, including the functions of the various embodiments described below. Some wireless devices may have multiple processors, such as one processor in SOC 204 dedicated to wireless communication functions and one processor in SOC 202 dedicated to running other applications. Typically, software applications may be stored in memories 1206, 1216 before they are accessed and loaded into the processor. The processor may have internal memory sufficient to store application software instructions.

[0110]

[0123] As used in this application, terms such as “component,” “module,” and “system” are intended to include, but are not limited to, computer-related entities such as hardware, firmware, hardware-software combinations, software, or running software configured to perform a particular operation or function. For example, a component may be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. As an example, both an application running on a wireless device and the wireless device itself may be referred to as a component. One or more components may reside within a process and / or an execution thread, and components may be localized on one processor or core and / or distributed across two or more processors or cores. Furthermore, these components may be executed from various non-temporary computer-readable media storing various instructions and / or data structures. Components may communicate via local and / or remote processes, function or procedure calls, electronic signals, data packets, memory reads / writes, and other known network, computer, processor, and / or process-related communication methods.

[0111]

[0124] Several different cellular and mobile communication services and standards are expected to become available or be planned for the future, all of which will implement and benefit from various embodiments. Such services and standards include, for example, the Third Generation Partnership Project (3GPP), the Long-Term Evolution (LTE) system, the Third Generation Wireless Mobile Communication Technology (3G), the Fourth Generation Wireless Mobile Communication Technology (4G), the Fifth Generation Wireless Mobile Communication Technology (5G), the Global System for Mobile Communications (GSM), the Universal Mobile Telecommunications System (UMTS), 3GSM, General Purpose Packet Radio Service (GPRS), and the Code Division Multiple Access (CDMA) system (e.g., cdmaOne). This includes CDMA1020(registered trademark), GSM Advanced High-Speed ​​Data Rate (EDGE), Advanced Mobile Phone Systems (AMPS), Digital AMPS (IS-136 / TDMA), Evolution Data Optimized (EV-DO), Digital Extended Cordless Telecommunications (DECT), Worldwide Interoperability for Microwave Access (WiMAX), Wireless Local Area Networks (WLAN), Wi-Fi Protected Access I and II (WPA, WPA2), and Integrated Digital Extended Network (iDEN). Each of these technologies involves, for example, the transmission and reception of voice, data, signaling, and / or content messages. Any references to terms and / or technical details relating to individual telecommunications standards or technologies are for illustrative purposes only and are not intended to limit the claims to any particular communication system or technology unless explicitly stated in the language of the claims.

[0112]

[0125] The various embodiments illustrated and described are provided merely as examples to illustrate the various features of the claims. However, features illustrated and described with respect to any given embodiment are not necessarily limited to the embodiment in question, but may be used in conjunction with or in combination with other embodiments illustrated and described. Furthermore, the claims are not intended to be limited by any single exemplary embodiment. For example, one or more operations of methods 400, 500, 900, and / or 1000 may be replaced by or in combination with one or more operations of methods 400, 500, 900, and / or 1000.

[0113]

[0126] The above description of the method and process flow diagram are given only as illustrative examples and are not intended to require or imply that the operations of the various embodiments must be performed in the order presented. As will be understood by those skilled in the art, the order of operations in the above embodiments may be performed in any order. Words such as “then,” “next,” and “then” are not intended to limit the order of operations and are used to guide the reader throughout the description of the method. Furthermore, references to singular claim elements using, for example, the articles “a,” “an,” or “the” should not be interpreted as limiting that element to the singular form.

[0114]

[0127] The various exemplary logic blocks, modules, components, circuits, and algorithmic operations described in relation to the embodiments disclosed herein may be implemented as electronic hardware, computer software, or a combination of both. To clearly demonstrate this hardware-software compatibility, various exemplary components, blocks, modules, circuits, and operations have generally been described above in terms of their functions. Whether such functions are implemented as hardware or as software depends on the specific application and design constraints imposed on the overall system. A person skilled in the art may implement the described functions in various ways for each specific application, but such a determination of embodiments should not be construed as resulting in a departure from the claims.

[0115]

[0128] The hardware used to implement the various exemplary logics, logic blocks, modules, and circuits described in relation to the embodiments disclosed herein may be implemented or carried out using general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of receiver smart objects, for example, a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors working with a DSP core, or any other such configuration. Alternatively, some operations or methods may be carried out by circuits specific to a given function.

[0116]

[0129] In one or more embodiments, the described functions may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or codes on a non-temporary computer-readable storage medium or a non-temporary processor-readable storage medium. The operation of the methods or algorithms disclosed herein may be performed in processor-executable software modules or processor-executable instructions that may reside on a non-temporary computer-readable storage medium or a non-temporary processor-readable storage medium. A non-temporary computer-readable storage medium or a non-temporary processor-readable storage medium may be any storage medium accessible by a computer or processor. Such a non-temporary computer-readable storage medium or a non-temporary processor-readable storage medium may include, but are not limited to, RAM, ROM, EEPROM®, FLASH® memory, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage smart objects, or any other medium that may be used to store desired program code in the form of instructions or data structures and that may be accessible by a computer. The terms "disk" and "disc" as used herein include Compact Disc (CD), LaserDisc® (disc), Optical Disc (disc), Digital Multipurpose Disc (disc) (DVD), Floppy Disk (disk), and Blu-ray® Disc (disc), where a Disk typically reproduces data magnetically, and a Disc (disc) reproduces data optically using a laser. Combinations of the above also fall within the scope of non-temporary computer-readable media and non-temporary processor-readable media. Furthermore, the operation of a method or algorithm may exist as one or any combination, or set thereof, of code and / or instructions on a non-temporary processor-readable storage medium and / or non-temporary computer-readable storage medium that can be incorporated into a computer program product.

[0117]

[0130] Prior descriptions of the disclosed embodiments are provided to enable those skilled in the art to construct or use the claims. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the claims. Accordingly, this disclosure is not intended to be limited to the embodiments shown herein and should be given the broadest scope that coincides with the following claims and the principles and novel features disclosed herein. The invention described in the original claims of this application is listed below. [C1] A method performed by the processor of a wireless device which is one of several participant computing devices operating in an immersive three-dimensional group session, Receiving a scene graph for an immersive 3D group session, wherein the scene graph comprises at least its own graphical output node assigned to be controlled by the wireless device, and each of the other graphical output nodes assigned to be controlled by each of the other of the plurality of participant computing devices. Controlling the components of the unique graphical output node with respect to the three-dimensional space of the immersive three-dimensional group session, Sending the components of the proprietary graphical output node in the first media stream to the other of the plurality of participant computing devices, Receiving, in the media stream, the components of the other graphical output node from each of the other participant computing devices, Rendering the immersive 3D group session on the wireless device's display based at least partially on the components of the proprietary graphical output node and the components of the other graphical output node, A method that includes [a certain feature]. [C2] Receiving a scene graph update, which includes instructions for a new participant computing device for the immersive 3D group session and instructions for a new graphical output node assigned to be controlled by the new participant computing device. The new participant computing device receives the components of the new graphical output node in a second media stream, Rendering the immersive 3D group session on the display of the wireless device based at least partially on the components of the proprietary graphical output node, the components of the other graphical output node, and the components of the new graphical output node, A method of C1 that further includes the following: [C3] Receiving a session description protocol (SDP) for the immersive 3D group session that indicates the address of the data channel in which the scene graph will be shared, Furthermore, The method according to C1, wherein receiving the scene graph comprises downloading the scene graph via the data channel. [C4] As part of setting up the Session Initiation Protocol (SIP) for the immersive 3D group session, sending an offer to send or receive the scene graph to the other of the multiple participant computing devices, A method of C1 that further includes the following: [C5] Further comprising sending an offer to send or receive the scene graph to the other of the plurality of participant computing devices, wherein the offer directs the unique graphical output node. The method described in C1. [C6] The method according to C1, wherein the immersive 3D group session is a Web Real-Time Communication (WebRTC) session. [C7] The method of C1, wherein controlling the components of the proprietary graphical output node with respect to the three-dimensional space of the immersive three-dimensional group session is at least partially based on a determined position of the wireless device with respect to the three-dimensional space of the immersive three-dimensional group session. [C8] Controlling the components of the custom graphical output node based at least in part on the determined position of the wireless device with respect to the three-dimensional space of the immersive three-dimensional group session is further comprising controlling the components of the custom graphical output node based at least in part on the determined position of the wireless device with respect to the three-dimensional space of the immersive three-dimensional group session and the determined orientation of the wireless device with respect to the three-dimensional space of the immersive three-dimensional group session, according to the method of C7. [C9] Display and, A processor coupled to the aforementioned display, A wireless device comprising, the processor is Receiving a scene graph for an immersive 3D group session, wherein the scene graph comprises at least its own graphical output node assigned to be controlled by the wireless device, and each of the other graphical output nodes assigned to be controlled by each of the multiple participant computing devices. Controlling the components of the unique graphical output node with respect to the three-dimensional space of the immersive three-dimensional group session, Sending the components of the proprietary graphical output node in a first media stream to another of the plurality of participant computing devices, and receiving the components of the other graphical output nodes in a media stream from each of the other of the plurality of participant computing devices. Rendering the immersive 3D group session on the display of the wireless device based at least partially on the components of the proprietary graphical output node and the components of the other graphical output node, A wireless device consisting of processor-executable instructions for performing a certain action. [C10] The aforementioned processor is Receiving a scene graph update, which includes instructions for a new participant computing device for the immersive 3D group session and instructions for a new graphical output node assigned to be controlled by the new participant computing device, The new participant computing device receives the components of the new graphical output node in a second media stream, Rendering the immersive 3D group session on the display of the wireless device based at least partially on the components of the proprietary graphical output node, the components of the other graphical output node, and the components of the new graphical output node, A wireless device as described in C9, further comprising processor-executable instructions for performing the following actions. [C11] The aforementioned processor is The scene graph will be shared, and a session description protocol (SDP) for the immersive 3D group session will be received which indicates the address of the data channel. It is further composed of processor-executable instructions for performing the following: The wireless device according to C9, wherein the processor is further comprised of processor-executable instructions for receiving the scene graph by downloading the scene graph via the data channel. [C12] The aforementioned processor is As part of setting up the Session Initiation Protocol (SIP) for the immersive 3D group session, an offer to send or receive the scene graph is sent to the other of the multiple participant computing devices. A wireless device as described in C9, further comprising processor-executable instructions for performing the following actions. [C13] The aforementioned processor is The processor executable instructions further comprise a process for sending an offer to send or receive the scene graph to the other of the plurality of participant computing devices, wherein the offer directs the unique graphical output node. Wireless device as described in C9. [C14] The immersive 3D group session is a Web Real-Time Communication (WebRTC) session, as described in C9. [C15] The method according to C9, wherein the processor further comprises processor-executable instructions for controlling the components of the proprietary graphical output node with respect to the three-dimensional space of the immersive three-dimensional group session, by controlling the components of the proprietary graphical output node with respect to the three-dimensional space of the immersive three-dimensional group session, at least in part based on the determined position of the wireless device with respect to the three-dimensional space of the immersive three-dimensional group session. [C16] The wireless device according to C15, wherein the processor further comprises processor-executable instructions for controlling the components of the proprietary graphical output node based at least partially on the determined position of the wireless device in the three-dimensional space of the immersive three-dimensional group session and the determined orientation of the wireless device in the three-dimensional space of the immersive three-dimensional group session. [C17] A non-temporary processor-readable medium storing processor-executable instructions, wherein the processor-executable instructions are transmitted to the processor of a wireless device. Receiving a scene graph for an immersive 3D group session, wherein the scene graph comprises at least its own graphical output node assigned to be controlled by the wireless device, and each of the other graphical output nodes assigned to be controlled by each of the multiple participant computing devices. Controlling the components of the unique graphical output node with respect to the three-dimensional space of the immersive three-dimensional group session, Sending the components of the proprietary graphical output node in the first media stream to another of the multiple participant computing devices, Receiving, in the media stream, the components of the other graphical output node from each of the other participant computing devices, Rendering the immersive 3D group session on the wireless device's display based at least partially on the components of the proprietary graphical output node and the components of the other graphical output node, A non-transient processor-readable medium configured to perform an operation comprising the following: [C18] The stored processor-executable instructions are sent to the processor of the wireless device. Receiving a scene graph update, which includes instructions for a new participant computing device for the immersive 3D group session and instructions for a new graphical output node assigned to be controlled by the new participant computing device, The new participant computing device receives the components of the new graphical output node in a second media stream, Rendering the immersive 3D group session on the display of the wireless device based at least partially on the components of the proprietary graphical output node, the components of the other graphical output node, and the components of the new graphical output node, A non-transient processor-readable medium as described in C17, configured to perform an operation that further includes the above. [C19] The stored processor-executable instructions are sent to the processor of the wireless device. The scene graph will be shared, and a session description protocol (SDP) for the immersive 3D group session will be received which indicates the address of the data channel. It is configured to perform an action that further includes the following: The processor executable instruction is configured to cause the processor of the wireless device to perform an operation that includes receiving the scene graph and downloading the scene graph via the data channel. Non-temporary processor-readable media as described in C17. [C20] The stored processor-executable instructions are sent to the processor of the wireless device. As part of setting up the Session Initiation Protocol (SIP) for the immersive 3D group session, an offer to send or receive the scene graph is sent to the other of the multiple participant computing devices. A non-transient processor-readable medium as described in C17, configured to perform an operation that further includes the above. [C21] The stored processor-executable instructions are sent to the processor of the wireless device. The device is configured to perform an operation that further comprises sending an offer to send or receive the scene graph to one of the other participant computing devices, wherein the offer directs the device to its own graphical output node. Non-temporary processor-readable media as described in C17. [C22] The immersive three-dimensional group session is a Web Real-Time Communication (WebRTC) session, a non-temporary processor-readable medium as described in C17. [C23] The non-temporary processor-readable medium according to C17, wherein the stored processor-executable instructions are configured to cause the processor of the wireless device to perform an operation which comprises controlling the components of the unique graphical output node with respect to the three-dimensional space of the immersive three-dimensional group session, or controlling the components of the unique graphical output node with respect to at least a portion of the determined position of the wireless device with respect to the three-dimensional space of the immersive three-dimensional group session. [C24] The non-temporary processor-readable medium according to C17, wherein the stored processor-executable instructions are configured to cause the processor of the wireless device to perform an operation which comprises controlling the components of the unique graphical output node based at least partially on the determined position of the wireless device with respect to the three-dimensional space of the immersive three-dimensional group session, and controlling the components of the unique graphical output node based at least partially on the determined position of the wireless device with respect to the three-dimensional space of the immersive three-dimensional group session and the determined orientation of the wireless device with respect to the three-dimensional space of the immersive three-dimensional group session. [C25] A wireless device, Means for receiving a scene graph for an immersive 3D group session, wherein the scene graph comprises at least its own graphical output node assigned to be controlled by the wireless device, and each of the other graphical output nodes assigned to be controlled by each of the multiple participant computing devices. Means for controlling the components of the unique graphical output node with respect to the three-dimensional space of the immersive three-dimensional group session, Means for sending the components of the proprietary graphical output node in the first media stream to other of the plurality of participant computing devices, Means for receiving, in a media stream, components of the other graphical output node from each of the other participant computing devices, A means for rendering the immersive 3D group session on the display of the wireless device, based at least in part on the components of the proprietary graphical output node and the components of the other graphical output node, A wireless device equipped with the following features. [C26] Means for receiving scene graph updates, including instructions for a new participant computing device for the immersive 3D group session and instructions for a new graphical output node assigned to be controlled by the new participant computing device, Means for receiving components of the new graphical output node in a second media stream from the new participant computing device, A means for rendering the immersive 3D group session on the display of the wireless device, based at least in part on the components of the proprietary graphical output node, the components of the other graphical output node, and the components of the new graphical output node, A wireless device as described in C25, further equipped with the features mentioned above. [C27] The means further comprises receiving a session description protocol (SDP) for the immersive 3D group session which indicates the address of a data channel in which the scene graph will be shared, The means for receiving the scene graph is a wireless device according to C25, comprising means for downloading the scene graph via the data channel. [C28] The wireless device according to C25, further comprising means for sending an offer to send or receive the scene graph to the other of the plurality of participant computing devices as part of the setup of a Session Initiation Protocol (SIP) for the immersive three-dimensional group session. [C29] The system further comprises means for sending an offer to send or receive the scene graph to the other of the plurality of participant computing devices, wherein the offer directs the unique graphical output node. Wireless devices as described in C25. [C30] The immersive 3D group session is a Web Real-Time Communication (WebRTC) session, as described in C25.

Claims

1. A method implemented by a processor in a wireless device to support immersive 3D group sessions, For the immersive 3D group session associated with the scene, receive a scene description representing an object-based hierarchy of the scene's geometry, wherein the scene description indicates that at least its own graphical output node is controlled by the wireless device, and the scene description indicates that each of the one or more other graphical output nodes is controlled by each of the one or more participant computing devices. Rendering the scene on the wireless device's display based at least partially on one or more components of the proprietary graphical output node and one or more other components of the one or more other graphical output nodes, A method that includes [a certain feature].

2. The method according to claim 1, wherein the scene description for the immersive three-dimensional group session refers to an external media stream for each of the one or more other graphical output nodes.

3. Receiving a session description protocol (SDP) for the immersive three-dimensional group session, Furthermore, The method according to claim 2, wherein the SDP declares the external media stream for each of the one or more other graphical output nodes.

4. The SDP indicates the address of a data channel in which the scene description will be shared, The method according to claim 3, wherein receiving the scene description comprises downloading the scene description via the data channel.

5. The method according to claim 1, further comprising controlling one or more components of the unique graphical output node with respect to the three-dimensional space of the immersive three-dimensional group session.

6. Sending one or more components of the unique graphical output node in the first media stream to the other of the one or more participant computing devices, Receiving, in the media stream, the components of the other graphical output node from each of the other participant computing devices, The method according to claim 5, further comprising:

7. The method according to claim 5, wherein controlling one or more components of the proprietary graphical output node with respect to the three-dimensional space of the immersive three-dimensional group session comprises controlling one or more components of the proprietary graphical output node based at least in part on a determined position of the wireless device with respect to the three-dimensional space of the immersive three-dimensional group session.

8. The method according to claim 7, wherein controlling one or more components of the proprietary graphical output node based at least partially on the determined position of the wireless device in the three-dimensional space of the immersive three-dimensional group session comprises controlling one or more components of the proprietary graphical output node based at least partially on the determined position of the wireless device in the three-dimensional space of the immersive three-dimensional group session and the determined orientation of the wireless device in the three-dimensional space of the immersive three-dimensional group session.

9. The method according to claim 1, wherein the immersive three-dimensional group session is a web real-time communication (WebRTC) session.

10. A wireless device, For an immersive 3D group session associated with a scene, receive a scene description representing an object-based hierarchy of the scene's geometry, wherein the scene description indicates that at least its own graphical output node is controlled by the wireless device, and the scene description indicates that each of the one or more other graphical output nodes is controlled by each of the one or more participant computing devices. Rendering the scene on the wireless device's display based at least partially on one or more components of the proprietary graphical output node and one or more other components of the one or more other graphical output nodes, A wireless device equipped with a processor consisting of processor-executable instructions for performing the following actions.

11. The wireless device according to claim 10, wherein the scene description for the immersive three-dimensional group session refers to an external media stream for each of the one or more other graphical output nodes.

12. The processor is Receiving the Session Description Protocol (SDP) for the aforementioned immersive 3D group session, It is further composed of processor-executable instructions for performing the following: The wireless device according to claim 11, wherein the SDP declares the external media stream for each of the one or more other graphical output nodes.

13. The SDP indicates the address of a data channel in which the scene description will be shared, The wireless device according to claim 12, wherein receiving the scene description comprises downloading the scene description via the data channel.

14. The processor is The wireless device according to claim 10, further comprising processor-executable instructions for controlling one or more components of the proprietary graphical output node with respect to the three-dimensional space of the immersive three-dimensional group session.

15. The processor is Sending one or more components of the proprietary graphical output node in the first media stream to the other of the one or more participant computing devices, Receiving, in the media stream, the components of the other graphical output node from each of the other participant computing devices, The wireless device according to claim 14, further comprising processor-executable instructions for performing the following.

16. The wireless device according to claim 14, wherein the processor is further comprised of processor-executable instructions for controlling one or more components of the proprietary graphical output node with respect to the three-dimensional space of the immersive three-dimensional group session, by controlling one or more components of the proprietary graphical output node with respect to the three-dimensional space of the immersive three-dimensional group session, at least in part based on a determined position of the wireless device with respect to the three-dimensional space of the immersive three-dimensional group session.

17. The wireless device according to claim 16, wherein the processor is further comprised of processor-executable instructions for controlling one or more components of the proprietary graphical output node based at least partially on the determined position of the wireless device in the three-dimensional space of the immersive three-dimensional group session and the determined orientation of the wireless device in the three-dimensional space of the immersive three-dimensional group session.

18. The wireless device according to claim 10, wherein the immersive three-dimensional group session is a web real-time communication (WebRTC) session.

19. A non-temporary processor-readable medium storing processor-executable instructions, wherein the processor-executable instructions are used by the processor of a wireless device. For an immersive 3D group session associated with a scene, receive a scene description representing an object-based hierarchy of the scene's geometry, wherein the scene description indicates that at least its own graphical output node is controlled by the wireless device, and the scene description indicates that each of the one or more other graphical output nodes is controlled by each of the one or more participant computing devices. Rendering the scene on the wireless device's display based at least partially on one or more components of the proprietary graphical output node and one or more other components of the one or more other graphical output nodes, A non-transient processor-readable medium configured to perform an operation comprising the following:

20. The non-temporary processor-readable medium according to claim 19, wherein the scene description for the immersive three-dimensional group session refers to an external media stream for each of the one or more other graphical output nodes.

21. The stored processor-executable instructions are transmitted to the processor of the wireless device. Receiving the Session Description Protocol (SDP) for the aforementioned immersive 3D group session, It is configured to perform an action that further includes the following: The non-temporary processor-readable medium according to claim 20, wherein the SDP declares the external media stream for each of the one or more other graphical output nodes.

22. The SDP indicates the address of a data channel in which the scene description will be shared, The non-temporary processor-readable medium according to claim 21, wherein receiving the scene description comprises downloading the scene description via the data channel.

23. The stored processor-executable instructions are transmitted to the processor of the wireless device. A non-temporary processor-readable medium according to claim 19, configured to perform an operation further comprising controlling one or more components of the proprietary graphical output node with respect to the three-dimensional space of the immersive three-dimensional group session.

24. The stored processor-executable instructions are transmitted to the processor of the wireless device. Sending one or more components of the proprietary graphical output node in the first media stream to the other of the one or more participant computing devices, Receiving, in the media stream, the components of the other graphical output node from each of the other participant computing devices, A non-temporary processor-readable medium according to claim 23, configured to perform an operation further comprising the above.

25. The non-temporary processor-readable medium according to claim 23, wherein the stored processor-executable instructions are configured to cause the processor of the wireless device to perform an operation which comprises controlling one or more components of the proprietary graphical output node with respect to the three-dimensional space of the immersive three-dimensional group session, or controlling one or more components of the proprietary graphical output node with respect to at least a portion of the determined position of the wireless device with respect to the three-dimensional space of the immersive three-dimensional group session.

26. The non-temporary processor-readable medium according to claim 25, wherein the stored processor-executable instructions are configured to cause the processor of the wireless device to perform an operation which comprises controlling one or more components of the unique graphical output node based at least partially on the determined position of the wireless device with respect to the three-dimensional space of the immersive three-dimensional group session, and controlling one or more components of the unique graphical output node based at least partially on the determined position of the wireless device with respect to the three-dimensional space of the immersive three-dimensional group session and the determined orientation of the wireless device with respect to the three-dimensional space of the immersive three-dimensional group session.

27. ​​The non-temporary processor-readable medium according to claim 19, wherein the immersive three-dimensional group session is a web real-time communication (WebRTC) session.

28. A wireless device, Means for receiving a scene description representing an object-based hierarchy of the geometry of a scene for an immersive three-dimensional group session associated with a scene, wherein the scene description indicates that at least its own graphical output node is controlled by the wireless device, and the scene description indicates that each of the one or more other graphical output nodes is controlled by each of the one or more participant computing devices. Means for rendering the scene on the display of the wireless device, based at least in part on one or more components of the proprietary graphical output node and one or more other components of the one or more other graphical output nodes, A wireless device equipped with the following features.

29. The wireless device according to claim 28, wherein the scene description for the immersive three-dimensional group session refers to an external media stream for each of the one or more other graphical output nodes.

30. Means for receiving a session description protocol (SDP) for the immersive three-dimensional group session, Furthermore, The wireless device according to claim 29, wherein the SDP declares the external media stream for each of the one or more other graphical output nodes.

Images