Shared split rendering for augmented reality

Abstract

A shared split rendering for augmented reality is provided. A method of shared split rendering for augmented reality may include analyzing (1410) data related to coordinates within a three-dimensional space in which a plurality of augmented reality devices are located, and generating (1420) one or more device groups from the plurality of augmented reality devices. The method may also include generating (1430) one or more shared media content for each of the one or more groups of devices.

Description

TECHNICAL FIELD

[0001] Some example embodiments can generally relate to mobile or wireless telecommunication technologies and systems, such as Long Term Evolution (LTE) or Fifth Generation (5G) New Radio (NR) access technologies, or 5G beyond, or Sixth Generation (6G) access technologies or other communication systems. For example, certain example embodiments can relate to shared split rendering for extended reality. BACKGROUND

[0002] Examples of mobile or wireless telecommunication technologies and systems can include Universal Mobile Telecommunication System (UMTS) Terrestrial Radio Access Network (UTRAN), Long Term Evolution (LTE) Evolved Universal Terrestrial Radio Access Network (E-UTRAN), LTE-Advanced (LTE-A), MulteFire, LTE-A Pro, Fifth Generation (5G) Radio Access Technology or New Radio (NR) Access Technology, and / or Sixth Generation (6G) Radio Access Technology. The fifth generation (5G) and sixth generation (6G) wireless systems refer to the next generation (NG) of radio systems and network architecture. The 5G and 6G network technologies are mainly based on new radio (NR) technology, but 5G / 6G (or NG) networks can also be built on E-UTRAN radios. It is estimated that NR can provide bitrates on the order of 10-20 Gbit / s or higher and can support at least enhanced mobile broadband (eMBB) and ultra-reliable low-latency communications (URLLC), as well as massive machine type communications (mMTC). NR is expected to offer extreme broadband and ultra-reliable, low-latency connections, as well as massive networking capacity to support the Internet of Things (IoT). SUMMARY

[0003] Various example embodiments can provide an apparatus comprising means for analyzing data related to coordinates within a three-dimensional space in which a plurality of extended reality devices are located and means for generating one or more groups of devices from the plurality of extended reality devices. The apparatus can further comprise means for generating one or more shared media contents for each of the one or more groups of devices.

[0004] Certain example embodiments can provide an apparatus comprising means for transmitting data related to a shared session identifier of the apparatus and at least one other extended reality device and means for receiving one or more encoded shared media contents and one or more user-specific media contents. The apparatus can further comprise means for assembling a data stream by merging the one or more encoded shared media contents and the one or more user-specific media contents and means for inputting the assembled data stream to a data decoder for displaying the assembled data stream.

[0005] Some example embodiments can provide a method comprising analyzing data related to coordinates within a three-dimensional space in which a plurality of extended reality devices are located and generating one or more groups of devices from the plurality of extended reality devices. The method can further comprise generating one or more shared media contents for each of the one or more groups of devices.

[0006] Various example embodiments can provide a method comprising transmitting data related to a shared session identifier of a device and at least one other extended reality device and receiving one or more encoded shared media contents and one or more user-specific media contents. The method can further comprise assembling a data stream by merging the one or more encoded shared media contents and the one or more user-specific media contents and inputting the assembled data stream to a data decoder for displaying the assembled data stream. BRIEF DESCRIPTION OF DRAWINGS

[0007] For proper understanding of the example embodiments, reference should be made to the drawings, wherein: Figure 1 An example scenario for split rendering is shown; Figure 2 An example of a signal diagram for initiating a split rendering setup according to certain example embodiments is shown; Figure 3 An example of a signal diagram for establishing a split rendering session according to various example embodiments is shown; Figure 4A An example of a signal diagram for sharing a split rendering session according to some example embodiments is shown; Figure 4B An example of a continuation of the signal diagram of Figure 4A according to some example embodiments is shown; Figure 5A An example of a rendering viewport of a three-dimensional object according to various example embodiments is shown; Figure 5B Another example of a rendering viewport of a three-dimensional object according to certain example embodiments is shown; Figure 6A An example experimental simulation according to some example embodiments is shown; Figure 6B An example performance metric of the simulation of Figure 6A according to various example embodiments is shown; Figure 7A Another example experimental simulation according to certain example embodiments is shown; Figure 7B An example performance metric of the simulation of Figure 7A according to some example embodiments is shown; Figure 8AAnother example experimental simulation is shown according to certain example embodiments; Figure 8B Example performance metrics for simulations of Figure 8A are shown according to various example embodiments; Figure 9A Another example experimental simulation is shown according to certain example embodiments; Figure 9B Example performance metrics for simulations of Figure 9A are shown according to various example embodiments; Figure 10A Another example experimental simulation is shown according to certain example embodiments; Figure 10B Example performance metrics for simulations of Figure 10A are shown according to various example embodiments; Figure 11A Another example experimental simulation is shown according to certain example embodiments; Figure 11B Example performance metrics for simulations of Figure 11A are shown according to various example embodiments; Figure 12A Another example experimental simulation is shown according to certain example embodiments; Figure 12B Example performance metrics for simulations of Figure 12A are shown according to various example embodiments; Figure 13A Another example experimental simulation is shown according to certain example embodiments; Figure 13B Example performance metrics for simulations of Figure 13A are shown according to various example embodiments; Figure 14 is an example of a flow diagram of a method according to certain example embodiments; Figure 15 is an example of a flow diagram of another method according to certain example embodiments; and Figure 16 is an example of a set of apparatuses according to various example embodiments. DETAILED DESCRIPTION

[0008] It will be readily understood that the components of certain example embodiments, as generally described and illustrated in the figures herein, can be arranged and designed in a wide variety of different configurations. The following is a detailed description of certain example embodiments of systems, methods, apparatuses, and non-transitory computer program products for shared split rendering for extended reality. Although the devices discussed below and illustrated in the figures relate to 5G / 6G or next generation NodeB (gNB) network entities and / or devices and user equipment (UE) devices, the present disclosure is not limited to only network entities / devices and UEs.

[0009] It will be readily understood that the components of certain example embodiments, as generally described and illustrated in the figures herein, can be arranged and designed in a wide variety of different configurations. The following is a detailed description of certain example embodiments of systems, methods, apparatuses, and non-transitory computer program products for shared split rendering for extended reality. Although the devices discussed below and illustrated in the figures relate to 5G / 6G or next generation NodeB (gNB) network entities and / or devices and user equipment (UE) devices, the present disclosure is not limited to only network entities / devices and UEs.

[0010] The Third Generation Partnership Project (3GPP) has defined specifications for immersive reality technologies. Immersive technologies extend the reality experienced by a person by fusing a virtual world with the real world or by creating a fully immersive experience. For example, immersive technologies can include augmented reality (AR), in which virtual objects are superimposed on the real world; virtual reality (VR), in which a person is fully immersed in a virtual environment; and mixed reality (MR), in which virtual and real-world objects can interact in real time. The term extended reality (XR) can be used to refer to all of these immersive technologies. In XR, audio and / or video data can need to be transmitted and / or received at periodic or aperiodic occasions while staying within the bandwidth limitations of the local network in which the XR devices are located.

[0011] A goal of 3GPP is to try to optimize the efficient use of collaborative XR applications in which multiple users equipped with XR devices (e.g., XR headsets) can share the same experience and view similar or identical content. One example use of sharing an XR experience between multiple users using XR devices can be a classroom in which students use XR devices for shared or common XR simulations.

[0012] Figure 1An example scenario is shown in which multiple classrooms of a university campus are equipped with XR devices for students in the classrooms. In this example, a university campus is equipped with an XR-enabled lecture hall. A teacher can use XR to show additional virtual content to students to enhance teaching. The XR headsets are preferably lightweight and affordable, which can be achieved by, for example, offloading costly and energy-consuming processing to a server. For example, media rendering can be offloaded to a server, such as a cloud server or other type of server on a standalone non-public network (S-NPN), which can be performed by a process called split rendering.

[0013] Split rendering can require high throughput and low latency connections. Congestion in a local network, such as an S-NPN, can cause service interruptions and / or a low quality of experience (QoE) if the number of parallel sessions is too high. Split rendering can also require higher available computing power on a cloud server, which can result in higher infrastructure costs. In Figure 1 In the example of a lecture hall, multiple students located in close proximity to each other in the lecture hall can request the server to provide the same or similar media rendering, as the students’ XR devices have similar positions or poses.

[0014] In Figure 1 the scenario, two types of content can be distributed. First, offline-generated content can be sent from a streaming server depending on the user’s position in the room, and second, real-time three-dimensional scenes can require split rendering on a server. In both cases, the delivery of content can be performed by exploiting stereoscopic encoding. However, in this case, the views and poses of proximate users are expected to be relevant, which cannot be exploited by traditional stereoscopic encoding. Therefore, an additional level of optimization can be required by exploiting an additional common base layer view that will be used as a common reference for proximate users. Users can send their own pose, which will be used to generate left and right views that can be encoded using the common reference view. This additional base layer reference view can save a significant amount of bitrate in each user’s viewport encoding and can be sent through a multicast / broadcast mechanism to, for example, absorb the additional bitrate for transmission. New video codecs can have the potential to exploit the common base view efficiently and can be able to save bitrate in the overall transmission. This can be achieved by using a multiview codec that is able to handle three views, such as a common view and two left and right stereoscopic views. Performance evaluation of the positive impact on streaming can be determined by the resource savings achieved on the transmission. The anchor point can be a full 3D high-efficiency video coding (HEVC) / 3D-AVC simulcast delivery of the rendered video stream. A configuration compared to a simulcast can occur in the case of using the multicast reference view as a reference to encode the left or right eye views generated for users, which are sent through unicast. This can be used for several types of user clustering (e.g., in terms of the number of users, distance, cluster size).

[0015] 3GPP specifications hope to be able to exploit correlation and redundancy among multiple users using split rendering architecture. Various example embodiments can provide technical advantages to support one or more procedures for exploiting correlation, redundancy, and / or dependencies to optimize delivery of shared XR experiences. Certain example embodiments can provide for optimized encoding and delivery of split rendering sessions for XR implementations.

[0016] Various example embodiments can provide that multiple XR split rendering sessions can be executed in parallel, each session can have its own allocated resources, communication pipes, encoders, and / or packagers. Certain example embodiments can exploit correlation among multiple users and / or XR devices used by users to jointly optimize split rendering sessions and reduce bandwidth required for N split rendering sessions by sharing parts of the split rendering workflow.

[0017] Figure 2 A signal diagram example of initiating a split rendering setup is shown in accordance with various example embodiments. Certain example embodiments can provide for configuration of, for example, an application 201, a media session handler / device 202, a split rendering client (SRC) 203, a split rendering server (SRS) 204, a real-time communication application function (RTC AF / SRF) 205, and an application service provider 206. Certain example embodiments can provide that at 210, the application service provider 206 can request the SRF 205 to provide a split management session, at 215, the application service provider 206 can request the SRF 205 to attach a newly created shared split management session to an existing session already running for a shared XR service.

[0018] At 220, the shared split management session can be announced to the application 201 as part of the service access information, at 225, the application 201 can request a split of the client media function from the SRC 203. At 230, the SRC 203 can query the media session handler / device 202 for media capabilities of the SRC 203, at 235, the SRC 203 can send initial viewing conditions of the SRC 203 to the SRS 204, including, for example, pose and / or position in a three-dimensional space. At 240, the SRS 204 can evaluate different potential split sharing options based on the initial viewing conditions sent by the SRC 203. For example, the SRS 204 can evaluate whether the SRC 203 can reuse at least a portion of an already configured session, or can evaluate which existing data streams can be reused (e.g., already generated views, audio tracks, or metadata that are currently to be sent over a multicast broadcast service (MBS)). At 245, the SRC 203 and the SRS 204 can negotiate acceptable capabilities of the SRC 203 and agree on a split sharing option, which can include existing reusable data streams, assets, media, and / or bitstreams already available through the MBS.

[0019] At 250, the SRS 204 can initiate a split rendering process, at 255, the SRC 203 can establish a shared session, such as a WebRTC session. At 260, the SRC 203 can subscribe to an available MBS session for delivery of common data streams, assets, media, and / or bitstreams, at 265, the SRC 203 can notify the application 201 that split rendering on the edge is running. At 270, the SRC 203 can send uplink metadata, such as pose and / or motion information, at 275, the SRS 204 can send rendered assets or media to the SRC 203. At 280, the SRS 204 can send shared media / assets to the SRC 203 over the MBS.

[0020] Figure 3Signal diagram examples are shown that establish a split rendering session in accordance with various example embodiments. Certain example embodiments can provide for configuration of, for example, an XR runtime application 301, a presentation engine 302 of a SRC, an XR source management function 303 of a SRC, a media access function 304 of a SRC, and an SRS 305. Certain example embodiments can provide one or more procedures including, at 310, the presentation engine 302 can discover the SRS 305 and can set up a connection to the SRS 305. The presentation engine 302 can provide information about its rendering capabilities and XR runtime configuration, for example, an OpenXR configuration. If a shared split rendering session is established, the session can provide session related parameters to enable optimizations (e.g., a shared session ID, a scene ID, and / or similar parameters). At 315, the SRS 305 can respond by creating a description of the split rendering output and input what the SRS 305 expects to receive from the SRC. If a shared split rendering session is established, the SRS 305 can provide information about existing shared data streams, assets, media, and / or bitstreams that are to be reused by the MBS.

[0021] At 320, the presentation engine 302 can request a buffered stream from the media access function 304, which in turn can establish a connection to the SRS 305 to stream, for example, poses, and can retrieve a split rendering buffer. At 325, the XR source management function 303 can retrieve poses and user input from the XR runtime 301, at 330, the XR source management function 303 can share information, for example, pose predictions and user input actions, with the SRS 305. At 335, the SRS 305 can use the shared information to render frames, at 340, the SRS 305 can encode and send the rendered frames to the XR source management function 303. If a shared split rendering session is established, the rendered frames can consist of a common shared view and, for example, additional user-specific information, for example, additional views and / or auxiliary data. At 345, the media access function 304 can decode the encoded frames and can process the buffered frames, at 350, the media access function 304 can transfer the decoded or original buffered frames to the presentation engine 302 and / or the XR runtime application 301 of the SRC for display. At 355, the XR runtime application 301 can compose and render the frames.

[0022] Certain example embodiments can provide a data model for split rendering configuration. For example, Table 1 below provides a definition of a split rendering configuration resource.

[0023]

[0024] Table 1 For example, some example embodiments can provide a sharedStatus parameter, which can indicate whether a split rendering session is part of a shared setup. As a further example, certain example embodiments can provide a sharedParameters parameter, which can describe shared parameters of a session. The shared parameters can include a session ID and status, and an existing public stream or session cluster that can be joined.

[0025] Some example embodiments can provide a policy template for independent split rendering. QoS specifications can include, for example, at least one or more of two configurations for left and right eye buffer streams, one optional configuration for depth buffer streams, and one configuration for audio streams. Independent split rendering can use WebRTC to transport rendered media in real-time, which can apply Real-Time Transport Protocol (RTP) restrictions for WebRTC. Certain example embodiments can also or instead provide a policy template for shared split rendering. QoS specifications can include, for example, at least one or more of one configuration for shared view, two configurations for left and right eye buffer streams, one optional configuration for depth buffer streams, one configuration for shared audio streams, and one configuration for audio streams. For shared split rendering, an MBS can transmit at least a common portion of rendered data or media, including, for example, a common rendered view or audio. WebRTC for user-specific additional data (e.g., additional views) can apply RTP restrictions for WebRTC.

[0026] Figure 4A and 4B An example is shown of a signal diagram for one or more processes for a shared split rendering session, in accordance with various example embodiments. Certain example embodiments can provide, for example, a configuration of multiple XR devices, such as a first XR device 401, an i-th XR device 402, and / or an n-th XR device 403. The configuration can also include, for example, a network manager 404, a server 405, and an application 406. Certain example embodiments can provide one or more processes, including, at 410, the application 406 can announce a shared XR split rendering session to all devices connected on a network, including, for example, the first XR device 401, the i-th XR device 402, the n-th XR device 403, the network manager 404, and the server 405. At 411-413, the first XR device 401, the i-th XR device 402, and the n-th XR device 403 can join the shared split rendering session, and can provide information about their capabilities, such as, for example, capabilities for display, decoding, rendering, and / or the like.

[0027] At 414, the application 406 can configure resources on the local servers (such as one or more local edge servers) to handle as many users as possible that can register. The application 406 can use any method to allocate the edge servers, or can allow the MNO to set up the edge servers to run the split rendering process. At 415, the shared split rendering session information can be shared among all devices (e.g., the first XR device 401, the ithXR device 402, the nthXR device 403, the network manager 404, and the application 406), including, for example, the relevant network IP addresses, the split rendering server IP addresses, and one or more associated ports. At 416-418, each of the first XR device 401, the ithXR device 402, and the nthXR device 403 can send initial configuration information, such as their positions in the three-dimensional space, current velocities, and / or current poses, to the server 405.

[0028] At 419, the server 405 can pre-configure the split rendering session. The server 405 can pre-configure by clustering each device into a number of user pools and identifying the commonalities that can be shared. The server 405 can then instantiate the required renderers, encoders, and packagers, and can decide which generated media streams can be sent to which user pool by PTM and PTP, and can configure the network manager 404 accordingly. At 420, the server 405 can send information to all devices (e.g., the first XR device 401, the ithXR device 402, the nthXR device 403, and the network manager 404) about the user pool they belong to respectively, and which public and user-specific data each device can need to receive. At 421-423, the first XR device 401, the ithXR device 402, and the nthXR device 403 can establish transport connections in order to receive the public and user-specific data streams, which can include, for example, webRTC session establishment, and subscribe to the corresponding multicast groups to receive the public data.

[0029] At 424-426, the first XR device 401, the i-th XR device 402, and the n-th XR device 403 can send their pose information and user actions to the server 405. At 427, the server 405 can perform rendering for the requested poses for each user pool and generate an appropriate number of public and user-specific media data streams. At 428, the server 405 can perform encoding and packing of the public and user-specific media streams, including, for example, video multi-view encoding. At 429, the public media data streams can be sent to different user pools through PTM, such as, for example, through the multicast groups to which they subscribe. At 430-432, the first XR device 401, the i-th XR device 402, and the n-th XR device 403 can each receive their user-specific media data streams using PTP, such as, for example, through their own webRTC unicast sessions. At 433, the first XR device 401, the i-th XR device 402, and the n-th XR device 403 can assemble, decode, and display the media data streams received through PTM and PTP.

[0030] Various example embodiments can provide an enablement process for reducing bandwidth used to distribute multi-user split rendering sessions. The process can include, for example, analyzing a space, such as a three-dimensional space, using two or three dimensional coordinates, and evaluating information related to the space, such as density and / or pose similarity. Each user can have an XR device, such as, for example, an XR headset. Users (e.g., the users’ XR devices) can be grouped or clustered based on the evaluation of the information related to the space.

[0031] A single reference content or media, e.g., a single reference view, can be generated for each cluster or group. The reference content / media can be an encoded multi-view reference media / content. For example, the users’ views can be encoded using multi-view with the reference view of the cluster as a base layer. The reference media (e.g., view) can be sent to the users’ XR devices using one or more point-to-multipoint (PTM) data streams, and one or more user-specific media (e.g., views) can be sent using point-to-point (PTP) data streams. Each XR device or user can subscribe to the PTM stream and can receive their PTP stream in parallel.

[0032] The bitstream can be assembled by merging the PTM and PTP sub-streams, which can then be fed to a decoder. The decoder can output multiple media, e.g., three views, where only the left and right views can be transmitted for display in the XR device, while the base view can not be displayed. By encoding the left / right views using a common proximate reference media (e.g., view), bandwidth savings can be achieved, as the encoding benefits from an enhanced predictor. Efficiency can be improved by, e.g., introducing multiple references. However, in this example, three views can be a more practical implementation for encoding and decoding complexity.

[0033] Simulations and experiments have been performed in accordance with various example embodiments. In the examples discussed herein, example encoding techniques can be used, e.g., anchor point: MV-HEVC reference software HTM 16.30 used in stereo mode. One view (left) is encoded as an I-frame, and the other (right) is encoded as a P-frame. The simulations can employ MV-HEVC reference software HTM 16.30, 3-view configuration, where the base view can be a common shared view, while the other views can be stereo views. The common view can be encoded as an I-frame, and the two views are encoded as P-frames that reference the common view. For example, four bitrate points can be generated for each configuration, using quantization parameters (QPs) = {25, 30, 35, 40}. A +3 delta QP can be applied to the P-frames, and the I-frame can be encoded using the base QP. For each user, the Bjontegaard delta rate (BD-RATE) can be computed by taking the average Y PSNR of the stereo views as the distortion metric, and the sum of the stereo data rates as the bitrate. If the selected common view can not be selected as a user view, but rather as a "virtual" view, additional data rate can be needed to perform the virtual view, which can be allocated among proximate users for the BD-RATE computation.

[0034] Figure 5A and 5B Examples of rendering viewports of a three-dimensional object are illustrated in accordance with certain example embodiments. Figure 5A and 5B Rendering views that can be presented to an encoder from different locations in space are illustrated. Figure 5A may be left and right views from a first location, e.g., (1, 1), Figure 5B may be left and right views from a second location, e.g., (5, 8).

[0035] Figure 6A and 6B Example experimental simulations of a case of N = 54 students are illustrated in accordance with some example embodiments. Figure 6AThis example simulation illustrates an experiment applicable to 54 students equipped with XR devices, with each student spaced, for example, 1 meter apart and a 1-meter height difference between rows. An object, such as a 3D object, can be positioned 2 meters away from the center of the first row of students. The students (with XR devices) can be clustered or grouped into 3x3 groups. The center student / XR device can be used as a reference in the left-eye view for encoding the surrounding students within the cluster / group.

[0036] Figure 6B Examples illustrating various data streams sent to users, from Figure 6A The layout provided in the codec. The rendered video can be encoded and then packaged before being sent to the user via a data stream. The public media stream can be sent using multicast transmission, while user-specific media streams can be sent via unicast. For example, user 0.0 can receive the public media stream via multicast and its two additional views via unicast. All three media streams can be assembled and fed into a multi-view decoder (MV-DEC), then pose-corrected using asynchronous time warp (ATW) and displayed.

[0037] Figure 7A and 7B Another example experimental simulation is illustrated based on certain example embodiments. Figure 7A This example simulation demonstrates that a constant user density can be provided on a 3x3 cluster. The reference view can be identified as the center user's left eye, which is likely the closest available view for surrounding users / XR devices. The public view can be delivered via multicast, while other views can be distributed specifically to each user via unicast. Each XR device can receive both multicast and unicast data streams simultaneously and can reconstruct a stereoscopic view based on the data streams. Figure 7B Explanation Figure 7A Example performance metrics for a shared segmentation rendering session for the clustering shown. On average, this configuration can reduce bandwidth (e.g., bit rate) on the transport link by 18.25%. It can be observed that 0% is achieved for the left-eye center user view used as a reference. The encoding configuration may not change compared to co-cast with the center user.

[0038] Figure 8A and 8B Another example experimental simulation is illustrated based on certain example embodiments. Figure 8A This describes a clustering method where users and XR devices can be grouped or clustered into 1x3 clusters. Figure 8B Explanation Figure 8A An example can provide a performance metric where bandwidth (e.g., bit rate) can be reduced by 17.17%.

[0039] Figure 9A and 9BAnother example experimental simulation is described in accordance with certain example embodiments. Figure 9A A clustering is described in which users and XR devices can be grouped or clustered into 2x3 constant clusters using alternative clustering and spatial analysis. Figure 9B A clustering is described in which users and XR devices can be grouped or clustered into 2x3 constant clusters using alternative clustering and spatial analysis. Figure 9A Examples of the above can provide a performance metric in which bandwidth (e.g., bitrate) can be reduced by 18.42%.

[0040] Figure 10A A clustering is described in which users and XR devices can be grouped or clustered into 2x3 constant clusters using alternative clustering and spatial analysis. 10B Another example experimental simulation is described in accordance with certain example embodiments. Figure 10A A clustering is described in which users and XR devices can be grouped or clustered according to the distance of the users to the display objects. As the distance to the objects distance increases, the left / right view displacement can decrease. Figure 10B A clustering is described in which users and XR devices can be grouped or clustered according to the distance of the users to the display objects. As the distance to the objects distance increases, the left / right view displacement can decrease. Figure 10A Examples of the above can provide a performance metric in which bandwidth (e.g., bitrate) for full transmission can be reduced by 19.38%.

[0041] In various example embodiments, the user used as a reference can be mobile, e.g., the user can leave the room, move away, or disconnect, which can, for example, disrupt the session of other users, thereby reducing the QoE of surrounding users in the cluster or group. Accordingly, some example embodiments can deploy a virtual user / view.

[0042] Figure 11A A clustering is described in which users and XR devices can be grouped or clustered into 2x3 constant clusters using alternative clustering and spatial analysis. 11B Another example experimental simulation is described in accordance with certain example embodiments using a virtual view. Figure 11A A clustering is described in which users and XR devices can be grouped or clustered into 2x3 constant clusters using alternative clustering and spatial analysis. Figure 11B A clustering is described in which users and XR devices can be grouped or clustered into 2x3 constant clusters using alternative clustering and spatial analysis. Figure 11A Examples of the above can provide a performance metric in which bandwidth (e.g., bitrate) for full transmission can be reduced by 26.95%. Simulating a virtual view / user can require additional bandwidth / bitrate, which can result in a reduction amount limited to 11.07%.

[0043] Figure 12A A clustering is described in which users and XR devices can be grouped or clustered into 2x3 constant clusters using alternative clustering and spatial analysis. 12B Another example experimental simulation is described in accordance with certain example embodiments using a virtual view. Figure 12A A clustering is described in which users and XR devices can be grouped or clustered into 2x3 constant clusters using alternative clustering and spatial analysis. Figure 12B A clustering is described in which users and XR devices can be grouped or clustered into 2x3 constant clusters using alternative clustering and spatial analysis. Figure 12AThe example can provide a performance metric for aligned views / users, where bandwidth (e.g., bit rate) can be reduced by 89.74%. Simulating virtual views / users may require additional bandwidth / bit rate, which could result in an overall system reduction of only 17.24%.

[0044] Figure 13A and 13B Another example experimental simulation using a virtual view is illustrated based on certain example implementations. Figure 13A This illustrates a clustering approach where users and XR devices can be grouped or clustered into 5x3 clusters or groups. In this example, the distance between users can be reduced to 0.5 meters, and the height difference between rows can be reduced to 0.5 meters. This configuration increases user density, which in turn increases the relevance between the views of neighboring users. Figure 13B Explanation Figure 13A An example can provide a performance metric where bandwidth (e.g., bit rate) can be reduced by 22.50% when considering simulated virtual views / users.

[0045] Various example embodiments can provide one or more methods for use, such as by utilizing k-means clustering with distance to virtual scene vectors as the criterion or by using pose similarity. Various other codecs or communication protocols, or combinations thereof, can be used. MV-HEVC may be advantageous because hardware HEVC encoders can be reused through software updates to support multi-layer decoding (e.g., multi-layer support only requires high-level syntax operations). As an example, 5G MBS can be used as a PTM transport protocol.

[0046] Figure 14 Example flowcharts of the method are illustrated based on certain example embodiments. In the example embodiments, Figure 14 The method can be performed by network elements or network entities in a 3GPP system (such as LTE or 5G-NR). For example, in an example embodiment, Figure 14 The method can be executed by the server, such as SRS 204 / 305, similar to... Figure 16 The device 1610 shown is shown.

[0047] According to various example embodiments, Figure 14 The method may include, at 1410, analyzing data relating to coordinates in a three-dimensional space in which the plurality of extended reality devices are located, and at 1420, generating one or more device groups from the plurality of extended reality devices. The method may also include, at 1430, generating one or more shared media contents for each of the one or more device groups.

[0048] Some example embodiments can provide that the method further comprises receiving data related to a plurality of extended reality devices located within a three-dimensional space, and evaluating a distribution and a pose of each of the plurality of extended reality devices within the three-dimensional space based on the data. The method can further comprise encoding media content for each of the one or more device groups, and transmitting the encoded media content to the one or more device groups.

[0049] Certain example embodiments can provide that the method further comprises encoding a plurality of perspective views in combination with a reference view as media content, the reference view as a base view for each of the one or more device groups, and transmitting the encoded plurality of perspective views and the reference view. The encoded media content can be transmitted through one or more point-to-multipoint data streams together with one or more user-specific media content transmitted via one or more point-to-point data streams, or the encoded plurality of perspective views and reference view can be transmitted through one or more point-to-multipoint data streams together with one or more user-specific views transmitted via one or more point-to-point data streams. Figure 1 One of the one or more point-to-multipoint data streams and the point-to-point data stream are transmitted in parallel to each of the plurality of extended reality devices.

[0050] Figure 15 An example flowchart of a method is illustrated in accordance with certain example embodiments. In example embodiments, Figure 15 The method of can be performed by a user equipment or user device in a 3GPP system, such as LTE or 5G-NR. For example, in example embodiments, Figure 15 The method of can be performed by a UE, XR device, or similar device, such as SRC 203, similar to the apparatus 1620 shown in Figure 16 . Figure 16 The method of can be performed by a UE, XR device, or similar device, such as SRC 203, similar to the apparatus 1620 shown in Figure 16 .

[0051] In accordance with various example embodiments, Figure 15 The method of can comprise, at 1510, transmitting data related to a shared session identifier of the apparatus and at least one other extended reality device, and, at 1520, receiving one or more encoded shared media content and one or more user-specific media content. The method can further comprise, at 1530, assembling a data stream by merging the one or more encoded shared media content and one or more user-specific media content, and, at 1540, inputting the assembled data stream to a data decoder for displaying the assembled data stream.

[0052] Certain example embodiments can provide sharing media content including multiple perspective views and a reference view. The one or more encoded shared media content are transmitted over one or more point-to-multipoint data streams along with the one or more user-specific media content transmitted via one or more point-to-point data streams, or the multiple perspective views and the reference view encoded are received over one or more point-to-multipoint data streams and one or more user-specific views are received via point-to-point data streams. The method can further include subscribing to the one or more point-to-multipoint data streams and point-to-point data streams. The one or more encoded shared media content and one or more user-specific media content can be received in parallel. The method can further include displaying the assembled data streams.

[0053] Figure 16 Apparatuses 1610 and 1620 are illustrated in accordance with various example embodiments. In various example embodiments, apparatus 1610 can be an element in or associated with a network, or a network entity, such as a server, S-NPN, or similar device. As noted above, SRS 204 / 305 can be an example of apparatus 1610 in accordance with various example embodiments. It should be noted that one of ordinary skill in the art would understand that apparatus 1610 can include components or features not shown in FIG. 16 or described above. Figure 16 In addition, apparatus 1620 can be a user device or other similar device connected to a network, such as a UE, XR device, XR headset, or similar device. SRC (e.g., SRC 203) can be an example of apparatus 1620 in accordance with various example embodiments described above. It should be noted that one of ordinary skill in the art would understand that apparatus 1620 can include components or features not shown in FIG. 17 or described above. Figure 16 In addition, apparatus 1620 can be a user device or other similar device connected to a network, such as a UE, XR device, XR headset, or similar device. SRC (e.g., SRC 203) can be an example of apparatus 1620 in accordance with various example embodiments described above. It should be noted that one of ordinary skill in the art would understand that apparatus 1620 can include components or features not shown in FIG. 17 or described above.

[0054] In accordance with various example embodiments, apparatuses 1610 and / or 1620 can include one or more processors, one or more computer-readable storage media (e.g., memory, storage, etc.), one or more radio access components (e.g., modems, transceivers, etc.), and / or user interfaces. In some example embodiments, apparatuses 1610 and / or 1620 can be configured to operate using one or more radio access technologies, such as GSM, LTE, LTE-A, NR, 5G, WLAN, WiFi, NB-IoT, Bluetooth, NFC, MulteFire, and / or any other radio access technology.

[0055] As Figure 16As shown by way of example, the apparatus 1610 and / or 1620 can include or be coupled to processors 1612 and 1622, respectively, for processing information and executing instructions or operations. The processors 1612 and 1622 can be any type of general or specific purpose processor. In fact, the processors 1612 and 1622 can include one or more of general -purpose computers, special purpose computers, microprocessors, digital signal processors (DSPs), field- programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), and processors based on a multi-core processor architecture, as examples. While two processors 1612 and 1622 are used in the illustrated example, other examples can use a single processor, three or more processors, or an integrated circuitic system with more than one processor. Figure 16 While a single processor 1612 (and 1622) is shown for each of the apparatus 1610 and / or 1620, multiple processors can be used according to other example embodiments. For example, it will be understood that, in certain example embodiments, the apparatus 1610 and / or 1620 can include two or more processors that can form a multi-processor system (e.g., in which case the processors 1612 and 1622 can represent a multi-processor), which can support multiprocessing. According to certain example embodiments, the multi-processor system can be tightly coupled or loosely coupled, e.g., forming a computer cluster.

[0056] The processors 1612 and 1622 can perform functions related to the operation of the apparatus 1610 and / or 1620, respectively, including, as some examples, precoding of antenna gain / phase parameters, encoding and decoding of individual bits making up a communication message, formatting of information, and overall control of the apparatus 1610 and / or 1620, including processes described in the detailed description. Figures 2-15

[0057] The apparatus 1610 and / or 1620 can further include or be coupled to memories 1614 and / or 1624 (internal or external), which can be coupled to the processors 1612 and / or 1622, respectively, for storing information and instructions that can be executed by the processors 1612 and 1622. The memories 1614 (and 1624) can be one or more memories and can be of any type suitable to the local application environment, and can employ any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory and / or removable memory. The memories 1614 (and 1624) can comprise any combination of any type of volatile or non-volatile memory suitable for use in a local application environment, such as random access memory (RAM), read only memory (ROM), static storage such as a magnetic or optical disk, hard disk drive (HDD), or any other type of non-transitory machine or computer readable media. The instructions stored in the memories 1614 and 1624 can include programs or computer program code that, when executed by the processors 1612 and 1622, enable the apparatus 1610 and / or 1620 to perform the tasks described herein.​

[0058] In certain example embodiments, the apparatus 1610 and / or 1620 can further include or be coupled to (internal or external) drives or ports configured to accept and read external computer readable storage media, such as an optical disc, USB drive, flash drive, or any other storage medium. For example, the external computer readable storage medium can store a computer program or software for execution by the processors 1612 and 1622 and / or the apparatus 1610 and / or 1620 to perform any of the methods illustrated in FIG. 16. Figures 2-15

[0059] In some example embodiments, the apparatus 1610 and / or 1620 can also include or be coupled to one or more antennas 1615 and 1625 for receiving downlink signals and transmitting through the uplink from the apparatus 1610 and / or 1620, respectively. The apparatus 1610 and / or 1620 can further include transceivers 1616 and 1626 configured to transmit and receive information, respectively. The transceivers 1616 and 1626 can also include radio interfaces (e.g., modems) coupled to the antennas 1615 and 1625, respectively. The radio interfaces can correspond to one or more of GSM, LTE, LTE-A, 5G, NR, WLAN, NB-IoT, Bluetooth, BT-LE, NFC, RFID, UWB, etc. The radio interfaces can include other components, such as filters, converters (e.g., digital-to-analog converters, etc.), symbol demappers, signal shaping components, inverse fast Fourier transform (IFFT) modules, etc., to process symbols, such as OFDMA symbols, carried by the downlink or uplink.

[0060] For example, the transceivers 1616 and 1626 can be configured to modulate information on to carrier wave(s) to be transmitted through the antennas 1615 and 1625, respectively, and demodulate information received through the antennas 1615 and 1625, respectively, for further processing by other elements of the apparatus 1610 and / or 1620. In other example embodiments, the transceivers 1616 and 1626 can be capable of transmitting and receiving signals or data directly. Additionally or alternatively, in some example embodiments, the apparatus 1610 and / or 1620 can include input and / or output devices (I / O devices). In certain example embodiments, the apparatus 1610 and / or 1620 can further include a user interface, such as a graphical user interface or touchscreen.

[0061] ​In certain example embodiments, the memories 1614 and 1624 can store software modules that provide functionality when executed by the processors 1612 and 1622, respectively. The modules can include, for example, an operating system that provides operating system functionality for the apparatuses 1610 and / or 1620. The memories can also store one or more functional modules, such as an application program or program, to provide additional functionality for the apparatuses 1610 and / or 1620. The components of the apparatuses 1610 and / or 1620 can be implemented in hardware, or as any suitable combination of hardware and software. According to certain example embodiments, the apparatus 1610 can be selectively configured to communicate with the apparatus 1620 according to any radio access technology (e.g., NR) over a wireless or wired communication link 1630.

[0062] According to certain example embodiments, the processors 1612 and 1622 and the memories 1614 and 1624 can be included in or can form a part of processing circuitry or control circuitry. Furthermore, in some example embodiments, the transceivers 1616 and 1626 can be included in or can form a part of transceiving circuitry.

[0063] For example, in certain example embodiments, the apparatus 1610 can be controlled by the memory 1614 and the processor 1612 to analyze data related to coordinates within a three-dimensional space in which a plurality of extended reality devices are located and to generate one or more groups of devices from the plurality of extended reality devices. The apparatus 1610 can also be controlled to generate one or more shared media contents for each of the one or more groups of devices.

[0064] In some example embodiments, the apparatus 1620 can be controlled by the memory 1624 and the processor 1622 to transmit data related to a shared session identifier of the apparatus and at least one other extended reality device and to receive one or more encoded shared media contents and one or more user-specific media contents. The apparatus 1620 can also be controlled to assemble a data stream by merging the one or more encoded shared media contents and the one or more user-specific media contents and to input the assembled data stream into a data decoder for displaying the assembled data stream.

[0065] In some example embodiments, an apparatus (e.g., apparatus 1610 and / or apparatus 1620) can include means for performing the methods, processes, or any variants discussed herein. Examples of means can include one or more processors, memories, controllers, transmitters, receivers, and / or computer program codes for causing the operations to be performed.

[0066] Various example embodiments can relate to an apparatus, such as apparatus 1610, comprising means for analyzing data related to coordinates within a three-dimensional space in which a plurality of extended reality devices are located, and means for generating one or more groups of devices from the plurality of extended reality devices. The apparatus 1610 can further comprise means for generating one or more shared media contents for each of the one or more groups of devices.

[0067] Certain example embodiments can relate to an apparatus, such as apparatus 1620, comprising means for transmitting data related to a shared session identifier of the apparatus and at least one other extended reality device, and means for receiving one or more encoded shared media contents and one or more user-specific media contents. The apparatus 1620 can further comprise means for assembling a data stream by merging the one or more encoded shared media contents and the one or more user-specific media contents, and means for inputting the assembled data stream into a data decoder for displaying the assembled data stream.

[0068] As used herein, the term "circuitry" can refer to a purely hardware circuit implementation (e.g., analog and / or digital circuitry), combinations of hardware circuits and software, combinations of analog and / or digital hardware circuits with software / firmware, any portions of hardware processor(s) including digital signal processors that co-operate to cause an apparatus (e.g., apparatus 1610 and / or 1620) to perform various functions, and / or hardware circuitry and / or processors, or portions thereof, that use software for operation but can not require software during operation when not in active use. As a further example, as used herein, the term "circuitry" can also cover an implementation that is solely a hardware circuit or processor or multiple processors, or portions of hardware circuitry or processor(s), and accompanying software and / or firmware. The term circuitry can also cover, for example, a baseband integrated circuit in a server, cellular network node or device, or other computing or network device.

[0069] A computer program product can comprise one or more computer-executable components which, when the program is run, are configured to carry out some of the example embodiments. The one or more computer-executable components can be at least one software code or portions thereof. Modifications and configurations needed for implementing certain example embodiments can be executed as routines which can be implemented as added or updated software routines. The software routines can be downloaded to the apparatus.

[0070] As an example, software or computer program code or portions thereof can be in source code form, object code form, or in some intermediate form, and it can be stored in some form or medium alone or in combination with other data. A "computer-readable medium," "computer-readable storage medium," or "physical computer-readable medium" can be, for example, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable medium include an electrical connection, hard-wired digital gate, random access memory (RAM), read-only memory (ROM), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), compact disk (CD) ROM, digital tape, magnetic disk storage or other computer storage devices, including memories, accumulators, and registers of the described circuitry. The computer-readable medium can also be, for example, but is not limited to, a carrier wave, a transmitted signal, or a computer data base, software, and / or distributed quantum information sent on or via a communication network and / or through a communication link. The computer-readable medium can be a non-transitory medium. The term "non-transitory" can also be used in some examples to describe a computer-readable medium. The term "non-transitory" can also be used herein to more particularly exclude plausibly transitory computer-readable media, such as signals, waves, and other forms that are not stored or persistently stored on a computer-readable medium, such as a memory, a storage device, or a computer.

[0071] In other example embodiments, the functions can be performed by hardware or circuitry contained in an apparatus (e.g., the apparatuses 1610 and / or 1620), such as by using an application specific integrated circuit (ASIC), a programmable gate array (PGA), a field programmable gate array (FPGA), or any other combination of hardware and software. In yet another example embodiment, the functions can be implemented as a signal, a non-tangible item that can be carried by an electromagnetic signal downloaded from the Internet or other network.

[0072] According to certain example embodiments, an apparatus, such as a node, device, or corresponding component, can be configured as a circuit, computer, or microprocessor, such as a single-chip computer element, or as a chipset, including at least a memory for providing storage capacity to the arithmetic operation and an operation processor for performing the arithmetic operation.

[0073] The features, structures, or characteristics of the example embodiments described in this specification can be combined in any suitable manner in one or more example embodiments. For example, the usage of the phrases "certain embodiments", "one example embodiment", "some embodiments" or other similar language throughout this specification refers to the fact that a particular feature, structure, or characteristic described in connection with an embodiment can be included in at least one embodiment. Thus, appearances of the phrases "in certain embodiments", "in one example embodiment", "in some embodiments", "in other embodiments", or other similar language throughout this specification may, but do not necessarily, all refer to the same group of embodiments, and the described features, structures, or characteristics can be combined in any suitable manner in one or more example embodiments. In addition, the terms "cell", "node", "gNB", or other similar language can be used interchangeably throughout this specification.

[0074] As used herein, "at least one of " and "one or more of " and similar phrases, where the list of two or more elements is connected by "and" or "or", means at least the first element, or at least the second element, or at least one of the first or second element, or at least one of the first, second, or third element, and so on.

[0075] Those of ordinary skill in the art will readily understand that the present disclosure as described above can be practiced with different orders of procedures and / or different hardware element configurations than those disclosed. Accordingly, although the present disclosure has been described based upon these example embodiments, it would be apparent to those of skill in the art that certain modifications, variations, and alternative constructions would be apparent, while remaining within the spirit and scope of the example embodiments. Although the above embodiments relate to 5G NR and LTE technologies, the above embodiments can also be applicable to any other now or future 3GPP technologies, such as LTE-advanced and / or fourth generation (4G) and / or sixth generation (6G) technologies.

[0076] Partial Glossary: 3GPP third generation partnership project 5G fifth generation 6G sixth generation AF application function BD-RATE Bjontegaard delta rate DL downlink EMBB enhanced mobile broadband gNB 5G or next generation NodeB HEVC high efficiency video coding ID identifier LTE long term evolution MBS multicast broadcast service NR new radio PTM point to multipoint PTP point to point QoE quality of experience QoS quality of service RTC real time communication RTP real time transport protocol SRC segmentation rendering client SRS segmentation rendering server UE user equipment UL uplink XR extended reality

Claims

1. An apparatus comprising: A component used to analyze data related to the coordinates within the three-dimensional space where multiple extended reality devices are located; Components for generating one or more device groups from the plurality of extended reality devices; as well as A component for generating one or more shared media contents for each of the one or more groups of devices.

2. The apparatus according to claim 1, further comprising: Components for receiving data associated with the plurality of extended reality devices located within the three-dimensional space.

3. The apparatus according to claim 1, further comprising: A component used to evaluate the distribution and orientation of each of the plurality of extended reality devices in the three-dimensional space based on the data.

4. The apparatus according to any one of claims 1 to 3, further comprising: Components for encoding media content for each of the one or more groups of devices; as well as Components for transmitting the encoded media content to the one or more device groups.

5. The apparatus according to any one of claims 1 to 4, further comprising: A component for encoding multiple perspective views into media content by combining a reference image, the reference image serving as a base view for each of the one or more groups of devices; as well as Components used for transmission encoding of the plurality of viewpoints and the reference view.

6. The apparatus according to claim 4 or claim 5, wherein: The encoded media content is transmitted together with one or more user-specific media content transmitted via one or more point-to-multipoint data streams; or The encoded multiple viewpoints and the reference view are transmitted together with one or more user-specific views transmitted via one or more point-to-multipoint data streams.

7. The apparatus of claim 6, wherein one of the one or more point-to-multipoint data streams and the point-to-point data stream are transmitted in parallel to each of the plurality of extended reality devices.

8. An apparatus comprising: Components for transmitting data associated with a shared session identifier of the device and at least one other extended reality device; A component for receiving one or more encoded shared media contents and one or more user-specific media contents; Components for assembling a data stream by combining one or more encoded shared media content and one or more user-specific media content; as well as A component for inputting the assembled data stream into a data decoder for displaying the assembled data stream.

9. The apparatus of claim 8, wherein the one or more shared media contents include a plurality of perspective views and reference views.

10. The apparatus according to claim 8 or claim 9, wherein: The one or more encoded shared media content is transmitted together with the one or more user-specific media content transmitted via one or more peer-to-peer data streams through one or more peer-to-peer data streams; or The encoded multiple viewpoints and the reference view are received via one or more point-to-multipoint data streams, and one or more user-specific views are received via point-to-point data streams.

11. The apparatus of claim 10, further comprising: Components for subscribing to the one or more point-to-multipoint data streams and the point-to-point data streams.

12. The apparatus according to any one of claims 8 to 11, wherein the one or more encoded shared media content and the one or more user-specific media content are received in parallel.

13. The apparatus according to any one of claims 8 to 12, further comprising: Components used to display the assembled data stream.

14. A method comprising: Analyze data related to the coordinates of multiple extended reality devices in the three-dimensional space. Generate one or more device groups from the plurality of extended reality devices; as well as Generate one or more shared media contents for each of the one or more device groups.

15. The method of claim 14, further comprising: Receive the data associated with the plurality of extended reality devices located within the three-dimensional space.

16. The method of claim 14, further comprising: Based on the data, the distribution and orientation of each of the plurality of extended reality devices in the three-dimensional space are evaluated.

17. The method according to any one of claims 14 to 16, further comprising: Encode media content for each of the one or more groups of devices; as well as Transmit the encoded media content to the one or more device groups.

18. The method according to any one of claims 14 to 17, further comprising: Multiple perspective views are combined with a reference view and encoded into media content, the reference view serving as the base view for each of the one or more device groups; as well as The multiple viewpoints and the reference view of the transmission encoding.

19. The method according to claim 17 or claim 18, wherein: The encoded media content is transmitted along with one or more user-specific media contents transmitted via one or more peer-to-peer data streams; or The encoded multiple viewpoints and the reference view are transmitted together with one or more user-specific views transmitted via one or more point-to-multipoint data streams.

20. The method of claim 19, wherein one of the one or more point-to-multipoint data streams and the point-to-point data stream are transmitted in parallel to each of the plurality of extended reality devices.

21. A method comprising: Transmit data associated with a shared session identifier of the device and at least one other extended reality device; Receive one or more encoded shared media content and one or more user-specific media content; The data stream is assembled by merging the one or more encoded shared media content and the one or more user-specific media content; as well as The assembled data stream is input into a data decoder for display.

22. The method of claim 21, wherein the one or more shared media contents include a plurality of perspective views and reference views.

23. The method according to claim 21 or claim 22, wherein: The one or more encoded shared media content is transmitted together with the one or more user-specific media content transmitted via one or more peer-to-peer data streams through one or more peer-to-peer data streams; or The encoded multiple viewpoints and the reference view are received via one or more point-to-multipoint data streams, and one or more user-specific views are received via point-to-point data streams.

24. The method of claim 23, further comprising: Subscribe to one or more point-to-multipoint data streams and the point-to-point data stream.

25. The method according to any one of claims 21 to 24, wherein the one or more encoded shared media content and the one or more user-specific media content are received in parallel.

26. The method according to any one of claims 21 to 25, further comprising: The assembled data stream is displayed.

Images