Provision of detailed device capabilities for split rendering negotiation

Abstract

The method and apparatus include computer code configured to cause a processor or processors to determine the capabilities of an augmented reality (AR) device—the capabilities indicate either an audio encoder or decoder of the AR device and a video encoder or decoder of the AR device as part of the AR device acting as a split rendering client (SRC)—; to acquire AR data of at least one media component of audio and video based on a request to signal the capabilities of the AR device to a server—the server acts as a split rendering server (SRS) for split rendering of the media component between the SRS and the SRC and provides the AR data—; and to process the AR data based on the capabilities of the AR device signaled to the SRC.

Description

Technology Field

[0001] Cross-reference regarding related applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 623,763 filed January 22, 2024 and U.S. Application No. 19 / 028,708 filed January 17, 2025, the contents of which are hereby expressly incorporated by reference in their entirety into this application.

[0003] field

[0004] The present disclosure provides a method for providing device capabilities for negotiation between a device and a network to set up a split-rendering session. Background Technology

[0005] 3GPP has a work item on split rendering of media delivery services, where client media functions are split between the device and the network edge. Consequently, clients can run less demanding processes more lightly and receive more complex applications and services. In turn, the edge network will receive media, decode it, and render it in a partially simpler form, enabling clients to run lighter processes.

[0006] 5G augmented reality devices need to perform intensive processing including multiple parallel media decoding and possible media encoding, scene composition, and augmented reality rendering.

[0007] If an Application and / or Application Service Provider decide to operate client media functions using a split-rendering method, they must replace this functionality with the following two new modules: 1. an edge-dependent light media service client, and 2. a media processing application running on a 5GMS AS.

[0008] The current specification defines split rendering configuration parameters that include desired view information. The device is expected to transmit view position information for those views back to the network. However, the configuration parameters do not define any specifications regarding how often and at what increments that position information should be captured and transmitted to the network.

[0009] To solve one or more different technical problems, the present disclosure provides technical solutions that enable some of the utility and technical signaling features to be improved in using such operations by providing options for applying various operations to the solved elements while reducing network overhead and server computation overhead.

[0010] Therefore, due to any of these reasons, there is a demand for technical solutions to these problems arising in video coding technology.

[0011] A method and apparatus are provided, comprising memory configured to store computer program code, and a processor or processors configured to access computer program code and operate as instructed by computer program code. The computer program is configured to enable the processor to implement code acquisition, the code is configured to enable at least one processor to perform a conversion between a visual media file and a bitstream of visual media data according to a format rule, the format rule being: determining the capabilities of an augmented reality (AR) device—the capabilities being indicated as either an audio encoder or decoder of the AR device and a video encoder or decoder of the AR device as part of the AR device acting as a split rendering client (SRC)—; acquiring AR data of at least one media component of audio and video based on a request signaling the capabilities of the AR device to a server—the server acting as a split rendering server (SRS) for split rendering of the media component between the SRS and the SRC and providing the AR data—; and instructing the SRC to process the AR data based on the capabilities of the AR device signaled to the SRC.

[0012] A request to signal the capabilities of an AR device is in the form of a uniform resource indicator (URI) and may include a profile identifier containing the syntax of "deviceType".

[0013] Profile identifiers can indicate extended reality (XR) runtime and scene manager capabilities.

[0014] A request to signal the capabilities of an AR device may include a list of at least some of the capabilities of the AR device, and at least some of the capabilities of the AR device indicate optional codecs of the AR device.

[0015] At least some of the capabilities of the AR device may additionally specify the number of interfaces and instances of the AR device’s optional codecs, Extended Reality (XR) runtime profiles, optional extensions, scene description formats, and profiles supported by the scene description formats.

[0016] A request signaling the capabilities of an AR device is in the form of a uniform resource indicator (URI) and may additionally include a profile identifier containing the syntax of "deviceType", the syntax indicating a list of at least some of the capabilities is "deviceDetailedCapabilities", and the profile identifier may indicate Extended Reality (XR) runtime and scene manager capabilities other than at least some of the capabilities of the AR device. Brief explanation of the drawing

[0017] Additional features, nature, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings. FIG. 1 is a simplified block diagram of a communication system according to embodiments. FIG. 2 is a simplified example of an encoder and decoder environment according to embodiments. FIG. 3 is a simplified block diagram of decoders according to embodiments. FIG. 4 is a simplified block diagram of encoders according to embodiments. FIG. 5 is a simplified block diagram of an AR system according to embodiments. FIG. 6 is a simplified block diagram of a 5G system according to embodiments. FIG. 7 is a simplified block diagram in a 5G UE environment according to embodiments. FIG. 8 is a simplified block diagram in an EDGAR environment according to embodiments. FIG. 9 is a simplified block diagram in an EDGAR UE environment according to embodiments. FIG. 10 is a simplified diagram of AR usage according to embodiments. FIG. 11 is a simplified block diagram of a mixed AR and non-AR system according to embodiments. FIG. 12 is a simplified block diagram of a non-AR UE according to embodiments. FIG. 13 is a simplified block and timing diagram according to embodiments. FIG. 14 is a simplified block diagram of a 5GMS architecture according to embodiments. FIG. 15 is a simplified block diagram of an MSE framework according to embodiments. FIG. 16 is a simplified block diagram of a 5GMSd-Aware Application according to embodiments. FIG. 17 is a simplified flowchart according to embodiments. FIG. 18 is a simplified block diagram according to embodiments. FIG. 19 is a schematic example according to embodiments. Specific details for implementing the invention

[0018] The proposed features discussed below may be used individually or combined in any order. Additionally, embodiments may be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored on a non-transient computer-readable medium.

[0019] FIG. 1 illustrates a simplified block diagram of a communication system (100) according to one embodiment of the present disclosure. The communication system (100) may include at least two terminals (102 and 103) interconnected via a network (105). For unidirectional transmission of data, the first terminal (103) may encode video data at a local location to transmit to another terminal (102) via the network (105). The second terminal (102) may receive the coded video data from the other terminal from the network (105), decode the coded data, and display the recovered video data. Unidirectional data transmission may be common in media serving applications, etc.

[0020] FIG. 1 illustrates a second pair of terminals (101 and 104) provided to support bidirectional transmission of coded video that may occur, for example, during a video conference. For bidirectional transmission of data, each terminal (101 and 104) may code video data captured at a local location to transmit to another terminal via a network (105). Each terminal (101 and 104) may also receive coded video data transmitted by another terminal, decode the coded data, and display the recovered video data on a local display device.

[0021] In FIG. 1, terminals (101, 102, 103, and 104) may be exemplified as servers, personal computers, and smartphones, but the principles of the present disclosure are not so limited. Embodiments of the present disclosure are applied in laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network (105) represents any number of networks that transmit coded video data between terminals (101, 102, 103, and 104), including, for example, wired and / or wireless communication networks. The communication network (105) may exchange data on circuit-switched and / or packet-switched channels. Representative networks include communication networks, local area networks, wide area networks, and / or the Internet. For the purposes of this discussion, the architecture and topology of the network (105) may not be important to the operation of the present disclosure unless described herein.

[0022] FIG. 2 illustrates the arrangement of a video encoder and a decoder in a streaming environment as an example of an application for the disclosed subject. The disclosed subject may be equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0023] A streaming system may include a capture subsystem (203), which may include a video source (201), for example, a digital camera, that generates an uncompressed video sample stream (213). The sample stream (213) may be highlighted as having a high data volume compared to encoded video bitstreams and may be processed by an encoder (202) coupled to the camera (201). The encoder (202) may include hardware, software, or a combination thereof to enable or implement the embodiments of the disclosed subject matter as described in more detail below. An encoded video bitstream (204), which may be highlighted as having a lower data volume compared to the sample stream, may be stored on a streaming server (205) for future use. One or more streaming clients (212 and 207) may access the streaming server (205) to retrieve copies (208 and 206) of the encoded video bitstream (204). The client (212) may include a video decoder (211) that decodes an incoming copy (208) of an encoded video bitstream and generates an outgoing video sample stream (210) that can be rendered on a display (209) or another rendering device (not described). In some streaming systems, the video bitstreams (204, 206 and 208) may be encoded according to specific video coding / compression standards. Examples of such standards have been mentioned above and are further described herein.

[0024] FIG. 3 may be a functional block diagram of a video decoder (300) according to an embodiment of the present invention.

[0025] A receiver (302) may receive one or more codec video sequences to be decoded by a decoder (300); in the same or other embodiments, may receive one coded video sequence at a time, wherein the decoding of each coded video sequence is independent of other coded video sequences. A coded video sequence may be received from a channel (301), which may be a hardware / software link to a storage device storing encoded video data. The receiver (302) may receive the encoded video data along with other data, e.g., encoded audio data and / or auxiliary data streams, which may be forwarded to their respective use entities (not described). The receiver (302) may separate the encoded video sequence from the other data. To prevent network jitter, a buffer memory (303) may be combined between the receiver (302) and the entropy decoder / parser (304) (hereinafter "parser"). When the receiver (302) receives data from a storage / forward device with sufficient bandwidth and controllability, or from an isosynchronous network, the buffer (303) may not be needed or may be small. For use on best effort packet networks such as the Internet, the buffer (303) may be required, may be relatively large, and advantageously may be of an adaptive size.

[0026] The video decoder (300) may include a parser (304) to reconstruct symbols (313) from an entropy-coded video sequence. The categories of the symbols include information used to manage the operation of the decoder (300), and information for controlling a rendering device, such as a display (312), which may be coupled to the decoder but is not an integral part of the decoder. The control information for the rendering device(s) may be in the form of Supplementary Enhancement Information (SEI) messages or Video Usability Information (VUI) parameter set fragments (not described). The parser (304) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may follow video coding techniques or standards and may follow principles well known to those skilled in the art, including variable-length coding, Huffman coding, and arithmetic coding with or without context sensitivity. The parser (304) may extract a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to a group. The subgroups may include Group of Pictures (GOPs), pictures, tiles, slices, macroblocks, Coding Units (CUs), blocks, Transform Units (TUs), Prediction Units (PUs), etc. The entropy decoder / parser may also extract information such as transform coefficients, quantizer parameter values, motion vectors, etc. from the coded video sequence.

[0027] The parser (304) can generate symbols (313) by performing an entropy decoding / parsing operation on a video sequence received from the buffer (303). The parser (304) can receive encoded data and selectively decode specific symbols (313). Additionally, the parser (304) can determine whether specific symbols (313) will be provided to a motion compensation prediction unit (306), a scaler / inverse transformation unit (305), an intra prediction unit (307), or a loop filter (311).

[0028] The reconstruction of the symbols (313) may involve a number of different units depending on the type of the coded video picture or its parts (e.g., inter- and intra-picture, inter- and intra-block), and other factors. Which units are involved, and the manner in which, may be controlled by subgroup control information parsed from the coded video sequence by the parser (304). The flow of this subgroup control information between the parser (304) and the number of units below is not described for clarity.

[0029] In addition to the functional blocks already mentioned, the decoder (300) may be conceptually subdivided into a number of functional units as described below. In actual implementations operating under commercial constraints, many of these units may interact closely with one another and be at least partially integrated with one another. However, for the purpose of illustrating the disclosed subject, the conceptual subdivision into the functional units below is appropriate.

[0030] The first unit is a scaler / inverse transform unit (305). The scaler / inverse transform unit (305) receives quantized transform coefficients as well as control information, including which transform to use, block size, quantization factor, quantization scaling matrices, etc., as symbol(s) (313) from the parser (304). This can output blocks containing sample values ​​that can be input to an aggregator (310).

[0031] In some cases, the output samples of the scaler / inverse transform (305) may relate to an intra-coded block; that is, a block that uses prediction information from previously reconstructed parts of the current picture rather than prediction information from previously reconstructed pictures. Such prediction information may be provided by an intra-picture prediction unit (307). In some cases, the intra-picture prediction unit (307) uses surrounding already reconstructed information fetched from the current (partially reconstructed) picture (309) to generate a block of the same size and shape as the block being reconstructed. In some cases, the aggregator (310) adds the prediction information generated by the intra-prediction unit (307) to the output sample information provided by the scaler / inverse transform unit (305) on a per-sample basis.

[0032] In other cases, the output samples of the scaler / inverse unit (305) may be intercoded and potentially associated with a motion-compensated block. In such cases, the motion compensation prediction unit (306) may access the reference picture memory (308) to fetch samples used for prediction. After motion-compensating the fetched samples according to the symbols (313) associated with the block, these samples may be added to the output of the scaler / inverse unit (in this case, called residual samples or residual signals) by the aggregator (310) to generate output sample information. The addresses in the reference picture memory where the motion compensation unit fetches prediction samples may be controlled by motion vectors available to the motion compensation unit in the form of symbols (313) that may have, for example, X, Y, and reference picture components. Motion compensation may also include interpolation of sample values ​​fetched from the reference picture memory when sub-sample accurate motion vectors are in use, motion vector prediction mechanisms, etc.

[0033] Various loop filtering techniques within the loop filter unit (311) may be performed on the output samples of the aggregator (310). Video compression techniques may include in-loop filter techniques that are made available to the loop filter unit (311) as symbols (313) from the parser (304) and are controlled by parameters included in the coded video bitstream, but which respond not only to meta-information acquired during the decoding of previous (in decoding order) parts of the coded picture or coded video sequence, but also to previously reconstructed and loop-filtered sample values.

[0034] The output of the loop filter unit (311) may be a sample stream that is not only output to a display (312) which may be a rendering device, but may also be stored in a reference picture memory (557) for use in future inter-picture predictions.

[0035] Certain coded pictures, when fully reconstructed, can be used as reference pictures for future prediction. When a coded picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (304)), the current reference picture (309) can become part of the reference picture buffer (308), and a new current picture memory can be reallocated before starting the reconstruction of the next coded picture.

[0036] The video decoder (300) can perform decoding operations according to a predetermined video compression technique that may be documented in a standard such as ITU-T Rec. H.265. The coded video sequence may comply with the syntax specified by the video compression technique or standard in use, in that it adheres to the syntax of the video compression technique or standard as specified in the video compression technique document or standard and specifically in the profile document within it. Additionally, for compliance, it may be necessary for the complexity of the coded video sequence to be within the boundaries defined by the levels of the video compression technique or standard. In some cases, the levels limit the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the levels may, in some cases, be further limited through HRD (Hypothetical Reference Decoder) specifications and metadata for HRD buffer management signaled in the coded video sequence.

[0037] In one embodiment, the receiver (302) may receive additional (redundant) data along with the encoded video. This additional data may be included as part of the encoded video sequence(s). This additional data may be used by the video decoder (300) to properly decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0038] FIG. 4 may be a functional block diagram of a video encoder (400) according to an embodiment of the present disclosure.

[0039] The encoder (400) can receive video samples from a video source (401) (not part of the encoder) capable of capturing video image(s) to be coded by the encoder (400).

[0040] A video source (401) may provide a source video sequence to be coded by an encoder (303) in the form of a digital video sample stream, which may have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media serving system, the video source (401) may be a storage device that stores previously prepared video. In a video conferencing system, the video source (401) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of individual pictures that impart motion when viewed sequentially. The pictures themselves may be organized as a spatial array of pixels, where each pixel may contain one or more samples depending on the sampling structure, color space, etc. being used. A person skilled in the art can easily understand the relationship between pixels and samples. The following description focuses on the samples.

[0041] According to one embodiment, the encoder (400) can code and compress pictures of a source video sequence into a coded video sequence (410) in real time or under any other time constraints required by the application. Implementing an appropriate coding rate is one function of the controller (402). The controller controls other functional units as described below and is functionally coupled to these units. The coupling is not described for clarity. Parameters set by the controller may include rate control related parameters (picture skip, quantizer, lambda values ​​of rate-distortion optimization techniques, ...), picture size, picture group (GOP) layout, maximum motion vector search range, etc. A person skilled in the art can easily identify the other functions of the controller (402) because they may be related to the video encoder (400) optimized for a specific system design.

[0042] Some video encoders operate in what a person skilled in the art would easily recognize as a "coding loop." By oversimplification, the coding loop may consist of an encoding portion of an encoder (e.g., a source coder (403)) (responsible for generating symbols based on the input picture to be coded and reference picture(s)), and a (local) decoder (406) embedded in an encoder (400) that reconstructs the symbols to generate sample data that is also generated by a (remote) decoder (since any compression between the symbols and the coded video bitstream is lossless in the video compression techniques considered in the subject of disclosure). The reconstructed sample stream is input into the reference picture memory (405). Because the decoding of the symbol stream yields bit-exact results independently of the decoder location (local or remote), the reference picture buffer content is also bit exact between the local encoder and the remote encoder. In other words, the prediction part of the encoder "sees" as reference picture samples exactly the same sample values ​​that the decoder "sees" when using prediction during decoding. This basic principle of reference picture synchronicity (and the resulting drift when synchronicity cannot be maintained, for example, due to channel errors) is well known to those skilled in the art.

[0043] The operation of the “local” decoder (406) may be the same as the operation of the “remote” decoder (300) already described in detail above in relation to FIG. 3. However, with brief reference to FIG. 4 as well, since symbols are available and the encoding / decoding of symbols into a coded video sequence by the entropy coder (408) and parser (304) may be lossless, the entropy decoding parts of the decoder (300), including the channel (301), receiver (302), buffer (303), and parser (304), may not be fully implemented in the local decoder (406).

[0044] An observation that can be made at this point is that any decoder technique, excluding the parsing / entropy decoding present in the decoder, also inevitably needs to exist in the corresponding encoder in substantially the same functional form.

[0045] Since encoder technologies are the inverse of the comprehensively described decoder technologies, their descriptions may be condensed. More detailed explanations are required only in specific areas and are provided below.

[0046] As part of the operation, the source coder (403) may perform motion-compensated predictive coding, which predictively codes an input frame by referencing one or more previously coded frames from a video sequence designated as "reference frames." In this way, the coding engine (407) codes the differences between pixel blocks of the input frame and pixel blocks of the reference frame(s) that can be selected as predictive reference(s) for the input frame.

[0047] The local video decoder (406) can decode the coded video data of frames that can be designated as reference frames based on the symbols generated by the source coder (403). The operations of the coding engine (407) may advantageously be lossy processes. If the coded video data can be decoded by a video decoder (not shown in FIG. 4), the reconstructed video sequence may be a replica of the source video sequence, typically having some errors. The local video decoder (406) can replicate the decoding processes that can be performed by the video decoder on the reference frames and allow the reconstructed reference frames to be stored in a reference picture memory (405), which may be, for example, a cache. In this way, the encoder (400) can locally store copies of the reconstructed reference frames having common content as reconstructed reference frames to be acquired by the far-end video decoder (without transmission errors).

[0048] The predictor (404) can perform prediction searches for the coding engine (407). That is, for a new frame to be coded, the predictor (404) can search the reference picture memory (405) for specific metadata or sample data (as candidate reference pixel blocks), such as reference picture motion vectors, block shapes, etc., which can serve as appropriate prediction references for the new pictures. The predictor (404) can operate on a sample block-by-pixel block basis to find appropriate prediction references. In some cases, as determined by the search results obtained by the predictor (404), the input picture may have prediction references drawn from a number of reference pictures stored in the reference picture memory (405).

[0049] The controller (402) can manage the coding operation of the video coder (403), including, for example, the settings of parameters and subgroup parameters used to encode video data.

[0050] The outputs of all the aforementioned functional units may undergo entropy coding in an entropy coder (408). The entropy coder converts the symbols generated by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques known to a person skilled in the art, such as, for example, Huffman coding, variable-length coding, and arithmetic coding.

[0051] The transmitter (409) may buffer the coded video sequence(s) generated by the entropy coder (408) to prepare for transmission over a communication channel (411), which may be a hardware / software link to a storage device for storing the encoded video data. The transmitter (409) may merge the coded video data from the video coder (403) with other data to be transmitted, e.g., coded audio data and / or auxiliary data streams (sources not shown).

[0052] The controller (402) can manage the operation of the encoder (400). During coding, the controller (405) can assign a specific coded picture type to each coded picture, which can influence the coding techniques that can be applied to each picture. For example, pictures can often be assigned as one of the following frame types:

[0053] An Intra Picture (I Picture) may be one that can be coded and decoded without using any other frame in the sequence as a source of prediction. Some video codecs allow different types of Intra Pictures, including, for example, Independent Decoder Refresh Pictures. A person skilled in the art recognizes the corresponding variations of I Pictures and their respective applications and features.

[0054] The predictive picture (P picture) may be coded and decoded using intra prediction or inter prediction, using at most one motion vector and reference index to predict sample values ​​of each block.

[0055] A bidirectionally predictive picture (B picture) may be coded and decoded using intra-prediction or inter-prediction, using at most two motion vectors and reference indices to predict sample values ​​of each block. Similarly, multi-predictive pictures may use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0056] Source pictures are generally spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples, respectively) and can be coded on a block-by-block basis. Blocks can be predictively coded by referencing other (already coded) blocks determined by the coding assignment applied to each of the blocks' pictures. For example, blocks of I pictures can be non-predictively coded, or they can be predictively coded by referencing already coded blocks of the same picture (spatial prediction or intra prediction). Pixel blocks of P pictures can be coded via spatial prediction, temporal prediction, or non-predictively by referencing one previously coded reference picture. Blocks of B pictures can be coded via spatial prediction, temporal prediction, or non-predictively by referencing one or two previously coded reference pictures.

[0057] The video coder (400) can perform coding operations according to a predetermined video coding technology or standard, such as ITU-T Rec. H.265. In the operation, the video coder (400) can perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancies in the input video sequence. Thus, the coded video data can comply with the syntax specified by the video coding technology or standard in use.

[0058] In one embodiment, the transmitter (409) may transmit additional data along with the encoded video. The source coder (403) may include such data as part of the encoded video sequence. The additional data may include other forms of redundant data such as temporal / spatial / SNR enhancement layers, redundant pictures and slices, Supplementary Enhancement Information (SEI) messages, Visual Usability Information (VUI) parameter set fragments, etc.

[0059] FIG. 5 is an example (500) of an end-to-end architecture for a STAR device according to exemplary embodiments showing a 5G stand-alone AR (STAR) user equipment (UE) receiver (600), a network / cloud (501), and a 5G UE (transmitter) (700). FIG. 6 is a more detailed example (600) of one or more configurations for a STAR UE receiver (600) according to exemplary embodiments, and FIG. 7 is a more detailed example (700) of one or more configurations for a 5G UE transmitter (700) according to exemplary embodiments. 3GPP TR 26.998 defines support for glass-type augmented reality / mixed reality (AR / MR) devices in 5G networks. And, according to exemplary embodiments in this specification, at least two classes of devices are considered: 1) devices capable of fully decoding and playing complex AR / MR content (standalone AR or STAR), and 2) devices having smaller computational resources and / or a smaller physical size (and thus battery) and capable of running such applications only when a significant portion of the computation is performed on a 5G edge server, network, or cloud rather than on the device (edge ​​dependent AR or EDGAR).

[0060] And, according to exemplary embodiments as described below, all participants of a shared AR conversation experience have AR devices, and each participant sees other participants in an AR scene, where the participants are overlays of a local physical scene, and the arrangement of participants within the scene is consistent across all receiving devices, for example, people in each local space have the same position / seating arrangement relative to each other, and such a virtual space creates the sense of being in the same space, but a shared conversation use case can be experienced where the room varies for each participant because the room is a real room or the space where each person is physically located.

[0061] For example, according to exemplary embodiments illustrated in connection with FIGS. 5 through 7, an immersive media processing function on a network / cloud (501) receives uplink streams from various devices and constructs a scene description that defines the arrangement of individual participants within a single virtual conference room. As well as the scene description, the encoded media streams are delivered to each receiving participant. The receiving participant's 5G STAR UE (600) receives, decodes, and processes 3D video and audio streams, and renders them using the received scene description and information received from its AR runtime to create an AR scene of the virtual conference room with all other participants. Although the virtual room for the participants is based on their own physical space, the seating / position arrangement of all other participants within the room matches the virtual room of all other participants in this session.

[0062] Refer also to FIG. 8, which illustrates an example (800) of an EDGAR device architecture according to exemplary embodiments, wherein the device itself, such as a 5G EDGAR UE (900), is not capable of heavy processing. Therefore, scene parsing and media parsing for the received content are performed at the cloud / edge (801), and then a simplified AR scene having a small number of media components is delivered to the device for processing and rendering. FIG. 9 illustrates a more detailed example of a 5G EDGAR UE (900) according to exemplary embodiments.

[0063] FIG. 10 illustrates an example (1000) in which User A (10), User B (11) and User T (12) will participate in an AR conference room, and one or more of the users may not have an R device. As illustrated, User A (10) is in an office (1001) and is sitting in a conference room with a varying number of chairs, and User A (10) is occupying one of the chairs. User B (11) is in a living room (1002) and is sitting in a two-person love seat, and in his living room there is also one or more couches for two people, as well as other furniture such as chairs and tables. User T (12) is in an airport lounge (1003) and is on a bench across from a coffee table among one or more other coffee tables.

[0064] And, in the office (1001), the AR of user A (10) shows user A (10) virtual user B (11v1) corresponding to user B (11) and virtual user T (12v1) corresponding to user T (12), and thereby, virtual user B (11v1) and virtual user T (12v1) refer to the AR environment shown to user A (10) as user A (10) sitting on the furniture and office chair of the office (1001) just like user A (10). And, the AR for user B (11) refers to the living room (1202) of example (1200) showing virtual user T (12v2) corresponding to user T (12) but sitting on the sofa of the living room (1202) and virtual user A (10v1) corresponding to user A (10) but also sitting on the furniture of the living room (1202) rather than the office chair in the office (1201). Additionally, the AR for user T (12) refers to the airport lounge (1203) which corresponds to user A (10) but shows virtual user A (10v2) sitting at a table in the airport lounge (1203), and virtual user B (11v2) also sitting at a table across from virtual user A (10v2). And, in each of such office (1201), living room (1202), and airport lounge (1203), the updated scene description of each room corresponds to the other rooms with respect to the location / seat arrangements. For example, per room, user A (10) is depicted counterclockwise relative to user (11) or his virtual representations, and user (11) or his virtual representations are depicted clockwise relative to user T (12) or his virtual representations.

[0065] However, AR technology has been limited in any attempts to integrate the creation and use of virtual spaces for devices that do not support AR but can parse VR or 2D video, and embodiments in this specification provide an improved technical procedure for creating a virtual scene that matches an AR scene when such devices participate in shared AR conversation services.

[0066] FIG. 11 illustrates an example (1100) of a end-to-end architecture having a non-AR device (1101) and a cloud / edge (1102) according to exemplary embodiments. FIG. 12 illustrates a more detailed block diagram example of a non-AR device (1101).

[0067] As illustrated in FIGS. 11 and 12, a non-AR UE (1101) is a device capable of rendering 360 video or 2-D video but without any AR capabilities. However, edge capabilities on the cloud / edge (1102) enable AR rendering of scenes received from a virtual room selected from a library, rendering scenes, and immersive visual and audio objects. The entire video is then encoded and delivered to the device (1101) for decoding and rendering.

[0068] In this way, AR processing on the edge / cloud (1102) may have multiview capabilities, such as being able to generate multiple videos of the same virtual room from different angles and using different viewports. And the device (1101) may receive one or more of these videos, switch between them if desired, or send commands to the edge / cloud processing to stream only the desired viewport / angle.

[0069] Additionally, the background capability can be changed, and the user of the device (1101) can select one of the desired room backgrounds from the provided library, for example, different conference rooms, or even living rooms and layouts. Then, the cloud / edge (1102) uses the selected background and creates a virtual room accordingly.

[0070] FIG. 13 illustrates an exemplary timing diagram (1300) for an exemplary call flow for an immersive AR conversation for a receiving non-AR UE (1101). For exemplary purposes, only one sender is shown in this diagram without showing its detailed call flow.

[0071] AR application module (21), media player module (22), and media access function module (23), which can be considered as modules of a receiving non-AR UE (1101), are illustrated. Additionally, a cloud / edge split rendering module (24) is illustrated. Additionally, a media delivery module (25) and a scene graph composer module (26) of each network cloud (1102) are illustrated. A 5G transmitter UE module (700) is also illustrated.

[0072] S1-S6 can be considered as session establishment phases. The AR application module (21) may request the start of a session for the media access function module (23) in S1, and the media access function module (23) may request the start of a session for the cloud / edge split rendering module (24) in S2.

[0073] The cloud / edge split rendering module (24) can implement session negotiation with the scene graph composer module (26) at S3, and the scene graph composer module (26) can negotiate with the 5G transmitter UE (700) accordingly. If successful, at S5, the cloud / edge split rendering module can send an acknowledgment to the media access function module (23), and the media access function module (23) can send an acknowledgment to the AR application module (21).

[0074] After that, S7 can be considered as a media pipeline configuration stage where the media access function module (23) and the cloud / edge split rendering module (24) each configure their respective pipelines. Next, after the pipeline configuration, a session can be initiated by a signal from the AR application module to the media player module (22) at S8, from the media player module (22) to the media access function module (23) at S9, and from the media access function module (23) to the cloud / edge split rendering module (24) at S10.

[0075] Next, there may be pose loop stages from S11 to S13, where in S11, pose data may be provided from the media player module (22) to the AR application module (21), and in S12, the AR application module may provide the pose data (12) to the media access function module (23), and thereafter, the media access function module (23) may provide the pose data to the cloud / edge split rendering module (24).

[0076] S14 through S16 may be considered as shared experience stream stages, wherein at S14, a 5G transmitter UE (700) may provide media streams to a media delivery module (25) at S14 and provide AR data to a scene graph compositor module (26) at S15. Subsequently, the scene graph compositor module (25) may construct one or more scenes based on the received AR data and provide scenes and scene updates to a cloud / edge split rendering module (24) at S16, and the media delivery module (25) may also provide media streams to a cloud / edge split rendering module at S17. This may include obtaining an AR scene descriptor from a non-AR device that does not render an AR scene according to exemplary embodiments, and generating a virtual scene by a cloud device by parsing and rendering the scene description obtained from the non-AR device.

[0077] S18 to S19 may be considered as media uplink stages, where a media player module (22) captures and processes media data from its local user, and in S18, provides the media data to a media access function module (23). Next, the media access module (23) encodes the media, and in S19, provides the media streams to a cloud / edge split rendering module (24).

[0078] The period between S19 and S20 can be considered as a media downlink stage, where a cloud / edge split rendering module (24) can implement scene parsing and complete AR rendering, and then S20 and S21 can be considered as constituting a media stream loop stage. In S20, the cloud / edge split rendering module (24) can provide media streams to a media access function module (23), the media access function module (23) can then decode the media, and in S21, provide media rendering to a media player (22).

[0079] By such features according to exemplary embodiments, a non-AR UE (1101) can utilize its display capable of rendering VR or 2-D video, even though it does not have a see-through display and therefore cannot generate an AR scene. As such, its immersive media processing function generates only a common scene description that describes each participant's relative position to other people and the scene. The scene itself needs to be adjusted with pose information on each device before being rendered as an AR scene, as described above. And an AR rendering process on the edge or in the cloud can parse the AR scene and generate a simplified VR-2D scene.

[0080] According to exemplary embodiments, the present disclosure uses a similar split rendering process of an EDGAR device for non-AR devices, such as VR or 2-D video devices, in which case characteristics such as edge / cloud AR rendering processes do not generate any AR scene. Instead, it generates a virtual scene by parsing and rendering a scene description received from an immersive media processing function for a given background (such as a conference room), and then renders each participant at the location described by the scene description within the conference room.

[0081] Additionally, the resulting video may be a 360 video or a 2-D video depending on the capabilities of the receiving non-AR device, and the resulting video is generated by taking into account force information received from the non-AR device according to exemplary embodiments.

[0082] Additionally, each other participant with a non-AR device is added as a 2-D video overlay on the 360 / 2D video of the conference room as shown in FIG. 10, and the conference room may have dedicated areas where these overlays are used, such as one of the furniture on which virtual images are overlaid as shown in FIG. 10.

[0083] Additionally, audio signals from all participants may be mixed to generate single-channel audio carrying the voice of the conference room if necessary, and video may be encoded as a single 360 ​​video or 2-D video and delivered to the device, and optionally, multiple video (multiview) sources may be generated, each of which captures the same virtual conference room from different views and provides those views to the device according to exemplary embodiments.

[0084] Additionally, the non-AR UE device (1101) can receive selected 360 video and / or one or more multiview videos along with audio and render them on the device display, and the user can switch between different views or change the viewport of the 360 ​​video by moving or rotating the view device and thus navigate in the virtual room while watching the video.

[0085] The embodiments described above provide these 5G media stream architecture (5GMS) extensions for using edge servers in their architectures, and while the specifications may have many features, these features could not technically be deployed as a set of software development kits (SDKs) on a device or as a set of microservices in the cloud, and these technical limitations are resolved by the embodiments further described below.

[0086] For example, the current Media Service Enabler technical report does not define a framework that relates specifications to SDKs, nor does it include any concept of microservices.

[0087] Refer to the example (1400) of FIG. 14, which illustrates a 5G media streaming architecture having edge extensions according to exemplary embodiments. As illustrated, there is a user device (UE) (1401) and a data network (DN) (1411). The UE (1401) may include a 5GMS client (1403) and a 5GMS-aware application (1405), which are similar to the AR or non-AR embodiments described above, but are not limited thereto. The 5GMS client (1403) may also include a media stream handler (1403) and a media session handler. The DN (1411) may include a 5GMS application server (AS) (1412), a 5GMS application function (AF) (1414), and a 5GMS application provider (1413). The 5GMS AF (1414) can also communicate with the network exposure function (NEF) (1415) and the policy and charging function (PCF) (1416).The improvements described in this specification may be understood in the context of at least one of the UE (1401), 5GMS-aware application (1405), 5GMS application provider (1413), NEF (1415) and PCF (1416); that is, instead of having a monolithic specification without definitions of a media service enabler (MSE) for each function or group of functions, one or more of such elements may generate their own specifications and, upon appropriate request, provide such specifications to other elements among such elements, which in turn may further configure the initial element specifications according to various possibilities such as those further described below, and such processing may be done through one or more of the illustrated exposed application programming interfaces (API) (1420) and interfaces M1, M2, M4, M5, M6, M7, M8, N33, and N5.

[0088] According to exemplary embodiments, FIG. 15 illustrates an example (1500) of an MSE framework consisting of two parts: an MSE specification (1501) and an MSE implementation (1502) (which includes an MSE SDK abstraction (1510), an MSE SDK (platform-dependent) instantiation (1520), and an MSE microservice (1530). The MSE SDK abstraction (1510) may have a configuration API abstraction (1513), control interfaces (1512), and media interfaces (1511), each of which may notify the configuration API (1523) of the MSE SDK (platform-dependent) instantiation (1520) and its control interfaces (1522) and media interfaces (1521). The MSE microservice (1530) may also have a configuration API (1533), control interfaces (1532), and media interfaces (1531).

[0089] According to exemplary embodiments, the MSE specification (1501) defines (a) media modalities and (b) MSE configurations, which provide these technical advantages. (a) Media modalities may indicate any of: (i) a functional description of the MSE including mandatory and optional features; (ii) control interfaces such as provisioning, authentication used by an application, and other functions for interacting with the MSE; (iii) media interfaces including all input and output formats and protocols; (iv) network interfaces including system and radio networks; (v) events, notifications, reporting, monitoring; and (vi) error handling. (b) An MSE configuration may include (i) an MSE description document (MDD) (which describes (1) the functions supported by the MSE implementation and their configuration parameters, and (2) optionally performance / cost metrics for different features / options), (ii) an MSE Configuration API (MCA) abstraction (which directs information for (1) retrieving the description document, (2) configuring an MSE instantiation, and (3) retrieving the state and status of the MSE instantiation), and (iii) a service API for the MSE Configuration API (MCA) abstraction.The MSE implementation (1502) is (a) an MSE SDK abstraction (1510) (which directs (i) media modalities that comply with the media modalities of the MSE specification (1501), and (ii) the MSE configuration API (MCA) abstraction of the MSE configuration described above in the MSE specification (1501), the MSE documentation (MDD), and the MSE configuration API (MCA) abstraction of the MSE configuration described above in the MSE specification (1501), (b) an MSE SDK (platform-dependent) instantiation (1520) (which directs an SDK implementation in a specific environment, and, unlike the MSE documentation (MDD) and MSE SDK abstraction (1530), only the MSE configuration API (MCA) abstraction of the MSE configuration described above in the MSE specification (1501), and (c) an MSE microservice (1510) (which is an MSE implementation as a microservice, and (i) MSE It may include any of the following: media modes that comply with the media modes of the specification (1501), and the MSE configuration API (MCA) abstraction for the MSE description document (MDD) and the service API for the MSE configuration API (MCA) abstraction of the MSE configuration described above of the MSE specification (1501). Determining conformity may be, for example, by matching formats or syntax, or by a flag indicating such formats or syntax.

[0090] For example, according to this advantageous MSE specification not found in the 3GPP SA4 specifications, for any SDK or microservice compliant with the MSE specification, descriptions of features and their configuration parameters may be retrieved by an external function or service according to exemplary embodiments, and this external function or service may set a specific configuration for operating the SDK, and the status and condition of the SDK in operation may be retrieved at any time. The status and condition may indicate available, running, busy, idle, offline, etc.

[0091] In this regard, FIG. 16 illustrates an exemplary embodiment (1600) in which a 5G Media Streaming downlink (5GMSd)-aware application (1405) communicates with a 5GMSd-client (1402') having a media session handler (1404) that communicates with a 5GMSd AF (1414'). Information on methods (1601), notifications and errors (1602), and circumstances (1603) can be communicated between a 5GMSd-aware application (1405') and a 5GMSd client (1402'), and information on circumstances (1604), subscriptions and notifications (1605), and settings and configuration information (1606) can be communicated to and from a media session handler (1404), and the media session handler (1404) itself can communicate to a 5GMSd AF (1414') via a link (M5d). According to exemplary embodiments, the MSE specification for the Media Session Handler (MSH) example (1600) illustrated in FIG. 16 comprises (a) media modalities (e.g., (i) functional descriptions ((1) service access information, (2) consumption reporting, (3) metric reporting, (4) dynamic policies, and (5) network auxiliary information), and (ii) M5d, M6d, and M7d information), and (b) MSE configuration information, e.g., (i) MDD (which describes (1) an identifier showing that this MSE complies with the media modalities, (2) optional features of the functional descriptions and M5d, M6d, and M7d along with their configuration parameters, and (3) optionally performance / cost metrics for different features / options), and (ii) API abstraction (which provides information for (1) retrieving the description document MDD, (2) configuring the MSE instantiation, and (3) retrieving the status and context of the MSE instantiation). (iii) will describe the service API for the above API abstraction, and (iii) the service API for the above API abstraction.For example, according to exemplary embodiments, an MSE SDK implementation of the above MSE specification that may exist on Android will support the following: (i) media modalities that comply with the above-mentioned media modalities of the MSE specification for MSE, and (ii) a specific implementation of the API abstraction of the MSE configuration of the MSE specification for the above-mentioned MDD and MSE configuration of the MSE specification for MSH.

[0092] According to exemplary embodiments, refer to the flowchart (1700) of FIG. 17, where MDD describes features implemented by the MSE SDK. The APIs of the MSE configuration mentioned above may allow an external Android process to retrieve this document in S1701 and configure the SDK with a set of configurable parameters described in the MDD in S1713 and the state and condition of the operating SDK that can be determined in S1714. The language and syntax of the MDD and the general framework of the MCA may be uniformly defined for all SA4 Media Service Enabler specifications in S1701, and only specific code points for each specification are defined in that specification. An external function or application that not only supports the MCA but also understands the MDD language and syntax may retrieve information from the MSE implementation in S1701, and in S1702, if it recognizes the MSE specification identifier, the external function or application understanding may parse and process the MDD and its configuration parameters. For example, the MSE specification and its identifier may be generated in S1701, and then, after a request for this specification is made, a conformity check may determine whether the desired identifier is provided, and then the MSE may be deployed according to the parameters generated in S1703 based on various capabilities and cost indexing in S1704, and in S1705, it may be determined whether to deploy the SDK on a device in S1706 or on a server in S1708, where, if it is S1708, the service API for MDD and MCA may be configured as mentioned above, or if it is S1710, it may be determined whether to proceed with MDD and general MCA or MDD and specific MCA. Regardless of this, this SDK may then be configured by an external device or function in S1713, and this may then request a situation in S1714 at any time according to the implementations configured in S1715.

[0093] Examples of MDDs can be found in ISO / IEC 23090-8. A function description document is a JSON document that describes the functionalities and features provided by a function and its configuration parameters.

[0094] As such, exemplary embodiments provide a method for defining media service enabler SDKs and services, wherein the functionalities and features of the media service enabler are described in a document that can be retrieved by an external function, application, or service, as well as configuration parameters for each feature, and the external entity can set up a specific configuration of the media service enabler using the same interface used to retrieve information, and the complexity or other metrics of the various functions of the media service enabler are also described in the document, and the media service enabler specification defines a unique identifier for that specific media service enabler as well as a format for the description document.

[0095] According to the embodiments, the example (1800) of FIG. 18 provides split rendering configuration parameters including a 3GPP SR_MSE split rendering architecture and desired view information. Additionally, the embodiments provide device capabilities added to the split rendering configuration information, as shown in Table 1 below:

[0096] Table 1

[0097]

[0098]

[0099] According to the embodiment, detailed device capabilities indicate capabilities at two levels: high-level device types and low-level device detailed capabilities as shown in Table 2 below:

[0100] Table 2

[0101]

[0102]

[0103] According to the embodiments, during negotiation between the split rendering client (SRC) and the split rendering server (SRS), the SRC provides device capabilities to the SRS as part of the split rendering configuration. Through the analysis of the device capabilities, the SRS defines how to split sessions between the network and the device, regardless of whether it provides high-level device types or additional detailed capabilities along with the splitRenderingProfile. Because device capabilities define more information than the splitRenderingProfile, the SRS can optimize split sessions for each device capability and does not restrict split rendering to a defined split rendering profile.

[0104] Accordingly, a method is provided for signaling augmented reality device capabilities to the network as part of the split rendering configuration by the device during device-network negotiation to set up a split rendering session, wherein the split rendering client provides the device types / profiles supported by the device, and optionally provides XR runtime and scene manager capabilities, as well as details regarding the number of codecs and the maximum number of instances it supports, thereby enabling the split rendering server to determine what the best option is for the split rendering session.

[0105] The XR runtime includes the functions and hardware components present in the XR device. However, these functions and hardware components are not directly exposed to the XR application. Instead, the XR runtime provides its functions and hardware components through the XR system. A single XR runtime can expose more than one XR system to target different purposes; for example, a handheld device can have two XR systems: one when the user is holding the device and one when the device is inserted into an HMD. At the start of the XR application, it is expected that the XR application will query which XR systems are available on the XR device and select one of them to create an XR session.

[0106] The split rendering client establishes a local XR session based on device configuration and user selection. The SR client defines view configurations (e.g., mono or stereo views), projection formats (e.g., projected, square, quad, or cubemap), swap chain image configurations, etc. Additionally, XR spaces and action configurations are negotiated between the SR client and the server. This includes defining common XR spaces and defining and selecting actions and action sets. This format is extensible to support the exchange of additional / future configuration information.

[0107] The techniques described above may be implemented as computer software that uses computer-readable instructions and is physically stored on one or more computer-readable media, or may be implemented by one or more specifically configured hardware processors. For example, FIG. 19 illustrates a computer system (1900) suitable for implementing specific embodiments of the disclosed subject matter.

[0108] Computer software may be coded using any suitable machine code or computer language in which assembly, compilation, linking, or similar mechanisms can be performed to generate code containing instructions that can be executed directly or through interpretation, micro-code execution, etc., by a computer central processing unit (CPU), graphics processing unit (GPU), etc.

[0109] The commands can be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0110] The components of the computer system (1900) illustrated in FIG. 19 are exemplary in nature and are not intended to imply any limitation on the scope of use or functionality of the computer software implementing the embodiments of the present disclosure. The configuration of the components should not be interpreted as having any dependency or requirement in relation to any one or a combination thereof of the components illustrated in the exemplary embodiments of the computer system (1900).

[0111] The computer system (1900) may include specific human interface input devices. Such human interface input devices may respond to input by one or more human users through, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), and olfactory input (not described). Human interface devices may also be used to capture specific media that are not necessarily directly related to conscious input by humans, such as audio (e.g., voice, music, ambient sound), images (e.g., scanned images, photographic images obtained from a still image camera), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0112] Input human interface devices may include one or more of a keyboard (1901), a mouse (1902), a trackpad (1903), a touch screen (1910), a joystick (1905), a microphone (1906), a scanner (1908), and a camera (1907) (each only one of which is depicted).

[0113] The computer system (1900) may also include certain human interface output devices. Such human interface output devices may stimulate one or more human user’s senses through, for example, tactile output, sound, light, and smell / taste. Such human interface output devices may include tactile output devices (e.g., a touch-screen (1910), or tactile feedback via a joystick (1905), but there may also be tactile feedback devices that do not serve as input devices), audio output devices (e.g., speakers (1909), headphones (not described)), visual output devices (e.g., screens (1910) including CRT screens, LCD screens, plasma screens, and OLED screens, each with or without touch-screen input capability and with or without tactile feedback capability—some of which may output two-dimensional visual output or output beyond three dimensions through means such as stereographic output—); virtual reality glasses (not described), holographic displays and smoke tanks (not described)), and printers (not described).

[0114] The computer system (1900) may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW (1920) having media such as CD / DVD (1911), thumb-drive (1922), removable hard drive or solid-state drive (1923), legacy magnetic media such as tape and floppy disk (not depicted), specialized ROM / ASIC / PLD-based devices such as security dongle (not depicted).

[0115] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the subject matter disclosed herein does not include transmission media, carriers, or other transient signals.

[0116] The computer system (1900) may also include an interface (1999) to one or more communication networks (1998). The networks (1998) may be, for example, wireless, wireline, optical. The networks (1998) may additionally be local, wide-area, metropolitan, automotive and industrial, real-time, delay-tolerant, etc. Examples of networks (1998) include local area networks, such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., TV wired or wireless wide-area digital networks including cable TV, satellite TV and terrestrial broadcast TV, automotive and industrial including CANBus, etc. Certain networks (1998) generally require external network interface adapters attached to specific general-purpose data ports or peripheral buses (1950 and 1951) (e.g., USB ports of the computer system (1900)); others are generally integrated into the core of the computer system (1900) by attachment to a system bus as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks (1998), the computer system (1900) can communicate with other entities. Such communication may be unidirectional receive-only (e.g., broadcast TV), unidirectional transmit-only (e.g., CANbus to specific CANbus devices), or bidirectional with other computer systems using, for example, local area or wide area digital networks. Specific protocols and protocol stacks may be used for each of the networks and network interfaces described above.

[0117] The aforementioned human interface devices, human-accessible storage devices, and network interfaces can be attached to the core (1940) of the computer system (1900).

[0118] The core (1940) may include one or more central processing units (CPUs) (1941), graphics processing units (GPUs) (1942), graphics adapters (1917), specialized programmable processing units in the form of field programmable gate areas (FPGAs) (1943), hardware accelerators (1944) for specific tasks, etc. These devices may be connected via a system bus (1948), along with internal mass storage (1947), such as read-only memory (ROM) (1945), random access memory (1946), internal non-user accessible hard drives, SSDs, etc. In some computer systems, the system bus (1948) may be accessible in the form of one or more physical plugs to enable expansion by additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus (1948) or via a peripheral bus (1951). Architectures for peripheral buses include PCI, USB, etc.

[0119] CPUs (1941), GPUs (1942), FPGAs (1943), and accelerators (1944) can be combined to execute specific instructions that can constitute the aforementioned computer code. The computer code may be stored in ROM (1945) or RAM (1946). Transient data may also be stored in RAM (1946), while permanent data may be stored, for example, in internal mass storage (1947). High-speed storage and retrieval of any of the memory devices may be made possible through the use of cache memory, which may be closely associated with one or more CPUs (1941), GPUs (1942), mass storage (1947), ROM (1945), RAM (1946), etc.

[0120] A computer-readable medium may have computer code for performing various computer-implemented operations. The medium and the computer code may be those specifically designed and configured for the purposes of this disclosure, or they may be of a kind well known and available to those skilled in the field of computer software technology.

[0121] As an example rather than a limitation, an architecture corresponding to a computer system (1900), and specifically a core (1940), may provide functionality as a result of processor(s) (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software implemented on one or more types of tangible computer-readable media. Such computer-readable media may be media associated with specific storage of the core (1940) that is of a non-transient nature, such as core-internal mass storage (1947) or ROM (1945), as well as user-accessible mass storage as described above. Software implementing various embodiments of the present disclosure may be stored on such devices and executed by the core (1940). The computer-readable media may include one or more memory devices or chips as needed. Software may enable the core (1940) and, specifically, the processors within it (including a CPU, GPU, FPGA, etc.) to execute specific processes or specific parts of specific processes described herein, including defining data structures stored in RAM (1946) and modifying such data structures according to processes defined by the software. Additionally or alternatively, the computer system may provide functionality as a result of logic hardwired or otherwise implemented in a circuit (e.g., accelerator (1944)) that can operate instead of or with the software to execute specific processes or specific parts of specific processes described herein. Reference to software may include logic, where appropriate, and vice versa. Reference to a computer-readable medium may include, where appropriate, a circuit storing software for execution (e.g., an integrated circuit (IC)), or a circuit implementing logic for execution, or both. The present disclosure includes any suitable combination of hardware and software.

[0122] Although the present disclosure describes various exemplary embodiments, modifications, substitutions, and various alternative equivalents exist within the scope of the present disclosure. Accordingly, it will be acknowledged that those skilled in the art will be able to devise a number of systems and methods that implement the principles of the present disclosure and are therefore within the true meaning and scope thereof, even though they are not expressly illustrated or described in this specification.

Claims

Claim 1 A method for video decoding, wherein the method is performed by at least one processor and comprises the steps of: determining the capabilities of an augmented reality (AR) device—the capabilities indicating either an audio encoder or decoder of the AR device and a video encoder or decoder of the AR device as part of the AR device acting as a split rendering client (SRC)—; acquiring AR data of at least one media component among audio and video based on a request to signal the capabilities of the AR device to a server—the server acting as a split rendering server (SRS) for split rendering of the media component between the SRS and the SRC and providing the AR data—and decoding the AR data based on the capabilities of the AR device signaled to the SRC. Claim 2 A method according to claim 1, wherein a request for signaling the capabilities of the AR device is in the form of a uniform resource indicator (URI) and includes a profile identifier containing the syntax of "deviceType". Claim 3 In paragraph 2, the method wherein the profile identifier indicates an eXtended reality (XR) runtime and scene manager capabilities. Claim 4 A method according to claim 1, wherein a request for signaling the capabilities of the AR device comprises a list of at least some of the capabilities of the AR device, and said at least some of the capabilities of the AR device indicate optional codecs of the AR device. Claim 5 A method according to claim 4, wherein at least some of the capabilities of the AR device further indicate the number of interfaces and instances of optional codecs of the AR device, extended reality (XR) runtime profiles, optional extensions, scene description formats, and profiles supported by said scene description formats. Claim 6 A method according to claim 5, wherein a request for signaling the capabilities of the AR device is in the form of a uniform resource indicator (URI) and further includes a profile identifier including the syntax of "deviceType", and the syntax indicating a list of at least some of the capabilities is "deviceDetailedCapabilities". Claim 7 In claim 6, the method wherein the profile identifier indicates extended reality (XR) runtime and scene manager capabilities other than at least some of the capabilities of the AR device. Claim 8 A method for video encoding, wherein the method is performed by at least one processor and comprises the steps of: determining the capabilities of an augmented reality (AR) device—wherein the capabilities indicate either an audio encoder or decoder of the AR device and a video encoder or decoder of the AR device as part of the AR device acting as a split rendering client (SRC); acquiring AR data of at least one media component among audio and video based on a request to signal the capabilities of the AR device to a server—wherein the server acts as a split rendering server (SRS) for split rendering of the media component between the SRS and the SRC and provides the AR data—and encoding the AR data based on the capabilities of the AR device signaled to the SRC. Claim 9 In claim 8, the method wherein the request for signaling the capabilities of the AR device is in the form of a uniform resource indicator (URI) and includes a profile identifier containing the syntax of "deviceType". Claim 10 In claim 9, the method wherein the profile identifier indicates an Extended Reality (XR) runtime and scene manager capabilities. Claim 11 In paragraph 8, a request for signaling the capabilities of the AR device comprises a list of at least some of the capabilities of the AR device, and the at least some of the capabilities of the AR device indicate optional codecs of the AR device. Claim 12 A method according to claim 11, wherein at least some of the capabilities of the AR device further indicate the number of interfaces and instances of optional codecs of the AR device, extended reality (XR) runtime profiles, optional extensions, scene description formats, and profiles supported by said scene description formats. Claim 13 A method according to claim 12, wherein a request for signaling the capabilities of the AR device is in the form of a uniform resource indicator (URI) and further includes a profile identifier containing the syntax of "deviceType", and the syntax indicating a list of at least some of the capabilities is "deviceDetailedCapabilities". Claim 14 In paragraph 13, the method wherein the profile identifier indicates extended reality (XR) runtime and scene manager capabilities other than at least some of the capabilities of the AR device. Claim 15 A method for processing visual media data, wherein the method is performed by at least one processor and comprises the step of performing a conversion between a visual media file and a bitstream of visual media data according to a format rule, wherein the format rule comprises: determining the capabilities of an augmented reality (AR) device—the capabilities indicate either an audio encoder or decoder of the AR device and a video encoder or decoder of the AR device as part of the AR device acting as a split rendering client (SRC)—; acquiring AR data of at least one media component of audio and video based on a request signaling the capabilities of the AR device to a server—the server acts as a split rendering server (SRS) for split rendering of the media component between the SRS and the SRC and provides the AR data—; and instructing the SRC to process the AR data based on the capabilities of the AR device signaled to the SRC. Claim 16 In paragraph 15, a method wherein a request to signal the capabilities of the AR device is in the form of a uniform resource indicator (URI) and includes a profile identifier containing the syntax of "deviceType". Claim 17 In paragraph 16, the above profile identifier indicates an Extended Reality (XR) runtime and scene manager capabilities, a method. Claim 18 In paragraph 15, a request for signaling the capabilities of the AR device comprises a list of at least some of the capabilities of the AR device, and the at least some of the capabilities of the AR device indicate optional codecs of the AR device. Claim 19 In paragraph 18, a method further indicating at least some of the capabilities of the AR device, the number of interfaces and instances of optional codecs of the AR device, extended reality (XR) runtime profiles, optional extensions, scene description formats, and profiles supported by said scene description formats. Claim 20 In claim 19, the request for signaling the capabilities of the AR device further comprises a profile identifier in the form of a uniform resource indicator (URI) and including the syntax of "deviceType", the syntax indicating a list of at least some of the capabilities is "deviceDetailedCapabilities", and the profile identifier indicates extended reality (XR) runtime and scene manager capabilities other than at least some of the capabilities of the AR device.

Images