Procedure for discovering the capabilities and performance of 5G EDGAR devices

The method and apparatus for controlling and discovering 5G EDGAR device characteristics through APIs reduce network and server overheads, enabling efficient utilization and monitoring of device capabilities, including media processing and hardware features.

JP2024543932AActive Publication Date: 2024-11-26TENCENT AMERICA LLC
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2024531535
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-03-29
Filing Date
2023-04-04
Publication Date
2024-11-26
Estimated Expiration
2043-04-04

AI Technical Summary

Technical Problem

Current 5G EDGAR devices lack the ability to receive and respond to capability queries made by authorized applications over a wireless network, leading to network and server computation overheads.

Method used

A method and apparatus that include a memory and processors configured to execute computer program code for controlling and discovering characteristics of 5G EDGAR devices through application programming interfaces, acquiring indexing, and implementing 5G media streaming services based on these characteristics.

Benefits of technology

Reduces network and server computation overheads while enabling efficient discovery and utilization of device capabilities, including media encoders, decoders, capture, rendering, and hardware features, with real-time evaluation and battery consumption monitoring.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024543932000001
    Figure 2024543932000001
  • Figure 2024543932000002
    Figure 2024543932000002
  • Figure 2024543932000003
    Figure 2024543932000003
Patent Text Reader

Abstract

Methods and apparatuses are included that include computer code configured to cause one or more processors to control a 5G Edge-Aware Augmented Reality (EDGAR) device, discover at least some characteristics of the 5G EDGAR device, obtain an indexing of at least some characteristics of the 5G EDGAR device, and control implementation of a 5G Media Streaming (5GMS) service according to the indexing of the at least some characteristics of the 5G EDGAR device.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is based on and claims priority to U.S. Provisional Patent Application No. 63 / 338,767, filed May 5, 2022, and U.S. Patent Application No. 18 / 192,328, filed March 29, 2023, the disclosures of which are incorporated by reference in their entireties herein.

[0002] The present disclosure provides procedures for discovery of the capabilities and dynamic performance of 5G EDGAR devices by applications on the device or applications running on the network (edge ​​applications on the edge, or application service providers) so that complex parts of the application can be run on the edge and only optical processing is performed on the device. [Background technology]

[0003] 3GPP TR 26.998 defines support for glasses-type Augmented Reality / Mixed Reality (AR / MR) devices in 5G networks. Mainly, two device classes are considered: 1) devices capable of fully decoding and playing complex AR / MR content (Standalone AR, i.e. STAR), and 2) devices with smaller computational resources and / or smaller physical size (hence battery) capable of running such applications only if the majority of the computation is performed on the 5G edge server, network, or cloud rather than on the device (Edge-Dependent AR, i.e. EDGAR). Recent activities on profiling EDGAR devices have been started in 3GPP.

[0004] Even though the current TR 26.998 defines an architecture for EDGAR devices receiving streaming content by performing some of the required extensive computations on the cloud / edge, it does not have the ability to receive and respond to capability queries made by any authorized application, such as an application on the device, an application on the edge network, or an application service provider over the wireless network. Summary of the Invention [Problem to be solved by the invention]

[0005] To address one or more different technical problems, the present disclosure provides technical solutions that reduce network overhead and server computation overhead, while providing options to apply various operations to the resolved elements, so that their use may improve their utility and some of the technical signaling characteristics. [Means for solving the problem]

[0006] Methods and apparatuses are included that include a memory configured to store computer program code and one or more processors configured to access the computer program code and operate as instructed by the computer program code. The computer program code includes a first control code configured to cause the at least one processor to control a 5G Edge-Aware Augmented Reality (EDGAR) device and discover at least some features of the 5G EDGAR device, an acquisition code configured to cause the at least one processor to acquire an indexing of the at least some features of the 5G EDGAR device, and a second control code configured to cause the at least one processor to control implementation of a 5G Media Streaming (5GMS) service according to the indexing of the at least some features of the 5G EDGAR device.

[0007] According to an example embodiment, the first control code is configured to cause the at least one processor to control the 5G EDGAR device to discover at least some characteristics via one of a plurality of application programming interfaces (APIs) of the 5G EDGAR device.

[0008] According to an exemplary embodiment, the first control code is configured to cause the at least one processor to control the 5G EDGAR device via a network API; and an acquisition code configured to cause at least one processor to acquire the indexing via the network API.

[0009] According to an example embodiment, at least some of the characteristics include parameters of at least one of a media encoder and a media decoder.

[0010] According to an example embodiment, at least some of the features include a device capture capability of the 5G EDGAR device, the device capture capability including any of a camera capability and a microphone capability.

[0011] According to an example embodiment, at least some of the features include device rendering capabilities of the 5G EDGAR device, the device rendering capabilities including any of speaker capabilities and display capabilities.

[0012] According to an example embodiment, at least some of the features include raw hardware capabilities of the 5G EDGAR device, the raw hardware capabilities including any of a CPU benchmark, a GPU benchmark, memory bandwidth, and on-device storage.

[0013] According to an example embodiment, at least some of the characteristics include functional efficiency of the 5G EDGAR device, which indicates a relative amount of power consumption per function and a relative amount for each function relative to a parameter.

[0014] According to an example embodiment, at least some of the characteristics include any of the power state of the 5G EDGAR device and the rate at which the battery of the 5G EDGAR device drains.

[0015] According to an example embodiment, the indexing indicates either whether at least some combination of features can be executed in real time on a 5G EDGAR device and the estimated overall battery consumption of that combination.

[0016] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Brief description of the drawings]

[0017] [Figure 1] FIG. 1 is a simplified schematic diagram according to an embodiment. [Diagram 2] FIG. 1 is a simplified schematic diagram according to an embodiment. [Diagram 3] FIG. 2 is a simplified block diagram of a decoder according to an embodiment. [Figure 4] FIG. 2 is a simplified block diagram of an encoder according to an embodiment; [Diagram 5] FIG. 2 is a simplified block diagram according to an embodiment. [Figure 6] FIG. 2 is a simplified block diagram according to an embodiment. [Figure 7] FIG. 2 is a simplified block diagram according to an embodiment. [Figure 8] FIG. 2 is a simplified block diagram according to an embodiment. [Figure 9] FIG. 2 is a simplified block diagram according to an embodiment. [Figure 10] 1 is a simplified diagram according to an embodiment. [Figure 11] FIG. 2 is a simplified block diagram according to an embodiment. [Figure 12] FIG. 2 is a simplified block diagram according to an embodiment. [Figure 13]1 is a simplified block diagram and timing diagram according to an embodiment. [Figure 14] FIG. 2 is a simplified block diagram according to an embodiment. [Figure 15] FIG. 2 is a simplified block diagram according to an embodiment. [Figure 16] FIG. 2 is a simplified flow diagram according to an embodiment. [Figure 17] FIG. 2 is a simplified flow diagram according to an embodiment. [Figure 18] FIG. 2 is a schematic diagram according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] The proposed features described below may be used separately or combined in any order. Furthermore, the embodiments may be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute a program stored on a non-transitory computer-readable medium.

[0019] 1 shows a simplified block diagram of a communication system 100 according to one embodiment of the present disclosure. The communication system 100 may include at least two terminals 102 and 103 interconnected via a network 105. For unidirectional transmission of data, a first terminal 103 may code video data at a local location for transmission to the other terminal 102 via the network 105. The second terminal 102 may receive the coded video data of the other terminal from the network 105, decode the coded data, and display the restored video data. Unidirectional data transmission may be common in media serving applications, etc.

[0020] 1 illustrates a second pair of terminals 101 and 104 provided to support bidirectional transmission of coded video, such as may occur during a video conference. For the bidirectional transmission of data, each terminal 101 and 104 may code video data captured at a local location for transmission to the other terminal over network 105. Each terminal 101 and 104 may also receive coded video data transmitted by the other terminal, decode the coded data, and display the recovered video data on a local display device.

[0021] In FIG. 1, terminals 101, 102, 103, and 104 may be depicted as servers, personal computers, and smartphones, although the principles of the present disclosure are not so limited. Embodiments of the present disclosure find application with laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network 105 represents any number of networks that convey coded video data between terminals 101, 102, 103, and 104, including, for example, wired and / or wireless communication networks. Communication network 105 may exchange data in circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of this description, the architecture and topology of network 105 may not be important to the operation of the present disclosure unless described below herein.

[0022] 2 shows an arrangement of a video encoder and a video decoder in a streaming environment as an example of an application of the disclosed subject matter. The disclosed subject matter can be equally applicable to other video-enabled applications including, for example, video conferencing, digital TV, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0023] The streaming system may include a capture subsystem 203 that may include a video source 201, such as a digital camera, that creates an uncompressed video sample stream 213. The sample stream 213 may be emphasized as a high data volume compared to an encoded video bitstream and may be processed by an encoder 202 coupled to the camera 201. The encoder 202 may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video bitstream 204 may be emphasized as a lower data volume compared to the sample stream and may be stored on a streaming server 205 for future use. One or more streaming clients 212 and 207 may access the streaming server 205 and retrieve copies 208 and 206 of the encoded video bitstream 204. The client 212 may include a video decoder 211 that decodes the copy 208 of the input encoded video bitstream and creates an output video sample stream 210 that may be rendered on a display 209 or other rendering device (not shown). In some streaming systems, the video bitstreams 204, 206, and 208 may be encoded according to a particular video coding / compression standard, examples of which are mentioned above and further described herein.

[0024] FIG. 3 may be a functional block diagram of a video decoder 300 according to one embodiment of the present invention.

[0025] The receiver 302 may receive one or more codec video sequences to be decoded by the decoder 300, in the same or other embodiments, one coded video sequence at a time, with the decoding of each coded video sequence being independent of the other coded video sequences. The coded video sequences may be received from a channel 301, which may be a hardware / software link to a storage device that stores the encoded video data. The receiver 302 may receive the encoded video data together with other data, e.g., coded audio data and / or auxiliary data streams, that may be forwarded to a respective usage entity (not shown). The receiver 302 may separate the coded video sequence from the other data. To combat network jitter, a buffer memory 303 may be coupled between the receiver 302 and the entropy decoder / parser 304 (hereinafter, "parser"). If the receiver 302 is receiving data from a storage / forwarding device with sufficient bandwidth and controllability, or from an isosynchronous network, the buffer 303 may not be needed or may be small. For use over a best effort packet network such as the Internet, buffer 303 may be required and may be relatively large, and may advantageously be of adaptive size.

[0026] The video decoder 300 may include a parser 304 for reconstructing symbols 313 from the entropy coded video sequence. These categories of symbols include information used to manage the operation of the decoder 300 and potentially information for controlling a rendering device, such as a display 312, that is not an integral part of the decoder but may be coupled to it. The rendering device control information may be in the form of supplemental enhancement information (SEI message) or a video usability information parameter set fragment (not shown). The parser 304 may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may be according to a video coding technique or standard and may follow principles well known to those skilled in the art, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser 304 may extract from the coded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder based on at least one parameter corresponding to the group. The subgroups may include groups of pictures (GOPs), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), etc. The entropy decoder / parser may also extract information from the coded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, etc.

[0027] The parser 304 may perform an entropy decoding / parsing operation on the video sequence received from the buffer 303 to create symbols 313. The parser 304 may receive the encoded data and selectively decode particular symbols 313. Additionally, the parser 304 may determine whether a particular symbol 313 should be provided to the motion compensated prediction unit 306, the scaler / inverse transform unit 305, the intra prediction unit 307, or the loop filter 311.

[0028] The reconstruction of symbols 313 may involve several different units depending on the type of coded video picture or part thereof (inter-picture and intra-picture, inter-block and intra-block, etc.), as well as other factors. Which units are involved and how may be governed by subgroup control information parsed from the coded video sequence by parser 304. The flow of such subgroup control information between parser 304 and the following units is not shown for clarity.

[0029] In addition to the functional blocks already mentioned, the decoder 300 can be conceptually subdivided into several functional units, as described below. In a practical implementation operating under commercial constraints, many of these units will interact closely with each other and may be at least partially integrated with each other. However, for purposes of describing the disclosed subject matter, the following conceptual subdivision into functional units is appropriate:

[0030] The first unit is a scalar / inverse transform unit 305. The scalar / inverse transform unit 305 receives quantized transform coefficients and control information including the transform to use, block size, quantization factor, quantization scaling matrix, etc. as symbols 313 from the parser 304. The scalar / inverse transform unit 305 may output blocks containing sample values ​​that may be input to an aggregator 310.

[0031] In some cases, the output samples of the scalar / inverse transform unit 305 may also relate to intra-coded blocks, i.e. blocks that do not use prediction information from a previously reconstructed picture, but can use prediction information from a previously reconstructed part of the current picture. Such prediction information may be provided by the intra-picture prediction unit 307. In some cases, the intra-picture prediction unit 307 uses surrounding already reconstructed information fetched from the current (partially reconstructed) picture 309 to generate a block of the same size and shape as the block being reconstructed. The aggregator 310 adds, possibly on a sample-by-sample basis, the prediction information generated by the intra-prediction unit 307 to the output sample information provided by the scalar / inverse transform unit 305.

[0032] In other cases, the output samples of the scalar / inverse transform unit 305 may relate to an inter-coded, potentially motion-compensated block. In such a case, the motion compensation prediction unit 306 may access the reference picture memory 308 to fetch samples used for prediction. After motion compensating the fetched samples according to the symbols 313 relating to the block, these samples may be added to the output of the scalar / inverse transform unit by the aggregator 310 to generate output sample information (in this case referred to as residual samples or residual signals). The addresses in the reference picture memory format from which the motion compensation unit fetches the prediction samples may be controlled by the motion vector and made available to the motion compensation unit in the form of symbols 313, which may have, for example, X, Y, and reference picture components. Motion compensation may also include interpolation of sample values ​​fetched from the reference picture memory when a sub-sample accurate motion vector is in use, motion vector prediction mechanisms, etc.

[0033] The output samples of the aggregator 310 may be subjected to various loop filtering techniques in a loop filter unit 311. Video compression techniques may include in-loop filter techniques controlled by parameters contained in the coded video bitstream and made available to the loop filter unit 311 as symbols 313 from the parser 304, but may also be responsive to meta-information obtained during decoding of a previous part (in decoding order) of the coded picture or coded video sequence, or may be responsive to previously reconstructed, loop filtered sample values.

[0034] The output of the loop filter unit 311 may be a sample stream that can be output to a display 312, which may be a rendering device, or may be stored in a reference picture memory 557 for use in future inter-picture prediction.

[0035] Once a particular coded picture is fully reconstructed, it can be used as a reference picture for future prediction. Once a coded picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by parser 304), the current reference picture 309 can become part of reference picture buffer 308, and a new current picture memory can be reallocated before starting reconstruction of a subsequent coded picture.

[0036] The video decoder 300 may perform decoding operations according to a given video compression technique documented in a standard such as ITU-T Rec. H.265. The coded video sequence may conform to the syntax specified by the video compression technique or standard being used in the sense of adhering to the syntax of the video compression technique or standard as specified in the video compression technique document or standard, specifically in a profile document therein. It may also be necessary for compliance that the complexity of the coded video sequence be within a range defined by the level of the video compression technique or standard. In some cases, the level limits a maximum picture size, a maximum frame rate, a maximum reconstructed sample rate (e.g., measured in megasamples per second), a maximum reference picture size, etc. The limits set by the level may in some cases be further limited by the specification of a hypothetical reference decoder (HRD) and metadata for HRD buffer management signaled within the coded video sequence.

[0037] In one embodiment, the receiver 302 may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the coded video sequence. The additional data may be used by the video decoder 300 to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, a temporal layer, a spatial layer, or a signal-to-noise ratio (SNR) enhancement layer, redundant slices, redundant pictures, forward error correction codes, etc.

[0038] FIG. 4 may be a functional block diagram of a video encoder 400 according to one embodiment of the present disclosure.

[0039] The encoder 400 may receive video samples from a video source 401 (not part of the encoder), which may capture video images that are coded by the encoder 400 .

[0040] The video source 401 may provide a source video sequence to be coded by the encoder (303) in the form of a digital video sample stream that may be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...) and suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media serving system, the video source 401 may be a storage device that stores previously prepared video. In a video conferencing system, the video source 401 may be a camera that captures local image information as a video sequence. The video data may be provided as a number of individual pictures that convey motion when viewed in sequence. The pictures themselves may be organized as a spatial array of pixels, each of which may contain one or more samples, depending on the sampling structure, color space, etc., in use. Those skilled in the art can easily understand the relationship between pixels and samples. The following description focuses on samples.

[0041] According to an embodiment, the encoder 400 may code and compress pictures of a source video sequence into a coded video sequence 410 in real-time or under any other time constraint as required by an application. Providing an appropriate coding rate is one function of the controller 402. The controller controls and is operatively coupled to other functional units as described below. Coupling is not shown for clarity. Parameters set by the controller may include rate control related parameters (picture skip, quantizer, lambda value of rate distortion optimization technique, etc.), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. Those skilled in the art can easily identify other functions of the controller 402 as they may relate to the video encoder 400 being optimized for a particular system design.

[0042] Some video encoders operate in what those skilled in the art will easily recognize as a "coding loop." As an oversimplified explanation, the coding loop may consist of an encoding part of the encoder (e.g., source coder 403) (responsible for creating symbols based on the input picture to be coded and reference pictures), and a (local) decoder 406 built into the encoder 400 that reconstructs the symbols and creates sample data that the (remote) decoder would also create (since any compression between the symbols and the coded video bitstream is lossless in the video compression techniques considered in the disclosed subject matter). The reconstructed sample stream is input to a reference picture memory 405. The contents of the reference picture buffer are also bit-exact between the local and remote encoders, since the decoding of the symbol stream leads to bit-exact results independent of the location of the decoder (local or remote). In other words, the predictive part of the encoder "sees" exactly the same sample values ​​as the decoder would "see" when using prediction during decoding as reference picture samples. This basic principle of reference picture synchrony (and the resulting drift when synchrony cannot be maintained, eg, due to channel errors) is well known to those skilled in the art.

[0043] The operation of the "local" decoder 406 may be the same as that of the "remote" decoder 300, which has already been described in detail above in relation to Figure 3. However, with brief reference also to Figure 4, because symbols are available and the encoding / decoding of symbols into a coded video sequence by the entropy coder 408 and the parser 304 may be lossless, the entropy decoding portion of the decoder 300, including the channel 301, the receiver 302, the buffer 303, and the parser 304, may not be fully implemented in the local decoder 406.

[0044] At this point, it can be said that any decoder technology, other than analysis / entropy decoding, present in the decoder must necessarily also be present in the corresponding encoder in substantially the same functional form. A description of the encoder technology can be omitted, since it is the inverse of the decoder technology, which has been described generically. Only in certain areas is a more detailed description necessary, which is provided below.

[0045] As part of its operation, the source coder 403 may perform motion-compensated predictive coding, which predictively codes an input frame with reference to one or more previously coded frames from the video sequence, designated as “reference frames.” In this manner, the coding engine 407 codes differences between pixel blocks of the input frame and pixel blocks of reference frames that may be selected as predictive references to the input frame.

[0046] The local video decoder 406 may decode the coded video data of the frames that may be designated as reference frames based on the symbols created by the source coder 403. The operation of the coding engine 407 may advantageously be a lossy process. If the coded video data may be decoded in a video decoder (not shown in FIG. 4), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder 406 may reproduce the decoding process that may be performed by the video decoder on the reference frames and store the reconstructed reference frames in the reference picture memory 405, which may be, for example, a cache. In this way, the encoder 400 may locally store copies of the reconstructed reference frames that have common content as the reconstructed reference frames that will be obtained by the far-end video decoder (without transmission errors).

[0047] The predictor 404 may perform a predictive search for the coding engine 407. That is, for a new frame to be coded, the predictor 404 may search the reference picture memory 405 for sample data (as candidate reference pixel blocks) or specific metadata such as reference picture motion vectors, block shapes, etc., that may serve as suitable predictive references for the new picture. The predictor 404 may operate on sample blocks, pixel block by pixel block, to find suitable predictive references. In some cases, as determined by the search results obtained by the predictor 404, the input picture may have predictive references drawn from multiple reference pictures stored in the reference picture memory 405.

[0048] The controller 402 may manage the coding operations of the video coder 403, including, for example, setting parameters and subgroup parameters used to encode the video data.

[0049] The output of all the aforementioned functional units may be subjected to entropy coding in an entropy coder 408. The entropy coder converts the symbols produced by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques known to those skilled in the art, such as, for example, Huffman coding, variable length coding, arithmetic coding, etc.

[0050] The transmitter 409 may buffer the coded video sequence created by the entropy coder 408 in preparation for transmission over a communication channel 411, which may be a hardware / software link to a storage device that will store the encoded video data. The transmitter 409 may merge the coded video data from the video coder 403 with other data to be transmitted, such as coded audio data and / or auxiliary data streams (sources not shown).

[0051] The controller 402 may manage the operation of the encoder 400. During coding, the controller 405 may assign a particular coded picture type to each coded picture, which may affect the coding technique that may be applied to the respective picture. For example, pictures may often be assigned as one of the following frame types:

[0052] An intra picture (I-picture) may be a picture that can be coded and decoded without using any other frame in a sequence as a source of prediction. Some video codecs allow various types of intra pictures, including, for example, independent decoder refresh pictures. Those skilled in the art are aware of these variations of I-pictures and their respective uses and characteristics.

[0053] A predictive picture (P picture) may be a picture that can be coded and decoded using intra- or inter-prediction, which uses at most one motion vector and reference index to predict sample values ​​for each block.

[0054] A bidirectionally predicted picture (B-picture) may be a picture that can be coded and decoded using intra- or inter-prediction that uses up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, a multi-predicted picture may use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0055] A source picture may generally be spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and coded block by block. A block may be predictively coded with reference to other (already coded) blocks as determined by the coding assignment applied to the block's respective picture. For example, a block of an I picture may be non-predictively coded or predictively coded with reference to already coded blocks of the same picture (spatial prediction or intra prediction). A pixel block of a P picture may be non-predictively coded via spatial prediction or via temporal prediction with reference to one previously coded reference picture. A pixel block of a B picture may be non-predictively coded via spatial prediction or via temporal prediction with reference to one or two previously coded reference pictures.

[0056] Video coder 400 may perform coding operations in accordance with a given video coding technique or standard, such as ITU-T Rec. H.265. In its operations, video coder 400 may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy in an input video sequence. Thus, the coded video data may conform to a syntax specified by the video coding technique or standard being used.

[0057] In one embodiment, the transmitter 409 may transmit additional data along with the encoded video. The source coder 403 may include such data as part of the coded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures or slices, Supplemental Enhancement Information (SEI) messages, Visual Usability Information (VUI) parameter set fragments, etc.

[0058] Figure 5 is an example end-to-end architecture 500 for a standalone AR (STAR) device according to an example embodiment showing a 5G STAR user equipment (UE) receiver 600, a network / cloud 501, and a 5G UE (sender) 700. Figure 6 is a further detailed example 600 of one or more configurations for a STAR UE receiver 600 according to an example embodiment, and Figure 7 is a further detailed example 700 of one or more configurations for a 5G UE sender 700 according to an example embodiment. 3GPP TR 26.998 defines support for glasses-type augmented reality / mixed reality (AR / MR) devices in 5G networks. Also, in accordance with example embodiments herein, at least two device classes are contemplated: 1) devices capable of fully decoding and playing complex AR / MR content (standalone AR, i.e., STAR), and 2) devices that have smaller computational resources and / or smaller physical size (and therefore battery) and are capable of running such applications only if the majority of the computation is performed on the 5G edge server, network, or cloud rather than on the device (edge-dependent AR, i.e., EDGAR).

[0059] Also, according to example embodiments, as described below, a shared conversation use case can be experienced, in which all participants in a shared AR conversation experience have an AR device, each participant sees the other participants in the AR scene, the participants are overlays in the local physical scene, and the placement of participants in the scene is consistent at all receiving devices, e.g., people in each local space have the same positions / seating arrangements relative to each other, and such a virtual space creates the sense of being in the same space, but the room is different for each participant, as the room is an actual room or space in which each person is physically located.

[0060] For example, according to the exemplary embodiment shown with respect to Figures 5-7, an immersive media processing function on the network / cloud 501 receives uplink streams from various devices and composes a scene description that defines the arrangement of individual participants in a single virtual conference room. The scene description as well as the encoded media streams are distributed to each receiving participant. The receiving participant's 5G STAR UE 600 receives, decodes, processes the 3D video and audio streams and renders them using the received scene description and the information received from its AR runtime to create an AR scene of the virtual conference room with all other participants. The virtual room for a participant is based on his or her own physical space, but the seating / position arrangement of all other participants in the room matches the virtual rooms of all other participants in the session.

[0061] See also FIG. 8 showing an example 800 of an EDGAR device architecture according to an exemplary embodiment, where a device such as a 5G EDGAR UE 900 itself cannot perform a large amount of processing. Therefore, scene analysis and media analysis on received content is performed in the cloud / edge 801, and then a simplified AR scene with a small number of media components is delivered to the device for processing and rendering. FIG. 9 also shows an AR / MR (mixed reality) application provider 802. FIG. 9 shows a more detailed example of a 5G EDGAR UE 900 according to an exemplary embodiment.

[0062] However, even with capabilities such as those of the exemplary embodiments of Figures 5-9, there may be one or more technical issues with constructing a common virtual space scene description for immersive media functions. Also, as described below, such embodiments are technically improved in the context of immersive media processing functions to generate a scene description that is provided to all participants so that all participants experience the same relative placement of the participants within the local AR scene.

[0063] 10 illustrates an example 1000 in which users A10, B11, and T12 are to participate in an AR conference room, where one or more of the users may not have an R device. As shown, user A10 is in his office 1001, sitting in a conference room with a variable number of chairs, where user A10 is sitting in a chair. User B11 is in his living room 1002, sitting on a two-seater sofa, where there are also one or more couches for two people in his living room, as well as other furniture such as chairs and tables. User T12 is on a bench in the airport lobby 1003, but across a coffee table among one or more other coffee tables is the bench.

[0064] See also the AR environment in office 1001 where user A10's AR shows user A10 virtual user B11v1 corresponding to user B11 and virtual user T12v1 corresponding to user T12, with virtual user B11v1 and virtual user T12v1 shown to user A10 as sitting on furniture, office chairs, in office 1001, just like user A10. See also living room 1202 in example 1200 where the AR for user B11 shows virtual user T12v2 corresponding to user T12, but sitting on a couch in living room 1202, and virtual user A10v1 corresponding to user A10, sitting on furniture in living room 1202 rather than in an office chair in office 1201. See also airport lobby 1203, where the AR for user T12 shows virtual user A10v2, corresponding to user A10, sitting at a table in the airport lobby 1203, and virtual user B11v2, also sitting at a table opposite virtual user A10v2. And in each of those offices 1201, living room 1202, and airport lobby 1203, the updated scene description of each room matches the other rooms in terms of position / seating arrangement. For example, user A10 is shown with its virtual representation being counterclockwise relative to user T11, or clockwise relative to user T12, or its virtual representation room by room.

[0065] However, AR technology has been limited in any attempt to incorporate the creation and use of virtual spaces for devices that do not support AR but can analyze VR or 2D video, and the embodiments herein provide improved technical procedures for creating virtual scenes that match the AR scenes when such devices participate in a shared AR conversation service.

[0066] 11 illustrates an example end-to-end architecture 1100 having a non-AR device 1101 and a cloud / edge 1102 according to an exemplary embodiment. FIG. 12 also illustrates a more detailed block diagram example of the non-AR device 1101.

[0067] As shown in Figures 11 and 12, the non-AR UE 1101 is a device that is capable of rendering 360 or 2D video but does not have AR capabilities. However, the edge function on the cloud / edge 1102 is capable of AR rendering the received scene, rendering scene, and immersive visual and audio objects in a virtual room selected from a library. The entire video is then encoded and delivered to the device 1101 for decoding and rendering.

[0068] Thus, there may be multi-view capabilities, such as the AR processing on the edge / cloud 1102 being able to generate multiple videos of the same virtual room from different angles, with different viewports, etc. The device 1101 can then receive one or more of these videos and switch between them as needed, or send commands to the edge / cloud processing to stream only the desired viewport / angle.

[0069] Also, the background capability can be changed, where the user on the device 1101 can select the desired room background from a provided library, for example, one of the different conference rooms, or a living room and layout. The cloud / edge 1102 then uses the selected background and creates the virtual room accordingly.

[0070] 13 shows an example timing diagram 1300 for an example call flow for an immersive AR conversation for a receiving non-AR UE 1101. For illustrative purposes, only one sender is shown in this figure without showing a detailed call flow.

[0071] Shown are an AR application module 21, a media playback module 22, and a media access function module 23, which may be considered modules of a receiving non-AR UE 1101. Also shown is a cloud / edge partitioned rendering module 24. Also shown are a media delivery module 25 and a scene graph composer module 26, each of the network clouds 1102. Also shown is a 5G sender UE module 700.

[0072] S1 to S6 can be considered as a session establishment phase. The AR application module 21 can request to start a session to the media access function module 23 in S1, and the media access function module 23 can request to start a session to the cloud / edge split rendering module 24 in S2.

[0073] The cloud / edge partitioned rendering module 24 may perform session negotiation with the scene graph composer module 26 in S3, which may negotiate accordingly with the 5G sender UE 700. If successful, the cloud / edge partitioned rendering module may send an acknowledgement to the media access function module 23 in S5, which may send an acknowledgement to the AR application module 21.

[0074] Thereafter, S7 can be considered to be a media pipeline configuration stage where the media access function module 23 and the cloud / edge partitioned rendering module 24 each configure the pipeline. Then, after that pipeline configuration, a session can be initiated by signals from the AR application module to the media player module 22 at S8, from the media player module 22 to the media access function module 23 at S9, and from the media access function module 23 to the cloud / edge partitioned rendering module 24 at S10.

[0075] Next, there may be a pose loop stage from S11 to S13, where in S11 pose data may be provided from the media player module 22 to the AR application module 21, and in S12 the AR application module may provide the pose data 12 to the media access function module 23, which may then provide the pose data to the cloud / edge split rendering module 24.

[0076] S14-S16 can be considered as a shared experience stream stage in which the 5G sender UE 700 can provide a media stream to the media delivery module 25 at S14 and provide AR data to the scene graph synthesis module 26 at S15. The scene graph synthesis module 25 can then compose one or more scenes based on the received AR data and provide scenes and scene updates to the enablement / edge partitioned rendering module 24 at S16, and the media delivery module 25 can provide the media stream to the cloud / edge partitioned rendering module at S17. This can include, according to an exemplary embodiment, obtaining an AR scene descriptor from a non-AR device that does not render an AR scene, and generating a virtual scene by the cloud device by parsing and rendering the scene description obtained from the non-AR device.

[0077] S18-S19 can be considered a media uplink stage in which the media player module 22 captures and processes media data from its local user and provides the media data, at S18, to the media access function module 23. The media access module 23 can then encode the media and provide the media stream, at S19, to the cloud / edge partitioned rendering module 24.

[0078] Between S19 and S20 can be considered a media downlink phase in which the cloud / edge split rendering module 24 can perform scene analysis and full AR rendering, after which S20 and S21 can be considered to constitute a media stream loop phase. In S20, the cloud / edge split rendering module 24 can provide a media stream to the media access function module 23, which can then decode the media and provide a media rendering to the media player 22 in S21.

[0079] With these features according to the example embodiment, a non-AR UE 1101 can utilize its display capable of rendering VR or 2-D video even if it does not have a see-through display and therefore cannot create an AR scene. In this way, its immersive media processing capabilities generate only a common scene description that describes the scene and the relative positions of each participant with respect to the other participants. The scene itself needs to be adjusted with the pose information of each device before being rendered as an AR scene as described above. Also, an AR rendering process on the edge or cloud can analyze the AR scene and create a simplified VR-2D scene.

[0080] According to an example embodiment, the present disclosure uses a similar split rendering process of the EDGAR device for non-AR devices such as VR or 2-d video devices, where the edge / cloud AR rendering process does not generate an AR scene. Instead, it generates a virtual scene by parsing and rendering a scene description received from an immersive media processing capability for a given background (such as a conference room), and then renders each participant in the location described by the scene description of the conference room.

[0081] Additionally, the resulting video can be 360 ​​video or 2-D video depending on the capabilities of the receiving non-AR device, and the resulting video is generated taking into account position information received from the non-AR device in accordance with an exemplary embodiment.

[0082] Additionally, each other participant with a non-AR device is added as a 2D video overlay to the 360 / 2D video of the conference room, as shown in FIG. 10, and the room can have areas dedicated to the use of these overlays, such as furniture on which virtual images are overlaid, as shown in FIG. 10.

[0083] Additionally, audio signals from all participants can be mixed as needed to create single channel audio carrying the voices within the room, video can be encoded as a single 360 ​​or 2-D video and distributed to the devices, and optionally multiple video (multi-view) sources can be created, each capturing the same virtual conference room from different views and providing those views to devices according to an exemplary embodiment.

[0084] Additionally, the non-AR UE device 1101 can receive and render on the device display the 360 ​​video and / or one or more selected multi-view videos along with audio, and the user can change the viewport of the 360 ​​video by switching between different views or moving or rotating the view device, and thus navigate within the virtual room while watching the video.

[0085] The above-described embodiments include such 5G Media Stream Architecture (5GMS) extensions for the use of edge servers in their architecture, the specifications of which may have many features, but such features are technically not possible to deploy as a set of software development kits (SDKs) on a device or as a set of microservices on the cloud, and such technical deficiencies are addressed by the embodiments described further below.

[0086] For example, the current Media Services Enabler technical report does not define a framework that ties the specification to an SDK and does not include the concept of microservices.

[0087] Please refer to the example 1400 of FIG. 14 illustrating a 5G media streaming architecture with edge extension according to an exemplary embodiment. As shown, there is a user equipment (UE) 1401 and a data network (DN) 1411. The UE 1401 can include a 5GMS client 1403 and a 5GMS-enabled application 1405, such as, but not limited to, the AR or non-AR embodiments described above. The 5GMS client 1403 can also include a media stream handler 1403 and a media session handler. The DN 1411 can include a 5GMS application server (AS) 1412, a 5GMS application function (AF) 1414, and a 5GMS application provider 1413. The 5GMS AF 1414 may also communicate with a network exposure function (NEF) 1415 and a policy and charging function (PCF) 1416. The improvements described herein can be understood in the context of at least one or more of the UE 1401, the 5GMS-enabled application 1405, the 5GMS application provider 1413, the NEF 1415, and the PCF 1416, i.e., rather than having a monolithic specification in which there is no Media Services Enabler (MSE) definition for each function or group of functions, one or more of those elements can generate their own specifications and, upon matching requirements, can provide such specifications to another of those elements, which can further configure its initial element specification according to various possibilities as further described below, and such processing can be via the multiple exposed application programming interfaces (APIs) 1420 shown as well as any one or more of interfaces M1, M2, M4, M5, M6, M7, M8, N33, and N5.

[0088] FIG. 15 shows an example 1500 of a technical defect caused by the inability of the 5G server itself, as well as applications running through the 5G server, to obtain discovery via a new Capability and Performance Discovery (CPD) module 1501 added to the EDGAR client. This CPD module 1501 is responsible for providing a list of features, the parameters supported by the features, their performance, and the overall power state of the device (such as battery remaining). The module can collect statistics on each available feature, such as various encoders and decoders, measure resource consumption, i.e. cpu / gpu cycles, memory bandwidth, memory, and battery consumption, and provide detailed characteristics of the features, such as maximum video encoder profile and level, maximum supported width and height, and other related information. The CPD module 1501 of the EDGAR device architecture also allows queries made by any authorized application, such as an application on the device, an application on the edge network, or an application service provider over the wireless network.

[0089] 16 illustrates an example of a CPD module 1501 according to an example embodiment having a local application programming interface (API) for native applications on the device to access the device as well as for providing information to applications on an edge server or to the application service provider itself rather than the application via a 5G system interface. For example, controlling the 5G EDGAR device to discover features of the 5G EDGAR device includes controlling the 5G EDGAR device via a network API, and obtaining an indexing of features of the 5G EDGAR device includes obtaining the indexing via the network API.

[0090] The CPD module 1501, according to an exemplary embodiment, is configured to provide, via its API, indexing of any of the following functions, which are non-exhaustive examples of functionality: (1) providing a list of features supported by the device: (a) media encoders and decoders and their maximum supported profiles and levels (image, video, audio); (b) device capture capabilities (camera, microphone, and other sensors), which may be manufacturer specifications, current status such as whether they are used, application capabilities, etc.; (c) device rendering capabilities (speakers, display, and other sensors), which may be manufacturer specifications, current status such as whether they are used, application capabilities, etc.; (d) device rendering capabilities (speakers, display, and other sensors), which may be manufacturer specifications, current status such as whether they are used, application capabilities, etc.; (e) device performance information such as the device's performance, device model, and device specifications; (e) the device network protocol stack (various communication protocols), which may be manufacturer specifications, current status, application functionality, etc.; (f) raw hardware capabilities such as CPU and GPU benchmarks, memory, memory bandwidth, and on-device storage, which may be manufacturer specifications, current status, application functionality, etc.; (g) the efficiency of each function, i.e., the relative energy consumption of each function based on the parameters of each function, which may be determined by the EDGAR device by monitoring while the function is executed for each parameter; (h) the device's power state and battery drain rate, and (2) a combination profile assessment, such as whether a particular configuration of functions can be executed in real time, and the overall battery consumption of that combination.

[0091] Thus, example embodiments herein provide a method for discovering functionality supported by a 5G device using a discovery and performance function having a local device interface, as well as a network interface that can provide a list of media functions supported by the device, including capture, rendering, networking protocol stacks, raw hardware capabilities, the efficiency of each function, and finally power state and consumption, which can be requested to evaluate whether a particular combination of functions can be performed in real time, and if so, report the power consumption.

[0092] For example, referring to the example flowchart 1700 of FIG. 17, a 5G service in S1701 can request discovery in S1702 from the CPD module 1501, and in response, the CPD module 1501, via its various APIs, can discover and index, in S1704, according to an example embodiment, via its APIs, any of the following, which are non-exhaustive examples of functionality: (1) providing a list of features supported by the device: (a) media encoders and decoders and their maximum supported profiles and levels (image, video, audio), (b) device capture capabilities (camera, microphone, and other sensors), (c) device rendering capabilities (speakers, display, and other sensors); (d) device network protocol stack (various communication protocols); (e) raw hardware capabilities such as CPU and GPU benchmarks, memory, memory bandwidth, and on-device storage; (f) the efficiency of each feature, i.e., the relative energy consumption for each feature based on the parameters of that feature (this could be any particular codec, or any particular media stream, or its objects that are provided to the device being discovered); (g) the device's power state and battery drain rate; and (2) a combination profile assessment such as whether a particular configuration of features can be executed in real time, and the overall battery consumption of that combination.

[0093] A report may then be generated in S1705 regarding the index, or in S1707 as one or more combinations of the indexed functionality requested by the networked application or by the networked server itself, rather than by a networked application running via the networked server.

[0094] Additionally, the discovery in S1702 may be provided on an ongoing basis as live reports and indexing upon request, and in S1706 the discovered information may be provided to a network server or an application running via a networked server, any of which may then determine whether to offload the 5G service or a portion thereof to or between one of additional 5G networked devices as described above in the AR and non-AR contexts, although embodiments herein are not so limited.

[0095] The techniques described above may be implemented using computer readable instructions, as computer software physically stored on one or more computer readable media, or by one or more tangibly configured hardware processors. For example, FIG. 18 illustrates a computer system 1800 suitable for implementing certain embodiments of the disclosed subject matter.

[0096] The computer software can be coded using any suitable machine code or computer language that can undergo mechanisms such as assembly, compilation, linking, etc. to produce code containing instructions that can be executed directly by a computer central processing unit (CPU), graphics processing unit (GPU), etc., or via interpretation, microcode execution, etc.

[0097] The instructions may be executed on various types of computers or computer components including, for example, personal computers, tablet computers, servers, smart phones, gaming consoles, Internet of Things devices, and the like.

[0098] 18 for computer system 1800 are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing embodiments of the present disclosure. The arrangement of components should not be interpreted as having a dependency or requirement regarding any one or combination of components illustrated in the exemplary embodiment of computer system 1800.

[0099] The computer system 1800 may include certain human interface input devices. Such human interface input devices may be responsive to input by one or more human users via, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), olfactory input (not shown). The human interface devices may also be used to capture certain media not necessarily directly associated with conscious human input, such as audio (e.g., voice, music, ambient sounds), images (e.g., scanned images, photographic images, still images obtained from a camera, etc.), video (two-dimensional video, three-dimensional video including stereoscopic video, etc.).

[0100] The input human interface devices may include one or more of a keyboard 1801, a mouse 1802, a trackpad 1803, a touch screen 1810, a joystick 1805, a microphone 1806, a scanner 1808, a camera 1807 (only one of each is shown).

[0101] The computer system 1800 may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the human user's senses, for example, through haptic output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via touch screen 1810 or joystick 1805, although there may also be haptic feedback devices that do not function as input devices), audio output devices (e.g., speakers 1809, headphones (not shown)), visual output devices (e.g., screens 1810 including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capability, each with or without haptic feedback capability, some of which may be capable of outputting two-dimensional visual output or output in excess of three dimensions via means such as stereoscopic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0102] The computer system 1800 may also include human accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW 1820 having media such as CD / DVD 1811, thumb drives 1822, removable hard drives or solid state drives 1823, legacy magnetic media such as tapes and floppy disks (not shown), dedicated ROM / ASIC / PLD based devices such as security dongles (not shown), etc.

[0103] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the subject matter of this disclosure does not encompass transmission media, carrier waves, or other transitory signals.

[0104] The computer system 1800 may also include an interface 1899 to one or more communications networks 1898. The networks 1898 may be, for example, wireless, wired, optical. The networks 1898 may further be local, wide area, metropolitan, vehicular and industrial, real-time, delay tolerant, etc. Examples of networks 1898 include local area networks such as Ethernet, WLAN, etc., cellular networks including GSM, 3G, 4G, 5G, LTE, etc., TV wired or wireless wide area digital networks including cable TV, satellite TV, and terrestrial broadcast TV, vehicular and industrial including CANBus, etc. Particular networks 1898 generally require an external network interface adapter attached to a particular general-purpose data port or peripheral bus (1850 and 1851) (e.g., a USB port of the computer system 1800, etc.), while other networks are generally built into the core of the computer system 1800 by connection to the system bus as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks 1898, the computer system 1800 can communicate with other entities. Such communications may be unidirectional receive only (e.g., broadcast TV), unidirectional transmit only (e.g., CANbus to certain CANbus devices), or bidirectional, such as to other computer systems using local or wide area digital networks. Particular protocols and protocol stacks may be used with each of those networks and network interfaces, as described above.

[0105] The aforementioned human interface devices, human accessible storage devices, and network interfaces may be attached to the core 1840 of the computer system 1800.

[0106] The core 1840 may include one or more central processing units (CPUs) 1841, graphics processing units (GPUs) 1842, graphics adapters 1817, dedicated programmable processing units in the form of field programmable gate areas (FPGAs) 1843, hardware accelerators for specific tasks 1844, and the like. These devices may be connected via a system bus 1848, along with read only memory (ROM) 1845, random access memory 1846, internal mass storage 1847 such as an internal hard drive, SSD, etc. that is not accessible to the user. In some computer systems, the system bus 1848 may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus 1848 or via a peripheral bus 1851. Architectures for peripheral buses include PCI, USB, and the like.

[0107] The CPU 1841, GPU 1842, FPGA 1843, and accelerator 1844 can execute certain instructions that may combine to constitute the aforementioned computer code. That computer code can be stored in ROM 1845 or RAM 1846. Persistent data can be stored, for example, in internal mass storage 1847, while temporary data can also be stored in RAM 1846. Rapid storage and retrieval in any of the memory devices can be made possible by the use of cache memories that can be closely associated with one or more of the CPU 1841, GPU 1842, mass storage 1847, ROM 1845, RAM 1846, etc.

[0108] The computer-readable medium can bear computer code for performing various computer-implemented operations. The medium and computer code can be those specially designed and constructed for the purposes of the present disclosure, or they can be of the available kind well known to those skilled in the computer software arts.

[0109] By way of example and not limitation, the architecture corresponding to computer system 1800, specifically core 1840, may provide functionality as a result of a processor (including CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media may be user-accessible mass storage as previously described, as well as media associated with specific storage of core 1840 that is non-transitory in nature, such as core internal mass storage 1847 or ROM 1845. Software implementing various embodiments of the present disclosure may be stored in such devices and executed by core 1840. Computer-readable media may include one or more memory devices or chips, depending on the particular needs. The software may cause core 1840, specifically the processor therein (including CPU, GPU, FPGA, etc.) to perform certain processes or certain parts of certain processes described herein, including defining data structures stored in RAM 1846 and modifying such data structures according to the software-defined processes. Additionally or alternatively, the computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator 1844) that may operate in place of or in conjunction with software to perform particular processes or particular portions of particular processes described herein. References to software may encompass logic, where appropriate, and vice versa. References to computer-readable media may encompass circuitry (such as integrated circuits (ICs)) that stores software for execution, circuitry that embodies logic for execution, or both, where appropriate. The present disclosure encompasses any suitable combination of hardware and software.

[0110] While this disclosure describes several exemplary embodiments, there are alterations, permutations, and various substitute equivalents that fall within the scope of this disclosure. It will thus be appreciated that those skilled in the art will be able to devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within the spirit and scope of the present disclosure. [Explanation of symbols]

[0111] 10 User A, 10v1 Virtual User A, 10v2 Virtual User A, 11 User B, 11v1 Virtual User B, 11v2 Virtual User B, 12 User T, 12v1 Virtual User T, 12v2 Virtual User T, 12 Pose data, 21 AR application module, 22 Media playback module, media player module, 23 Media access function module, 24 Cloud / edge split rendering module, 25 Media delivery module, 26 Scene graph composer module, scene graph synthesis module, 100 Communication system, 101 Terminal, 102 Terminal, second terminal, 103 Terminal, first terminal, 104 Terminal, 105 Communication network, 201 Video source, camera, 202 Encoder, 203 Capture subsystem, 204 Encoded video bitstream, video bitstream, 205 Streaming server, 206 Copy of encoded video bitstream, video bitstream, 207 Streaming client, 208 Copy of encoded video bitstream, video bitstream, 209 Display, 210 Video sample stream, 211 Video decoder, 212 Streaming client, 213 Uncompressed video sample stream, 300 Video decoder, 301 Channel, 302 Receiver, 303 Buffer memory, Encoder, 304 Entropy decoder, Parser, 305 Scaler / inverse transform unit, 306 Motion compensation prediction unit, 307 Intra picture prediction unit, 308 Reference picture memory, Reference picture buffer, 309 Current reference picture, 310 Aggregator, 311 Loop filter unit, 312 Display, 313 Symbol, 400 Video encoder, Video coder, 401 Video source, 402 Controller, 403 Source coder, Video coder, 404 Predictor, 405 Reference picture memory, 406 Local video decoder, 407 Coding engine, 408 Entropy coder, 409 Transmitter, 410 Coded video sequence, 411 Communication channel, 500 Example, 501 Network / cloud, 600STAR User Equipment (UE) receiver, example, 700 5G sender UE module, example, 800 Example, 801 Cloud / Edge, 802 AR / MR (Mixed Reality) application provider, 900 5G EDGAR UE, 1000 Example, 1001 Office, 1002 Living room, 1003 Airport lobby, 1100 Example, 1101 Non-AR device / non-AR UE, 1102 Cloud / Edge, 1200 Example, 1201 Office, 1202 Living room, 1203 Airport lobby, 1300 Timing diagram, 1400 Example, 1401 User Equipment (UE), 1402 5GMS client, 1403 Media stream handler, 1405 5GMS-enabled application, 1412 5GMS Application Server (AS), 1413 5GMS application provider, 1414 5GMS Application Functions (AF), 1415 Network Exposed Functions (NEF), 1416 Policy and Charging Functions (PCF), 1420 Exposed Application Programming Interfaces (APIs), 1500 Example, 1501 Capability and Performance Discovery (CPD) module, 1700 Flowchart, 1800 Computer system, 1801 Keyboard, 1802 Mouse, 1803 Trackpad, 1805 Joystick, 1806 Microphone, 1807 Camera, 1808 Scanner, 1809 Speaker, 1810 Touch screen, 1811 CD / DVD, 1817 Graphics adapter, 1820 CD / DVD ROM / RW, 1822 Thumb drive, 1823 Removable hard drive, Solid state drive, 1840 Core, 1841 Central Processing Unit (CPU), 1842 Graphics Processing Unit (GPU), 1843 field programmable gate area (FPGA), 1844 hardware accelerator, 1845 read only memory (ROM), 1846 random access memory, 1847 internal mass storage, 1848 system bus, 1850 peripheral bus, 1851 peripheral bus, 1898 communication network, 1899 interface, M1 interface, M2 interface, M4 interface, M5 interface, M6 interface, M7interface, M8 interface, N5 interface, N33 interface, S1 session establishment phase, S2 session establishment phase, S3 session establishment phase, S4 session establishment phase, S5 session establishment phase, S6 session establishment phase, S7 media pipeline configuration phase, S11 pause loop phase, S12 pause loop phase, S13 pause loop phase, S14 shared experience stream phase, S15 shared experience stream phase, S16 shared experience stream phase, S18 media uplink phase, S19 media uplink phase, S20 media stream loop phase, S21 media stream loop phase

Claims

1. 1. A method for media streaming, the method comprising: Controlling a 5G Edge-Based Augmented Reality (EDGAR) device and discovering at least some characteristics of the 5G EDGAR device; obtaining an index of at least some of the characteristics of the 5G EDGAR device; controlling implementation of a 5G Media Streaming (5GMS) service according to the indexing of the at least some characteristics of the 5G EDGAR device; A method comprising:

2. The step of controlling the 5G EDGAR device to discover the at least some characteristics of the 5G EDGAR device includes controlling the 5G EDGAR device to discover the at least some characteristics via one of a plurality of application programming interfaces (APIs) of the 5G EDGAR device. The method of claim 1.

3. The step of controlling the 5G EDGAR device to discover at least some of the characteristics of the 5G EDGAR device includes controlling the 5G EDGAR device via a network API; The step of obtaining the indexing of at least some of the characteristics of the 5G EDGAR device includes obtaining the indexing via the network API. The method of claim 2.

4. the at least some characteristics include parameters of at least one of a media encoder and a media decoder; The method of claim 3.

5. At least some of the features include a device capture capability of the 5G EDGAR device, the device capture capability including any of a camera capability and a microphone capability; The method of claim 3.

6. At least some of the features include a device rendering capability of the 5G EDGAR device, the device rendering capability including any of a speaker capability and a display capability; The method of claim 3.

7. At least some of the features include raw hardware capabilities of the 5G EDGAR device, the raw hardware capabilities including any of a CPU benchmark, a GPU benchmark, memory bandwidth, and on-device storage; The method of claim 3.

8. At least some of the features include a functional efficiency of the 5G EDGAR device, the functional efficiency indicating a relative amount of power consumption per function and a relative amount for each parameter of the function; The method of claim 3.

9. At least some of the characteristics include a power state of the 5G EDGAR device and a rate at which the battery of the 5G EDGAR device is depleted. The method of claim 3.

10. The indexing indicates either whether a combination of at least some of the features can be executed in real time on the 5G EDGAR device and an estimated overall battery consumption of the combination. The method of claim 3.

11. 1. An apparatus for media streaming, comprising: at least one memory configured to store computer program code; at least one processor configured to access the computer program code and operate as instructed by the computer program code, the computer program code comprising: first control code configured to cause the at least one processor to control a 5G Edge-based Augmented Reality (EDGAR) device and discover at least some characteristics of the 5G EDGAR device; and acquiring code configured to cause the at least one processor to acquire an indexing of the at least some characteristics of the 5G EDGAR device. second control code configured to cause the at least one processor to control implementation of a 5G Media Streaming (5GMS) service according to the indexing of the at least some characteristics of the 5G EDGAR device; At least one processor including: An apparatus comprising:

12. The first control code is configured to cause the at least one processor to control the 5G EDGAR device to discover the at least some of the characteristics via one of a plurality of application programming interfaces (APIs) of the 5G EDGAR device.

12. The apparatus of claim 11.

13. The first control code is configured to cause the at least one processor to control the 5G EDGAR device via a network API; the obtaining code configured to cause the at least one processor to obtain the indexing via the network API; 13. The apparatus of claim 12.

14. the at least some characteristics include parameters of at least one of a media encoder and a media decoder; 14. The apparatus of claim 13.

15. At least some of the features include a device capture capability of the 5G EDGAR device, the device capture capability including any of a camera capability and a microphone capability; 14. The apparatus of claim 13.

16. At least some of the features include a device rendering capability of the 5G EDGAR device, the device rendering capability including any of a speaker capability and a display capability; 14. The apparatus of claim 13.

17. At least some of the features include raw hardware capabilities of the 5G EDGAR device, the raw hardware capabilities including any of a CPU benchmark, a GPU benchmark, memory bandwidth, and on-device storage; 14. The apparatus of claim 13.

18. At least some of the features include a functional efficiency of the 5G EDGAR device, the functional efficiency indicating a relative amount of power consumption per function and a relative amount for each parameter of the function; 14. The apparatus of claim 13.

19. At least some of the characteristics include a power state of the 5G EDGAR device and a rate at which the battery of the 5G EDGAR device is depleted.

14. The apparatus of claim 13.

20. When executed by at least one processor, the method causes the at least one processor to: Controlling a 5G Edge-Based Augmented Reality (EDGAR) device and discovering at least some characteristics of the 5G EDGAR device; obtaining an index of at least some of the characteristics of the 5G EDGAR device; controlling implementation of a 5G Media Streaming (5GMS) service according to the indexing of the at least some characteristics of the 5G EDGAR device; A non-transitory computer readable medium storing a program for executing the method.

Citation Information

Patent Citations

  • To provide users with feedback on power consumption within battery-powered electronic devices.

    JP2013502181A

  • Systems and methods for providing zone functionality in networked media systems

    US20100292818A1

  • Edge computing local breakout

    US20220038554A1

  • Methods for discovery of media capabilities of 5g edge

    WO2021225765A1