Method for describing and configuring a 5G media service enabler

A framework for defining media service enablers as SDKs or microservices in the 5GMS architecture addresses deployment limitations, reducing overhead and enhancing operational efficiency.

JP2024543901AInactive Publication Date: 2024-11-26TENCENT AMERICA LLC
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2024531146
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-03-30
Filing Date
2023-04-04
Publication Date
2024-11-26
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The existing 5G Media Streaming (5GMS) architecture is inadequate as it cannot be deployed as a set of software development kits (SDKs) or microservices, leading to network and server computational overhead, and lacks a framework for defining media service enablers per feature or group of features.

Method used

A framework is developed to define media service enablers as SDKs or microservices, using an MSE specification that includes an MSE description document (MDD) and configuration API (MCA) to ensure compliance and configuration of SDKs and microservices, allowing for reduced network and server overhead.

Benefits of technology

This approach reduces network and server computational overhead while enabling flexible deployment of media service enablers, improving practicality and technical signaling capabilities.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024543901000001
    Figure 2024543901000001
  • Figure 2024543901000002
    Figure 2024543901000002
  • Figure 2024543901000003
    Figure 2024543901000003
Patent Text Reader

Abstract

The invention includes a method and apparatus comprising computer code configured such that one or more processors generate a Media Services Enabler (MSE) configuration of the MSE specification for 5G media streaming (5GMS) services, determine whether at least one of a software development kit (SDK) and a microservice conforms to the MSE specification, configure at least one of a function and a service according to the MSE specification and based on the determination that at least one of the SDK and a microservice conforms to the MSE specification, control the implementation of the 5G media streaming (5GMS) service according to the configuration of at least one of the SDK and a microservice, and at least one of the functions and a service is outside the implementation of the 5GMS service.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is based on and claims priority to U.S. Provisional Patent Application No. 63 / 338,338, filed May 4, 2022, and U.S. Patent Application No. 18 / 192,921, filed March 30, 2023, the disclosures of which are incorporated by reference in their entireties herein.

[0002] This disclosure defines a framework for developing specifications and deploying them as media service enablers, as SDKs, or as microservices on cloud processors. [Background technology]

[0003] The current design is a monolithic specification that does not define media service enablers per feature or group of features, even though the 5G Media Streaming (5GMS) architecture provides an extension to use edge servers in the architecture. The 5GMS architecture is insufficient because it cannot be deployed as a set of SDKs on the device or as a set of microservices on the cloud. Summary of the Invention [Problem to be solved by the invention]

[0004] To address one or more different technical problems, the present disclosure provides technical solutions that reduce network overhead and server computation overhead, while providing options to apply various operations to the resolved elements, such that their utility and some of the technical signaling functions may be improved when using these operations. [Means for solving the problem]

[0005] Methods and apparatuses are included that include a memory configured to store computer program code and one or more processors configured to access the computer program code and operate when instructed by the computer program code. The computer program code includes: a generation code configured to cause the at least one processor to perform a generation code configured to cause the at least one processor to perform a generation of a media service enabler (MSE) configuration of an MSE specification for a 5G media streaming (5GMS) service; a determination code configured to cause the at least one processor to perform a determination of whether a software development kit (SDK) and at least one of the microservices conform to the MSE specification; a setting code configured to cause the at least one processor to perform a setting of a configuration of the SDK and at least one of the microservices according to the MSE specification and based on determining that the SDK and at least one of the microservices conform to the MSE specification by at least one of the functions and services; and a control code configured to cause the at least one processor to control an implementation of the 5G media streaming (5GMS) service according to the configuration of the SDK and at least one of the microservices, where at least one of the functions and services is external to the implementation of the 5GMS service.

[0006] According to an example embodiment, the implementation of the 5GMS service includes implementation of at least one of MSE SDK abstraction, MSE SDK instantiation, and MSE microservices.

[0007] According to an example embodiment, an MSE SDK abstraction represents media aspects, an MSE configuration including an MSE description document (MDD), and an MSE configuration application programming interface (API) (MCA), an MSE SDK instantiation represents a specific implementation of the media aspects, the MSE configuration including the MDD, and the MCA, and an MSE microservice represents a service API for the media aspects, the MSE configuration including the MDD, and the specific implementation of the MCA.

[0008] According to an example embodiment, the media aspect refers to any of service access information, consumption reporting, metric reporting, dynamic policies, and network assistance. According to an example embodiment, the MDD indicates a performance metric and whether the MSE complies with the media aspects.

[0009] According to an example embodiment, when the function and / or service determines that the SDK and / or microservice complies with the MSE specification, it sets the configuration of the SDK and / or microservice via the same interface through which the function and / or service received the information.

[0010] According to an example embodiment, when the function and / or service determines that the SDK and / or microservice complies with the MSE specification, it sets the configuration of the SDK and / or microservice via the same interface through which the function and / or service received the information.

[0011] According to an example embodiment, determining that at least one of the SDK and the microservice conforms to the MSE specification includes determining that at least a format of the MSE description document (MDD) conforms to a format of at least one of the SDK and the microservice.

[0012] According to an example embodiment, generating an MSE configuration for an MSE specification for a 5G media streaming (5GMS) service includes setting an identifier of the MSE.

[0013] According to an example embodiment, the information received by the function and / or service includes an identifier of the MSE.

[0014] According to an example embodiment, controlling an implementation of the 5GMS service in accordance with configuration of the SDK and / or microservices includes indicating to at least one of functions and services external to the implementation of the 5GMS service that the function and / or service may, in some cases, request a status of the 5GMS service.

[0015] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Brief description of the drawings]

[0016] [Figure 1] FIG. 2 is a simplified schematic diagram according to an embodiment. [Diagram 2] FIG. 2 is a simplified schematic diagram according to an embodiment. [Diagram 3] FIG. 2 is a simplified block diagram of a decoder according to an embodiment. [Figure 4] FIG. 2 is a simplified block diagram of an encoder according to an embodiment. [Diagram 5] FIG. 2 is a simplified block diagram according to an embodiment. [Figure 6] FIG. 2 is a simplified block diagram according to an embodiment. [Figure 7] FIG. 2 is a simplified block diagram according to an embodiment. [Figure 8] FIG. 2 is a simplified block diagram according to an embodiment. [Figure 9] FIG. 2 is a simplified block diagram according to an embodiment. [Figure 10]FIG. 2 is a simplified diagram according to an embodiment. [Figure 11] FIG. 2 is a simplified block diagram according to an embodiment. [Figure 12] FIG. 2 is a simplified block diagram according to an embodiment. [Figure 13] FIG. 2 is a simplified block and timing diagram according to an embodiment. [Figure 14] FIG. 2 is a simplified block diagram according to an embodiment. [Figure 15] FIG. 2 is a simplified block diagram according to an embodiment. [Figure 16] FIG. 2 is a simplified block diagram according to an embodiment. [Figure 17] FIG. 2 is a simplified flow diagram according to an embodiment. [Figure 18] FIG. 2 is a schematic diagram according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] The proposed features discussed below may be used separately or combined in any order. Furthermore, the embodiments may be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute a program stored on a non-transitory computer-readable medium.

[0018] FIG. 1 illustrates a simplified block diagram of a communication system 100 according to one embodiment of the present disclosure. The communication system 100 may include at least two terminals 102, 103 interconnected via a network 105. For unidirectional transmission of data, a first terminal 103 may encode video data at a local location for transmission to the other terminal 102 via the network 105. The second terminal 102 may receive the other terminal's encoded video data from the network 105, decode the encoded data, and display the restored video data. Unidirectional data transmission may be common in media serving applications, etc.

[0019] 1 illustrates a second pair of terminals 101 and 104 provided to support bidirectional transmission of encoded video, such as may occur during a video conference. For the bidirectional transmission of data, each terminal 101 and 104 may encode captured video data at a local location for transmission to the other terminal over network 105. Each terminal 101 and 104 may also receive encoded video data transmitted by the other terminal, decode the encoded data, and display the recovered video data on a local display device.

[0020] In FIG. 1, terminals 101, 102, 103, and 104 may be illustrated as a server, a personal computer, and a smartphone, although the principles of the present disclosure are not so limited. The embodiments of the present disclosure apply to laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network 105 represents any number of networks that convey encoded video data between terminals 101, 102, 103, and 104, including, for example, wired and / or wireless communication networks. Communication network 105 may exchange data in circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of this discussion, the architecture and topology of network 105 may not be important to the operation of the present disclosure unless described below in this specification.

[0021] 2 illustrates the arrangement of a video encoder and a video decoder in a streaming environment as one example of an application of the disclosed subject matter. The subject matter of this disclosure is equally applicable to other video-enabled applications including, for example, video conferencing, digital television, storage of compressed video on digital media including CDs, DVDs, memory sticks, and the like.

[0022] The streaming system may include a capture subsystem 203, which may include a video source 201, such as a digital camera, that creates an uncompressed video sample stream 213. The sample stream 213 may be emphasized as a large data volume compared to an encoded video bitstream and may be processed by an encoder 202 coupled to the camera 201. The encoder 202 may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video bitstream 204 may be emphasized as a smaller data volume compared to the sample stream and may be stored on a streaming server 205 for future use. One or more streaming clients 212 and 207 may access the streaming server 205 to retrieve copies 208 and 206 of the encoded video bitstream 204. The client 212 may include a video decoder 211 that decodes the incoming copy 208 of the encoded video bitstream and creates an outgoing video sample stream 210 that may be rendered on a display 209 or other rendering device (not shown). In some streaming systems, the video bitstreams 204, 206, and 208 may be encoded according to a particular video encoding / compression standard, examples of which are mentioned above and further described herein.

[0023] FIG. 3 may be a functional block diagram of a video decoder 300 according to one embodiment of the present invention.

[0024] The receiver 302 may receive one or more codec video sequences to be decoded by the decoder 300, and in the same or another embodiment, may receive one coded video sequence at a time, with the decoding of each coded video sequence being independent of the other coded video sequences. The coded video sequences may be received from a channel 301, which may be a hardware / software link to a storage device that stores the coded video data. The receiver 302 may receive the coded video data along with other data, such as coded audio data and / or auxiliary data streams, that may be forwarded to a respective using entity (not shown). The receiver 302 may separate the coded video sequences from the other data. To combat network jitter, a buffer memory 303 may be coupled between the receiver 302 and the entropy decoder / parser 304 (hereinafter, "parser"). If the receiver 302 is receiving data from a storage / forwarding device with sufficient bandwidth and controllability, or from an isosynchronous network, the buffer 303 may not be necessary or may be small. For use in a best effort packet network such as the Internet, buffer 303 may be required and may be relatively large, and may advantageously be of adaptive size.

[0025] The video decoder 300 may include a parser 304 for reconstructing symbols 313 from the entropy coded video sequence. These categories of symbols include information used to manage the operation of the decoder 300 and potentially information for controlling a rendering device, such as a display 312, that is not an integral part of the decoder but may be coupled to it. The rendering device control information may be in the form of supplemental enhancement information (SEI message) or a video usability information parameter set fragment (not shown). The parser 304 may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may be according to a video coding technique or standard and may be according to principles well known to those skilled in the art, including variable length coding, Huffman coding, arithmetic coding with or without contextual dependency, etc. The parser 304 may extract from the coded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder based on at least one parameter corresponding to that group. The subgroups may include groups of pictures (GOPs), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), etc. The entropy decoder / parser may also extract information such as transform coefficients, quantizer parameter values, motion vectors, etc. from the encoded video sequence.

[0026] The parser 304 may perform an entropy decoding / parsing operation on the video sequence received from the buffer 303 to create symbols 313. The parser 304 may receive the encoded data and selectively decode particular symbols 313. Additionally, the parser 304 may determine whether a particular symbol 313 is provided to the motion compensated prediction unit 306, the scaler / inverse transform unit 305, the intra prediction unit 307, or the loop filter 311.

[0027] The reconstruction of symbols 313 may involve several different units depending on the type of coded video picture or part thereof (inter-picture and intra-picture, inter-block and intra-block, etc.) as well as other factors. Which units are involved and how may be governed by subgroup control information parsed from the coded video sequence by parser 304. The flow of such subgroup control information between parser 304 and subsequent units is not shown for clarity.

[0028] In addition to the functional blocks already mentioned, the decoder 300 can be conceptually subdivided into several functional units, as described below. In an actual implementation operating under commercial constraints, many of these units will interact closely with each other and may be at least partially integrated with each other. However, for purposes of describing the disclosed subject matter, the following conceptual subdivision into functional units is appropriate:

[0029] The first unit is a scalar / inverse transform unit 305. The scalar / inverse transform unit 305 receives quantized transform coefficients and control information including the transform to use, block size, quantization coefficients, quantization scaling matrix, etc. as symbols 313 from the parser 304. The scalar / inverse transform unit 305 may output blocks containing sample values ​​that may be input to an aggregator 310.

[0030] In some cases, the output samples of the scalar / inverse transform 305 may relate to intra-coded blocks, i.e., blocks that do not use prediction information from a previously reconstructed picture, but can use prediction information from a previously reconstructed part of the current picture. Such prediction information may be provided by the intra-picture prediction unit 307. In some cases, the intra-picture prediction unit 307 uses surrounding already reconstructed information fetched from the current (partially reconstructed) picture 309 to generate a block of the same size and shape as the block being reconstructed. The aggregator 310 adds, possibly on a sample-by-sample basis, the prediction information generated by the intra-prediction unit 307 to the output sample information provided by the scalar / inverse transform unit 305.

[0031] In other cases, the output samples of the scalar / inverse transform unit 305 may relate to an inter-coded, potentially motion-compensated block. In such cases, the motion compensation prediction unit 306 may access the reference picture memory 308 to fetch samples used for prediction. After motion compensating the fetched samples according to the symbols 313 related to the block, these samples may be added by the aggregator 310 to the output of the scalar / inverse transform unit to generate output sample information (in this case referred to as residual samples or residual signals). The addresses in the reference picture memory format from which the motion compensation unit fetches the prediction samples may be controlled by a motion vector and may be made available to the motion compensation unit in the form of symbols 313 that may have, for example, X, Y, and reference picture components. Motion compensation may also include interpolation of sample values ​​fetched from the reference picture memory when sub-sample accurate motion vectors are used, motion vector prediction mechanisms, etc.

[0032] The output samples of the aggregator 310 may be subjected to various loop filtering techniques in a loop filter unit 311. Video compression techniques may include in-loop filter techniques controlled by parameters contained in the encoded video bitstream and made available to the loop filter unit 311 as symbols 313 from the parser 304, but may also be responsive to meta-information obtained during decoding of previous (in decoding order) parts of the encoded picture or encoded video sequence, or to previously reconstructed and loop filtered sample values.

[0033] The output of the loop filter unit 311 may be a sample stream that may be output to a display 312, which may be a rendering device, and may also be stored in a reference picture memory 557 for use in future inter-picture prediction.

[0034] Once a particular coded picture is fully reconstructed, it can be used as a reference picture for future predictions. Once a coded picture is fully reconstructed and the coded picture has been identified as a reference picture (e.g., by the parser 304), the current reference picture 309 can become part of the reference picture buffer 308, and a new current picture memory can be reallocated before starting reconstruction of a subsequent coded picture.

[0035] The video decoder 300 may perform decoding operations according to a given video compression technique, which may be documented in a standard, such as ITU-T Rec. H.265. The encoded video sequence may conform to the syntax specified by the video compression technique or standard being used, in the sense of adhering to the syntax of the video compression technique or standard, as specified in the video compression technique document or standard, specifically in a profile document therein. Also, what is required for conformance is that the complexity of the encoded video sequence is within a range defined by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstructed sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the level may be further constrained in some cases by a hypothetical reference decoder (HRD) specification, and metadata for HRD buffer management signaled in the encoded video sequence.

[0036] In one embodiment, the receiver 302 may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the encoded video sequence. The additional data may be used by the video decoder 300 to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, a temporal layer, a spatial layer, or a signal-to-noise ratio (SNR) enhancement layer, redundant slices, redundant pictures, forward error correction codes, etc.

[0037] FIG. 4 may be a functional block diagram of a video encoder 400 according to one embodiment of the present disclosure.

[0038] The encoder 400 may receive video samples from a video source 401 (not part of the encoder), which may capture video images that are encoded by the encoder 400 .

[0039] The video source 401 may provide a source video sequence to be encoded by the encoder (303) in the form of a digital video sample stream that may be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...) and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media delivery system, the video source 401 may be a storage device that stores previously prepared video. In a video conferencing system, the video source 401 may be a camera that captures local image information as a video sequence. The video data may be provided as a number of separate pictures that give motion when viewed in sequence. The pictures themselves may be organized as a spatial array of pixels, each pixel may contain one or more samples depending on the sampling structure, color space, etc. in use. Those skilled in the art can easily understand the relationship between pixels and samples. The following description focuses on samples.

[0040] According to an embodiment, the encoder 400 may encode and compress pictures of a source video sequence into an encoded video sequence 410 in real-time or under any other time constraint as required by an application. Providing an appropriate encoding rate is one function of the controller 402. The controller controls and is functionally coupled to other functional units as described below. For clarity, coupling is not depicted. Parameters set by the controller may include rate control related parameters (picture skip, quantizer, lambda value of rate distortion optimization technique, etc.), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. Those skilled in the art may easily identify other functions of the controller 402 as they may relate to the video encoder 400 optimized for a particular system design.

[0041] Some video encoders operate in what those skilled in the art will easily recognize as a "coding loop." As an oversimplified explanation, the coding loop may consist of an encoding part of the encoder (e.g., source coder 403) (responsible for creating symbols based on the input picture to be coded and reference pictures), and a (local) decoder 406 built into the encoder 400 that reconstructs the symbols to create sample data that the (remote) decoder will also create (since any compression between the symbols and the coded video bitstream is lossless in the video compression techniques considered in the disclosed subject matter). The reconstructed sample stream is input to a reference picture memory 405. The decoding of the symbol stream results in bit-exact results regardless of the location of the decoder (local or remote), so the reference picture buffer contents are also bit-exact between the local and remote encoders. In other words, the predictive part of the encoder "sees" exactly the same sample values ​​as the decoder "sees" when using prediction during decoding as reference picture samples. This basic principle of reference picture synchrony (and the resulting drift when synchrony cannot be maintained, for example due to channel errors) is well known to those skilled in the art.

[0042] The operation of the "local" decoder 406 may be the same as that of the "remote" decoder 300, which has already been described in detail above in relation to Figure 3. However, with brief reference also to Figure 4, because symbols are available and the encoding / decoding of symbols into an encoded video sequence by the entropy coder 408 and the parser 304 may be lossless, the entropy decoding portion of the decoder 300, including the channel 301, the receiver 302, the buffer 303, and the parser 304, may not be fully implemented in the local decoder 406.

[0043] An observation that can be made in this respect is that any decoder technique other than analysis / entropy decoding present in a decoder must necessarily be present in the corresponding encoder, in substantially the same functional form. A description of the encoder technique can be omitted, since it is the inverse of the decoder technique described generically. Only in certain areas is a more detailed description necessary, which is given below.

[0044] As part of its operation, the source coder 403 may perform motion-compensated predictive coding, which predictively codes an input frame with reference to one or more previously coded frames from the video sequence, designated as “reference frames.” In this manner, the coding engine 407 codes differences between pixel blocks of the input frame and pixel blocks of reference frames that may be selected as predictive references for the input frame.

[0045] The local video decoder 406 may decode the encoded video data of the frames that may be designated as reference frames based on the symbols created by the source coder 403. The operation of the encoding engine 407 may advantageously be a lossy process. When the encoded video data may be decoded in a video decoder (not shown in FIG. 4), the reconstructed video sequence may be a copy of the source video sequence, usually with some errors. The local video decoder 406 may reproduce the decoding process that may be performed by the video decoder on the reference frames, and cause the reconstructed reference frames to be stored in the reference picture memory 405, which may be, for example, a cache. In this way, the encoder 400 may locally store copies of the reconstructed reference frames that have common content with the reconstructed reference frames obtained by the far-end video decoder (without transmission errors).

[0046] The predictor 404 may perform a prediction search for the coding engine 407. That is, for a new frame to be coded, the predictor 404 may search the reference picture memory 405 for sample data (as candidate reference pixel blocks) or specific metadata, such as reference picture motion vectors, block shapes, etc., that may serve as suitable prediction references for the new picture. The predictor 404 may operate on pixel block by pixel block sample blocks to find suitable prediction references. In some cases, as determined by the search results obtained by the predictor 404, the input picture may have prediction references drawn from multiple reference pictures stored in the reference picture memory 405.

[0047] The controller 402 may manage the encoding operations of the video coder 403, including, for example, setting parameters and subgroup parameters used to encode the video data.

[0048] The output of all the aforementioned functional units may undergo entropy coding in an entropy coder 408. The entropy coder converts the symbols produced by the various functional units into an encoded video sequence by losslessly compressing the symbols according to techniques known to those skilled in the art, such as, for example, Huffman coding, variable length coding, arithmetic coding, etc.

[0049] The transmitter 409 may buffer the encoded video sequence created by the entropy coder 408 and prepare it for transmission over a communication channel 411, which may be a hardware / software link to a storage device that stores the encoded video data. The transmitter 409 may merge the encoded video data from the video coder 403 with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (sources not shown).

[0050] The controller 402 may manage the operation of the encoder 400. During encoding, the controller 405 may assign a particular encoding picture type to each encoded picture, which may affect the encoding technique that may be applied to the respective picture. For example, pictures may often be assigned as one of the following frame types:

[0051] An intra picture (I-picture) may be a picture that can be coded and decoded without using any other frame in a sequence as a source of prediction. Some video codecs allow various types of intra pictures, including, for example, independent decoder refresh pictures. Those skilled in the art are aware of these variations of I-pictures and their respective uses and characteristics.

[0052] A predictive picture (P picture) may be a picture that can be encoded and decoded using intra- or inter-prediction, which uses at most one motion vector and reference index to predict sample values ​​for each block.

[0053] A bidirectionally predicted picture (B-picture) may be a picture that can be coded and decoded using intra- or inter-prediction, which uses up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, a multi-prediction picture may use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0054] A source picture may generally be spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples, respectively) and coded block by block. A block may be predictively coded with reference to other (already coded) blocks, as determined by the coding assignment applied to the respective picture of the block. For example, a block of an I-picture may be non-predictively coded or may be predictively coded with reference to already coded blocks of the same picture (spatial or intra prediction). A pixel block of a P-picture may be non-predictively coded via spatial prediction or via temporal prediction with reference to one previously coded reference picture. A block of a B-picture may be non-predictively coded via spatial prediction or via temporal prediction with reference to one or two previously coded reference pictures.

[0055] Video coder 400 may perform encoding operations in accordance with a given video encoding technique or standard, such as ITU-T Rec. H.265. In its operations, video coder 400 may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the encoded video data may conform to a syntax specified by the video encoding technique or standard being used.

[0056] In one embodiment, the transmitter 409 may transmit additional data along with the encoded video. The source coder 403 may include such data as part of the encoded video sequence. The additional data may include other forms of redundant data such as temporal / spatial / SNR enhancement layers, redundant pictures and slices, Supplemental Enhancement Information (SEI) messages, Visual Usability Information (VUI) parameter set fragments, etc.

[0057] FIG. 5 is an example end-to-end architecture 500 for a standalone AR (STAR) device according to an example embodiment, showing a 5G STAR user equipment (UE) receiver 600, a network / cloud 501, and a 5G UE (transmitter) 700. FIG. 6 is a further detailed example 600 of one or more configurations for a STAR UE receiver 600 according to an example embodiment, and FIG. 7 is a further detailed example 700 of one or more configurations for a 5G UE transmitter 700 according to an example embodiment. 3GPP TR 26.998 specifies support for glasses-based augmented reality / mixed reality (AR / MR) devices in 5G networks. Also, according to example embodiments herein, at least two classes of devices are considered: 1) devices that can fully decode and play complex AR / MR content (standalone AR, i.e., STAR), and 2) devices that have smaller computational resources and / or smaller physical size (hence battery) and can only run such applications when the majority of the computations are performed on the 5G edge server, network, or cloud rather than on the device (edge-dependent AR, i.e., EDGAR).

[0058] Also, according to exemplary embodiments, as described below, a shared conversation use case can be experienced where all participants in a shared AR conversation experience have an AR device, each participant sees the other participants in the AR scene, the participants are overlays in the local physical scene, and the placement of participants in the scene is consistent at all receiving devices, e.g., people in each local space have the same positions / seat arrangement relative to each other, and such a virtual space creates the sense of being in the same space, but the room is different for each participant because the room is an actual room or space in which each person is physically located.

[0059] For example, according to the exemplary embodiment illustrated with respect to FIGS. 5-7, an immersive media processing function on the network / cloud 501 receives uplink streams from various devices and composes a scene description that defines the placement of individual participants in a single virtual conference room. The scene description and encoded media streams are distributed to each receiving participant. The receiving participant's 5G STAR UE 600 receives, decodes, processes the 3D video and audio streams and renders them using the received scene description and information received from its AR runtime to create an AR scene of the virtual conference room with all other participants. The room for the participant is based on his / her physical space, but the seating / positioning of all other participants in the room matches the virtual room of all other participants in the session.

[0060] See also FIG. 8, which illustrates an example 800 of an EDGAR device architecture, according to an exemplary embodiment, where a device such as a 5G EDGAR UE 900 itself cannot perform a large amount of processing. Therefore, scene analysis and media analysis on received content is performed in the cloud / edge 801, and then a simplified AR scene with a small number of media components is delivered to the device for processing and rendering. FIG. 9 illustrates a more detailed example of a 5G EDGAR UE 900 according to an exemplary embodiment.

[0061] 10 illustrates an example 1000 in which users A10, B11, and T12 are to participate in an AR conference room, where one or more of the users may not have an R device. As shown, user A10 is in an office 1001, sitting in a conference room with a variable number of chairs, with user A10 sitting in a chair. User B11 is in his living room 1002, sitting on a two-seater couch, and there are also one or more two-seater couches in his living room, as well as other furniture such as chairs and tables. User T12 is on a bench in an airport lounge 1003, with another bench across a coffee table among one or more other coffee tables.

[0062] See the AR environment in office 1001 where user A10's AR shows user A10 virtual user B11v1 corresponding to user B11 and virtual user T12v1 corresponding to user T12, where virtual user B11v1 and virtual user T12v1 are shown to user A10 as sitting on furniture, office chairs, in office 1001, just like user A10. See also living room 1202 in example 1200 where the AR for user B11 shows virtual user T12v2 corresponding to user T12, but sitting on a couch in living room 1202, and virtual user A10v1 corresponding to user A10, also sitting on furniture in living room 1202, rather than in an office chair in office 1201. See also airport lobby 1203, where user T12's AR shows virtual user A10v2, corresponding to user A10, sitting at a table in airport lounge 1203, and virtual user B11v2, also sitting at a table opposite virtual user A10v2. Also, in each of these offices 1201, living room 1202, and airport lounge 1203, the updated scene description of each room matches the other rooms in terms of location / seating arrangement. For example, user A10 is shown, room by room, as counterclockwise relative to user T11, or his virtual representation, who is also clockwise relative to user T12, or his virtual representation.

[0063] However, AR technology has been limited in any attempt to incorporate the creation and use of virtual spaces for devices that do not support AR but can analyze VR or 2D video, and the embodiments herein provide improved technical procedures for creating virtual scenes that match the AR scenes when such devices participate in a shared AR conversation service.

[0064] 11 illustrates an example end-to-end architecture 1100 having a non-AR device 1101 and a cloud / edge 1102 according to an exemplary embodiment. FIG. 12 also illustrates a more detailed block diagram example of the non-AR device 1101.

[0065] As shown in Figures 11 and 12, the non-AR UE 1101 is a device that can render 360 or 2D video but does not have AR capabilities. However, the edge capabilities on the cloud / edge 1102 can AR render the received scene, rendered scene, and immersive visual and audio objects in a virtual room selected from a library. The entire video is then encoded and delivered to the device 1101 for decoding and rendering.

[0066] Thus, there may be multi-view functionality, such as AR processing on the edge / cloud 1102 generating multiple videos of the same virtual room from different angles, with different viewports, etc. Device 1101 can then receive one or more of these videos and switch between them as needed, or send commands to the edge / cloud processing to stream only the desired viewport / angle.

[0067] Also, the background feature can be changed, where the user on the device 1101 can select a desired room background from a provided library, for example, one of the different conference rooms, or even a living room and layout, and the cloud / edge 1102 will use the selected background and create the virtual room accordingly.

[0068] 13 shows an example timing diagram 1300 of an example call flow for an immersive AR conversation for a receiving non-AR UE 1101. For illustrative purposes, only one transmitter is shown in this figure without showing a detailed call flow.

[0069] Shown are an AR application module 21, a media playback module 22, and a media access function module 23, which can be considered modules of a receiving non-AR UE 1101. Also shown is a cloud / edge partitioned rendering module 24. Also shown are a media delivery module 25 and a scene graph composer module 26 of each of the network clouds 1102. Also shown is a 5G transmitting UE module 700.

[0070] S1 to S6 can be considered as session establishment phases. The AR application module 21 can request to initiate a session to the media access function module 23 in S1, and the media access function module 23 can request to initiate a session to the cloud / edge split rendering module 24 in S2.

[0071] The cloud / edge partitioned rendering module 24 may perform session negotiation with the scene graph composer module 26 in S3, which may negotiate accordingly with the 5G sender UE 700. If successful, the cloud / edge partitioned rendering module may send an acknowledgement to the media access function module 23, which may send an acknowledgement to the AR application module 21 in S5.

[0072] Thereafter, S7 can be considered to be a media pipeline configuration stage where the media access function module 23 and the cloud / edge partitioned rendering module 24 configure their respective pipelines, and after that pipeline configuration, sessions can be initiated by signals from the AR application module to the media player module 22 at S8, from the media player module 22 to the media access function module 23 at S9, and from the media access function module 23 to the cloud / edge partitioned rendering module 24 at S10.

[0073] There may then be a pose loop stage from S11 to S13, where in S11 pose data may be provided from the media player module 22 to the AR application module 21, and in S12 the AR application module may provide the pose data 12 to the media access function module 23, which may then provide the pose data to the cloud / edge split rendering module 24.

[0074] S14-S16 can be considered as a shared experience stream stage in which the 5G sender UE 700 can provide a media stream to the media delivery module 25 at S14 and provide AR data to the scene graph compositor module 26 at S15. The scene graph compositor module 25 can then compose one or more scenes based on the received AR data and provide scenes and scene updates to the enablement / edge partitioned rendering module 24 at S16, and the media delivery module 25 can provide a media stream to the cloud / edge partitioned rendering module at S17. This can include obtaining an AR scene descriptor from a non-AR device that does not render an AR scene, and generating a virtual scene by the cloud device by parsing and rendering the scene description obtained from the non-AR device according to an example embodiment.

[0075] S18-S19 can be considered a media uplink stage in which the media player module 22 captures and processes media data from its local user, and provides the media data, at S18, to the media access function module 23. The media access module 23 can then encode the media and provide the media stream, at S19, to the cloud / edge partitioned rendering module 24.

[0076] Between S19 and S20 can be considered a media downlink phase in which the cloud / edge split rendering module 24 can perform scene analysis and full AR rendering, after which S20 and S21 can be considered to constitute a media stream loop phase. In S20, the cloud / edge split rendering module 24 can provide a media stream to the media access function module 23, which can then decode the media and provide a media rendering to the media player 22 in S21.

[0077] With these features according to the example embodiment, the non-AR UE 1101 can utilize its display capable of rendering VR or 2D video even if it does not have a see-through display and therefore cannot create an AR scene. Thus, its immersive media processing capability generates only a common scene description that describes the scene and the relative position of each participant to the other participants. The scene itself needs to be adjusted with the pose information of each device before being rendered as an AR scene as described above. Also, the AR rendering process on the edge or cloud can analyze the AR scene and create a simplified VR-2D scene.

[0078] According to an example embodiment, the present disclosure uses a similar split rendering process of the EDGAR device for non-AR devices such as VR or 2d video devices, where the edge / cloud AR rendering process does not generate an AR scene. Instead, it generates a virtual scene by parsing and rendering a scene description received from an immersive media processing capability for a given background (such as a conference room), and then renders each participant in the location described by the scene description of the conference room.

[0079] Additionally, the resulting video can be 360 ​​video or 2D video depending on the capabilities of the receiving non-AR device, and the resulting video is generated taking into account the location information received from the non-AR device according to an exemplary embodiment.

[0080] Additionally, each other participant with a non-AR device would be added as a 2D video overlay to the conference room's 360 / 2D video, as shown in FIG. 10, and the room could have dedicated areas where these overlays are used, such as furniture with virtual images overlaid on them, as shown in FIG. 10.

[0081] Additionally, audio signals from all participants can be mixed as needed to create a single channel audio carrying the audio in the room, video can be encoded as a single 360 ​​or 2D video and delivered to the device, and possibly multiple video (multi-view) sources can be created, each of which captures the same virtual conference room from different views and provides those views to the device in accordance with an exemplary embodiment.

[0082] Additionally, the non-AR UE device 1101 can receive and render the 360 ​​video and / or one or more selected multi-view videos along with audio on the device display, and the user can change the viewport of the 360 ​​video by switching between different views or moving or rotating the view device, thus navigating within the virtual room while watching the video.

[0083] The above-described embodiments provide for such 5G Media Stream Architecture (5GMS) extensions for use with edge servers in their architecture, but while the specifications may have many features that are technically impossible to deploy as a set of software development kits (SDKs) on a device or as a set of microservices on the cloud, and such technical deficiencies are addressed by the embodiments described further below.

[0084] For example, the current Media Services Enabler technical report does not define a framework that ties the specification to an SDK and does not include the concept of microservices.

[0085] Please refer to the example 1400 of FIG. 14 illustrating a 5G media streaming architecture with edge extensions according to an exemplary embodiment. As shown, there is a user equipment (UE) 1401 and a data network (DN) 1411. The UE 1401 may include a 5GMS client 1403 and a 5GMS-enabled application 1405, such as, but not limited to, the AR or non-AR embodiments described above. The 5GMS client 1403 may also include a media stream handler 1403 and a media session handler. The DN 1411 may include a 5GMS application server (AS) 1412, a 5GMS application function (AF) 1414, and a 5GMS application provider 1413. The 5GMS AF 1414 may also communicate with a network exposure function (NEF) 1415 and a policy and charging function (PCF) 1416. The improvements described herein may be understood in the context of at least one of the UE 1401, the 5GMS aware application 1405, the 5GMS application provider 1413, the NEF 1415, and the PCF 1416. That is, rather than having a monolithic specification where there is no Media Services Enabler (MSE) definition for each function or group of functions, one or more of those elements may generate its own specification and, upon matching requirements, provide such specification to another of those elements which may further configure its initial element specification according to various possibilities as further described below, such processing may be via the multiple exposed application programming interfaces (APIs) 1420 shown as well as any one or more of interfaces M1, M2, M4, M5, M6, M7, M8, N33, and N5.

[0086] 15 shows an example 1500 of a two-part MSE framework of MSE specification 1501 and MSE implementation 1502, including MSE SDK abstraction 1510, and MSE SDK (platform dependent) instantiation 1520, and MSE microservices 1530. MSE SDK abstraction 1510 can have a configuration API abstraction 1513, a control interface 1512, and a media interface 1511, which can inform the configuration API 1523 of MSE SDK (platform dependent) instantiation 1520 and its control interface 1522 and media interface 1521, respectively. MSE microservices 1530 can also have a configuration API 1533, a control interface 1532, and a media interface 1531.

[0087] According to an example embodiment, MSE specification 1501, herein the "MSE specification," defines (a) media aspects and (b) MSE configuration, such definitions and use providing the technical advantages described herein. (a) Media Aspects may refer to any of: (i) a functional description of the MSE including required and optional features; (ii) control interfaces such as provisioning, authentication, and other functions used by applications that interact with this MSE; (iii) media interfaces including all input and output formats and protocols; (iv) network interfaces including system and wireless networks; (v) events, notifications, reporting, and monitoring; and (vi) error handling. (b) MSE Configuration. As used herein, “MSE Configuration” refers to any of: (i) an MSE Description Document (MDD), which describes (1) the features and their configuration parameters supported by an MSE implementation, and (2) possibly performance / cost metric criteria for various features / options; (ii) an MSE Configuration API (MCA) abstraction, which provides information for (1) retrieving the description document, (2) configuring an MSE instantiation, and (3) retrieving the state and status of an MSE instantiation; and (iii) a service API for that MSE Configuration API (MCA) abstraction, such information being referenced herein.The MSE implementation 1502 includes (a) an MSE SDK abstraction 1510, which represents (i) media aspects that conform to the media aspects of the MSE specification 1501, and (ii) an MSE description document (MDD) and an MSE configuration API (MCA) abstraction of the above MSE configuration of the MSE specification 1501; (b) an MSE SDK (platform dependent) instantiation 1520, which represents an SDK implementation in a particular environment and conforms to: (i) media aspects that conform to the media aspects of the MSE specification 1501, and (ii) an MSE description document (MDD) and an MSE configuration API (MCA) abstraction of the above MSE configuration of the MSE specification 1501; The SDK abstraction 1530 may include either (i) a media aspect conforming to the media aspect of the MSE specification 1501, an MSE configuration API (MCA) abstraction for an MSE description document (MDD) of the above-mentioned MSE configuration of the MSE specification 1501, or an MSE microservice 1510 that indicates a service API for the MSE configuration API (MCA) abstraction. Determining compliance may be done, for example, by matching a format or syntax or by a flag indicating such format or syntax.

[0088] For example, according to such advantageous MSE specification, which is not found in the 3GPP SA4 specification, for any SDK or microservice conforming to the MSE specification, a description of the features and their configuration parameters can be obtained by an external function or service according to an example embodiment, which can set a specific configuration for running that SDK, and can obtain the state and status of the running SDK at any time. The state and status can indicate available, running, busy, idle, offline, etc.

[0089] In that regard, Figure 16 illustrates an implementation 1600 of an example embodiment in which there is a 5G media streaming downlink (5GMSd) aware application 1405 communicating with a 5GMSd client 1402' that has therein a media session handler 1404 that communicates with a 5GMSd AF 1414'. Method 1601, notification and error 1602, and status 1603 information can be communicated between the 5GMSd aware application 1405' and the 5GMSd client 1402', and status 1604, subscription and notification 1605, and settings and configuration information 1606 can be communicated between the media session handler 1404, which itself can communicate to the 5GMSd AF 1414' via link M5d. According to an example embodiment, the MSE specification for the example media session handler (MSH) 1600 shown in FIG. 16 describes (a) media aspects ((i) functional description (among (1) service access information, (2) consumption reporting, (3) metrics reporting, (4) dynamic policy, and (5) network assistant information), and (ii) M5d, M6d, and M7d information), and (b) (i) MDD (which describes (1) an identifier indicating that this MSE complies with the media aspects, (2) optional features of the functional description, and M5d, M6d, and M7d with their configuration parameters, (3) possibly performance / cost metric criteria of different features / options), (ii) API abstraction (which indicates information for (1) obtaining the description document MDD, (2) configuring the MSE instantiation, and (3) obtaining the state and status of the MSE instantiation), and (iii) MSE configuration information such as service APIs for the API abstraction. For example, according to an exemplary embodiment, an MSE SDK implementation of the above MSE specification, which may be present in Android, supports the following: (i) media aspects that conform to the above media aspects of the MSE specification for MSE, and (ii) a specific implementation of the above MDD of the MSE configuration of the MSE specification, and API abstractions of the MSE configuration of the MSE specification for MSH.

[0090] According to an exemplary embodiment, see flow diagram 1700 in FIG. 17. Here, the MDD describes the functions implemented by the MSE SDK. The API of the MSE configuration above can enable an external Android process to obtain this document in S1701 and configure the SDK with a set of configurable parameters described in the MDD in S1713 as well as the state and status of the running SDK, which can be determined in S1714. The language and syntax of the MDD and the general framework of the MCA can be defined uniformly for all SA4 media service enabler specifications in S1701, and only the specific code points of each specification are defined in that specification. An external function or application that understands the MDD language and syntax, as well as an external function or application that supports the MCA, can retrieve information from the MSE implementation in S1701, and if it recognizes the MSE specification identifier, the external function or application understanding can parse and process the MDD and its configuration parameters in S1702. For example, an MSE specification and its identifier may be generated in S1701, then after a request for such specification is made, a compliance check can determine whether the desired identifier is provided, after which the MSE can be deployed according to the parameters generated in S1703 depending on various capabilities and cost indexing in S1704 to determine whether to deploy an SDK in S1705 to the device in S1706 or to the server in S1708, where the service APIs of the MDD and MCA can be set as above, or whether to proceed with the MDD and MCA in general, or the MDD and a specific MCA in S1710. Nevertheless, such an SDK can be configured by an external device or function in S1713, which can request a status S1714 at any time according to the implementation configured in S1715.

[0091] An example of an MDD can be found in ISO / IEC 23090-8. A capability description document is a JSON document that describes the functionality and features that a capability provides as well as its configuration parameters.

[0092] Thus, the exemplary embodiment provides a method for defining a media service enabler SDK and service, where the functionality and features of the media service enabler are described in a document that can be retrieved by an external function, application, or service, as well as configuration parameters for each feature, an external entity can set up a specific configuration of the media service enabler using the same interface used to retrieve the information, the complexity or other metrics of the various features of the media service enabler are also described in the document, and the media service enabler specification defines the format of the description document, as well as a unique identifier for that particular media service enabler.

[0093] The techniques described above may be implemented using computer readable instructions, as computer software physically stored on one or more computer readable media, or by one or more tangibly configured hardware processors. For example, Figure 18 illustrates a computer system (1800) suitable for implementing certain embodiments of the disclosed subject matter.

[0094] Computer software can be encoded using any suitable machine code or computer language and can be subject to assembly, compilation, linking, or similar mechanisms to create code containing instructions that can be executed by a computer central processing unit (CPU), graphics processing unit (GPU), or the like, directly, or through interpretation, microcode execution, or the like.

[0095] The instructions may be executed on various types of computers or components thereof including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, Internet of Things devices, and the like.

[0096] 18 with respect to computer system 1800 are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing the embodiments of the present disclosure, nor should the arrangement of components be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary embodiment of computer system 1800.

[0097] The computer system 1800 may include certain human interface input devices. Such human interface input devices may be responsive to input by one or more human users, for example, via tactile input (keystrokes, swipes, data glove movements, etc.), audio input (voice, clapping, etc.), visual input (gestures, etc.), olfactory input (not shown). The human interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (speech, music, ambient sounds, etc.), images (scanned images, photographic images obtained from a still image camera, etc.), video (two-dimensional video, three-dimensional video including stereoscopic video, etc.).

[0098] The input human interface devices may include one or more of a keyboard 1801, a mouse 1802, a trackpad 1803, a touch screen 1810, a joystick 1805, a microphone 1806, a scanner 1808, and a camera 1807 (only one of each is shown).

[0099] The computer system 1800 may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the human user's senses, for example, through haptic output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via touch screen 1810 and joystick 1805, although there may also be haptic feedback devices that do not function as input devices), audio output devices (e.g., speakers 1809, headphones (not shown)), visual output devices (e.g., screens 1810 including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capability, each with or without haptic feedback capability, some of which may be capable of outputting two-dimensional visual output or output in excess of three dimensions via means such as stereoscopic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0100] The computer system 1800 may also include human accessible storage devices and their associated media, such as optical media including a CD / DVD ROM / RW 1820 having media such as CD / DVD 1811, a thumb drive 1822, a removable hard drive or solid state drive 1823, legacy magnetic media such as tape or floppy disks (not shown), dedicated ROM / ASIC / PLD based devices such as security dongles (not shown), etc.

[0101] Also, those skilled in the art will appreciate that the term "computer-readable medium" as used in connection with the subject matter disclosed herein does not encompass transmission media, carrier waves, or other transitory signals.

[0102] The computer system 1800 may also include an interface 1899 to one or more communications networks 1898. The networks 1898 may be, for example, wireless, wired, optical. The networks 1898 may further be local, wide area, metropolitan, vehicular and industrial, real-time, delay tolerant, etc. Examples of networks 1898 include local area networks such as Ethernet, wireless LAN, etc., cellular networks including GSM, 3G, 4G, 5G, LTE, etc., TV wired or wireless wide area digital networks including cable TV, satellite TV, and terrestrial broadcast TV, vehicular and industrial including CANBus, etc. Particular networks 1898 generally require an external network interface adapter attached to a particular general-purpose data port or peripheral bus (1850 and 1851) (e.g., a USB port of the computer system 1800, etc.), while other networks are generally built into the core of the computer system 1800 by attachment to the system bus as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks 1898, the computer system 1800 can communicate with other entities. Such communications may be unidirectional receive only (e.g., broadcast TV), unidirectional transmit only (e.g., CANbus to certain CANbus devices), or bidirectional, such as to other computer systems using local or wide area digital networks. Particular protocols and protocol stacks may be used with each of those networks and network interfaces, as described above.

[0103] The aforementioned human interface devices, human accessible storage devices, and network interfaces may be attached to the core 1840 of the computer system 1800.

[0104] The core 1840 may include one or more central processing units (CPUs) 1841, graphics processing units (GPUs) 1842, graphics adapters 1817, dedicated programmable processing units in the form of field programmable gate areas (FPGAs) 1843, hardware accelerators for specific tasks 1844, and the like. These devices may be connected via a system bus 1848, along with read-only memory (ROM) 1845, random access memory 1846, and internal mass storage 1847, such as internal hard drives and SSDs, that are not accessible to the user. In some computer systems, the system bus 1848 may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, and the like. Peripherals may be attached directly to the core's system bus 1848 or via a peripheral bus 1851. Architectures for peripheral buses include PCI, USB, and the like.

[0105] The CPU 1841, GPU 1842, FPGA 1843, and accelerator 1844 may execute a number of instructions that may be constructed in combination as described above as computer code. This computer code may be stored in ROM 1845 or RAM 1846. Transient data may also be stored in RAM 1846, while persistent data may be stored, for example, in internal mass storage 1847. Cache memory, which may be closely associated with one or more of the CPU 1841, GPU 1842, mass storage device 1847, ROM 1845, RAM 1846, etc., may be used to enable fast storage and fast retrieval of any of the memory devices.

[0106] The computer-readable medium can bear computer code for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the kind well known and available to those skilled in the computer software arts.

[0107] By way of example and not limitation, the architecture corresponding to computer system 1800, and in particular core 1840, may provide functionality as a result of processors (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media may be user-accessible mass storage as previously described, as well as media associated with specific storage of core 1840 that is non-transitory in nature, such as core internal mass storage 1847 or ROM 1845. Software implementing various embodiments of the present disclosure may be stored in such devices and executed by core 1840. Computer-readable media may include one or more memory devices or chips, depending on particular needs. The software may cause core 1840, and in particular the processors therein (including CPUs, GPUs, FPGAs, etc.) to perform certain processes or certain portions of certain processes described herein, including defining data structures stored in RAM 1846 and modifying such data structures according to processes defined by the software. Additionally or alternatively, a computer system may provide functionality as a result of hardwired or otherwise embodied logic in circuitry (e.g., accelerator 1844) that can operate in place of or in conjunction with software to perform certain processes or certain portions of certain processes described herein. References to software may encompass logic, and vice versa, where appropriate. References to computer-readable media may encompass circuitry (such as integrated circuits (ICs)) that stores software for execution, circuitry that embodies logic for execution, or both, where appropriate. The present disclosure encompasses any suitable combination of hardware and software.

[0108] While this disclosure has described several exemplary embodiments, there are alterations, permutations, and various substitute equivalents that fall within the scope of this disclosure. Thus, it will be appreciated that those skilled in the art can devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within its spirit and scope. [Explanation of symbols]

[0109] 10 User A, 10v1, 10v2 Virtual user A, 11 User B, 11v1, 11v2 Virtual user B, 12 User T, 12v1, 12v2 Virtual user T, 21 AR application module, 22 Media player module, 23 Media access function module, 24 Cloud / edge split rendering module, 25 Media delivery module, 26 Scene graph composer module, 100 Communication system, 101, 102, 103, 104 Terminal, 105, 1898 Network, 201, 1807 Camera, 202, 400 Encoder, 203 Capture subsystem, 204 Encoded video bitstream, 204, 206, 208 Video bitstream, 205 Streaming server, 207, 212 Streaming client, 208 Encoded video bitstream, 209, 312 Display, 210 Outgoing video sample stream, 211, 300 Video decoder, 212 Client, 213 Uncompressed video sample stream, 301 Channel, 302 Receiver, 303 Buffer, 304 Parser, 305 Scaler / inverse transform unit, 306 Motion compensation prediction unit, 307 Intra picture prediction unit, 308, 405, 557 Reference picture memory, 309 Current reference picture, 310 Aggregator, 311 Loop filter unit, 313 Symbol, 400 Video encoder, 401 Video source, 402 Controller, 403 Source coder, 404 Predictor, 406 Local video decoder, 407 Coding engine, 408 Entropy coder, 409 Transmitter, 410 Encoded video sequence, 411 Communication Channel, 500, 800, 1000, 1100, 1200, 1400, 1500, 1600 Example, 501 Network / Cloud, 600 User Equipment Receiver, 700 5G UE (Sender), 801, 1102 Cloud / Edge, 900 5G EDGAR UE, 1001, 1201 Office, 1002, 1202 Living Room, 1003, 1203 Airport Lounge, 1101 Non-AR Device, 1101 Non-AR UE, 1300 Exemplary Timing Diagram, 1401 User Equipment, 1402 5GMS Client, 1402'5GMSd Client, 1403 Media Stream Handler, 1404 Media Session Handler, 1405 5GMS Aware Application, 1405' 5GMSd Aware Application, 1411 Data Network, 1412 5GMS Application Server, 1413 5GMS Application Provider, 1414 5GMS Application Function, 1414' 5GMSd AF, 1415 Network Exposed Function, 1416 Policy and Charging Function, 1420 Multiple Exposed Application Programming Interface, 1501 MSE Specification, 1502 MSE Implementation, 1510 MSE SDK Abstraction, 1511, 1521, 1531 Media Interface, 1512, 1522, 1532 Control Interface, 1513 Configuration API Abstraction, 1520 MSE SDK (platform dependent) instantiation, 1523, 1533 Configuration API, 1530 MSE Microservices, 1601 Methods, 1602 notifications and errors, 1603, 1604 status, 1605 subscriptions and notifications, 1606 settings and configuration information, 1700 flowcharts, 1800 computer system, 1801 keyboard, 1802 mouse, 1803 trackpad, 1805 joystick, 1806 microphone, 1808 scanner, 1809 speaker, 1810 touch screen, 1811 CD / DVD, 1817 graphics adapter, 1820 ROM / RW, 1822 thumb drive, 1823 solid state drive, 1840 core, 1841 central processing unit, 1842 graphics processing unit, 1843 field programmable gate area, 1844 accelerator, 1845 read only memory, 1846 random access memory, 1847 internal mass storage, 1848 system bus, 1850 General purpose data port, 1851 peripheral bus, 1899 interface

Claims

1. 1. A method for media streaming, the method comprising: generating a Media Service Enabler (MSE) configuration for a 5G Media Streaming (5GMS) service MSE specification; determining whether at least one of a software development kit (SDK) and a microservice complies with the MSE specification; configuring the at least one of the SDK and the microservices according to the MSE specification, by at least one of functions and services based on determining that the at least one of the SDK and the microservices conforms to the MSE specification; controlling implementation of a 5G media streaming (5GMS) service according to the configuration of the at least one of the SDK and the microservices; Including, At least one of the functions and the services is external to the implementation of the 5GMS service; method.

2. The implementation of the 5GMS service includes an implementation of at least one of MSE SDK abstraction, MSE SDK instantiation, and MSE microservices; The method of claim 1.

3. the MSE SDK abstraction describes a media aspect, an MSE configuration including an MSE description document (MDD), and an MSE configuration application programming interface (API) (MCA); the MSE SDK instantiation indicates a particular implementation of the media aspects, the MSE configuration including the MDD, and the MCA; the MSE microservice represents a service API for the particular implementation of the media aspects, the MSE configuration including the MDD, and the MCA; The method of claim 2.

4. The media aspect indicates any one of service access information, consumption reporting, metric reporting, dynamic policy, and network assistance. The method of claim 3.

5. the MDD indicating a performance metric and whether the MSE complies with the media aspect; The method of claim 4.

6. and in the step of determining that the at least one of the SDK and the microservice conforms to the MSE specification, the at least one of the function and the service sets the configuration of the at least one of the SDK and the microservice through the same interface through which the at least one of the function and the service received information. The method of claim 1.

7. determining that the at least one of the SDK and the microservice conforms to the MSE specification includes determining that at least a format of an MSE description document (MDD) conforms to a format of the at least one of the SDK and the microservice; The method of claim 6.

8. The step of generating the MSE configuration of the MSE specification of the 5G media streaming (5GMS) service includes setting an identifier of an MSE. The method of claim 6.

9. the information received by the at least one of the function and the service includes the identifier of the MSE; The method of claim 8.

10. The step of controlling the implementation of the 5GMS service according to the configuration of the at least one of the SDK and the microservice includes a step of indicating to the at least one of the functions and services that are external to the implementation of the 5GMS service that the at least one of the functions and services may request a status of the 5GMS service from time to time. The method of claim 1.

11. 1. An apparatus for media streaming, comprising: at least one memory configured to store computer program code; at least one processor configured to access said computer program code and to act as instructed by said computer program code, said computer program code comprising: generation code configured to cause the at least one processor to generate a media services enabler (MSE) configuration for an MSE specification for 5G media streaming (5GMS) services; determining code configured to cause the at least one processor to determine whether at least one of a software development kit (SDK) and a microservice complies with the MSE specification; configuration code configured to cause the at least one processor to configure a configuration of the at least one of the SDK and the microservices according to the MSE specification and based on determining that the at least one of the SDK and the microservices conforms to the MSE specification by at least one of a function and a service; control code configured to cause the at least one processor to control implementation of a 5G media streaming (5GMS) service in accordance with the configuration of the at least one of the SDK and the microservices; At least one processor, Equipped with At least one of the functions and the services is external to the implementation of the 5GMS service; Device.

12. The implementation of the 5GMS service includes an implementation of at least one of MSE SDK abstraction, MSE SDK instantiation, and MSE microservices; 12. The apparatus of claim 11.

13. the MSE SDK abstraction describes a media aspect, an MSE configuration including an MSE description document (MDD), and an MSE configuration application programming interface (API) (MCA); the MSE SDK instantiation indicates a particular implementation of the media aspects, the MSE configuration including the MDD, and the MCA; the MSE microservice represents a service API for the particular implementation of the media aspects, the MSE configuration including the MDD, and the MCA; 13. The apparatus of claim 12.

14. The media aspect indicates any one of service access information, consumption reporting, metric reporting, dynamic policy, and network assistance.

14. The apparatus of claim 13.

15. the MDD indicating a performance metric and whether the MSE complies with the media aspect; 15. The apparatus of claim 14.

16. setting the configuration of the at least one of the SDK and the microservices through the same interface through which the at least one of the functions and services received information in determining that the at least one of the SDK and the microservices conforms to the MSE specification; 12. The apparatus of claim 11.

17. determining that the at least one of the SDK and the microservice conforms to the MSE specification includes determining that at least a format of an MSE description document (MDD) conforms to a format of the at least one of the SDK and the microservice; The method of claim 16.

18. generating the MSE configuration of the MSE specification of the 5G media streaming (5GMS) service includes setting an identifier of an MSE; The method of claim 16.

19. the information received by the at least one of the function and the service includes the identifier of the MSE; 20. The apparatus of claim 18.

20. A non-transitory computer-readable medium storing a program for causing a computer to execute a process, the process comprising: generating a Media Service Enabler (MSE) configuration for a 5G Media Streaming (5GMS) service MSE specification; determining whether at least one of a software development kit (SDK) and a microservice complies with the MSE specification; setting, by at least one of features and services, a configuration of the at least one of the SDK and the microservices in accordance with the MSE specification and based on determining that the at least one of the SDK and the microservices conforms to the MSE specification; controlling implementation of a 5G media streaming (5GMS) service according to the configuration of the at least one of the SDK and the microservices; Including, At least one of the functions and the services is external to the implementation of the 5GMS service; Non-transitory computer-readable medium.

Citation Information

Patent Citations

  • Streaming media network system, streaming media service realization method and streaming media service enabler

    US20080307108A1

  • Service enabler function

    US20170034015A1

  • Media streaming with edge computing

    US20220240061A1

  • A system, a device, and methods of streaming an event

    WO2022077066A1