Media access function / player initialization method for 5g media streaming unique session management

Abstract

A method and apparatus are described that include computer code configured to cause at least one processor to create a media player instance to deliver at least a portion of video content to a UE including a 5GMSd-aware application and a 5GMSd client, determine whether to provide either of a media transport session ID and an external service ID to a 5GMS MAF upon creating the media player instance, control the MSH to provide the media transport session ID to the 5GMS MAF, and stream the content to the UE based on the media transport session ID.

Description

[0001] Related applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 646,684, filed May 13, 2024, and U.S. Application No. 19 / 205,110, filed May 12, 2025, the entire disclosure of which is expressly incorporated herein by reference. Technical Field

[0003] This invention provides a novel method for initializing a media player on a 5G device, which ensures the creation of a unique media delivery session. Background Technology

[0004] The M7 interface can be used for 5G Media Streaming (5GMS) applications or for interaction between a Media Session Handler (MSH) and a 5GMS media stream processor. However, the defined functionality of the M7 interface is very limited, and it does not define how to manage media transport sessions. The current 5GMS architecture only defines a general architecture for uplink and downlink media streams. Furthermore, while a simple M7 / M6 interface is typically available for interacting with the MSH, its defined functionality is very simple and limited. These descriptions should not be considered as acknowledged prior art, but only as background to the disclosures below.

[0005] Therefore, for any of the above reasons, there is an urgent need for technical solutions to these problems in video codec technology. This document focuses on initializing the media player on 5G devices to ensure the creation of unique media transmission sessions. Summary of the Invention

[0006] The present invention includes a method and apparatus comprising: a memory configured to store computer program code; and at least one processor configured to access the computer program code and operate according to the instructions of the computer program code. The computer program is configured to cause the processor to: create a media player instance of a 5GMS Media Access Function (MAF), the media player instance being configured to deliver at least a portion of video content to a user equipment (UE) including a 5GMSd sensing application and a 5GMSd client; upon creating the media player instance, determining whether to provide either a media transport session identifier (ID) or an external service ID to the 5GMS MAF; based on the determination that neither the media transport session ID nor the external service ID is provided to the 5GMS MAF, controlling the MSH to provide the media transport session ID to the 5GMS MAF; and performing a conversion between a visual media file of the video content and a bitstream of visual media data of the video content according to format rules, and based on the conversion, performing at least one of streaming the video content to the UE and playing the video content by the UE.

[0007] A video decoding method is provided, executed by at least one processor, the method comprising: creating a media player instance of a 5G media stream 5GMS Media Access Function (MAF), the media player instance being configured to deliver at least a portion of video content to a user equipment (UE) including a 5G Media Service downlink 5GMSd awareness application and a 5GMSd client; determining, when creating the media player instance, whether to provide either a Media Transport Session Identifier (MTSI) or an External Service ID to the 5GMS MAF; based on the determination that neither the MTSID nor the External Service ID is provided to the 5GMS MAF, controlling a Media Session Processor (MSH) to provide the MTSID to the 5GMS MAF; and streaming the content to the UE based on the MTSID, and decoding the content at the UE.

[0008] A video encoding method is provided, executed by at least one processor, the method comprising: creating a media player instance of a 5G media stream 5GMS Media Access Function (MAF), the media player instance being configured to deliver at least a portion of video content to a user equipment (UE) including a 5G Media Service downlink 5GMSd awareness application and a 5GMSd client; determining, when creating the media player instance, whether to provide either a Media Transport Session Identifier (MTSI) or an External Service ID to the 5GMS MAF; based on the determination that neither the MTSID nor the External Service ID is provided to the 5GMS MAF, controlling a Media Session Processor (MSH) to provide the MTSID to the 5GMS MAF; and encoding the video content and streaming the video content to the UE based on the MTSID.

[0009] A method for processing visual media data is provided, executed by at least one processor, the method comprising: creating a media player instance of a 5G media stream 5GMS Media Access Function (MAF), the media player instance being configured to deliver at least a portion of video content to a user equipment (UE) including a 5G Media Service downlink 5GMSd sensing application and a 5GMSd client; determining, when creating the media player instance, whether to provide either a Media Transport Session Identifier (MTSI) or an External Service ID to the 5GMS MAF; based on the determination that neither the MSI nor the External Service ID is provided to the 5GMS MAF, controlling a Media Session Processor (MSH) to provide the MSH to the 5GMS MAF; and performing a conversion between a visual media file of the video content and a bitstream of the visual media data of the video content according to a formatting rule; and based on the conversion, performing at least one of the following operations: streaming the video content to the UE, or the UE playing the video content.

[0010] The process of controlling the MSH to provide the media transmission session ID to the 5GMS MAF includes: at the first moment when the MSH is controlled to provide the media transmission session ID to the 5GMS MAF, controlling the MSH to check whether the associated service access information SAI of the media player instance and the external service ID of the media player instance both do not exist.

[0011] Controlling the MSH to provide the media transmission session ID to the 5GMS MAF further includes: at the first time, determining whether the associated SAI already exists but the media transmission session ID does not yet exist; and performing any of the following operations: based on determining at the first time that either the external service ID or the media transmission session ID already exists, providing the media transmission session ID as an existing ID to the 5GMS MAF; and based on determining at the first time that the media transmission session ID does not yet exist, generating the media transmission session ID and providing the media transmission session ID to the 5GMS MAF.

[0012] Creating the media player instance includes enabling API communication via either the M7d Application Programming Interface API or the M11d API.

[0013] Creating the media player instance involves calling the initialize() function of the Dynamic Adaptive Streaming Media DASH Streaming API, which is based on the Hypertext Transfer Protocol (HTTP).

[0014] The input parameters of the initialize() function include the service ID parameter and the session ID parameter.

[0015] The sessionID parameter is at least one of the following: provided by the MSH at the M11d API, or omitted at the M6d API. Attached Figure Description

[0016] Other features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, wherein:

[0017] Figure 1 This is a simplified block diagram of a communication system according to an embodiment;

[0018] Figure 2 This is a simplified schematic diagram of the encoder and decoder environment according to an embodiment;

[0019] Figure 3 This is a simplified block diagram of the decoder according to an embodiment;

[0020] Figure 4 This is a simplified block diagram of the encoder according to an embodiment;

[0021] Figure 5 This is a simplified block diagram of an augmented reality (AR) system according to an embodiment;

[0022] Figure 6This is a simplified block diagram of a 5G system according to an embodiment;

[0023] Figure 7 This is a simplified block diagram in a 5G UE environment according to an embodiment;

[0024] Figure 8 This is a simplified block diagram in an edge-dependent AR (EDGAR) environment according to an embodiment;

[0025] Figure 9 This is a simplified block diagram of the EDGAR UE environment according to an embodiment;

[0026] Figure 10 This is a simplified diagram of AR used according to the embodiment;

[0027] Figure 11 This is a simplified block diagram of a hybrid AR and non-AR system according to an embodiment;

[0028] Figure 12 This is a simplified block diagram of a non-AR UE according to an embodiment;

[0029] Figure 13 These are simplified block diagrams and timing diagrams based on embodiments;

[0030] Figure 14 This is a simplified block diagram of the 5GMS architecture according to an embodiment;

[0031] Figure 15 This is a simplified flowchart according to an embodiment;

[0032] Figure 16 This is a schematic diagram according to an embodiment. Detailed Implementation

[0033] The proposed features discussed below can be used individually or in any combination in any order. Furthermore, embodiments can be implemented by processing circuitry (e.g., at least one processor or at least one integrated circuit (IC)). In one example, at least one processor executes a program stored in a non-volatile computer-readable storage medium.

[0034] Figure 1A simplified block diagram of a communication system 100 according to an embodiment of the present invention is shown. The communication system 100 may include at least two terminals 102 and 103 interconnected via a network 105. To achieve unidirectional data transmission, the first terminal 103 may encode video data locally for transmission to the other terminal 102 via the network 105. The second terminal 102 may receive the encoded video data from the other terminal from the network 105, decode the encoded data, and display the recovered video data. Unidirectional data transmission may be common in scenarios such as media service applications.

[0035] Figure 1 A second pair of terminals 101 and 104 is shown, configured to support bidirectional transmission of encoded video, which may occur, for example, during a video conference. To enable bidirectional data transmission, each terminal 101 and 104 can encode video data acquired at a local location for transmission to the other terminal via network 105. Each terminal 101 and 104 can also receive encoded video data transmitted by the other terminal, decode the encoded data, and display the recovered video data on a local display device.

[0036] exist Figure 1 In this invention, terminals 101, 102, 103, and 104 may be represented as servers, personal computers, and smartphones, but the principles of the invention are not limited thereto. Embodiments of the invention can be applied to laptops, tablets, media players, and / or dedicated video conferencing equipment. Network 105 refers to any number of networks, including, for example, wired and / or wireless communication networks, that transmit encoded video data between terminals 101, 102, 103, and 104. Communication network 105 may exchange data in circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks (LANs), wide area networks (WANs), and / or the Internet. For the purposes of this discussion, unless otherwise stated below, the architecture and topology of network 105 may not materially affect the operation of the invention.

[0037] Figure 2 As an application example of the disclosed subject matter, the deployment of a video encoder and decoder in a streaming media environment is shown. The disclosed subject matter is equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, storing compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0038] The streaming media system may include an acquisition subsystem 203, which may include a video source 201, such as a digital camera, which, for example, creates an uncompressed sample video stream 213. It should be emphasized that the sample stream 213 may have a higher data volume compared to the encoded video stream and may be processed by an encoder 202 coupled to the camera 201. The encoder 202 may include hardware, software, or a combination of hardware and software to enable or implement aspects of the disclosed subject matter described in more detail below. It should be emphasized that the encoded video stream 204 may have a lower data volume compared to the sample stream and may be stored on a streaming media server 205 for future use. At least one streaming media client 212 and 207 may access the streaming media server 205 to retrieve copies 208 and 206 of the encoded video stream 204. Client 212 may include video decoder 211, which decodes a copy 208 of the incoming encoded video stream and creates an output video sample stream 210, which can be rendered on display 209 or other rendering device (not depicted). In some streaming media systems, video streams 204, 206, and 208 may be encoded according to specific video encoding / compression standards. Examples of these standards have been mentioned above and will be further described in this application.

[0039] Figure 3 This can be a functional block diagram of the video decoder 300 according to an embodiment of the present invention.

[0040] Receiver 302 can receive at least one codec video sequence to be decoded by decoder 300; in the same embodiment or another embodiment, one encoded video sequence is received at a time, wherein the decoding of each encoded video sequence is independent of other encoded video sequences. The encoded video sequences can be received from channel 301, which can be a hardware / software link to a storage device storing the encoded video data. Receiver 302 can receive the encoded video data, as well as other data that can be forwarded to the corresponding user entity (not depicted), such as encoded audio data and / or auxiliary data streams. Receiver 302 can separate the encoded video sequences from other data. To combat network jitter, buffer memory 303 can be coupled between receiver 302 and entropy decoder / resolver 304 (hereinafter referred to as the "resolver"). Buffer 303 may not be necessary or may be small when receiver 302 receives data from a store / forward device with sufficient bandwidth and controllability or from an isosynchronous network. For use on best-effort packet networks such as the Internet, buffer 303 may be necessary, can be relatively large, and can advantageously have an adaptive size.

[0041] The video decoder 300 may include a parser 304 for reconstructing symbols 313 from an entropy-coded video sequence. These symbols may include information for managing the operation of the decoder 300, and information that may be used to control rendering devices, such as a display 312, which is not part of the decoder but may be coupled to it. Control information for one or more rendering devices may be in the form of Supplementary Enhancement Information (SEI) (SEI messages) or fragments of video availability information parameter sets (not depicted). The parser 304 may perform parsing / entropy decoding on the received encoded video sequence. The encoding of the encoded video sequence may be based on video coding techniques or standards and may follow principles well known to those skilled in the art, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser 304 may extract a subgroup parameter set from the encoded video sequence for at least one pixel subgroup in the video decoder based on at least one parameter corresponding to a group. Subgroups can include Group of Pictures (GOP), pictures, tiles, slices, macroblocks, coding units (CU), blocks, transform units (TU), prediction units (PU), etc. The entropy decoder / parser can also extract information such as transform coefficients, quantizer parameter values, and motion vectors from the encoded video sequence.

[0042] The parser 304 can perform entropy decoding / parsing operations on the video sequence received from the buffer 303 to create symbols 313. The parser 304 can receive encoded data and selectively decode specific symbols 313. Furthermore, the parser 304 can determine whether to provide specific symbols 313 to the motion compensation prediction unit 306, the scaler / inverse transform unit 305, the intra-frame prediction unit 307, or the loop filter 311.

[0043] Depending on the type of the encoded video image or its portions (e.g., inter-frame and intra-frame images, inter-frame and intra-frame blocks) and other factors, the reconstruction of symbol 313 may involve multiple different units. Which units are involved and how these units are involved can be controlled by subgroup control information parsed from the encoded video sequence by parser 304. For clarity, the flow of such subgroup control information between parser 304 and the multiple units described below is not depicted.

[0044] In addition to the functional blocks already mentioned, the decoder 300 can be conceptually subdivided into several functional units as described below. In practical implementations under commercial constraints, many of these units interact closely with each other and can be at least partially integrated with one another. However, for the purposes of describing the disclosed subject matter, it is appropriate to conceptually subdivide it into the functional units described below.

[0045] The first unit is the scaler / inverse transform unit 305. The scaler / inverse transform unit 305 receives from the parser 304 quantization transform coefficients as one or more symbols 313, along with control information including the transform to be used, block size, quantization factor, quantization scaling matrix, etc. This scaler / inverse transform unit 305 can output blocks containing sample values, which can be input to the aggregator 310.

[0046] In some cases, the output samples of the scaler / inverse transform unit 305 may involve intra-coded blocks; that is, the block does not use predictive information from previously reconstructed images, but instead uses predictive information from previously reconstructed portions of the current image. Such predictive information can be provided by the intra-image prediction unit 307. In some cases, the intra-image prediction unit 307 uses surrounding reconstructed information obtained from the current (partially reconstructed) image 309 to generate blocks with the same size and shape as the blocks in the reconstruction. In some cases, the aggregator 310 adds the predictive information already generated by the intra-prediction unit 307 to the output sample information provided by the scaler / inverse transform unit 305 on a per-sample basis.

[0047] In other cases, the output samples of the scaler / inverse transform unit 305 may involve inter-frame coded and possibly motion-compensated blocks. In this case, the motion-compensated prediction unit 306 can access the reference image memory 308 to obtain samples for prediction. After motion compensation of the obtained samples according to the block-related symbols 313, the aggregator 310 can add these samples to the output of the scaler / inverse transform unit (referred to in this case as residual samples or residual signals) to generate output sample information. The address at which the motion compensation unit obtains the predicted samples in the reference image memory can be controlled by motion vectors, which are provided to the motion compensation unit in the form of symbols 313, which may have, for example, X components, Y components, and reference image components. Motion compensation may also include interpolation of sample values ​​obtained from the reference image memory when using subsample precise motion vectors, motion vector prediction mechanisms, etc.

[0048] The output samples of aggregator 310 can undergo various loop filtering techniques in loop filter unit 311. Video compression techniques may include loop filtering techniques controlled by parameters included in the encoded video bitstream and used by loop filter unit 311 as symbols 313 from parser 304. However, loop filtering techniques may also respond to metadata obtained during decoding of previous (in decoding order) portions of the encoded image or encoded video sequence, as well as to previously reconstructed and loop-filtered sample values.

[0049] The output of the loop filter unit 311 can be a sample stream, which can be output to the display 312 (which can be a rendering device) and can also be stored in the reference image memory 557 for use in future inter-frame image prediction.

[0050] Some encoded images can be used as reference images for future predictions after full reconstruction. Once the encoded images are fully reconstructed and have been identified as reference images (e.g., by the parser 304), the current reference image 309 can become part of the reference image buffer 308, and the new current image memory can be reallocated before the reconstruction of subsequent encoded images begins.

[0051] The video decoder 300 can perform decoding operations according to a predetermined video compression technique, which may be documented in standards such as ITU-T Recommendation H.265. If the encoded video sequence follows the syntax of a video compression technique or standard, such as the syntax specified in a video compression technique document or standard, particularly in its configuration file, then the encoded video sequence may conform to the syntax specified by the used video compression technique or standard. For compliance, the complexity of the encoded video sequence must also be within the limits defined by the hierarchy of the video compression technique or standard. In some cases, the hierarchy limits the maximum image size, maximum frame rate, maximum reconstruction sampling rate (measured, for example, in megasamples per second), maximum reference image size, etc. In some cases, the limitations set by the hierarchy can be further limited by the Hypothetical Reference Decoder (HRD) specification and the HRD buffer management metadata indicated by signaling in the encoded video sequence.

[0052] In one embodiment, receiver 302 may receive supplemental (redundant) data along with encoded video. This supplemental data may be part of one or more encoded video sequences. The supplemental data may be used by video decoder 300 to properly decode the data and / or more accurately reconstruct the original video data. The supplemental data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant images, forward error correction codes, etc.

[0053] Figure 4 This can be a functional block diagram of a video encoder 400 according to an embodiment of the present invention.

[0054] The encoder 400 can receive video samples from a video source 401 (which is not part of the encoder), which can capture one or more video images to be encoded by the encoder 400.

[0055] Video source 401 can provide a source video sequence to be encoded by encoder 303 in the form of a digital video sample stream. This digital video sample stream can have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601 YCrCb, RGB, etc.), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media service system, video source 401 can be a storage device storing previously prepared video. In a video conferencing system, video source 401 can be a camera capturing local image information as a video sequence. Video data can be provided in the form of multiple individual images, which produce an animation effect when viewed sequentially. The images themselves can be organized as spatial pixel arrays, where each pixel can include at least one sample depending on the sampling structure, color space, etc., used. Those skilled in the art will readily understand the relationship between pixels and samples. Samples are described in detail below.

[0056] According to one embodiment, encoder 400 can encode and compress images of a source video sequence into an encoded video sequence 410 in real time or under any other time constraints required by the application. Enforcing an appropriate encoding rate is one of the functions of controller 402. The controller controls and is functionally coupled to other functional units as described below. For clarity, the coupling is not depicted. Parameters set by the controller may include rate control-related parameters (image skipping, quantizer, λ value of rate-distortion optimization techniques, etc.), image size, GOP layout, maximum motion vector search range, etc. Other functions of controller 402 can be readily identified by those skilled in the art, as these functions may be related to the video encoder 400 optimized for a particular system design.

[0057] Some video encoders operate in an “encoding loop” well known to those skilled in the art. In a very simplified description, the encoding loop may consist of the encoding portion of the encoder (e.g., source encoder 403) responsible for creating symbols based on the input image to be encoded and one or more reference images, and a (local) decoder 406 embedded in encoder 400 that reconstructs the symbols to create sample data that a (remote) decoder would also create (since in the video compression techniques considered in the disclosed subject matter, any compression between the symbols and the encoded video stream is lossless). The reconstructed sample stream is fed into reference image memory 405. Since decoding the symbol stream produces a bit-by-bit accurate result independent of the decoder’s location (local or remote), the contents of the reference image buffer are also bit-by-bit accurate between the local and remote encoders. In other words, the reference image samples “seen” by the encoder’s prediction portion are exactly the same as the sample values ​​“seen” by the decoder during prediction. This fundamental principle of reference image synchronization (and the drift that occurs, for example, when synchronization cannot be maintained due to channel errors) is well known to those skilled in the art.

[0058] The operation of the "local" decoder 406 is the same as that of the "remote" decoder 300, as described above. Figure 3 The "remote" decoder 300 is described in detail. However, a brief reference is provided. Figure 4 Since symbols are available, and the entropy encoder 408 and parser 304 can encode / decode symbols into an encoded video sequence in a lossless manner, the entropy decoding portion of decoder 300, including channel 301, receiver 302, buffer 303 and parser 304, may not be fully implemented in local decoder 406.

[0059] It can be seen from this that any decoder technique other than parsing / entropy decoding, which exists in the decoder, must also necessarily exist in the corresponding encoder in essentially the same functional form. The description of encoder techniques can be simplified, as encoder techniques are the reverse operation of a fully described decoder technique. More detailed descriptions are only required in certain areas, and these descriptions are provided below.

[0060] As part of the operation of the source encoder 403, the source encoder 403 may perform motion-compensated predictive coding on the input frame, which predictively codes the input frame with reference to at least one previously encoded frame designated as a "reference frame" in the video sequence. In this way, the encoding engine 407 encodes the differences between pixel blocks of the input frame and pixel blocks of one or more reference frames, which may be selected as one or more predictive references for the input frame.

[0061] The local video decoder 406 can decode encoded video data of frames that can be designated as reference frames based on symbols generated by the source encoder 403. Advantageously, the operation of the encoding engine 407 can be a lossy process. When the encoded video data can be decoded by the video decoder ( Figure 4 During decoding (not shown), the reconstructed video sequence can typically be a copy of the source video sequence with some errors. The local video decoder 406 replicates the decoding process performed by the video decoder on the reference frame, and allows the reconstructed reference frame to be stored in a reference image memory 405, which can be, for example, a cache. In this way, the encoder 400 can store copies of the reconstructed reference frames locally, these copies having the same content (no transmission errors) as the reconstructed reference frames obtained by the remote video decoder.

[0062] Predictor 404 can perform a prediction search against encoding engine 407. That is, for a new frame to be encoded, predictor 404 can search the reference image memory 405 for sample data (as candidate reference pixel blocks) or certain metadata, such as reference image motion vectors, block shapes, etc., that can be used as suitable prediction references for the new image. Predictor 404 can operate pixel-by-pixel based on sample blocks to find appropriate prediction references. In some cases, as determined by the search results obtained by predictor 404, the input image may have prediction references obtained from multiple reference images stored in reference image memory 405.

[0063] The controller 402 can manage the encoding operations of the video encoder 403, including, for example, setting parameters and subgroup parameters for encoding video data.

[0064] The outputs of all the aforementioned functional units can be entropy encoded in the entropy encoder 408. The entropy encoder converts the symbols generated by each functional unit into an encoded video sequence by lossless compression of the symbols according to techniques known to those skilled in the art (e.g., Huffman coding, variable-length coding, arithmetic coding, etc.).

[0065] Transmitter 409 may buffer one or more encoded video sequences created by entropy encoder 408 in preparation for transmission via communication channel 411, which may be a hardware / software link connected to a storage device where the encoded video data will be stored. Transmitter 409 may combine encoded video data from video encoder 403 with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (data sources not shown).

[0066] Controller 402 can manage the operation of encoder 400. During encoding, controller 405 can assign a specific encoded image type to each encoded image, which may affect the encoding techniques applicable to the corresponding image. For example, one of the following frame types can typically be assigned to an image:

[0067] An intra-frame image (I-image) can be an image that can be encoded and decoded without using any other frames in the sequence as a prediction source. Some video codecs allow different types of intra-frame images, including, for example, images refreshed by a standalone decoder. Those skilled in the art are aware of these variations of I-images and their corresponding applications and characteristics.

[0068] A predictive image (P-image) can be an image that is encoded and decoded by predicting sample values ​​for each block via intra-frame prediction or via inter-frame prediction using at most one motion vector and a reference index.

[0069] A bidirectional predictive image (B-image) can be an image encoded and decoded by predicting sample values ​​for each block via intra-frame prediction or via inter-frame prediction using up to two motion vectors and reference indices. Similarly, a multi-predictive image can reconstruct a single block using more than two reference images and associated metadata.

[0070] The source image can typically be spatially subdivided into multiple sample blocks (e.g., each block contains 4×4, 8×8, 4×8, or 16×16 samples) and encoded block by block. Blocks can be predictively coded with reference to other (already coded) blocks, which are determined by the coding assignments of the corresponding images applied to the block. For example, blocks of an I-image can be unpredictably coded, or these blocks can be predictively coded (spatial prediction or intra-frame prediction) with reference to already coded blocks of the same image. Pixel blocks of a P-image can be unpredictably coded, encoded via spatial prediction, or encoded via temporal prediction with reference to a previously coded reference image. Blocks of a B-image can be unpredictably coded, encoded via spatial prediction, or encoded via temporal prediction with reference to one or two previously coded reference images.

[0071] The video encoder 400 can perform encoding operations according to predetermined video coding technologies or standards, such as ITU-T H.265 Recommendation. In operation, the video encoder 400 can perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Therefore, the encoded video data can conform to the syntax specified by the video coding technology or standard used.

[0072] In one embodiment, transmitter 409 may transmit additional data along with the encoded video. Source encoder 403 may include such data as part of the encoded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant images and slices, SEI messages, fragments of Visual Usability Information (VUI) parameter sets, etc.

[0073] According to embodiments of this application, both the encoding and decoding processes can be considered as visual media data processing, which performs conversion between visual media files and the bitstream of visual media data according to format rules.

[0074] Figure 5 Example 500 is an end-to-end architecture for a stand-alone AR (STAR) device according to an exemplary embodiment, which illustrates a 5G STAR UE receiver 600, a network / cloud 501, and a 5G UE (transmitter) 700. Figure 6 This is a more detailed example 600 of at least one configuration for a STAR UE receiver 600 according to an exemplary embodiment, and Figure 7 This is a more detailed example 700 of at least one configuration for a 5G UE transmitter 700 according to an exemplary embodiment. 3GPP TR26.998 defines support for glasses-based augmented reality / mixed reality (AR / MR) devices in 5G networks. And according to an exemplary embodiment of this application, at least two types of devices are considered: 1) devices fully capable of decoding and playing complex AR / MR content (standalone AR or STAR), and 2) devices with limited computing resources and / or small physical size (and therefore smaller battery), which can only run such applications (edge-dependent AR or EDGAR) when most of the computation is performed on a 5G edge server, network, or cloud rather than on the device. In this application, the abbreviations Uu, Gnb, NEF, PCF, MSE, SDK, and 5GMSd can be considered as representing inter-user, 5G base station gNodeB, physical control format, mobile execution environment (MExE) service environment, source deployment kit, and 5G maximum sensitivity degradation, respectively. Furthermore, the M_d in this application (such as M6d and M7d) relates to UE MSH APIs, which allow 5GMS-aware applications to interact with the 5GMSd MSH. RTP and AVP can be viewed as real-time protocol and attribute value pairs, respectively.

[0075] Furthermore, according to the exemplary embodiments described below, a shared session use case can be experienced in which all participants in the shared AR session experience have AR devices, each participant sees other participants in the AR scene, wherein the participants are presented in an overlay manner in the local physical scene, and the arrangement of the participants in the scene is consistent across all receiving devices. For example, people in each local space have the same position / seat arrangement relative to each other, and this virtual space creates the feeling of being in the same space, but since the room is the real room or space in which each person is actually located, the room varies from participant to participant.

[0076] For example, according to the combination Figures 5 to 7 In the exemplary embodiment shown, the immersive media processing function on the network / cloud 501 receives uplink streams from various devices and constructs a scene description that defines the arrangement of each participant in a single virtual meeting room. This scene description, along with the encoded media stream, is transmitted to each receiving participant. The receiving participant's 5G STAR UE 600 receives, decodes, and processes the 3D video and audio streams, and renders them using the received scene description and information received from its AR runtime, thereby creating an AR scene of the virtual meeting room that includes all other participants. While each participant's virtual room is based on their own physical space, the seating / positioning of all other participants within that room remains consistent with the virtual room of every other participant in this session.

[0077] See also, according to an exemplary embodiment, Figure 8 The example 800 illustrates an EDGAR device architecture where devices such as the 5G EDGAR UE 900 cannot perform heavy processing on their own. Therefore, scene parsing and media parsing for the received content are performed in the cloud / edge 801, and then a simplified AR scene with a small number of media components is transmitted to the device for processing and rendering. Figure 9 A more detailed example of a 5G EDGAR UE 900 according to an exemplary embodiment is shown.

[0078] Figure 10 Example 1000 is shown, in which users A10, B11, and T12 participate in an AR conference room, and at least one of these users may not have an R device. As shown, user A10 is in his office 1001, sitting in a conference room with multiple chairs, and user A10 is sitting in one of the chairs. User B11 is in his living room 1002, sitting on a small two-seater sofa, which also contains at least one two-seater sofa and other furniture such as chairs and tables. User T12 is in an airport lounge 1003, sitting on a bench opposite another bench across a coffee table, and surrounded by at least one other coffee table.

[0079] Furthermore, it can be seen that in the AR environment, within office 1001, user A 10's AR presents to user A 10 a virtual user B 11v1 corresponding to user B 11 and a virtual user T 12v1 corresponding to user T 12, and makes virtual users B 11v1 and T 12v1 appear to user A 10 as if they were sitting on furniture or an office chair in office 1001, just like user A 10. It can also be seen that in living room 1202 of example 1200, user B 11's AR presents a virtual user T 12v2 corresponding to user T 12 but sitting on a sofa in living room 1202, and a virtual user A 10v1 corresponding to user A 10 and also sitting on furniture in living room 1202 instead of an office chair in office 1201. See also airport lounge 1203, where the AR representation of user T 12 corresponds to a virtual user A10v2 of user A 10, but seated at a table in airport lounge 1203, and a virtual user B 11v2, also seated at the same table opposite virtual user A 10v2. Furthermore, in each of the offices 1201, living room 1202, and airport lounge 1203, the updated scene descriptions of each room remain consistent with the other rooms in terms of location / seating arrangement. For example, in each room, user A 10 is presented as being in a relatively counter-clockwise direction relative to user 11 or its virtual representation, while user 11 or its virtual representation is in a relatively clockwise direction relative to user T 12 or its virtual representation.

[0080] However, AR technology has always had limitations in attempting to introduce virtual space creation and use for devices that do not support AR but can parse VR or 2D video, and embodiments of this application provide an improved technical process for creating virtual scenes consistent with AR scenes when such devices participate in shared AR session services.

[0081] Figure 11 Example 1100 is shown, illustrating an end-to-end architecture with a non-AR device 1101 and a cloud / edge 1102 according to an exemplary embodiment. Figure 12 A more detailed block diagram example of a non-AR device 1101 is shown.

[0082] like Figure 11 and 12 As shown, the non-AR UE 1101 is a device capable of rendering 360-degree or 2D video but lacking any AR capabilities. However, edge functionality on the cloud / edge 1102 enables AR rendering of the received scene, the rendered scene, and immersive visual and audio objects in a virtual room selected from a library. The entire video is then encoded and transmitted to device 1101 for decoding and rendering.

[0083] Therefore, it can possess multi-view capability. For example, AR processing on the edge / cloud 1102 can generate multiple videos of the same virtual room: multiple videos with different angles and different viewports. And the device 1101 can receive at least one of these videos, switch between these videos as needed, or send commands to the edge / cloud processing to stream only the required viewport / angle.

[0084] In addition, it can also have background changing capabilities, where users on device 1101 can select the desired room background from a provided library, such as different meeting rooms or even living rooms and layouts. And the cloud / edge 1102 uses the selected background and creates a virtual room accordingly.

[0085] Figure 13 An exemplary timing diagram 1300 is shown for an exemplary call flow for receiving an immersive AR session for a non-AR UE 1101. For illustrative purposes, only one transmitter is shown in this diagram, and its detailed call flow is not shown.

[0086] The figure shows AR application module 21, media playback module 22, and MAF module 23, which can be considered as modules receiving non-AR UE 1101. The figure also shows cloud / edge segmentation rendering module 24. The figure further shows media delivery module 25 and scene graph combiner module 26 of network cloud 1102. The figure also shows 5G transmitter UE module 700.

[0087] S1-S6 can be considered as the session establishment phase. At S1, the AR application module 21 can request to initiate a session from the MAF module 23, and at S2, the MAF module 23 can request to initiate a session from the cloud / edge segmentation rendering module 24.

[0088] The cloud / edge segmentation rendering module 24 can conduct session negotiation with the scene graph combiner module 26 at S3, and the scene graph combiner module 26 can correspondingly negotiate with the 5G transmitter UE 700. If successful, at S5, the cloud / edge segmentation rendering module can send an acknowledgment to the MAF module 23, and the MAF module 23 can send an acknowledgment to the AR application module 21.

[0089] Subsequently, S7 can be considered the media pipeline configuration phase, in which the MAF module 23 and the cloud / edge segmentation rendering module 24 each configure their respective pipelines. Then, after this pipeline configuration, a session can be initiated via the following signals: a signal sent from the AR application module to the media player module 22 at S8, a signal sent from the media player module 22 to the MAF module 23 at S9, and a signal sent from the MAF module 23 to the cloud / edge segmentation rendering module 24 at S10.

[0090] Subsequently, there may be a pose loop stage from S11 to S13, where at S11, pose data can be provided from the media player module 22 to the AR application module 21; at S12, the AR application module can provide pose data 12 to the MAF module 23, and then the MAF module 23 can provide pose data to the cloud / edge segmentation rendering module 24.

[0091] S14 to S16 can be considered as the shared experience streaming phase, wherein at S14, the 5G transmitter UE 700 can provide the media stream at S14 to the media delivery module 25, and at S15, it provides AR data to the scene graph synthesizer module 26. Subsequently, the scene graph synthesizer module 25 can synthesize at least one scene based on the received AR data, and at S16, it provides the scene and scene updates to the cloud / edge segmentation rendering module 24; simultaneously, the media delivery module 25 can provide the media stream to the cloud / edge segmentation rendering module at S17. According to an exemplary embodiment, this can include: obtaining an AR scene descriptor from a non-AR device that does not render AR scenes, and generating a virtual scene by a cloud device by parsing and rendering the scene descriptor obtained from the non-AR device.

[0092] S18 to S19 can be considered the media uplink stage, in which the media player module 22 collects and processes media data from its local user and provides the media data to the MAF module 23 at S18. Then, the media access module 23 can encode the media and provide the media stream to the cloud / edge split rendering module 24 at S19.

[0093] The period between S19 and S20 can be considered the media downlink phase, in which the cloud / edge segmentation rendering module 24 performs scene parsing and completes AR rendering. S20 and S21 can then be considered the media stream loop phase. At S20, the cloud / edge segmentation rendering module 24 provides the media stream to the MAF module 23, which then decodes the media and provides the rendered media to the media player 22 at S21.

[0094] With the help of such features according to the exemplary embodiments, even if the non-AR UE 1101 lacks a perspective display and therefore cannot create an AR scene, its display can still be utilized to render VR or 2D video. Therefore, the immersive media processing function of the non-AR UE 1101 only generates a general scene description describing the relative position of each participant with respect to other participants and the scene. The scene itself needs to be adjusted at each device using pose information before it can be rendered as an AR scene as described above. Furthermore, the AR rendering process at the edge or in the cloud can parse the AR scene and create a simplified VR-2D scene.

[0095] According to an exemplary embodiment, the present invention employs a segmented rendering process similar to that of EDGAR devices for non-AR devices (such as VR or 2D video devices), characterized in that, in this case, the edge / cloud AR rendering process does not generate any AR scene. Instead, the edge / cloud AR rendering process generates a virtual scene by parsing and rendering a scene description received from an immersive media processing function for a given background (such as a conference room), and then renders each participant in the conference room at the location described by the scene description.

[0096] Furthermore, depending on the ability to receive non-AR devices, the resulting video can be a 360-degree video or a 2D video, and according to an exemplary embodiment, the generated video is generated based on the posture information received from the non-AR device.

[0097] In addition, each other participant with a non-AR device was added as a 2D video overlay to the 360-degree / 2D video of the conference room, such as... Figure 10 As shown, the meeting room can have a dedicated area for using these overlays, for example... Figure 10 Furniture areas with overlaid virtual images.

[0098] Furthermore, according to an exemplary embodiment, audio signals from all participants may be mixed as necessary to generate mono audio carrying indoor speech, video may be encoded as a single 360-degree video or 2D video and delivered to the device, and optionally, multiple video (multi-view) sources may be created, wherein each video source captures the same virtual conference room from different perspectives and provides these perspectives to the device.

[0099] In addition, the non-AR UE device 1101 can receive 360-degree video and / or at least one selected multi-view video and audio, and render them on the device display. Users can switch between different perspectives or change the viewport of the 360-degree video by moving or rotating the viewing device, thereby enabling them to roam in a virtual room while watching the video.

[0100] Although the above embodiments are equipped with such 5GMS architecture extensions to use edge servers in their architecture, and although their specifications may have many features, it has been technically impossible to deploy these features as a set of software development kits (SDKs) on the device or as a set of microservices in the cloud. This technical deficiency is addressed by the embodiments described further below.

[0101] For example, the current Media Service Enabler (MSE) technical report does not define a framework for associating specifications with SDKs, and does not include any microservices concept.

[0102] See Figure 14Example 1400 illustrates a 5GMS architecture with edge extension according to an exemplary embodiment. As shown, a UE 1401 and a data network (DN) 1411 are present. UE 1401 may include a 5GMS client 1403 and a 5GMS-aware application 1405, such as the AR or non-AR embodiments described above, but such applications are not limited thereto. The 5GMS client 1403 may also include a media stream processor 1403 and an MSH, and according to an embodiment, the media stream processor 1403 may be considered as a split rendering client (SRC) with an M11 interface. DN 1411 may include a 5GMS application server (AS) 1412, a 5GMS application function (AF) 1414, and a 5GMS application provider 1413. The 5GMS AF 1414 may also communicate with a network exposure function (NEF) 1415 and a policy and charging function (PCF) 1416. The improvements described in this application can be understood in the context of at least one of UE 1401, 5GMS Aware Application 1405, 5GMS Application Provider 1413, NEF 1415, and PCF 1416; that is, instead of employing a monolithic specification lacking an MSE definition for each function or functional group, at least one of these elements can generate its own specification and, upon receiving a compliance request, provide that specification to another element among these elements, which can then further configure the initial element specification according to various possibilities further described below, and such processing can be performed through at least one of the multiple open APIs 1420 shown and interfaces M1, M2, M4, M5, M6, M7, M8, M11, N33, and N5. According to embodiments, interfaces M1, M2, M4, M5, M6, M7, M8, and M11 can be considered as interfaces M1d, M2d, M4d, M5d, M6d, M7d, M8d, and M11d.

[0103] Figure 15 A workflow 1500 according to an exemplary embodiment is shown, wherein, at S1501, the application (App), for example... Figure 14 The 5GMS-aware application 1405 can have a SAI, or an entry point Uniform Resource Locator (URL), or a manifest from M8, or obtain and subscribe to the SAI via M6 using an external service ID or a 3GPP service URL.

[0104] At S1502, the application can pass one of the following parameters to MAF (media streamer / player, e.g.) Figure 14 5GMS AF 1414): (i.) a URL pointing to a manifest or entry point, or a document, (ii.) an external service ID, and / or (iii.) via a call to an MSH (e.g., Figure 14 The session ID obtained from MSH 1404.

[0105] At S1503, if the MAF does not receive a session ID, it calls the MSH to assign a media transmission session ID. According to an exemplary embodiment, the MSH is configured to avoid duplicate session IDs or assigning multiple session IDs to the same session.

[0106] Then, at S1504, MAF streams the media content via M4. Furthermore, at S1505, when MAF receives a reset() or destroy() response, it requests MSH to release the media transmission session ID.

[0107] According to various embodiments, the MSH maintains the latest SAI and active session IDs, and when the MSH receives an updated SAI (via request or notification plus request), it notifies the application and / or MAF. Furthermore, according to various embodiments, the MSH can allocate media transport session IDs based on application requests or via MAF.

[0108] Therefore, the functionality according to embodiments of this application may exist such that: all media requests sent by the media stream processor (media player or media streamer) at reference point M4 to the 5GMS AS shall reference the media transport session ID using the HTTP header specified in Clause 6.2.3.6, wherein the value of this ID shall be different for each media stream session, and the value of this ID shall be specified by the MSH when initiating a new media stream session by calling the method specified in Clause 13.2.3.2 at reference point M11; if the media transport session ID is omitted when initiating a new media stream session by calling the media streamer, the media stream processor shall call the method in Clause 10.2.X of TS 26.510 to obtain a new media transport session ID from the MSH, or assign its own ID for use at reference point M4, and shall notify the caller of the method which value it has selected for future interactions with the media streamer.

[0109] Furthermore, such functionality according to embodiments of this application is further improved technically such that: all media requests sent by the media stream processor (media player or media streamer) to the 5GMS AS at reference point M4 should use the HTTP header specified in Clause 6.2.3.6 to reference the media transport session ID, wherein the value of this ID should be different for each media stream session, and the value of this ID should be specified by the MSH when initiating a new media stream session by calling the method specified in Clause 13.2.3.2 at reference point M11; if the media transport session ID is omitted when initiating a new media stream session by calling the media streamer, the media stream processor should call the method in TS 26.510 to obtain a new media transport session ID from the MSH for use at reference point M4, and should notify the caller of the method which value has been selected for future interactions with the media streamer.

[0110] Clause 6.2.3.6 relates to "Media Transport Session ID," stipulating that at applicable reference points, the Media Transport Session ID should be transmitted in the HTTP header CMCD-Session:sid in accordance with Table 1 of CTA 5004. Therefore, the Media Transport Session ID for 5GMS should be a UUID. Example: CMCD-Session:sid: 8bf9f090-82fd-4686-aa4a-39e6a9381b76.

[0111] Clause 13.2.3.2 relates to "Initialize" and defines the `initialize()` method. No preconditions apply. This method is called to create a new media player instance. If the caller of this method does not provide a value, the media player may assign a media transfer session ID to the media transfer session and return that ID. The input parameters for this method are specified in Table 1.

[0112] Table 1: Input parameters of the initialize() method

[0113]

[0114] According to Clause 13.2.3.2, the following functions are initialized via `initialize()`: media playback management to enable API-based communication via M7d and / or M11d. Specifically, this establishes the notification and error API (see Clause 13.2.5) and status information (see Clause 13.2.6). If the `sessionId` parameter is provided by MSH at reference point M11d, the newly created media player instance should use this value in its subsequent interactions, particularly in interactions involving the notification and error API (see Clause 13.2.5) and status information (see Clause 13.2.6). If the `sessionId` parameter is omitted when the 5GMSd-aware application calls this method at reference point M6d, the newly created media player instance should rely on MSH to assign a media transport session ID as a side effect of calling the method specified in Clause 11.2.2.1 of TS 26.510 at reference point M11d. The media player should then use this value in its subsequent interactions, particularly in interactions covered by the notification and error API (see Clause 13.2.5) and status information (see Clause 13.2.6).

[0115] According to Clause 13.2.3.2, the return values ​​of the `initialize()` method are shown in Table 2 below:

[0116] Table 2: Return values ​​of the initialize() method

[0117]

[0118] The other media player methods specified below require the media transport session ID to be referenced as an input parameter.

[0119] Therefore, according to embodiments of this application, an extension to MAF initialization is provided, as shown in Table 3 below:

[0120] Table 3: Input parameters for extending the initialize() method

[0121]

[0122] And according to embodiments of this application and depending on the input provided:

[0123] - If a sessionId is provided, the media player uses this value in its subsequent interactions, and

[0124] - If an external service ID is provided, the media player uses this value as input and calls the method specified in Clause 10.2.X of TS 26.510 to request a media transfer session ID from the MSH and uses the obtained value in its subsequent interactions.

[0125] Therefore, according to embodiments of this application, a media session ID can be assigned during player initialization. In many cases, the application may acquire the SAI but not initiate playback. Assigning a media transport session ID prematurely is not beneficial and only increases the burden of managing unused IDs.

[0126] A further benefit of embodiments of this application is that applications can obtain SAI from M5 without going through MSH. For example, the application may have a manifest or entry point URL, or receive the complete SAI through M8, and the application can pass these values ​​to MAF. MAF, capable of obtaining the media session ID from MSH, can handle these use cases.

[0127] A further benefit of embodiments of this application is that MAF can request a session ID from MSH, thereby enabling it to verify the session ID and ensure that only one ID is assigned to the associated resource (such as SAI, media entry point, and external service ID).

[0128] Therefore, this invention provides extended functionality for 5GMS MAF / media streamer / media player, wherein when communicating a session to the MAF, one of the following values ​​may be carried: Media Transport Session ID, External Service ID, or none; wherein, in the absence of a Media Transport Session ID, the MAF may invoke a method to request the MSH to provide the Media Transport Session ID; in response, the MSH may check whether an associated SAI and External Service ID already exist in its existing list, and whether the entry already has an assigned Media Transport Session ID, or has an assigned SAI but no assigned Media Transport Session ID, or neither; wherein, in each case, the MSH looks up an existing ID or generates a new ID and returns these values, thereby ensuring that there are no duplicate records or multiple IDs assigned to the same session.

[0129] Therefore, the initialize() method has been improved through embodiments of this application, and this method, along with other methods according to embodiments of this application, is shown in Table 4:

[0130] Table 4: Methods defined for the DASH stream API

[0131]

[0132] Furthermore, it should be noted that Clause 7.3.2 cited above may relate to simple data types and specifies common simple data types used within Media Delivery APIs, including a brief description of each data type. References are provided where types from other specifications are reused. See Table 5 below:

[0133] Table 5: Simple Data Types

[0134]

[0135]

[0136]

[0137] Furthermore, Clause 5.4.2.1 above relates to the explicit initiation / termination of media session processing. For example, media session processing for a new media transport session can be explicitly initiated by a media-aware application or the MAF by calling the appropriate API method on the MSH at reference point M6 or M11, respectively.

[0138] The external service ID should be provided as an input parameter to the API method.

[0139] - The media entry point URL (e.g., obtained from the media application provider at reference point M8) can be optionally provided as an input parameter to initiate media delivery.

[0140] In response, the MSH should assign a globally unique Media Delivery Session ID for use by media clients in subsequent interactions with the Media AF and Media AS. If the latest copy is not already cached, the MSH should attempt to obtain the complete SAI for the specified external service ID from the Media AF using the operation defined in Clause 5.3.2.3, and if successful, should return the Media Delivery Session ID to the caller of the API method. If called by a media-aware application at reference point M6, the MSH should initialize a new MAF instance on behalf of the caller and should initiate media delivery by passing it the media entry point URL.

[0141] Subsequent interactions between the media-aware application at reference point M6 and MSH should reference the relevant media transport session ID.

[0142] Subsequent interactions between the media-aware application and the MAF at reference point M7 should reference the relevant media transport session ID. Subsequent interactions between the MSH and MAF at reference point M11 should reference the relevant media transport session ID. Subsequent interactions between the media access client and the media AS at reference point M4 should reference the relevant media transport session ID so that media access recorded by the media AS can be associated with media session processing operations recorded by the media AF.

[0143] Media-aware applications or MAFs can explicitly terminate media session processing of a media transport session by calling the appropriate API method on the MSH at reference point M6 or M11, using the target media transport session ID as an input parameter.

[0144] Furthermore, claim 10.1A relates to a media transport session identifier. All media requests sent from the media stream processor (media player or media streamer) to the 5GMS AS at reference point M4 shall reference the media transport session ID using the HTTP header specified in clause 6.2.3.6. The value of this ID shall be different for each media stream session.

[0145] The above-described technology can be implemented as computer software using computer-readable instructions and physically stored in at least one computer-readable medium, or implemented by at least one specially configured hardware processor. For example, Figure 16 A computer system 1600 is shown that is suitable for implementing certain embodiments of the disclosed subject matter.

[0146] Computer software can be written using any suitable machine code or computer language. This machine code or computer language may be processed by mechanisms such as assembly, compilation, and linking to generate code containing instructions. These instructions can be executed directly by the computer's central processing unit (CPU), graphics processing unit (GPU), or through interpretation, microcode execution, or other methods.

[0147] These instructions can be executed on various types of computers or their components, including personal computers, tablets, servers, smartphones, gaming devices, Internet of Things devices, and so on.

[0148] Figure 16 The components of the computer system 1600 shown are merely exemplary and are not intended to imply any limitation on the scope or functionality of the computer software implementing embodiments of the present invention. The configuration of the components should also not be construed as having any dependency or requirement on any component or combination of components shown in the exemplary embodiments of the computer system 1600.

[0149] Computer system 1600 may include certain human-machine interface input devices. Such human-machine interface input devices may respond to input from at least one human user via, for example, tactile input (e.g., key presses, swipes, data glove movements), audio input (e.g., voice, clapping sounds), visual input (e.g., gestures), or olfactory input (not depicted). Human-machine interface devices may also be used to acquire media that are not necessarily directly related to conscious human input, such as audio (e.g., voice, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still image camera), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0150] Human-machine interface input devices may include at least one of the following (only one of each is depicted): keyboard 1601, mouse 1602, touchpad 1603, touch screen 1610, joystick 1605, microphone 1606, scanner 1608, and camera 1607.

[0151] Computer system 1600 may also include certain human-machine interface output devices. Such human-machine interface output devices can stimulate the senses of at least one human user through, for example, tactile output, sound, light, and smell / taste. Such human-machine interface output devices may include tactile output devices (such as tactile feedback provided by touchscreen 1610 or joystick 1605, but there may also be tactile feedback devices that do not act as input devices), audio output devices (such as speakers 1609, headphones (not depicted)), visual output devices (such as screens 1610, including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touchscreen input functionality, each with or without tactile feedback functionality—some of which are capable of outputting two-dimensional or more than three-dimensional visual output through means such as stereoscopic output; virtual reality glasses (not depicted), holographic displays, and smoke boxes (not depicted)), and printers (not depicted).

[0152] The computer system 1600 may also include user-accessible storage devices and related media, such as optical media or similar media including CD / DVD ROM / RW 1620 equipped with CD / DVD 1611, flash memory drive 1622, removable hard disk drive or solid state drive (SSD) 1623, conventional magnetic media such as magnetic tape and floppy disk (not depicted), and dedicated devices based on ROM / ASIC / PLD such as security dongles (not depicted).

[0153] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not cover transmission media, carrier waves, or other transient signals.

[0154] Computer system 1600 may also include an interface 1699 connected to at least one communication network 1698. Network 1698 may be, for example, a wireless network, a wired network, or an optical network. Network 1698 may also be a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a vehicle-mounted and industrial network, a real-time network, a latency-tolerant network, etc. Examples of network 1698 include: local area networks such as Ethernet and wireless LAN; cellular networks including GSM, 3G, 4G, 5G, LTE, etc.; wired or wireless wide area digital networks including cable TV, satellite TV, and terrestrial broadcast TV; vehicle-mounted and industrial networks including CAN bus, etc. Some networks 1698 typically require an external network interface adapter (e.g., a USB port of computer system 1600) attached to certain general-purpose data ports or peripheral buses (1650 and 1651); other networks are typically integrated into the core of computer system 1600 via a system bus described below (e.g., integrated into a PC computer system via an Ethernet interface, or integrated into a smartphone computer system via a cellular network interface). Using any of these networks 1698, computer system 1600 can communicate with other entities. This communication can be one-way receiving (e.g., broadcasting TV), one-way transmitting (e.g., CAN bus to certain CAN bus devices), or bidirectional, such as connecting to other computer systems using a local area or wide area digital network. Certain protocols and protocol stacks can be used on each of the networks and network interfaces described above.

[0155] The aforementioned human-machine interface device, user-accessible storage device, and network interface can be attached to the core 1640 of the computer system 1600.

[0156] Core 1640 may include at least one CPU 1641, GPU 1642, graphics adapter 1617, dedicated programmable processing unit in the form of a Field Programmable Gate Array (FPGA) 1643, hardware accelerators 1644 for certain tasks, etc. These devices, along with read-only memory (ROM) 1645, random-access memory (RAM) 1646, and internal high-capacity storage devices 1647 such as internal non-user-accessible hard drives and solid-state drives, can be connected via system bus 1648. In some computer systems, system bus 1648 may be accessed via at least one physical connector, allowing for expansion through additional CPUs, GPUs, etc. Peripheral devices may be directly attached to the core's system bus 1648 or connected via peripheral bus 1651. Peripheral bus architectures include PCI, USB, etc.

[0157] CPUs 1641, GPUs 1642, FPGAs 1643, and accelerators 1644 can execute certain instructions, which, when combined, constitute the aforementioned computer code. This computer code can be stored in ROM 1645 or RAM 1646. Temporary data can also be stored in RAM 1646, while permanent data can be stored, for example, in an internal mass storage device 1647. Fast storage and retrieval of any of the storage devices can be implemented using a cache memory, which can be closely associated with at least one CPU 1641, GPU 1642, mass storage device 1647, ROM 1645, RAM 1646, etc.

[0158] The computer-readable medium may contain computer code for performing various computer-implemented operations. The medium and computer code may be designed and constructed specifically for the purposes of this invention, or may belong to a class well-known and available to those skilled in the art of computer software.

[0159] For example, but not as a limitation, the architecture corresponding to computer system 1600, particularly core 1640, can provide functionality generated when one or more processors (including CPUs, GPUs, FPGAs, accelerators, etc.) execute software embodied in at least one tangible computer-readable medium. Such a computer-readable medium can be a medium associated with certain storage devices (e.g., internal core mass storage 1647 or ROM 1645) that are user-accessible mass storage devices as described above and with the non-volatile nature of core 1640. Software implementing various embodiments of the invention can be stored in such devices and executed by core 1640. Depending on specific needs, the computer-readable medium may include at least one storage device or chip. The software can cause core 1640, particularly its processors (including CPUs, GPUs, FPGAs, etc.), to execute specific processes or specific portions of specific processes described in this application, including defining data structures stored in RAM 1646 and modifying such data structures according to the processes defined by the software. Alternatively or as an alternative, the computer system may provide functionality generated by hard-wired logic or otherwise embodied in circuitry (e.g., accelerator 1644), which may operate in place of or in conjunction with software to perform a particular process or a particular portion of a particular process described herein. Where appropriate, references to software may cover logic, and vice versa. Where appropriate, references to computer-readable media may cover circuitry (e.g., an IC) storing software for execution, circuitry embodying logic for execution, or both. This invention covers any suitable combination of hardware and software.

[0160] While several exemplary embodiments have been described in this invention, modifications, substitutions, and various alternative equivalents exist that fall within the scope of this invention. Therefore, it should be understood that those skilled in the art will be able to design numerous systems and methods that, while not explicitly stated or described herein, embody the principles of this invention and are therefore within its spirit and scope.

Claims

1. A video decoding method, executed by at least one processor, characterized in that, include: Create a media player instance for 5G media streaming 5GMS media access function (MAF), the media player instance being configured to deliver at least a portion of video content to a user equipment (UE) including a 5G media service downlink 5GMSd sensing application and a 5GMSd client. When creating the media player instance, determine whether to provide the 5GMS MAF with either a media transport session identifier ID or an external service ID; Based on the determination that neither the media transmission session ID nor the external service ID has been provided to the 5GMS MAF, the control media session processor MSH provides the media transmission session ID to the 5GMS MAF. as well as The content is streamed to the UE based on the media transmission session ID, and the content is decoded at the UE.

2. The method according to claim 1, characterized in that, Controlling the MSH to provide the media transmission session ID to the 5GMS MAF includes: At the first moment when the MSH is controlled to provide the media transmission session ID to the 5GMS MAF, the control of the MSH checks whether the associated service access information SAI of the media player instance and the external service ID of the media player instance both do not exist.

3. The method according to claim 2, characterized in that, Controlling the MSH to provide the media transmission session ID to the 5GMS MAF also includes: In the first instant, determine whether the associated SAI already exists but the media transmission session ID does not yet exist; and, Perform any of the following operations: Based on the fact that either the external service ID or the media transmission session ID already exists, the media transmission session ID is provided as an existing ID to the 5GMS MAF, and, Based on the fact that the media transmission session ID does not exist at the first time, the media transmission session ID is generated and provided to the 5GMS MAF.

4. The method according to claim 1, characterized in that, Creating the media player instance includes enabling API communication via either the M7d Application Programming Interface API or the M11d API.

5. The method according to claim 4, characterized in that, Creating the media player instance involves calling the initialize() function of the Dynamic Adaptive Streaming Media DASH Streaming API, which is based on the Hypertext Transfer Protocol (HTTP).

6. The method according to claim 5, characterized in that, The input parameters of the initialize() function include the service identifier (serviceID) and the session identifier (sessionID).

7. The method according to claim 6, characterized in that, The sessionID parameter is at least one of the following: provided by the MSH at the M11d API, or omitted at the M6d API.

8. A video encoding method, executed by at least one processor, characterized in that, include: Create a media player instance for 5G media streaming 5GMS media access function (MAF), the media player instance being configured to deliver at least a portion of video content to a user equipment (UE) including a 5G media service downlink 5GMSd sensing application and a 5GMSd client. When creating the media player instance, determine whether to provide the 5GMS MAF with either a media transport session identifier ID or an external service ID; Based on the determination that neither the media transmission session ID nor the external service ID has been provided to the 5GMS MAF, the control media session processor MSH provides the media transmission session ID to the 5GMS MAF; and... The video content is encoded and then streamed to the UE based on the media transmission session ID.

9. The method according to claim 8, characterized in that, Controlling the MSH to provide the media transmission session ID to the 5GMS MAF includes: At the first moment when the MSH is controlled to provide the media transmission session ID to the 5GMS MAF, the control of the MSH checks whether the associated service access information SAI of the media player instance and the external service ID of the media player instance both do not exist.

10. The method according to claim 9, characterized in that, Controlling the MSH to provide the media transmission session ID to the 5GMS MAF also includes: In the first instant, determine whether the associated SAI already exists but the media transmission session ID does not yet exist; and, Perform any of the following operations: Based on the fact that either the external service ID or the media transmission session ID already exists, the media transmission session ID is provided as an existing ID to the 5GMS MAF, and, Based on the fact that the media transmission session ID does not exist at the first time, the media transmission session ID is generated and provided to the 5GMS MAF.

11. The method according to claim 8, characterized in that, Creating the media player instance includes enabling API communication via either the M7d Application Programming Interface API or the M11d API.

12. The method according to claim 11, characterized in that, Creating the media player instance involves calling the initialize() function of the Dynamic Adaptive Streaming Media DASH Streaming API, which is based on the Hypertext Transfer Protocol (HTTP).

13. The method according to claim 12, characterized in that, The input parameters of the initialize() function include the service identifier (serviceID) and the session identifier (sessionID).

14. The method according to claim 13, characterized in that, The sessionID parameter is at least one of the following: provided by the MSH at the M11d API, or omitted at the M6d API.

15. A method for processing visual media data, executed by at least one processor, characterized in that, include: Create a media player instance for 5G media streaming 5GMS media access function (MAF), the media player instance being configured to deliver at least a portion of video content to a user equipment (UE) including a 5G media service downlink 5GMSd sensing application and a 5GMSd client. When creating the media player instance, determine whether to provide the 5GMS MAF with either a media transport session identifier ID or an external service ID; Based on the determination that neither the media transmission session ID nor the external service ID has been provided to the 5GMS MAF, the control media session processor MSH provides the media transmission session ID to the 5GMS MAF; and... According to the format rules, a conversion is performed between the visual media file of the video content and the bitstream of the visual media data of the video content, and based on the conversion, at least one of the following operations is performed: The video content is streamed to the UE, or the UE plays the video content.

16. The method according to claim 15, characterized in that, Controlling the MSH to provide the media transmission session ID to the 5GMS MAF includes: At the first moment when the MSH is controlled to provide the media transmission session ID to the 5GMS MAF, the control of the MSH checks whether the associated service access information SAI of the media player instance and the external service ID of the media player instance both do not exist.

17. The method according to claim 16, characterized in that, Controlling the MSH to provide the media transmission session ID to the 5GMS MAF also includes: In the first instant, determine whether the associated SAI already exists but the media transmission session ID does not yet exist; and, Perform any of the following operations: Based on the fact that either the external service ID or the media transmission session ID already exists, the media transmission session ID is provided as an existing ID to the 5GMS MAF, and, Based on the fact that the media transmission session ID does not exist at the first time, the media transmission session ID is generated and provided to the 5GMS MAF.

18. The method according to claim 15, characterized in that, Creating the media player instance includes enabling API communication via either the M7d Application Programming Interface API or the M11d API.

19. The method according to claim 18, characterized in that, Creating the media player instance involves calling the initialize() function of the Dynamic Adaptive Streaming Media DASH Streaming API, which is based on the Hypertext Transfer Protocol (HTTP).

20. The method according to claim 19, characterized in that, The input parameters of the initialize() function include the service identifier (serviceID) and the session identifier (sessionID); and, The sessionID parameter is at least one of the following: provided by the MSH at the M11d API, or omitted at the M6d API.

Images