Method for 5g streaming background data transmission with flexible time scheduling

Abstract

A method and apparatus, including computer code configured to cause one or more processors to: acquire augmented reality (AR) data of a media component, the media component including at least one of audio and video; determine, for 5G media streaming (5GMS) background data transfer (BDT), the following information: day of week, and occurrence number or end date; and an indication of a resource of a 5GMS media session handler (MSH) indicating one of started, stopped, and torn down, control media streaming of the media component by the 5GMS BDT according to the determination of the day of week and occurrence number or end date.

Description

[0001] Cross-references to related applications

[0002] This application claims provisional applications filed January 22, 2024, US 63 / 623,755; April 2, 2024, US 63 / 573,266; April 2, 2024, US 63 / 573,260; April 3, 2024, US 63 / 574,206; May 13, 2024, US 63 / 646,650; May 13, 2024, US 63 / 646,656; May 13, 2024, US 63 / 646,666; May 13, 2024, US 63 / 646,682; and January 17, 2025, US 63 / 646,682. Priority of 19 / 027,771, the contents of which are hereby expressly incorporated in their entirety by reference. Technical Field

[0003] This disclosure provides a method for providing device functionality for negotiating the establishment of a separate rendering session between a device and a network. Background Technology

[0004] 3GPP has a work project on the separation of rendering for media distribution services, in which client media functions are split between the device and the network edge. This allows the client to run lighter, less demanding processes and receive more complex applications and services. The edge network then receives the media, decodes it, and partially renders it into a simpler form, enabling the client to run the lighter-weight processes.

[0005] 5G augmented reality devices require intensive processing, including multi-parallel media decoding and possibly media encoding, scene compositing, and augmented reality rendering.

[0006] When application and / or application service providers decide to run client-side media functionality in a decoupled rendering manner, they must replace that functionality with two new modules: 1. an edge-dependent lightweight media service client, and 2. a media processing application running on 5GMS AS.

[0007] Current specifications define separate rendering configuration parameters, including the required view information. The device needs to send the view location information of these views back to the network. However, these configuration parameters do not define any specifications regarding the frequency and increment at which this location information should be captured and sent to the network.

[0008] Furthermore, 3GPP TS 26.512 defines the M5 interface as a pull interface, meaning the UE needs to request service access information via HTTP. Therefore, to view updates in the service access information, the UE needs to periodically request it. This invention implements a subscription method where the UE subscribes to and receives update notifications. Therefore, it only pulls service access information when a new update is available. In other words, although 3GPP TS 26.512 and S4-240505 define a simple M6 interface for interacting with the media session handler, the defined interface's functionality is very simple and limited.

[0009] While 3GPP S4-240505 defines the configuration for background data transfer, it can only signal the estimated amount of background data transfer. During the initial dynamic policy setup, the client may not be aware of the estimated data transfer amount. The client typically only learns these values ​​when it needs to use this feature, which may occur in the future and is not included in the initial dynamic policy setup. Although 3GPP S4-231194 defines the configuration for background data transfer, it only defines a single transfer window within a day.

[0010] To address one or more different technical problems, this disclosure provides technical solutions to reduce network overhead and server computational overhead, while also providing options for applying various operations to the parsed elements, thereby improving some of their usability and technical signaling characteristics when using these operations.

[0011] Therefore, for any of the above reasons, technical solutions are needed to address these problems in video coding technology. Summary of the Invention

[0012] This disclosure includes a method and apparatus comprising a memory and one or more processors, the memory being configured to store computer program code, the one or more processors being configured to access the computer program code and operate according to the instructions of the computer program code. The computer program is configured to cause the processors to: acquire augmented reality (AR) data of a media component, the media component including at least one of audio and video; determine the following information for a 5G Media Streaming (5GMS) background data transmission (BDT): day of the week, and the number of occurrences or end date; and an indication of resources based on a 5GMS Media Session Handler (MSH) indicating one of an initiated, stopped, or dismantled state, by which the 5GMS BDT controls the media streaming transmission of the media component based on the determination of the day of the week and the number of occurrences or end date.

[0013] Determining the day of the week and the number of occurrences or end date can include a call flow where the 5GMS client indicates a future time range, the 5GMS application function (AF) provides the 5GMS client with one or more windows within that future time range, and the 5GMS client selects a window from the one or more windows within that future time range, which is the time when the 5GMSBDT is implemented.

[0014] The resources of 5GMS MSH can be indicated through the EDGE-5 / M6 application programming interface (API) to indicate the edge resource status of the application.

[0015] Service access information for a user equipment (UE) can be updated based on the user equipment's (UE) subscription via the publish-subscribe channel URL, without the UE requesting an update from the 5GMS AF. The UE's subscription includes either automatic updates to the UE's service access information or requests initiated by the UE through the M5 interface.

[0016] This subscription can be a subscription from the UE to the 5GMS BDT.

[0017] The 5GMS MSH interface can be configured to respond to requests for streaming media access information, media distribution session identifiers, edge processing information, and subscriber acquisition, and the 5GMS BDT can be configured to be launched via either the EDGE-5 / M6 interface or the M11 interface. Attached Figure Description

[0018] Other features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, wherein:

[0019] Figure 1 This is a simplified block diagram of a communication system according to an embodiment.

[0020] Figure 2 This is a simplified illustration of the encoder and decoder environment according to an embodiment.

[0021] Figure 3 This is a simplified block diagram of the decoder according to an embodiment.

[0022] Figure 4 This is a simplified block diagram of an encoder according to an embodiment.

[0023] Figure 5 This is a simplified block diagram of an AR system according to an embodiment.

[0024] Figure 6 This is a simplified block diagram of a 5G system according to an embodiment.

[0025] Figure 7 This is a simplified block diagram in a 5G UE environment according to an embodiment.

[0026] Figure 8 This is a simplified block diagram of the EDGAR environment according to an embodiment.

[0027] Figure 9 This is a simplified block diagram of the EDGAR UE environment according to an embodiment.

[0028] Figure 10 This is a simplified diagram illustrating the AR application according to an embodiment.

[0029] Figure 11 This is a simplified block diagram of a hybrid AR and non-AR system according to an embodiment.

[0030] Figure 12 This is a simplified block diagram of a non-AR UE according to an embodiment.

[0031] Figure 13 These are simplified block diagrams and timing diagrams based on embodiments.

[0032] Figure 14 This is a simplified block diagram of the 5GMS architecture according to an embodiment.

[0033] Figure 15 This is a simplified block diagram of the MSE framework according to an embodiment.

[0034] Figure 16 This is a simplified block diagram of a 5GMSd sensing application according to an embodiment.

[0035] Figure 17 This is a simplified block diagram of the 5GMS architecture according to an embodiment.

[0036] Figure 18 This is a schematic diagram according to an embodiment. Detailed Implementation

[0037] The features discussed below can be used individually or in any order. Furthermore, embodiments can be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored on a non-volatile computer-readable medium.

[0038] Figure 1A simplified block diagram of a communication system 100 according to an embodiment of the present disclosure is shown. The communication system 100 may include at least two terminals 102 and 103 interconnected via a network 105. For unidirectional data transmission, the first terminal 103 may encode video data locally for transmission to the other terminal 102 via the network 105. The second terminal 102 may receive the encoded video data from the other terminal from the network 105, decode the encoded data, and display the recovered video data. Unidirectional data transmission may be common in media service applications, etc.

[0039] Figure 1 A second pair of terminals 101 and 104 is shown, which are used to support bidirectional transmission of encoded video, such as during a video conference. For bidirectional data transmission, each terminal 101 and 104 can encode video data captured at a local location for transmission to the other terminal via network 105. Each terminal 101 and 104 can also receive encoded video data transmitted by the other terminal, decode the encoded data, and display the recovered video data on a local display device.

[0040] exist Figure 1 In this disclosure, terminals 101, 102, 103, and 104 can represent servers, personal computers, and smartphones, but the principles of this disclosure are not limited thereto. Embodiments of this disclosure are applicable to laptops, tablets, media players, and / or dedicated video conferencing equipment. Network 105 represents any number of networks, including, for example, wired and / or wireless communication networks, that transmit encoded video data between terminals 101, 102, 103, and 104. Communication network 105 may exchange data on circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks (LANs), wide area networks (WANs), and / or the Internet. For the purposes of this discussion, the architecture and topology of network 105 may be irrelevant to the operation of this disclosure unless otherwise stated below.

[0041] Figure 2 Using the disclosed topic as an example, the placement of video encoders and decoders in a streaming environment is illustrated. The disclosed topic is also applicable to other video-enabled applications, including, for example, video conferencing, digital television, storing compressed video on digital media including CDs, DVDs, Memory Sticks, etc.

[0042] The streaming media system may include a capture subsystem 203, which may include a video source 201 (e.g., a digital camera) that creates, for example, an uncompressed video sample stream 213. This sample stream 213 may be emphasized as having a high data volume compared to an encoded video stream and may be processed by an encoder 202 coupled to the camera 201. The encoder 202 may include hardware, software, or a combination thereof to implement aspects of the disclosed subject matter as described in more detail below. An encoded video stream 204 may be stored on a streaming media server 205 for future use; this encoded video stream 204 may be emphasized as having a lower data volume compared to the sample stream. One or more streaming media clients 212 and 207 may access the streaming media server 205 to retrieve copies 208 and 206 of the encoded video stream 204. Client 212 may include a video decoder 211 that decodes the incoming copy 208 of the encoded video stream and creates an output video sample stream 210 that can be rendered on a display 209 or other rendering device (not depicted). In some streaming media systems, video streams can be encoded using 204, 206, and 208 bitrates according to certain video encoding / compression standards. Examples of these standards have been listed above, and a more detailed description follows.

[0043] Figure 3 This can be a functional block diagram of a video decoder 300 according to an embodiment of the present invention.

[0044] Receiver 302 can receive one or more encoded video sequences to be decoded by decoder 300; in the same or another embodiment, one encoded video sequence is decoded at a time, wherein the decoding of each encoded video sequence is independent of other encoded video sequences. Encoded video sequences can be received from channel 301, which can be a hardware / software connection to a storage device storing the encoded video data. Receiver 302 can receive encoded video data and other data (e.g., encoded audio data and / or auxiliary data streams), which may be forwarded to their respective user entities (not depicted). Receiver 302 can separate the encoded video sequences from other data. To combat network jitter, buffer memory 303 can be coupled between receiver 302 and entropy decoder / resolver 304 (hereinafter referred to as the "resolver"). Buffer memory 303 may not be necessary or may be very small when receiver 302 receives data from a store / forward device with sufficient bandwidth and controllability, or when receiving data from a synchronization network. For use on best-effort packet networks such as the Internet, a buffer memory 303 may be required. The buffer memory 303 can be relatively large and can advantageously have an adaptive size.

[0045] Video decoder 300 may include parser 304 to reconstruct symbols 313 from an entropy-encoded video sequence. These symbols may include information for managing the operation of decoder 300, and potentially information for controlling a rendering device, such as display 312, which is not part of the decoder but may be coupled to it. Control information for one or more rendering devices may be in the form of supplementary enhancement information (SEI messages) or fragments of video availability information parameter sets (not depicted). Parser 304 may perform parsing / entropy decoding on the received encoded video sequence. The encoding of the encoded video sequence may be based on video coding techniques or standards and may follow principles known to those skilled in the art, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. Parser 304 may extract a set of subgroup parameters from the encoded video sequence for at least one pixel subgroup in the video decoder based on at least one parameter corresponding to a group. Subgroups may include picture groups (GOPs), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), etc. Entropy decoders / parsers can also extract information such as transform coefficients, quantizer parameter values, and motion vectors from encoded video sequences.

[0046] The parser 304 can perform entropy decoding / parsing operations on the video sequence received from the buffer memory 303 to create symbols 313. The parser 304 can receive encoded data and selectively decode specific symbols 313. Furthermore, the parser 304 can determine whether to provide specific symbols 313 to the motion compensation prediction unit 306, the scaler / inverse transform unit 305, the intra-frame prediction unit 307, or the loop filter 311.

[0047] The reconstruction of symbol 313 can involve multiple different units, depending on the type of encoded video picture or its portions (e.g., inter-frame and intra-frame pictures, inter-frame and intra-frame blocks) and other factors. Which units are involved and how they are involved can be controlled by subgroup control information, which is parsed from the encoded video sequence by parser 304. For clarity, the flow of this subgroup control information between parser 304 and the multiple units is not described below.

[0048] In addition to the functional blocks already mentioned, the decoder 300 can be conceptually subdivided into multiple functional units as described below. In practical implementations operating under commercial constraints, many of these units interact closely with each other and can be at least partially integrated with one another. However, for the purposes of describing the disclosed subject matter, it is appropriate to conceptually subdivide it into the functional units described below.

[0049] The first unit is the scaler / inverse transform unit 305. This scaler / inverse transform unit 305 receives from the parser 304 quantization transform coefficients as one or more symbols 313, along with control information (including which transform to use, block size, quantization factor, quantization scaling matrix, etc.). It can output blocks containing sample values, which can be input into the aggregator 310.

[0050] In some cases, the output samples of the scaler / inverse transform unit 305 may belong to intra-coded blocks; that is, blocks that do not use predictive information from previously reconstructed images, but can use predictive information from previously reconstructed portions of the current image. Such predictive information can be provided by the intra-prediction unit 307. In some cases, the intra-prediction unit 307 uses surrounding already reconstructed information obtained from the current (partially reconstructed) image 309 to generate blocks with the same size and shape as the blocks in the reconstruction. In some cases, the aggregator 310 adds the predictive information generated by the intra-prediction unit 307 to the output sample information provided by the scaler / inverse transform unit 305 on a per-sample basis.

[0051] In other cases, the output samples of the scaler / inverse transform unit 305 may belong to inter-frame coded blocks that may have undergone motion compensation. In such cases, the motion compensation prediction unit 306 can access the reference image memory 308 to obtain samples for prediction. After motion compensation is performed on the obtained samples according to the symbols 313 belonging to that block, these samples can be added by the aggregator 310 to the output of the scaler / inverse transform unit (referred to as residual samples or residual signals in this case) to generate output sample information. The address in the reference image memory for the motion compensation unit to obtain prediction samples can be controlled by motion vectors, which are provided to the motion compensation unit in the form of symbols 313, which may have, for example, X, Y, and reference image components. Motion compensation may also include interpolation of sample values ​​obtained from the reference image memory when using subsample precise motion vectors, motion vector prediction mechanisms, etc.

[0052] The output samples of aggregator 310 can be used with various loop filtering techniques in loop filter unit 311. Video compression techniques may include loop filter techniques controlled by parameters included in the encoded video bitstream and provided to loop filter unit 311 as symbols 313 in parser 304, but may also be in response to metadata acquired during decoding of a previous (in decoding order) portion of the encoded picture or encoded video sequence, and in response to previously reconstructed and loop-filtered sample values.

[0053] The output of the loop filter unit 311 can be a sample stream, which can be output to the display 312 (which can be a rendering device) and stored in the reference image memory 557 for use in future inter-frame prediction.

[0054] Once a certain encoded image is fully reconstructed, it can be used as a reference image for future predictions. Once an encoded image has been fully reconstructed and has been identified as a reference image (e.g., by parser 304), the current reference image 309 can become part of the reference image buffer 308, and a new current image memory can be reallocated before starting to reconstruct the next encoded image.

[0055] The video decoder 300 can perform decoding operations according to a predetermined video compression technique, which may be specified in standards such as ITU-T Rec. H.265. The encoded video sequence can conform to the syntax specified by the video compression technique or standard used, as it follows the syntax of the video compression technique or standard, such as that specified in the video compression technique document or standard and, specifically, its configuration file. Furthermore, compliance requires that the complexity of the encoded video sequence be within limits defined by the level of the video compression technique or standard. In some cases, the level limits the maximum image size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference image size, etc. In some cases, the limitations set by the level can be further restricted by the hypothetical reference decoder (HRD) specification and the metadata managed by the HRD buffer, which are signaled in the encoded video sequence.

[0056] In one embodiment, receiver 302 may receive supplemental (redundant) data and encoded video. This supplemental data may be included as part of one or more encoded video sequences. Video decoder 300 may use this supplemental data to correctly decode the data and / or more accurately reconstruct the original video data. The supplemental data may be in the form of, for example, temporal enhancement layers, spatial enhancement layers or signal-to-noise ratio (SNR) enhancement layers, redundant fragments, redundant images, forward error correction codes, etc.

[0057] Figure 4 This may be a functional block diagram of a video encoder 400 according to an embodiment of the present disclosure.

[0058] The encoder 400 can receive video samples from a video source 401 (which is not part of the encoder), which can capture one or more video images to be encoded by the encoder 400.

[0059] Video source 401 may provide a source video sequence to be encoded by encoder (303) in the form of a digital video sample stream, which may have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601 YCrCb, RGB, etc.), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media service system, video source 401 may be a storage device storing previously prepared video. In a video conferencing system, video source 401 may be a camera capturing local image information as a video sequence. Video data may be provided as multiple individual pictures that produce motion effects when viewed sequentially. These pictures themselves may be organized as a spatial array of pixels, where each pixel may include one or more samples depending on the sampling structure, color space, etc., in use. The relationship between pixels and samples will be readily understood by those skilled in the art. The following description focuses on samples.

[0060] According to an embodiment, encoder 400 can encode and compress images of a source video sequence into an encoded video sequence 410 in real time or under any other time constraints required by the application. Implementing an appropriate encoding rate is a function of controller 402. The controller controls and is functionally coupled to other functional units described below. For clarity, the coupling is not depicted. Parameters set by the controller may include rate control-related parameters (image skipping, quantizer, λ value of rate distortion optimization techniques, etc.), image size, group of images (GOP) layout, maximum motion vector search range, etc. Other functions of controller 402 can be readily identified by those skilled in the art, as they may be related to the video encoder 400 optimized for a particular system design.

[0061] Some video encoders operate in a manner readily recognized by those skilled in the art as an "encoding loop." Simply put, an encoding loop can consist of the encoding portion of an encoder (e.g., source encoder 403) responsible for creating symbols based on the input image to be encoded and one or more reference images, and a (local) decoder 406 embedded in encoder 400 that reconstructs these symbols to create sample data that a (remote) decoder will also create (because any compression between the symbols and the encoded video stream is lossless in the video compression techniques considered in the disclosed subject matter). This reconstructed sample stream is fed into a reference image memory 405. Since the decoding of the symbol stream results in bit-accurate results regardless of the decoder's location (local or remote), the contents of the reference image buffer are also bit-accurate between the local and remote encoders. In other words, when prediction is used during decoding, the encoder's prediction portion will "see" the exact same sample values ​​that the decoder will "see" as reference image samples. The basic principles of reference image synchronization (and, if synchronization cannot be maintained, drift due to channel errors) are well known to those skilled in the art.

[0062] The operation of the "local" decoder 406 can be the same as that of the "remote" decoder 300, as already described above. Figure 3 The "remote" decoder 300 is described in detail. However, a brief reference is also provided. Figure 4 Since symbols are available, and the encoding / decoding of symbols in the encoded video sequence by the entropy encoder 408 and the parser 304 can be lossless, the entropy decoding part of the decoder 300 (including the channel 301, receiver 302, buffer memory 303 and parser 304) may not be implemented in the local decoder 406.

[0063] The observation that can be made in this respect is that any decoder technique other than parsing / entropy decoding, which exists in the decoder, must also necessarily exist in the corresponding encoder in essentially the same functional form. The description of encoder techniques can be simplified because they are exactly the opposite of the decoder techniques already described in detail. A more detailed description is only required in certain areas, and is provided below.

[0064] As part of its operation, the source encoder 403 can perform motion-compensated predictive coding, which predictively encodes the input frame with reference to one or more previously encoded frames from the video sequence designated as "reference frames". In this way, the encoding engine 407 encodes the difference between pixel blocks of the input frame and pixel blocks of one or more reference frames, which can be selected as one or more prediction references for the input frame.

[0065] The local video decoder 406 can decode encoded video data of frames that may be designated as reference frames based on symbols created by the source encoder 403. The operation of the encoding engine 407 can advantageously be a lossy process. When the video decoder (in...) Figure 4 When decoding encoded video data at a location (not shown), the reconstructed video sequence can typically be a copy of the source video sequence with some errors. The local video decoder 406 replicates the decoding process performed by the video decoder on the reference frame and can store the reconstructed reference frame in a reference image memory 405, which can be, for example, a cache memory. In this way, the encoder 400 can locally store copies of the reconstructed reference frames that share the same content (no transmission errors) as the reconstructed reference frames to be acquired by the remote video decoder.

[0066] Predictor 404 can perform a prediction search on encoding engine 407. That is, for a new frame to be encoded, predictor 404 can search in reference image memory 405 for sample data (as candidate reference pixel blocks) or certain metadata such as reference image motion vectors, block shapes, etc., that can be used as appropriate prediction references for the new image. Predictor 404 can operate on a sample block-to-pixel block basis to find appropriate prediction references. In some cases, as determined by the search results obtained by predictor 404, the input image may have prediction references extracted from multiple reference images stored in reference image memory 405.

[0067] The controller 402 can manage the encoding operations of the video encoder 403, including, for example, setting parameters and subgroup parameters for encoding video data.

[0068] The outputs of all the above-described functional units can undergo entropy encoding in the entropy encoder 408. The entropy encoder converts symbols generated by the various functional units into an encoded video sequence by losslessly compressing the symbols according to techniques known to those skilled in the art (e.g., Huffman coding, variable-length coding, arithmetic coding, etc.).

[0069] Transmitter 409 may buffer one or more encoded video sequences, such as those created by entropy encoder 408, to prepare them for transmission via communication channel 411, which may be a hardware / software link to a storage device that will store the encoded video data. Transmitter 409 may also combine encoded video data from video encoder 403 with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (sources not shown).

[0070] Controller 402 can manage the operation of encoder 400. During encoding, controller 405 can assign a specific encoded image type to each encoded image, which may affect the encoding technique that can be applied to the corresponding image. For example, images can typically be assigned to one of the following frame types:

[0071] An intra-frame picture (I-picture) can be a picture that can be encoded and decoded without using any other frames in the sequence as a prediction source. Some video codecs allow different types of intra-frame pictures, including, for example, pictures refreshed by a separate decoder. Those skilled in the art are aware of these variations of I-pictures, as well as their respective applications and characteristics.

[0072] A predicted image (P-image) can be an image that can be encoded and decoded using intra-frame prediction or inter-frame prediction, where intra-frame prediction or inter-frame prediction uses at most one motion vector and reference index to predict sample values ​​for each block.

[0073] A bidirectional prediction image (B-image) can be an image that can be encoded and decoded using either intra-frame prediction or inter-frame prediction, where intra-frame prediction or inter-frame prediction uses at most two motion vectors and a reference index to predict sample values ​​for each block. Similarly, a multi-prediction image can use more than two reference images and associated metadata to reconstruct a single block.

[0074] Source images can typically be spatially subdivided into multiple sample blocks (e.g., each block being 4×4, 8×8, 4×8, or 16×16 samples) and encoded on a block-by-block basis. Blocks can be predicted by referencing other (already encoded) blocks, such as those determined by the encoding assignment applied to the corresponding images of the blocks. For example, blocks of image I can be encoded unpredictably, or they can be predicted by referencing already encoded blocks of the same image (spatial prediction or intra-frame prediction). Pixel blocks of image P can be unpredictably encoded by spatial prediction or temporal prediction, referencing a previously encoded reference image. Blocks of image B can be unpredictably encoded by spatial prediction or temporal prediction, referencing one or two previously encoded reference images.

[0075] The video encoder 400 can perform encoding operations according to a predetermined video coding technology or standard (e.g., ITU-T Rec.H.265). In its operation, the video encoder 400 can perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Therefore, the encoded video data can conform to the syntax specified by the video coding technology or standard used.

[0076] In one embodiment, transmitter 409 may transmit additional data along with the encoded video. Source encoder 403 may include such data as part of the encoded video sequence. Additional data may include temporal enhancement layers / spatial enhancement layers / SNR enhancement layers, other forms of redundant data (e.g., redundant images and segments), supplementary enhancement information (SEI) messages, fragments of visual usability information (VUI) parameter sets, etc.

[0077] Figure 5 Example 500 of an end-to-end architecture for a standalone AR (STAR) device according to an exemplary embodiment shows a 5G STAR user equipment (UE) receiver 600, a network / cloud 501, and a 5G UE (transmitter) 700. Figure 6 This is a further detailed example 600 of one or more configurations of the STAR UE receiver 600 according to an exemplary embodiment. Figure 7 This is a further detailed example 700 of one or more configurations of a 5G UE transmitter 700 according to an exemplary embodiment. 3GPP TR 26.998 defines support for glasses-type augmented reality / mixed reality (AR / MR) devices in 5G networks. According to the exemplary embodiments herein, at least two types of devices are considered: 1) devices capable of fully decoding and playing complex AR / MR content (running AR or STAR independently), and 2) devices with small computing resources and / or physical size (and battery), and such applications can only run if most of the computation is performed on a 5G edge server, network, or cloud rather than on the device (edge-dependent AR or EDGAR).

[0078] According to the exemplary embodiments described below, a shared session usage scenario can be experienced, wherein all participants in the shared AR dialogue experience have AR devices, each participant can see other participants in the AR scene, these participants appear in the local physical scene as an overlay, and the arrangement of these participants in the scene is consistent across all receiving devices, for example, people in each local space have the same location / seat arrangement with each other, and such a virtual space creates a feeling of being in the same space, but the room is different for each person, because the room refers to the room or space that each person is actually in.

[0079] For example, according to Figures 5 to 7In the exemplary embodiment shown, the immersive media processing function on the network / cloud 501 receives uplink streams from various devices and composes a scene description that defines the arrangement of each participant in a single virtual meeting room. This scene description, along with the encoded media stream, is distributed to each receiving participant. The receiving participant's 5G STAR UE 600 receives, decodes, and processes the 3D video and audio streams, and renders them using the received scene description and information received from its AR runtime, thereby creating an AR scene containing the virtual meeting room of all other participants. While these participants' virtual rooms are based on their own physical spaces, the seating / positioning of all other participants in that room is consistent with the virtual rooms of all other participants in the session.

[0080] See also, according to an exemplary embodiment, Figure 8 The example 800 illustrates an EDGAR device architecture, where, for example, a 5G EDGAR UE 900, the device itself cannot handle heavy processing power. Therefore, scene parsing and media parsing of the received content are performed in the cloud / edge 801, and then a simplified AR scene containing a small number of media components is delivered to the device for processing and rendering. Figure 9 A more detailed example of a 5G EDGAR UE 900 according to an exemplary embodiment is shown.

[0081] Figure 10 Example 1000 is shown, in which users A 10, B 11, and T 12 will participate in an AR conference room, and one or more of these users may not have an R device. As shown, user A 10 is in his office 1001, sitting in a conference room with a different number of chairs, and user A 10 is sitting in one of the chairs. User B 11 is in his living room 1002, sitting on a two-seater sofa, which also contains one or more two-seater sofas, as well as other furniture such as chairs and tables. User T 12 is in airport VIP lounge 1003, sitting on a bench opposite another bench, separated by a coffee table, and surrounded by one or more other coffee tables.

[0082] In the AR environment of office 1001, user A 10's AR displays to user A 10 a virtual user B 11v1 corresponding to user B 11 and a virtual user T 12v1 corresponding to user T 12, and makes virtual users B 11v1 and T 12v1 appear to user A 10 as if user A 10 were sitting in an office chair in office 1001. See living room 1202 in example 1200, where user B 11's AR displays a virtual user T 12v2 corresponding to user T 12 but sitting on a sofa in living room 1202, and a virtual user A10v1 corresponding to user A 10 also sitting on furniture in living room 1202, instead of in an office chair in office 1201. See also airport lounge 1203, where user T12's AR display corresponds to virtual user A10v2 of user A10, but is seated at a table in airport lounge 1203, as is virtual user B11v2, who is also seated at the table opposite virtual user A10v2. In each of these offices 1201, lounges 1202, and airport lounge 1203, the updated scene description for each room is consistent with the other rooms in terms of location / seating arrangement. For example, user A10 is displayed as being in a relatively counter-clockwise direction relative to user 11 or its virtual representation, while user 11 is in a relatively clockwise direction relative to user T12 or its virtual representation (by room).

[0083] However, AR technology has always had limitations in trying to integrate the creation and use of virtual spaces onto devices that do not support AR but can resolve VR or 2D video. The embodiments in this article provide an improved technical process for creating virtual scenes consistent with AR scenes when such devices participate in shared AR dialogue services.

[0084] Figure 11 Example 1100 of an end-to-end architecture having a non-AR device 1101 and a cloud / edge 1102 according to an exemplary embodiment is shown. Figure 12 A more detailed block diagram example of a non-AR device 1101 is shown.

[0085] like Figure 11 and Figure 12 As shown, the non-AR UE 1101 is a device capable of rendering 360-degree or 2D video but without any AR capabilities. However, the edge functionality on the cloud / edge 1102 enables AR rendering of the received scene, rendering the scene, and presenting immersive visual and audio objects in a virtual room selected from a library. The entire video is then encoded and distributed to device 1101 for decoding and rendering.

[0086] Therefore, multi-view capability may exist, for example, AR processing on the edge / cloud 1102 can generate multiple videos of the same virtual room: from different angles and with different viewports. Device 1101 can receive one or more of these videos, switch between them as needed, or send commands to the edge / cloud processing to stream only the desired viewport / angle.

[0087] Furthermore, the ability to change the background may exist, where the user on device 1101 can select the desired room background from a provided library, such as one of different meeting rooms, or even a living room and layout. The cloud / edge 1102 then uses the selected background and creates a virtual room accordingly.

[0088] Figure 13 Example timing diagram 1300 shows an example call flow for an immersive AR conversation on a non-AR UE 1101 at the receiving end. For ease of illustration, only one sender is shown in this diagram, and its detailed call flow is not shown.

[0089] The figure shows the AR application module 21, media playback module 22, and media access function module 23, which can be considered as modules of the non-AR receiver UE 1101. The figure also shows the cloud / edge separation rendering module 24. The figure further shows the network / cloud 1102 media distribution module 25 and scene graph synthesis module 26. The figure also shows the 5G transmitter UE module 700.

[0090] S1 to S6 can be considered as the session establishment phase. The AR application module 21 can request the media access function module 23 to start the session at S1, and the media access function module 23 can request the cloud / edge separation rendering module 24 to start the session at S2.

[0091] The cloud / edge separation rendering module 24 can negotiate a session with the scene graph composition module 26 at S3, and the scene graph composition module 26 can negotiate with the 5G transmitter UE 700 accordingly. If successful, at S5, the cloud / edge separation rendering module can send an acknowledgment to the media access function module 23, and the media access function module 23 can send an acknowledgment to the AR application module 21.

[0092] Next, S7 can be considered the media pipeline configuration phase, in which the media access function module 23 and the cloud / edge separation rendering module 24 each configure their respective pipelines. Then, after configuring the pipeline, the session can be started by the AR application module sending a signal to the media player module 22 at S8, the media player module 22 sending a signal to the media access function module 23 at S9, and the media access function module 23 sending a signal to the cloud / edge separation rendering module 24 at S10.

[0093] Then, there can be a pose loop stage from S11 to S13, where at S11, pose data can be provided from the media player module 22 to the AR application module 21, and at S12, the AR application module can provide pose data 12 to the media access function module 23. After that, the media access function module 23 can provide pose data to the cloud / edge separation rendering module 24.

[0094] S14 to S16 can be considered as the shared experience streaming phase, where in S14, the 5G transmitter UE 700 can provide a media stream to the media distribution module 25, and in S15, it can provide AR data to the scene graph synthesis module 26. Then, the scene graph synthesis module 25 can synthesize one or more scenes based on the received AR data, and in S16, it provides the scenes and scene updates to the cloud / edge-separated rendering module 24. The media distribution module 25 can also provide the media stream to the cloud / edge-separated rendering module in S17. This can include, according to an exemplary embodiment, obtaining an AR scene descriptor from a non-AR device that does not render AR scenes, and generating a virtual scene by the cloud device through parsing and rendering the scene descriptor obtained from the non-AR device.

[0095] S18 to S19 can be considered the media uplink stage, in which the media player module 22 captures and processes media data from its local user and provides the media data to the media access function module 23 at S18. Then, the media access module 23 can encode the media at S19 and provide the media stream to the cloud / edge separation rendering module 24.

[0096] Between S19 and S20, a media downlink phase can be considered, in which the cloud / edge separation rendering module 24 can perform scene parsing and complete AR rendering. After this, S20 and S21 can be considered to form a media stream loop phase. At S20, the cloud / edge separation rendering module 24 can provide the media stream to the media access function module 23, which can then decode the media. At S21, media rendering is provided to the media player 22.

[0097] By means of such features according to the exemplary embodiments, the non-AR UE 1101 can still render VR or 2D video using its display even if it does not have a perspective display and therefore cannot create AR scenes. Thus, its immersive media processing capabilities only generate a general scene description, describing the relative position of each participant to other participants and the scene itself. The scene itself needs to be adjusted at each device using pose information before being rendered as an AR scene as described above. AR rendering processes at the edge or in the cloud can resolve the AR scene and create a simplified VR-2D scene.

[0098] According to an exemplary embodiment, this disclosure uses a separate rendering process similar to that of EDGAR devices for non-AR devices (e.g., VR or 2D video devices), characterized in that the edge / cloud AR rendering process does not generate any AR scene in this case. Instead, it generates a virtual scene by parsing and rendering a scene description for a given background (e.g., a conference room) received from an immersive media processing function, and then renders each participant to the location described in the scene description within that conference room.

[0099] Furthermore, the resulting video can be a 360-degree video or a 2D video, depending on the capabilities of the receiving non-AR device, and according to an exemplary embodiment, the generated video takes into account the pose information received from the non-AR device.

[0100] In addition, each other participant using a non-AR device will be added as a 2D video overlay to the 360 / 2D video in the conference room, such as... Figure 10 As shown, the room can have areas dedicated to using these overlays, such as furniture areas, where virtual images are overlaid, like... Figure 10 As shown.

[0101] Furthermore, according to an exemplary embodiment, if necessary, audio signals from all participants can be mixed to create mono audio carrying the speech in the room, video can be encoded into a single 360-degree video or 2D video and distributed to the device, and optionally, multiple video (multi-view) sources can be created, each of which captures the same virtual conference room from a different perspective and provides these views to the device.

[0102] In addition, the non-AR UE device 1101 can receive 360 ​​video and / or one or more selected multi-view videos and audio and render them on the device display, and the user can switch between different views or change the viewport of the 360 ​​video by moving or rotating the view device, thereby enabling navigation in a virtual room while watching the video.

[0103] While the above embodiments provide 5G Media Streaming Architecture (5GMS) extensions for using edge servers within their architecture, and while their specifications may have many features, these features are technically not deployable as a set of software development kits (SDKs) on devices or as a set of microservices in the cloud, and this technical limitation will be addressed in the embodiments described further below.

[0104] For example, the current Media Service Enablement Technology report does not define a framework for associating the specification with the SDK, nor does it include any concept of microservices.

[0105] See Figure 14 Example 1400 illustrates a 5G media streaming architecture with edge extension according to an exemplary embodiment. As shown, a User Equipment (UE) 1401 and a Data Network (DN) 1411 are present. The UE 1401 may include a 5GMS client 1403 and a 5GMS-aware application 1405, such as the AR or non-AR embodiments described above, and such applications are not limited thereto. The 5GMS client 1403 may also include a media streaming processor 1403 and a media session processor. The DN 1411 may include a 5GMS Application Server (AS) 1412, a 5GMS Application Function (AF) 1414, and a 5GMS Application Provider 1413. The 5GMS AF 1414 may also communicate with a Network Open Function (NEF) 1415 and a Policy and Charging Function (PCF) 1416. The improvements described herein can be understood in the context of at least one or more of UE 1401, 5GMS Aware Application 1405, 5GMS Application Provider 1413, NEF 1415, and PCF 1416; that is, instead of employing a single, monolithic specification lacking a definition for the Media Services Enabler (MSE) for each function or group of functions, one or more of these elements can generate their own specifications and, upon receiving a compliant request, provide this specification to another of these elements, which can then further configure the initial element specification according to various possibilities (e.g., further described below), and this process can be performed through the multiple open application programming interfaces (APIs) 1420 shown, as well as any one or more of interfaces M1, M2, M4, M5, M6, M7, M8, N33, and N5.

[0106] According to an embodiment, an improved method is provided to indicate the day of the week, the number of occurrences, and / or the end of data, thereby supporting not only a single transmission window in a single data transfer, but also multiple windows within a day for background data transfer. The M1BDTSpecification type definition according to Table 1 is provided:

[0107] Table 1

[0108]

[0109] According to the embodiments in this document, the M5BDTSpecification type definition based on Table 2 is also provided:

[0110] Table 2

[0111]

[0112] According to an exemplary embodiment, daysInWeek uses a parameter to define which day of the week the policy is active. This parameter can be used to set any combination of dates within the week.

[0113] The occurrence parameter defines the number of days the BDT policy can be repeated. For example, if daysInWeek=1 and occurrence=10, then according to the embodiment, this means that the policy is active for 10 consecutive Mondays.

[0114] The parameter `endTime` defines the end date and time of the BDT policy. Even if the required number of occurrences has not been reached, the BDT policy will be canceled on this date.

[0115] Therefore, according to an embodiment, a method for background data transmission in a 5GM media streaming architecture is provided for downloading / uploading media streams, wherein the day of the week to which the policy applies, the number of times the policy occurs, the number of sessions that can be used, and / or the end date and time of the policy are defined.

[0116] Media application providers use the Edge Resource Configuration API to configure edge resource usage for media streaming sessions associated with a parent configuration session. This information can be used as a template to select or instantiate an appropriate Media AS EAS instance that will provide media session services to the UE.

[0117] According to an embodiment, the edge resource API can be accessed via the following URL base path: {apiRoot} / 3gpp-maf-provisioning / {apiVersion} / provisioning-sessions / {provisioningSessionId} / .

[0118] According to the embodiments, a definition of the BdtPolicySchedule type, for example, according to Table 3, is also provided:

[0119] Table 3

[0120]

[0121] In addition, see Figure 14The exemplary embodiment provides for using different attributes to identify the future required range, thereby improving the existing design by including a new parameter (required window range) that the client signals the AF to the time range during which it may need to transmit background data in the future. Therefore, according to the embodiments herein, further M5BDTSpecification type definitions are also provided according to Table 4:

[0122] Table 4

[0123]

[0124] According to an embodiment, `desiredWindowRange` defines the start and end dates of the desired time interval for which a client might want to perform background data transfer. Because the current solution allows multiple background data transfer windows to be provided within a day, several days of a week, and several weeks or months, the AF (Application Function) is unaware of the time span over which the client is looking for an available window. `desiredWindowRange` provides this information to the AF, thus making the AF's response relevant to the client.

[0125] According to the embodiment, the desiredWindowRange parameter is used in conjunction with other parameters, including: when establishing background data transfer, the client needs to initially set a dynamic strategy for it. In the initial request, the client defines the desiredWindowRange in the M5BDTSpecification. At this stage, the client does not use the duration parameter of estimatedDataTransferVolume. Then, with Figure 14 Similarly, AF responds to clients by including windows configured from the M1 interface, the duration of which is defined by `desiredWindowRange`. AF does not need to include the maximum bitrate for each defined `AvailableWindow`. Then, when the client needs to set up background data transfer, it updates the dynamic policy based on the available windows, including the estimated data transfer volume (`estimatedDataTransferVolume`) and the required duration. It does not include `desiredWindowRange` because it only focuses on the moment background data transfer begins, while the duration represents the interval between transfers. AF then responds by updating `AvailableWindows`, which can accommodate background data transfer within the requested duration, and also includes the maximum bitrate, so that the client knows the available bandwidth limit for each window.

[0126] Furthermore, according to the embodiment, the media session handler M6 interface for background data transmission initialization and allocation requests is extended to initialize and request background data transmission through two defined methods: 1. Initialization: Background data transmission can be initialized by providing the required window range to the media session handler via reference point M6. The media session handler includes this value in its dynamic policy resource request sent to the media AF. And 2. Allocation Request: Allocated background data transmission can be requested by providing the estimated data transmission amount (including duration) to the media session handler at reference point M6. The media session handler includes this value in its request sent to the media AF.

[0127] Therefore, a method for background data transmission in a 5GM media streaming architecture is also provided for downloading / uploading media streams. The client defines the time range required for future background data transmission, so the AF can respond in advance to the available windows within that time range, allowing the client to decide whether it wants to perform background data transmission within that time range, and whether it can use any available window to complete the transmission when it enters that time range in the future. Both of these steps can be triggered using two methods of the Media Session Handler M6 interface.

[0128] According to the embodiments, the functionality of M6 is further extended to achieve interoperability between 5G media applications and 5GMS media session handlers, thereby enabling more flexible and richer 5G media application deployments on various devices through the defined extended interfaces. Table 5 shows the further extensions to the parameters of the media session handlers.

[0129] Table 5

[0130]

[0131] According to an exemplary embodiment, the following are provided: 1. In terms of configuration: a. Media access information provided by _mediaAccess includes: a configuration session type, which defines whether the configuration is for uplink streaming, downlink streaming, or real-time communication (RTC); and a media entry point, which defines one or more entry points for initiating any of the above services. Edge resource configuration implemented through edgeResource obtained from the service access information is used to establish edge processing. And 2. In terms of status: a. Edge resource status information provided by edgeResourcesStatus.

[0132] According to an exemplary embodiment, in addition to _edgeResource, which is provided as part of the EDGE-5 interface, the above information can be accessed via reference point M6.

[0133] According to an embodiment, a request to update service access information is provided. That is, the method at reference point M6 is used to request the media session handler to request the latest service access information from the media AF.

[0134] According to an embodiment, a method for retrieving media access information is provided. This method is used to retrieve configuration session type and streaming media access information at reference point M6.

[0135] It also provides the ability to dismantle media distribution sessions. M4 supports starting and stopping media distribution sessions. However, we propose a novel method for releasing allocated resources and media distribution session identifiers via reference point M6 after stopping a media distribution session.

[0136] It also provides media session handler information, as shown in Table 6, which expands the media session handler status information:

[0137] Table 6

[0138]

[0139] In addition, a new list of generic notification events available at reference point M6 is provided, such as the generic media session handler notification events in Table 7:

[0140] Table 7

[0141]

[0142] Therefore, extended functionality is also provided for the 5GMS media session handler interface, where the internal parameter set is expanded to include streaming media access information, configuration session type, edge processing information, and status. The extended interface allows the media session handler to obtain the latest service access information from the mobile network or to provide the current service access information through the interface. A new method for tearing down a media distribution session is provided, through which resources are released. The status of the media distribution session is expanded to three additional statuses, where start, stop, error, or teardown statuses can be retrieved from the interface. These events are similarly expanded to three new events.

[0143] According to an embodiment, Figure 15Example 1500 of the MSE framework is shown, which is divided into two parts: MSE specification 1501 and MSE implementation 1502. The MSE implementation 1502 includes an MSE SDK abstraction 1510, an MSE SDK (platform-dependent) instantiation 1520, and an MSE microservice 1530. The MSE SDK abstraction 1510 may have a configuration API abstraction 1513, a control interface 1512, and a media interface 1511, each of which can notify the configuration API 1523 of the MSE SDK (platform-dependent) instantiation 1520 and its control interface 1522 and media interface 1521. The MSE microservice 1530 may have a configuration API 1533, a control interface 1532, and a media interface 1531.

[0144] According to an exemplary embodiment, MSE specification 1501 defines (a) media aspects and (b) MSE configuration, which provides such technical advantages. (a) Media aspects may indicate any of the following: (i) a functional description of the MSE, including mandatory and optional functions; (ii) control interfaces, such as configuration, authentication used by applications, and other functions that interact with the MSE; (iii) media interfaces including all input and output formats and protocols; (iv) network interfaces including systems and radio networks; (v) events, notifications, reports, and controls; and (vi) error handling. (b) MSE configuration may indicate: (i) an MSE description document (MDD) (which describes (1) the functions supported by the MSE implementation and their configuration parameters, and (2) performance / cost metrics for the various optional features / options); (ii) an MSE configuration API (MCA) abstraction (indicating information for (1) retrieving the description document, (2) configuring the MSE instantiation, and (3) retrieving the status and health of the MSE instantiation); and (iii) a service API for the MSE configuration API (MCA) abstraction. MSE implementation 1502 may include any of the following: (a) MSE SDK abstraction 1510 (which indicates: (i) the media aspect, conforming to the media aspect of MSE specification 1501, and (ii) the MSE Description Document (MDD) and MSE Configuration API (MCA) abstractions, the latter being an abstraction of the MSE configuration described above in MSE specification 1501); (b) MSE SDK (platform-dependent) instantiation 1520 (which indicates an SDK implementation in a specific environment and conforms to: (i) the media aspect, conforming to the media aspect of MSE specification 1501, and (ii) the MSE Description Document (MDD), and is consistent with MSE... Unlike SDK Abstraction 1530, it only contains the MSE Configuration API (MCA) abstraction, which is a specific abstraction of the MSE configuration described above in MSE Specification 1501; and (c) MSE Microservices 1510, which is an MSE implementation as a microservice and indicates (i) the media aspect, conforming to the media aspect of MSE Specification 1501, and the service API, which is oriented towards the MSE Configuration API (MCA) abstraction of the MSE Description Document (MDD) and the MSE Configuration API (MCA) abstraction of the MSE configuration described above in MSE Specification 1501. Consistency can be determined by matching formats or syntax, for example, by using flags indicating such formats or syntax.

[0145] For example, according to this favorable MSE specification (not present in the 3GPP SA4 specification), for any SDK or microservice conforming to the MSE specification, an external function or service can retrieve descriptions of features and their configuration parameters according to exemplary embodiments. This external function or service can set specific configurations for running the SDK and can retrieve the status and condition of the running SDK at any time. This status and condition can indicate availability, running, busy, idle, offline, etc.

[0146] In view of this, Figure 16 An exemplary embodiment implementation example 1600 is shown, wherein a 5G media stream downlink (5GMSd) awareness application 1405 communicates with a 5GMSd client 1402', and a media session handler 1404 in the 5GMSd client 1402' communicates with a 5GMSd AF 1414'. Method 1601, notification and error 1602, and status 1603 information can be passed between the 5GMSd awareness application 1405' and the 5GMSd client 1402', and status 1604, subscription and notification 1605, and setting and configuration information 1606 can be passed bidirectionally with the media session handler 1404, which itself can communicate with the 5GMSd AF 1414' via link M5d. According to the exemplary embodiment, for Figure 16 The MSE specification of the Media Session Handler (MSH) example 1600 shown will describe: (a) media aspects (e.g., (i) functional descriptions (including (1) service access information, (2) consumption reports, (3) metric reports, (4) dynamic policies and (5) network auxiliary information) and (ii) M5d, M6d and M7d information), and (b) MSE configuration information, such as (i) MDD (which describes: (1) an identifier indicating that the MSE conforms to the media aspects, (2) the functional description and the optional features and configuration parameters of M5d, M6d and M7d, and (3) performance / cost metrics of the different optional features / options); (ii) API abstraction (indicating information for: (1) retrieving the description document MDD, (2) configuring the MSE instantiation, and (3) retrieving the status and condition of the MSE instantiation); and (iii) the service API for the API abstraction. For example, according to an exemplary embodiment, the MSE SDK implementation of the above MSE specification (which can be implemented in Android) will support the following: (i) media aspects, conforming to the media aspects defined for MSE in the above MSE specification; and (ii) the MDD of the MSE configuration of the above MSE specification, and a concrete implementation of the API abstraction of the MSE configuration in the MSE specification for MSH.

[0147] According to an embodiment, Figure 17Example 1700 provides a simple M6 interface for interacting with a media session handler. The defined interface has very simple and limited functionality and does not define any functionality for EDGE-5. According to an embodiment, the extended functionality of M6 includes EDGE-5, and the following is defined for M6.

[0148] An application can request the allocation of edge resources via a reference point EDGE-5 / M6. This method is typically used if EDGE_ELEGIBILITY is true (TRUE). The application then knows it has the right to use these resources and can then request the allocation and instantiation of the edge resources. Similarly, an application can request the termination of edge resources by sending a request via reference point EDGE-5 / M6.

[0149] Table 8 specifies the status information that can be obtained from the media session handler through the reference point EDGE-5 / M6.

[0150] Table 8

[0151]

[0152] Table 9 provides a list of general notification events that are available at reference point EDGE-5M6.

[0153] Table 9

[0154]

[0155] Table 10 provides a list of common error events that are open at reference point M6.

[0156] Table 10

[0157]

[0158] Therefore, extended functionality is provided for edge-enabled 5GMS architecture, where applications on the device can manage and control one or more edge resources through an extended EDGE-5 / M6 interface. If the application has the right to use the edge resource, it can view the status through the API and then request to use the edge resource through the same API. The activation and deactivation of the edge resource, as well as any errors that occur when running the deployed edge resource, are notified as events.

[0159] Back Figure 14 Or, for example, return to Figure 17Even though 3GPP TS 26.512 defines the M5 interface as a pull interface, meaning the UE will need to request service access information via HTTP, the UE needs to periodically request the service access information to view updates. Therefore, this embodiment also provides a subscription method where the UE subscribes to and receives update notifications, allowing the UE to pull service access information only when new updates are available.

[0160] In other words, M5's functionality has been extended to allow the Media Session Handler (MSH) to subscribe to service access information updates. The Media AF provides a subscription link as part of the service access information to obtain updates. The MSH uses the provided link to subscribe to the service.

[0161] The Service Access Information (SAI) resource has been expanded to include a link for subscribing to SAI updates, as shown in Table 11, "Extended Definition of ServiceAccessInformation Resource":

[0162] Table 11

[0163]

[0164]

[0165]

[0166]

[0167]

[0168]

[0169]

[0170]

[0171]

[0172] The notificationURL provides the URL for MSH to subscribe to. Then, whenever SAI gets an update, MSH receives the update via an MQTT channel.

[0173] According to an embodiment, the Service Access Information (SAI) resource is extended to include a link for subscribing to SAI update notifications, as shown in Table 12, for extending the definition of the ServiceAccessInformation resource:

[0174] Table 12

[0175]

[0176] According to one embodiment, the notification only includes the notification itself and does not contain updated service access information resources. The media session handler receiving the notification then needs to request the updated service access information from the media AF.

[0177] Therefore, an efficient method for receiving service access information updates is provided, wherein a publish-subscribe channel URL is provided to the UE in the service access information, and the UE can subscribe to service access information update notifications through the URL and receive service access information updates through the channel, without needing to request the update from the media AF, or the UE receives the update notification through the publish-subscribe channel and then uses the media session handler to request the updated service access information from the media AF through M5.

[0178] like Figure 14 As shown, according to the embodiment, the functionality of M6 has also been extended to further enhance the interoperability between 5G media applications and 5GMS media session handlers, thereby enabling more flexible and richer 5G media application deployments on various devices through the defined extended interfaces.

[0179] According to the embodiment, the `subscribeBDTInfo()` method is used to subscribe to background data transmission notification events. The input and return parameters of this method are defined in Tables 13 and 14.

[0180] Table 13

[0181]

[0182] Table 14

[0183]

[0184] Table 15 provides a list of notification events related to background data transmission that are enabled at reference point M6:

[0185] Table 15

[0186]

[0187] Therefore, according to the embodiment, an extended function for the 5GMS media session handler interface is provided, through which an application on the UE can subscribe to background data transmission information notification events by instructing an external service identifier, wherein if the request is accepted, it will be acknowledged, and then the application will receive background data transmission event notifications, which means that the application can take advantage of new background data transmission opportunities.

[0188] According to the embodiments, such as Figure 14As shown, the workflow for initiating a media distribution session has also been updated, as follows: The application (App) either already has Service Access Information (SAI) through the M8 interface, or obtains and subscribes to the SAI through the M6 ​​interface using an external service identifier or a 3GPP service URL; then, the App selects a media entry point and calls the MAF using initialize() or Preload() as described in TS26.512; then the MAF calls MSH to allocate a media distribution session identifier; then the MAF streams the media content through M4; and when the MAF receives reset() or destroy(), it requests MSH to release the media distribution session identifier.

[0189] It should also be noted that, according to the embodiment, the MSH needs to maintain the latest SAI. When the MSH receives an updated SAI (via request or notification + request), it notifies the application and / or MAF. Furthermore, the MSH assigns a media distribution session identifier to the entry point when the media stream actually begins and terminates the session when the media stream actually ends.

[0190] According to embodiments, the present invention extends the functionality of M6 and M11 to further enable interoperability between 5G media application / media access functions and 5GMS media session handlers, thereby enabling more flexible and richer 5G media application deployments on various devices through the defined extended interfaces.

[0191] According to an exemplary embodiment, the internal properties of the media session handler as defined in Table 16 have also been updated and expanded.

[0192] Table 16

[0193]

[0194] Therefore, according to the embodiment, 1. in terms of configuration: a. the overall service access information resource information includes: a configuration session type, which defines whether the configuration is for uplink streaming, downlink streaming, or real-time communication (RTC); and a media entry point, which defines one or more entry points for initiating any of the above services and other information. And b. a media distribution session identifier, which identifies the media session if a media session has been established. And 2. in terms of status: a. edge resource status information. According to the embodiment, in addition to _edgeResource provided as part of the EDGE-5 interface, this information can be accessed through reference point M6.

[0195] It also provides functionality to obtain service access information. Before initiating a media distribution session, 5GMS-aware applications or media access functions typically request service access information at reference points M6 or M11, respectively. Furthermore, these APIs can be used to retrieve the latest service access information and media distribution session identifiers.

[0196] The `getServiceAccessInformation()` method is used to request the media session handler to retrieve the latest service access information from the media access function (AF). The input and return parameters of this method are defined in Tables 17 (Input Parameters of the `getServiceAccessInformation()` Method) and 18 (Return Value of the `getServiceAccessInformation()` Method). Optionally, media-aware applications or media access functions can subscribe to events that provide service access information update notifications.

[0197] Table 17

[0198]

[0199] Table 18

[0200]

[0201] According to an embodiment, the `subscribeMediaAccessInformation()` method is used to subscribe to the service access information. A notification event is emitted whenever an update to the service access information becomes available. The input and return parameters of this method are defined in Table 19 (Input Parameters of the `subscribeMediaAccessInformation()` Method) and Table 20 (Return Values ​​of the `subscribeMediaAccessInformation()` Method):

[0202] Table 19

[0203]

[0204] Table 20

[0205]

[0206] Furthermore, according to an embodiment, a `requestDeliveryIdentifier()` method is provided for requesting the initiation of a media delivery session in a media session handler and obtaining the associated media delivery session identifier. The input and return parameters of this method are defined in Tables 21 (Input Parameters of the `requestDeliveryIdentifier()` Method) and 22 (Return Value of the `requestDeliveryIdentifier()` Method):

[0207] Table 21

[0208]

[0209] Table 22

[0210]

[0211] According to an embodiment, a `releaseDeliveryIdentifier()` method is also provided, which is used to release resources allocated for an assigned media distribution session identifier in a media session handler. Through this call, the media session handler does not maintain internal properties corresponding to the media distribution session identifier. The input and return parameters of this method are defined in Table 23 (Input Parameters of the `releaseDeliveryIdentifier()` Method) and Table 24 (Return Value of the `releaseDeliveryIdentifier()` Method):

[0212] Table 23

[0213]

[0214] Table 24

[0215]

[0216] According to the embodiment, updated general media session handler information is also provided. Table 25 (General Media Session Handler Status Information) specifies the status information that can be obtained from the media session handler through reference points M6 and M11.

[0217] Table 25

[0218]

[0219] Therefore, extended functionality is provided for the 5GMS media session handler interface, wherein the internal parameter set is extended to include streaming media access information and a media distribution session identifier, as well as edge processing information and status. The extended interface allows the media session handler to be instructed to obtain the latest service access information from the mobile network or to provide the current service access information through the interface, or to subscribe to service access information notification events and obtain the events so that it can request updated service access information. The media distribution session is initiated by the media session handler assigning a media distribution session identifier upon request from an application or media access function. A novel method for dismantling a media distribution session is provided, by which resources are released, also upon request from an application or media access function.

[0220] According to an embodiment, the Media Session Handler (MSH) maintains the latest Service Access Information (SAI), and when the MSH receives an updated SAI (either via request or notification + request), it notifies the application (App) and / or the Media Access Function (MAF). A media distribution session begins with the MSH assigning a Media Distribution Session Identifier (MDSI). The media distribution session ends with the MSH releasing the MDSI.

[0221] The workflow for initiating a media distribution session is updated in each embodiment as follows: 1. The app can optionally have a SAI, an external service ID, or a 3GPP service URL. 2. The app calls MSH via M6 to update the SAI and provides the information it possesses. 3. MSH initiates the media distribution session by allocating MDSI and provides it back to the app via M6. 4. The app calls MAF via M7 and provides MDSI. 5. At a later point, when MAF wants to transmit content, it uses MSDI and streams the content via M4. 6. When MAF receives a stop or reset playback command from the app, it requests MSH to release MSDI. 7. MSH releases MSDI and notifies MAF or the app that the media distribution session has ended.

[0222] The embodiments extend the functionality of M6 and M11 to further enhance interoperability between 5G media applications / media access functions and 5GMS media session handlers, thereby enabling more flexible and richer 5G media application deployments on various devices through the defined extended interfaces.

[0223] The `requestDeliveryIdentifier()` method is used to initiate a media delivery session in a media session handler and obtain the associated media delivery session identifier. The input and return parameters of this method are defined in Tables 26 (Input Parameters of the `requestDeliveryIdentifier()` Method) and 27 (Return Value of the `requestDeliveryIdentifier()` Method). Note: Media-aware applications or media access functions can also subscribe to events that provide service access information update notifications.

[0224] Table 26

[0225]

[0226] Table 27

[0227]

[0228] The `subscribeMediaAccessInformation()` method is used to subscribe to service access information. A notification event is emitted whenever an update to the service access information becomes available. The input and return parameters of this method are defined in Table 28 (Input Parameters of the `subscribeMediaAccessInformation()` Method) and Table 29 (Return Value of the `subscribeMediaAccessInformation()` Method).

[0229] Table 28

[0230]

[0231] Table 29

[0232]

[0233] The `releaseDeliveryIdentifier()` method is used in a media session handler to release the resources allocated for the assigned media distribution session identifier. Through this call, the media session handler does not maintain internal properties corresponding to the media distribution session identifier. The input and return parameters of this method are defined in Table 30 (Input Parameters of the `releaseDeliveryIdentifier()` Method) and Table 31 (Return Value of the `releaseDeliveryIdentifier()` Method).

[0234] Table 30

[0235]

[0236] Table 31

[0237]

[0238] According to the embodiment, Table 32 (General Media Session Processor State Information) defines the state information that can be obtained from the media session processor through reference points M6 and M11:

[0239] Table 32

[0240]

[0241] Therefore, extended functionality is provided for the 5GMS media session handler interfaces M6 and M11, whereby the extended interface allows a request to the media session handler to initiate a media distribution session. The media session handler assigns an identifier to the session and returns it, which means that the media distribution session is started. Whenever new updates to service access information become available, the media session handler notifies the application or media access function that it can obtain the updates. When the media distribution session ends, the application or media access function can request the media session handler to terminate the session. The media session handler cancels the assignment of the identifier and notifies the requester, thus terminating the media distribution session.

[0242] The above-described technologies can be implemented in the form of computer software using computer-readable instructions, and can be physically stored in one or more computer-readable media, or implemented by one or more specially configured hardware processors. For example, Figure 18 A computer system 1800 suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0243] Computer software can be encoded using any suitable machine code or computer language, which can be assembled, compiled, linked, or similar mechanisms to create code containing instructions that can be executed directly by a computer's central processing unit (CPU), graphics processing unit (GPU), or through interpretation, microcode execution, or other means.

[0244] These instructions can be executed on various types of computers or their components, including, for example, personal computers, tablets, servers, smartphones, gaming devices, IoT devices, etc.

[0245] Figure 18 The components of the computer system 1800 shown are merely illustrative and are not intended to impose any limitation on the scope or functionality of the computer software used to implement the embodiments of this disclosure. Nor should the configuration of the components be construed as having any dependency or requirement on one or more of the components shown in the exemplary embodiments of the computer system 1800.

[0246] Computer system 1800 may include certain human-machine interface input devices. Such human-machine interface input devices may respond to input from one or more human users through, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), and olfactory input (not shown). Human-machine interface devices may also be used to capture media that are not necessarily directly related to human conscious input, such as audio (e.g., voice, music, ambient sounds), images (e.g., scanned images, photographic images acquired from still image cameras), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0247] Input human-machine interface devices may include one or more of the following (only one of each is depicted): keyboard 1801, mouse 1802, touchpad 1803, touch screen 1810, joystick 1805, microphone 1806, scanner 1808, and camera 1807.

[0248] Computer system 1800 may also include certain human-machine interface output devices. Such human-machine interface output devices can stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell / taste. These human-machine interface output devices may include tactile output devices (e.g., tactile feedback via touchscreen 1810 or joystick 1805, but tactile feedback devices not used as input devices may also exist), audio output devices (e.g., speakers 1809, headphones (not shown)), visual output devices (e.g., screen 1810, including CRT screens, LCD screens, plasma screens, OLED screens (each screen may or may not have touchscreen input capability, each screen may or may not have tactile feedback capability), some of which are capable of outputting two-dimensional or higher-dimensional visual outputs through, for example, stereoscopic output; virtual reality glasses (not shown), holographic displays and smoke boxes (not shown), and printers (not shown).

[0249] The computer system 1800 may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW 1820 with CD / DVD 1811 or similar media, USB flash drives 1822, removable hard drives or solid-state drives 1823, conventional magnetic media such as magnetic tapes and floppy disks (not shown), dedicated ROM / ASIC / PLD devices such as security dongles (not shown), etc.

[0250] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the subject matter currently disclosed does not cover transmission media, carrier waves, or other volatile signals.

[0251] Computer system 1800 may also include an interface 1899 for one or more communication networks 1898. Network 1898 may be, for example, wireless, wired, or optical. Network 1898 may further be a local area network (LAN), wide area network (WAN), metropolitan area network (MAN), vehicular and industrial network, real-time network, latency-tolerant network, etc. Examples of network 1898 include, for example, LANs such as Ethernet and wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., wired or wireless wide area digital networks including cable TV, satellite TV, and terrestrial broadcast TV, vehicular and industrial networks including CANbus, etc. Some networks 1898 typically require external network interface adapters that are attached to certain general-purpose data ports or peripheral buses (1850 and 1851) (e.g., the USB port of computer system 1800); other networks are typically integrated into the core of computer system 1800 by being attached to system buses as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks 1898, computer system 1800 can communicate with other entities. This communication can be one-way (e.g., broadcasting TV), one-way (e.g., CANbus to certain CANbus devices), or bidirectional (e.g., communicating with other computer systems using a local area digital network or a wide area digital network). Certain protocols and protocol stacks can be used on each of those networks and network interfaces as described above.

[0252] The aforementioned human-machine interface devices, human-accessible storage devices, and network interfaces can be attached to the core 1840 of the computer system 1800.

[0253] The core 1840 may include one or more central processing units (CPUs) 1841, graphics processing units (GPUs) 1842, graphics adapters 1817, dedicated programmable processing units in the form of field-programmable gate arrays (FPGAs) 1843, hardware accelerators 1844 for certain tasks, etc. These devices, as well as read-only memory (ROM) 1845, random access memory 1846, and internal mass storage devices 1847 such as internal non-user-accessible hard disk drives (HDDs), SSDs, etc., can be connected via a system bus 1848. In some computer systems, the system bus 1848 may be accessed as one or more physical connectors to allow for expansion via additional CPUs, GPUs, etc. Peripheral devices can be directly attached to the core's system bus 1848, or attached to the core's system bus 1848 via a peripheral bus 1851. Architectures for the peripheral bus include PCI, USB, etc.

[0254] The CPU 1841, GPU 1842, FPGA 1843, and accelerator 1844 can execute certain instructions, which, when combined, constitute the aforementioned computer code. This computer code can be stored in ROM 1845 or RAM 1846. Transient data can also be stored in RAM 1846, while permanent data can be stored, for example, in an internal mass storage device 1847. Fast storage and retrieval of any memory device can be achieved by using a cache memory, which can be closely associated with one or more CPUs 1841, GPUs 1842, mass storage devices 1847, ROM 1845, RAM 1846, etc.

[0255] Computer-readable media may have computer code thereon for performing operations of various computer implementations. The media and computer code may be media and computer code specifically designed and constructed for the purposes of this disclosure, or they may be of a type well known and available to those skilled in the art of computer software.

[0256] By way of example and not limitation, the architecture corresponding to computer system 1800 (and particularly core 1840) can be functionally provided by one or more processors (including CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible, computer-readable media. Such computer-readable media can be media associated with user-accessible mass storage as described above and certain memories of core 1840 (e.g., internal mass storage 1847 or ROM 1845) having non-volatile properties. Software implementing the various embodiments of this disclosure can be stored in such devices and executed by core 1840. Depending on specific needs, the computer-readable media may include one or more memory devices or chips. The software can cause core 1840, and particularly the processors therein (including CPU, GPU, FPGA, etc.), to execute specific processes or specific portions of specific processes described herein, including defining data structures stored in RAM 1846 and modifying such data structures according to software-defined processes. Alternatively or concurrently, a computer system may provide functionality through hard-wired or otherwise embodied logic in circuitry (e.g., accelerator 1844), which may replace or operate with software to perform the specific process or a specific portion of the specific process described herein. Where appropriate, references to software may cover logic, and vice versa. Where appropriate, references to computer-readable media may cover circuitry storing software for execution (e.g., integrated circuits (ICs)), circuitry embodying logic for execution, or both. This disclosure covers any suitable combination of hardware and software.

[0257] While several exemplary embodiments have been described in this disclosure, there are changes, substitutions, and various alternative equivalents that fall within the scope of this disclosure. Therefore, it should be understood that those skilled in the art will be able to design numerous systems and methods that, while not expressly shown or described herein, embody the principles of this disclosure and are therefore within its spirit and scope.

Claims

1. A video data transmission method, characterized in that, The method is executed by at least one processor and includes: Acquire augmented reality (AR) data for media components, wherein the media components include at least one of audio and video; To determine the following information for 5G Media Streaming (5GMS) background data transmission (BDT): day of the week, and the number of occurrences or end date; and Indications based on resources of the 5GMS Media Session Handler (MSH), indicating one of the states of being started, stopped, or removed, are controlled by the 5GMS BDT based on the day of the week and the determination of the number of occurrences or end dates, to control the media streaming transmission of the media component.

2. The method according to claim 1, characterized in that, in, Determining the day of the week and the number of occurrences or end date includes a call flow where the 5GMS client indicates a future time range, the 5GMS Application Function (AF) provides the 5GMS client with one or more windows within the future time range, and the 5GMS client selects a window from the one or more windows within the future time range, the window being the time when the 5GMS BDT is implemented.

3. The method according to claim 2, characterized in that, The resource indication of the 5GMS MSH is an indication of the edge resource status of the application through the EDGE-5 / M6 application programming interface (API).

4. The method according to claim 3, characterized in that, in, The UE's service access information is updated based on the user equipment (UE)'s subscription through the publish-subscribe channel URL, without the UE requesting an update from the 5GMS AF, and the UE's subscription includes either automatic updates of the UE's service access information or requests initiated by the UE through the M5 interface.

5. The method according to claim 4, characterized in that, The subscription is the subscription made by the UE to the 5GMS BDT.

6. The method according to claim 4, characterized in that, in, The 5GMS MSH interface is configured to respond to requests for streaming media access information, media distribution session identifiers, edge processing information, and to acquire subscribers.

7. The method according to claim 6, characterized in that, The 5GMS BDT is configured to boot via either the EDGE-5 / M6 interface or the M11 interface.

8. A video data transmission device, characterized in that, The device includes: Memory for storing instructions; and At least one processor, the at least one processor being configured to execute the instructions to implement: Acquire augmented reality (AR) data for media components, wherein the media components include at least one of audio and video; To determine the following information for 5G Media Streaming (5GMS) background data transmission (BDT): day of the week, and the number of occurrences or end date; and Indications based on resources of the 5GMS Media Session Handler (MSH), indicating one of the states of being started, stopped, or removed, are controlled by the 5GMS BDT based on the day of the week and the determination of the number of occurrences or end dates, to control the media streaming transmission of the media component.

9. The apparatus according to claim 8, characterized in that, Determining the day of the week and the number of occurrences or end date includes a call flow where the 5GMS client indicates a future time range, the 5GMS Application Function (AF) provides the 5GMS client with one or more windows within the future time range, and the 5GMS client selects a window from the one or more windows within the future time range, the window being the time when the 5GMS BDT is implemented.

10. The apparatus according to claim 9, characterized in that, The indication of the resources of the 5GMS MSH is an indication of the edge resource status of the application via the EDGE-5 / M6 application programming interface (API).

11. The apparatus according to claim 10, characterized in that, The service access information of the user equipment (UE) is updated based on the subscription of the publish-subscribe channel URL, without the UE requesting the update from the 5GMS AF, and the subscription of the UE includes either automatic updates of the service access information of the UE or requests initiated by the UE through the M5 interface.

12. The apparatus according to claim 11, characterized in that, The subscription is the subscription made by the UE to the 5GMS BDT.

13. The apparatus according to claim 11, characterized in that, The 5GMS MSH interface is configured to respond to requests for streaming media access information, media distribution session identifiers, edge processing information, and to acquire subscribers.

14. The apparatus according to claim 13, characterized in that, The 5GMS BDT is configured to boot via either the EDGE-5 / M6 interface or the M11 interface.

15. A non-volatile computer-readable medium storing instructions, characterized in that, The instructions, when executed by the computer, cause the computer to perform the following: Acquire augmented reality (AR) data for media components, wherein the media components include at least one of audio and video; To determine the following information for 5G Media Streaming (5GMS) background data transmission (BDT): day of the week, and the number of occurrences or end date; and Indications based on resources of the 5GMS Media Session Handler (MSH), indicating one of the states of being started, stopped, or removed, are controlled by the 5GMS BDT based on the day of the week and the determination of the number of occurrences or end dates, to control the media streaming transmission of the media component.

16. The non-volatile computer-readable medium according to claim 15, characterized in that, The determination of the day of the week and the number of occurrences or end date includes a call flow, wherein the 5GMS client indicates a future time range, the 5GMS Application Function (AF) provides the 5GMS client with one or more windows within the future time range, and the 5GMS client selects a window from the one or more windows within the future time range, the window being the time for implementing the 5GMS BDT.

17. The non-volatile computer-readable medium according to claim 16, characterized in that, The resource indication of the 5GMS MSH is an indication of the edge resource status of the application through the EDGE-5 / M6 application programming interface (API).

18. The non-volatile computer-readable medium according to claim 17, characterized in that, The service access information of the user equipment (UE) is updated based on the subscription of the publish-subscribe channel URL, without the UE requesting the update from the 5GMSAF, and the subscription of the UE includes either automatic updates of the service access information of the UE or requests initiated by the UE through the M5 interface.

19. The non-volatile computer-readable medium according to claim 18, characterized in that, The subscription is the subscription made by the UE to the 5GMS BDT.

20. The non-volatile computer-readable medium according to claim 18, characterized in that, The 5GMS MSH interface is configured to respond to requests for streaming media access information, media distribution session identifiers, edge processing information, and subscriber acquisition, and the 5GMS BDT is configured to be launched via either the EDGE-5 / M6 interface or the M11 interface.

Images