Media processing method, device, and medium

Abstract

The embodiments of the present disclosure provide a scheme for media processing and propose a media processing method. The method includes performing a conversion between a media file of media and a media presentation of the media, the media presentation including a Main Stream Representation (MSR) and an External Stream Representation (ESR) associated with the MSR. A file conforming to a file format standard is generated by concatenating a first segment in the ESR, a second segment in the MSR corresponding to the first segment, and a subsequent segment following the second segment in the MSR. Compared with the conventional scheme, the proposed method has an advantage that it can more efficiently support Extended Dependent Random Access Point (EDRAP) based technology.

Description

[Technical Field]

[0001] FIELD Embodiments of the present disclosure relate generally to media coding techniques, and more particularly to an improved design for streaming based on Main Stream Representation (MSR) and External Stream Representation (ESR).

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 63 / 276,442, filed November 5, 2021, the entire contents of which are incorporated herein by reference. [Background technology]

[0003] Media streaming applications are typically based on Internet Protocol (IP), Transmission Control Protocol (TCP), and Hypertext Transfer Protocol (HTTP) transport methods and typically rely on file formats such as the International Organization for Standardization-based Media File Format (ISOBMFF). One such streaming system is HTTP-based dynamic adaptive streaming (DASH). In DASH, multiple representations of the video and / or audio data of multimedia content can exist, with the different representations corresponding to different coding characteristics (e.g., different profiles or levels of a video coding standard, different bit rates, different spatial resolutions, etc.). Extended dependent random access point (EDRAP) picture-based video coding and streaming has also been proposed. Therefore, MSR- and ESR-based streaming are worth investigating. Summary of the Invention

[0004] Embodiments of the present disclosure provide a solution for media processing.

[0005] In a first aspect, a media processing method is proposed, the method comprising: performing a conversion between a media file of media and a media presentation of the media, the media presentation including a Main Stream Representation (MSR) and an External Stream Representation (ESR) associated with the MSR, wherein a file conforming to a file format standard is generated by concatenation of a first segment in the ESR, a second segment in the MSR corresponding to the first segment, and a subsequent segment following the second segment in the MSR.

[0006] According to the method of the first aspect of the present disclosure, a file obtained by concatenating a segment in the ESR, its corresponding segment in the MSR, and all subsequent segments in the MSR conforms to a file format standard. Compared to conventional approaches that do not require such files to conform to a file format standard, the proposed method has the advantage of more efficiently supporting EDRAP-based random access techniques.

[0007] In a second aspect, an apparatus for processing media data is proposed, the apparatus comprising a processor and a non-transitory memory comprising instructions that, when executed by the processor, cause the processor to perform a method according to the first aspect of the present disclosure.

[0008] In a third aspect, a non-transitory computer-readable storage medium is proposed, the non-transitory computer-readable storage medium storing instructions for causing a processor to perform the method according to the first aspect of the present disclosure.

[0009] In a fourth aspect, another non-transitory computer-readable storage medium is proposed, the non-transitory computer-readable storage medium storing a media file of media generated by a method performed by a media processing device, the method including performing a conversion between the media file and a media presentation of the media, the media presentation including a Main Stream Representation (MSR) and an External Stream Representation (ESR) associated with the MSR, wherein a file conforming to a file format standard is generated by concatenating a first segment in the ESR, a second segment in the MSR corresponding to the first segment, and a subsequent segment following the second segment in the MSR.

[0010] In a fifth aspect, a method for storing a media file of media is proposed, the method comprising: performing a conversion between the media file and a media presentation of the media; and storing the media file in a non-transitory computer-readable storage medium, the media presentation including a Main Stream Representation (MSR) and an External Stream Representation (ESR) associated with the MSR, wherein concatenation of a first segment in the ESR, a second segment in the MSR corresponding to the first segment, and a subsequent segment following the second segment in the MSR generates a file conforming to a file format standard.

[0011] In a sixth aspect, another non-transitory computer-readable storage medium is proposed. The non-transitory computer-readable storage medium stores a media presentation of media generated by a method performed by a media processing device, the method including performing a conversion between a media file of the media and the media presentation. The media presentation includes a Main Stream Representation (MSR) and an External Stream Representation (ESR) associated with the MSR, and a file conforming to a file format standard is generated by concatenating a first segment in the ESR, a second segment in the MSR corresponding to the first segment, and a subsequent segment following the second segment in the MSR.

[0012] In a seventh aspect, a method for storing a media presentation of media is proposed, the method comprising: performing a conversion between a media file of the media and the media presentation; and storing the media presentation in a non-transitory computer-readable storage medium, the media presentation including a Main Stream Representation (MSR) and an External Stream Representation (ESR) associated with the MSR, wherein concatenation of a first segment in the ESR, a second segment in the MSR corresponding to the first segment, and a subsequent segment following the second segment in the MSR generates a file conforming to a file format standard.

[0013] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. [Brief explanation of the drawings]

[0014] The above and other objects, features, and advantages of exemplary embodiments of the present disclosure will become more apparent through the following detailed description taken in conjunction with the accompanying drawings, in which like reference numerals generally refer to like components. [Figure 1] 1 illustrates a block diagram of an exemplary video coding system according to some embodiments of the present disclosure. [Figure 2] 1 shows a block diagram of an exemplary video encoder according to some embodiments of the present disclosure. [Figure 3] 1 shows a block diagram of an exemplary video decoder according to some embodiments of the present disclosure. [Figure 4] FIG. 1 is a diagram for explaining a random access point (RAP for short). [Figure 5] FIG. 10 is another diagram for explaining the concept of RAP. [Figure 6] FIG. 1 is a diagram illustrating the concept of a dependent random access point (DRAP). [Figure 7] FIG. 10 is another diagram for explaining the concept of DRAP. [Figure 8] FIG. 1 is a diagram illustrating the concept of an Enhanced Dependent Random Access Point (EDRAP). [Figure 9] FIG. 10 is another diagram for explaining the concept of EDRAP. [Figure 10] FIG. 1 is a diagram illustrating EDRAP-based video streaming. [Figure 11] FIG. 10 is another diagram for explaining EDRAP-based video streaming. [Figure 12] 1 illustrates a flowchart of a media processing method according to some embodiments of the present disclosure. [Figure 13] FIG. 1 illustrates a block diagram of a computing device capable of implementing various embodiments of the present disclosure.

[0015] Throughout the drawings, the same or similar reference numbers typically refer to the same or similar elements. DETAILED DESCRIPTION OF THE INVENTION

[0016] The principles of the present disclosure will be described below with reference to several embodiments. These embodiments are merely described for the purpose of illustration and to help those skilled in the art understand and practice the present disclosure, and should not be construed as implying any limitations on the scope of the present disclosure. The disclosure described herein can be implemented in various ways other than those described below.

[0017] In the following description and claims, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0018] References in this disclosure to "one embodiment," "one embodiment," "exemplary embodiment," etc. indicate that the described embodiment may include a particular feature, structure, or characteristic, but not all embodiments necessarily include the particular feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in connection with an exemplary embodiment, it is noted that it is within the knowledge of one skilled in the art to affect such feature, structure, or characteristic in connection with other embodiments, whether or not explicitly stated.

[0019] Terms such as "first" and "second" may be used herein to describe various elements, but it should be understood that these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, a first element could be referred to as a second element. Similarly, a second element could be referred to as a first element without departing from the scope of the exemplary embodiments. As used herein, the term "and / or" includes any and all combinations of one or more of the listed terms.

[0020] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit example embodiments. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly dictates otherwise. It will be further understood that the terms "comprise," "comprise," "have," "have," "include," and / or "comprise," when used herein, specify the presence of stated features, elements, and / or components, etc., but do not exclude the presence or addition of one or more other features, elements, components, and / or combinations thereof.

[0021] Example Environment 1 is a block diagram illustrating an example video coding system 100 that may utilize the techniques of this disclosure. As shown, video coding system 100 may include a source device 110 and a destination device 120. Source device 110 may also be referred to as a video encoding device. Destination device 120 may also be referred to as a video decoding device. In operation, source device 110 may be configured to generate encoded video data, and destination device 120 may be configured to decode the encoded video data generated by source device 110. Source device 110 may include a video source 112, a video encoder 114, and an input / output (I / O) interface 116.

[0022] Video source 112 may include sources such as video capture devices. Examples of video capture devices include, but are not limited to, an interface that receives video data from a video content provider, a computer graphics system that generates video data, and / or combinations thereof.

[0023] The video data may include one or more pictures. The video encoder 114 encodes the video data from the video source 112 to generate a bitstream. The bitstream may include a series of bits forming a coded representation of the video data. The bitstream may include coded pictures and associated data. A coded picture is a coded representation of a picture. The associated data may include sequence parameter sets, picture parameter sets, and other syntax structures. The I / O interface 116 may include a modulator / demodulator and / or a transmitter. The encoded video data may be transmitted directly to the destination device 120 via the I / O interface 116 over the network 130A. The encoded video data may be stored on a storage medium / server 130B for access by the destination device 120.

[0024] Destination device 120 may include an I / O interface 126, a video decoder 124, and a display device 122. I / O interface 126 may include a receiver and / or a modem. I / O interface 126 may obtain encoded video data from source device 110 or storage medium / server 130B. Video decoder 124 may decode the encoded video data. Display device 122 may display the decoded video data to a user. Display device 122 may be integrated with destination device 120 or may be external to destination device 120 and configured to interface with an external display device.

[0025] Video encoder 114 and video decoder 124 may operate according to a video compression standard, such as the High Efficiency Video Coding (HEVC) standard, the Versatile Video Coding (VVC) standard, and other current and / or future standards.

[0026] FIG. 2 is a block diagram illustrating an example of a video encoder 200, which is an example of the video encoder 114 in the system 100 shown in FIG. 1, according to some embodiments of the present disclosure.

[0027] Video encoder 200 may be configured to implement any or all of the techniques of this disclosure. In the example of FIG. 2, video encoder 200 includes multiple functional components. The techniques described in this disclosure may be shared among various components of video encoder 200. In some examples, a processor may be configured to perform any or all of the techniques described in this disclosure.

[0028] In some embodiments, the video encoder 200 may include a division unit 201, a prediction unit 202, which may include a mode selection unit 203, a motion estimation unit 204, a motion compensation unit 205, and an intra prediction unit 206, a residual generation unit 207, a transform unit 208, a quantization unit 209, an inverse quantization unit 210, an inverse transform unit 211, a reconstruction unit 212, a buffer 213, and an entropy coding unit 214.

[0029] In other examples, video encoder 200 may include more, fewer, or different functional components. In one example, prediction unit 202 may include an intra block copy (IBC) unit. The IBC unit may perform prediction in an IBC mode in which at least one reference picture is the picture in which the current video block is located.

[0030] Furthermore, some components, such as the motion estimation unit 204 and the motion compensation unit 205, may be integrated but are depicted separately in the example of FIG. 2 for illustrative purposes.

[0031] Partition unit 201 may divide a picture into one or more video blocks. Video encoder 200 and video decoder 300 may support a variety of video block sizes.

[0032] The mode select unit 203 may, for example, select one of an intra-encoding mode or an inter-coding mode based on an error result, provide the resulting intra-coded or inter-coded block to the residual generation unit 207 to generate residual block data, and provide the encoded block to the reconstruction unit 212 to reconstruct and use as a reference picture. In some examples, the mode select unit 203 may select a combined intra-prediction and inter-prediction (CIIP) mode in which prediction is based on an inter-prediction signal and an intra-prediction signal. In the case of inter-prediction, the mode select unit 203 may select a resolution (e.g., sub-pixel or integer-pixel precision) of a motion vector for a block.

[0033] To perform inter prediction on a current video block, motion estimation unit 204 may generate motion information for the current video block by comparing one or more reference frames from buffer 213 with the current video block. Motion compensation unit 205 may determine a predictive video block for the current video block based on the motion information and decoded samples of pictures from buffer 213 other than the picture associated with the current video block.

[0034] Motion estimation unit 204 and motion compensation unit 205 may perform different operations on a current video block depending, for example, on whether the current video block is in an I slice, a P slice, or a B slice. As used herein, an "I slice" may refer to a portion of a picture composed of macroblocks, all of which are based on macroblocks within the same picture. Additionally, as used herein, in some aspects, "P slice" and "B slice" may refer to a portion of a picture composed of macroblocks that are independent of macroblocks within the same picture.

[0035] In some examples, motion estimation unit 204 may perform unidirectional prediction on the current video block, and motion estimation unit 204 may look up a reference picture in list 0 or list 1 for a reference video block of the current video block. Motion estimation unit 204 may then generate a reference index indicating a reference picture in list 0 or list 1 that contains the reference video block, and a motion vector indicating a spatial displacement between the current video block and the reference video block. Motion estimation unit 204 may output the reference index, prediction direction indicator, and motion vector as motion information for the current video block. Motion compensation unit 205 may generate a predictive video block for the current video block based on the reference video block indicated by the motion information of the current video block.

[0036] Alternatively, in other examples, motion estimation unit 204 may perform bidirectional prediction on the current video block. Motion estimation unit 204 may search a reference picture in list 0 for a reference video block of the current video block, or may search a reference picture in list 1 for another reference video block of the current video block. Motion estimation unit 204 may then generate a reference index indicating the reference picture in list 0 and list 1 that contains the reference video block, and a motion vector indicating the spatial displacement between the reference video block and the current video block. Motion estimation unit 204 may output the reference index and the motion vector for the current video block as motion information for the current video block. Motion compensation unit 205 may generate a predictive video block for the current video block based on the reference video block indicated by the motion information of the current video block.

[0037] In some examples, motion estimation unit 204 may output a full set of motion information for the decoder's decoding process. Alternatively, in some embodiments, motion estimation unit 204 may signal the motion information of the current video block by reference to the motion information of another video block. For example, motion estimation unit 204 may determine that the motion information of the current video block is sufficiently similar to the motion information of a neighboring video block.

[0038] In one example, motion estimation unit 204 may indicate, in a syntax structure associated with the current video block, a value that indicates to video decoder 300 that the current video block has the same motion information as another video block.

[0039] In another example, motion estimation unit 204 may identify another video block and a motion vector differential (MVD) in a syntax structure associated with the current video block. The motion vector differential indicates the difference between the motion vector of the current video block and the motion vector of the indicated video block. Video decoder 300 may use the motion vector and the motion vector differential of the indicated video block to determine the motion vector of the current video block.

[0040] As mentioned above, video encoder 200 may predictively signal motion vectors. Two examples of predictive signaling techniques that may be implemented by video encoder 200 include advanced motion vector prediction (AMVP) and merge mode signaling.

[0041] Intra prediction unit 206 may perform intra prediction on the current video block. When intra prediction unit 206 performs intra prediction on the current video block, intra prediction unit 206 may generate predictive data for the current video block based on decoded samples of other video blocks within the same picture. The predictive data for the current video block may include a predicted video block and various syntax elements.

[0042] Residual generation unit 207 may generate residual data for the current video block by subtracting (e.g., as indicated by a minus sign) the prediction video block of the current video block from the current video block. The residual data for the current video block may include residual video blocks that correspond to different sample components of the samples in the current video block.

[0043] In other examples, residual data for the current video block may not be present, for example, in skip mode, and residual generation unit 207 may not perform the subtraction operation.

[0044] Transform processing unit 208 may generate one or more transform coefficient video blocks for the current video block by applying one or more transforms to the residual video block associated with the current video block.

[0045] After transform processing unit 208 generates the transform coefficient video block associated with the current video block, quantization unit 209 may quantize the transform coefficient video block associated with the current video block based on one or more quantization parameter (QP) values ​​associated with the current video block.

[0046] Inverse quantization unit 210 and inverse transform unit 211 may apply inverse quantization and inverse transform, respectively, to the transform coefficient video block to reconstruct a residual video block from the transform coefficient video block. Reconstruction unit 212 may add the reconstructed residual video block to corresponding samples from one or more prediction video blocks generated by prediction unit 202 to generate a reconstructed video block associated with the current video block for storage in buffer 213.

[0047] After reconstruction unit 212 reconstructs the video blocks, a loop filtering operation may be performed to reduce video blocking artifacts in the video blocks.

[0048] The entropy encoding unit 214 may receive data from other functional components of the video encoder 200. Once the entropy encoding unit 214 receives the data, the entropy encoding unit 214 may perform one or more entropy encoding operations to generate entropy-coded data and output a bitstream including the entropy-coded data.

[0049] FIG. 3 is a block diagram illustrating an example of a video decoder 300, which is an example of the video decoder 124 in the system 100 shown in FIG. 1, according to some embodiments of the present disclosure.

[0050] Video decoder 300 may be configured to perform any or all of the techniques of this disclosure. In the example of FIG. 3, video decoder 300 includes multiple functional components. The techniques described in this disclosure may be shared among various components of video decoder 300. In some examples, a processor may be configured to perform any or all of the techniques described in this disclosure.

[0051] 3, video decoder 300 includes an entropy decoding unit 301, a motion compensation unit 302, an intra prediction unit 303, an inverse quantization unit 304, an inverse transform unit 305, a reconstruction unit 306, and a buffer 307. Video decoder 300 may, in some examples, perform a decoding pass that is generally inverse to the encoding pass described with respect to video encoder 200.

[0052] The entropy decoding unit 301 may retrieve an encoded bitstream. The encoded bitstream may include entropy-encoded video data (e.g., encoded blocks of video data). The entropy decoding unit 301 may decode the entropy-encoded video data. The motion compensation unit 302 may then determine motion information from the entropy-decoded video data, including motion vectors, motion vector precision, reference picture list indexes, and other motion information. The motion compensation unit 302 may determine such information by, for example, performing AMVP and merge mode. AMVP is used and includes deriving several most likely candidates based on data from neighboring PB and reference pictures. The motion information typically includes horizontal and vertical motion vector displacement values, one or two reference picture indexes, and, for prediction regions in B slices, identification of which reference picture list is associated with each index. As used herein, in some aspects, "merge mode" may refer to deriving motion information from spatially or temporally neighboring blocks.

[0053] The motion compensation unit 302 generates a motion-compensated block, which can perform interpolation based on an interpolation filter. An identifier of the interpolation filter used with sub-pixel accuracy can be included in a syntax element.

[0054] Motion compensation unit 302 may calculate interpolated values ​​for sub-integer pixels of the reference block using interpolation filters used by video encoder 200 during encoding of the video block. Motion compensation unit 302 may determine the interpolation filters used by video encoder 200 according to received syntax information and use the interpolation filters to generate the predictive block.

[0055] The motion compensation unit 302 may use at least a portion of the syntax information to determine the size of blocks used to encode frames and / or slices of the encoded video sequence, partition information describing how each macroblock of a picture of the encoded video sequence is divided, a mode indicating how each partition is encoded, one or more reference frames (and a reference frame list) for each inter-encoded block, and other information for decoding the encoded video sequence. As used herein, in some aspects, a "slice" may refer to a data structure that can be decoded separately from other slices of the same picture with respect to entropy coding, signal prediction, and residual signal reconstruction. A slice can be either an entire picture or a region of a picture.

[0056] The intra prediction unit 303 may form a prediction block from spatially adjacent blocks, for example, using an intra prediction mode received in the bitstream. The inverse quantization unit 304 inverse quantizes, or dequantizes, the quantized video block coefficients provided in the bitstream and decoded by the entropy decoding unit 301. The inverse transform unit 305 applies an inverse transform.

[0057] Reconstruction unit 306 may obtain a decoded block, for example, by adding the residual block and the corresponding prediction block generated by motion compensation unit 302 or intra prediction unit 303. Optionally, a deblocking filter may be applied to filter the decoded block to remove blockiness artifacts. The decoded video block is then stored in buffer 307, which provides reference blocks for subsequent motion compensation / intra prediction and also generates the decoded video for presentation on a display device.

[0058] Some exemplary embodiments of the present disclosure will be described in detail below. While section headings are used herein for ease of understanding, it should be understood that they do not limit the embodiments disclosed in a section to that section alone. Furthermore, while certain embodiments are described with reference to versatile video coding or other specific video codecs, the disclosed techniques are applicable to other video coding techniques. Furthermore, while some embodiments describe video coding steps in detail, it will be understood that the corresponding decoding steps that undo the coding are performed by a decoder. Furthermore, the term video processing encompasses video encoding or compression, video coding or decompression, and video trans-encoding, which represents video pixels from one compressed format to another compressed format or at a different compressed bit rate. 1. Overview The present disclosure relates to video streaming, and more particularly to the design of video streaming based on Main Stream Representation (MSR) and External Stream Representation (ESR). The ideas can be applied individually or in various combinations to media streaming systems based on the DASH standard and its extensions, etc. 2. Background 2.1 Video Coding Standards Video coding standards have evolved primarily through the development of well-known ITU-T and ISO / IEC standards. ITU-T developed H.261 and H.263, while ISO / IEC developed MPEG-1 and MPEG-4 Visual. These two organizations jointly developed the H.262 / MPEG-2 Video, H.264 / MPEG-4 AVC (Advanced Video Coding), and H.265 / HEVC standards. Since H.262, video coding standards have been based on a hybrid video coding architecture that utilizes temporal prediction plus transform coding. To explore future video coding technologies beyond HEVC, the Joint Video Exploration Team (JVET) was jointly established by VCEG and MPEG in 2015. Since then, many new methods have been adopted by the JVET and incorporated into reference software named the Joint Exploration Model (JEM). Later, when the Versatile Video Coding (VVC) project was officially launched, JVET was renamed the Joint Video Experts Team (JVET). VVC is a new coding standard that aims to reduce the bitrate by 50% compared to HEVC, and was finally completed by JVET at its 19th meeting, which ended on July 1, 2020. The Versatile Video coding (VVC) standard (ITU-T H.266 | ISO / IEC 23090-3) and the associated Versatile Supplemental Enhancement Information (VSEI) standard (ITU-T H.274 | ISO / IEC 23002-7) are designed for use in the widest range of applications, including both traditional uses such as television broadcasting, videoconferencing, or playback from storage media, and newer, more advanced uses such as adaptive bitrate streaming, video region extraction, content composition and combining from multiple coded video bitstreams, multiview video, scalable layered coding, and viewport-adaptive 360-degree immersive media. The Essential Video Coding (EVC) standard (ISO / IEC 23094-1) is another video coding standard recently developed by MPEG. 2.2 File Format Standards Media streaming applications are typically based on IP, TCP, and HTTP transport methods and rely on file formats such as the ISO Base Media File Format (ISOBMFF). One such streaming system is HTTP-based Dynamic Adaptive Streaming (DASH). When using video formats with ISOBMFF and DASH, video-format-specific file format specifications, such as the AVC file format or the HEVC file format, may be required for encapsulating video content in ISOBMFF tracks and DASH representations and segments. Important information about the video bitstream, such as profile, tier, level, and many other pieces of information, may need to be exposed as file format-level metadata and / or a DASH Media Presentation Description (MPD) for content selection purposes, e.g., selecting appropriate media segments for both initialization at the start of a streaming session and stream adaptation during the streaming session. Similarly, the use of image formats in ISOBMFF may require file format specifications specific to the image format, such as the AVC image file format and the HEVC image file format. The VVC video file format, which is a file format for storing VVC video content based on ISOBMFF, is currently being developed by MPEG. The VVC Image File Format, based on ISOBMFF and currently being developed by MPEG, is a file format for storing image content coded using VVC. 2.3 DASH In HTTP-based Dynamic Adaptive Streaming (DASH), multiple representations of video and / or audio data of multimedia content may exist, with different representations corresponding to different coding characteristics (e.g., different profiles or levels of a video coding standard, different bit rates, different spatial resolutions, etc.). A manifestation of such representations may be defined as a media presentation description (MPD) data structure. A media presentation may correspond to a structured collection of data accessible to a DASH streaming client device. A DASH streaming client device may request and download media data information to provide streaming services to a user of the client device. A media presentation may be described in the MPD data structure, including updates to the MPD. A media presentation may include a series of one or more periods. Each period may extend until the start of the next period or, in the case of the last period, until the end of the media presentation. Each period may include one or more representations of the same media content. A representation may be one of many alternative encoding versions of audio, video, timed text, or other such data. Representations may differ by encoding type, e.g., bitrate, resolution, and / or codec for video data, and bitrate, language, and / or codec for audio data. The term representation may be used to refer to a section of encoded audio or video data that corresponds to a particular period of multimedia content and is encoded in a particular manner. Representations for a particular time period may be assigned to a group indicated by an attribute in the MPD that indicates the adaptation set to which the representation belongs. Representations within the same adaptation set are generally considered to be substitutes for each other, in that a client device can dynamically and seamlessly switch between these representations to, for example, perform bandwidth adaptation. For example, each representation of video data for a particular time period may be assigned to the same adaptation set, but either representation may be selected for decoding to present media data, such as video data or audio data, of the multimedia content for the corresponding time period. Media content within a time period may, in some examples, be represented by either one representation from group 0 (if present) or a combination of at most one representation from each non-zero group. Timing data for each representation for a time period may be expressed relative to the start time of the time period. A representation may contain one or more segments. Each representation includes an initialization segment, and each segment of a representation may be self-initializing. If present, the initialization segment may contain initialization information for accessing the representation. Generally, initialization segments do not contain media data. A segment may be uniquely referenced by an identifier such as a uniform resource locator (URL), a uniform resource name (URN), or a uniform resource identifier (URI). The MPD may provide an identifier for each segment. In some examples, the MPD may provide a byte range in the form of a range attribute that may correspond to the data of a segment within a file accessible by a URL, URN, or URI. Different representations may be selected to search for different types of media data substantially simultaneously. For example, a client device may select an audio representation, a video representation, and a timed text representation for searching for a segment. In some examples, a client device may select a particular adaptation set for performing bandwidth adaptation. That is, the client device may select an adaptation set including a video representation, an adaptation set including an audio representation, and / or an adaptation set including timed text. Alternatively, a client device may select an adaptation set for a particular type of media (e.g., video) and directly select representations for other types of media (e.g., audio and / or timed text). The general DASH streaming procedure is shown in the following steps: 1) The client obtains the MPD. 2) The client estimates the downlink bandwidth and selects the video and audio representations according to the estimated downlink bandwidth and the codec, decoding capability, display size, and audio language settings. 3) Unless the end of the media presentation has been reached, the client requests media segments of the selected representation and presents the streaming content to the user. 4) The client continues to estimate the downlink bandwidth. If the bandwidth changes significantly in some direction (e.g., becomes lower), the client selects another video representation that matches the newly estimated bandwidth and proceeds to step 3. 2.4 Enhanced Dependent Random Access Point (EDRAP) Based Video Coding and Streaming Signaling of EDRAP pictures using Supplemental Enhancement Information (SEI) messages was proposed by JVET-U0084 and adopted into the VSEI specification at the 21st JVET meeting in January 2021. At the 133rd MPEG meeting in January 2021, the EDRAP sample group agreed based on the proposal in input document m56020. To support EDRAP-based video streaming, at the 134th MPEG meeting in April 2021, MPEG input document m56675 proposed the design of an External Stream Track (EST) for ISOBMFF. The MPEG output document MDS21030_WG03_N0425, titled "WD of ISO / IEC 23009-1 5th edition AMD2 EDRAP streaming and other extensions," contains the design of Main Stream Representation (MSR) and External Stream Representation (ESR) descriptors to support EDRAP-based streaming in DASH. 4 and 5 are diagrams illustrating the conventional concept of a random access point (RAP). An application (e.g., adaptive streaming) determines the frequency of the random access point (RAP), for example, the period of the RAP is 1 s or 2 s. Conventionally, the RAP is provided by coding an IRAP picture, as shown in FIG. 4. It should be noted that the inter-prediction references of non-key pictures between the RAP pictures are not shown, and the output order is from left to right. When randomly accessing from CRA6, the decoder receives and correctly decodes the pictures, as shown in FIG. 5. Figures 6 and 7 illustrate the concept of Dependent Random Access Point (DRAP). The DRAP approach improves coding efficiency by allowing DRAP pictures (and subsequent pictures) to reference their previous IRAP pictures for inter-prediction, as shown in Figure 6. Note that inter-prediction of non-key pictures between IRAP pictures is not shown, and the output order is from left to right. Upon random access from DRAP6, the decoder receives and correctly decodes the pictures, as shown in Figure 7. 8 and 9 illustrate the concept of Enhanced Dependent Random Access Point (EDRAP). The EDRAP approach provides a bit more flexibility by allowing EDRAP pictures (and subsequent pictures) to refer to some of the previous RAP pictures (IRAP or EDRAP), for example, as shown in FIG. 8. Note that inter-prediction of non-key pictures between RAP pictures is not shown, and the output order is from left to right. When randomly accessing from EDRAP6, the decoder receives and correctly decodes the pictures, as shown in FIG. 9. Figures 10 and 11 show diagrams illustrating EDRAP-based video streaming. When randomly accessing or switching to a segment starting from EDRAP6, the decoder receives and decodes the segment as shown in Figure 11. The design documents in the MPEG output document MDS21030_WG03_N0425 are shown below. 2.4.1 Definition Enhanced Dependent Random Access Point (EDRAP) pictures: Pictures in samples that are members of an EDRAP or DRAP sample group in an ISOBMFF track External elementary streams: Elementary streams containing access units with external pictures External Picture: A picture in an external elementary stream in the ESR that is required for inter prediction during decoding of an elementary stream in the MSR when randomly accessed from an EDRAP in the MSR. External Stream Representation (ESR): Representations including external elementary streams Mainstream Representation (MSR): Representations including video elementary streams 2.4.2 MSR and ESR Descriptors An adaptation set may contain an EssentialProperty descriptor with @schemeIdUri equal to urn:mpeg:dash:msr:2021. This descriptor is called an MSR descriptor. The presence of EssentialProperty indicates that each representation in the adaptation set is an MSR. The following applies to MSR: Each SAP in an MSR representation in an adaptation set can be used to access content in the representation if time-synchronized samples are available to the client if they are present in the track carried in the associated ESR. Each EDRAP picture in the MSR should be the first picture in a segment (i.e., each EDRAP picture should start a segment). An adaptation set may contain an EssentialProperty descriptor with @schemeIdUri equal to Urn:mpeg:dash:esr:2021. This descriptor is called an ESR descriptor. The presence of EssentialProperty indicates that each representation in the adaptation set is an ESR. An ESR should not be consumed or played by itself without other video representations. Each MSR is associated with an MSR through the (existing) representation level attributes @associationId and @associationType in the MSR as follows: the @id of the associated ESR should be referenced by the value contained in the attribute @associationId, and the corresponding value in the attribute @associationType is equal to "aest". The following constraints apply to MSRs and ESRs that are optionally associated with each other through the representation attributes @associationId and @associationType in the MSR: -For each segment in the MSR that starts with an EDRAP picture, there should be a segment in the ESR with the same segment start time (derived from the MPD) as the segment in the MSR, where the segment in the ESR carries the external pictures necessary to decode the EDRAP picture and subsequent pictures in decoding order in the bitstream carried in the MSR. - For each segment in the MSR that does not start at an EDRAP picture, there should not be a segment in the ESR with the same segment start time (derived from the MPD) as a segment in the MSR. 3. Problems The design of the MPEG output document MDS21030_WG03_N0425 has the following problems: 1) It is permitted for a Main Stream Representation (MSR) to have no associated External Stream Representations (ESR). 2) The term EDRAP picture is defined as "a picture within a sample that is a member of an EDRAP or DRAP sample group within an ISOBMFF track." However, the definition cannot be applied to representations that are not based on ISOBMFF, nor can the definition be applied to other types of media other than video. 3) There is no restriction requiring that the bitstream obtained by random access from the EDRAP samples in the MSR be a conforming bitstream. 4. Detailed Solutions To solve the above problems, the methods summarized as follows are disclosed. The solutions should be considered as examples to illustrate the general concept and should not be interpreted narrowly. Furthermore, these solutions can be combined and applied in any manner. 1) To solve the first problem, it is specified that a Main Stream Representation (MSR) should have an associated External Stream Representation (ESR). 2) To address the second problem, one or more of the following items are specified: a. For each media sample in the ESR with a particular presentation time, there should be a corresponding media sample in the MSR with the same presentation time. b. Each media sample in the MSR that has a corresponding ESR media sample is called an EDRAP sample. c. The first byte position of each EDRAP sample in the MSR is the I SAU which allows the media stream in the MSR to be played if the corresponding ESR media sample is provided to the media decoder immediately before the EDRAP sample and subsequent samples in the MSR. 3) To solve the third problem, it is specified that the concatenation of any segment in the ESR with the corresponding segment in the MSR and all subsequent segments should produce a conforming bitstream. 5. Implementation [Table 1] 5.1.1 MSR and ESR Descriptors [Table 2] [Table 3]

[0059] SUMMARY OF THE INVENTION The embodiments of the present disclosure relate to an improved design for streaming based on MSR and ESR. As used herein, the term "media" may include audio, video, and the like.

[0060] FIG. 12 shows a flowchart of a media processing method 1200 according to some embodiments of the present disclosure. The method 1200 may be implemented on a client or a server. As used herein, the term "client" may refer to computer hardware or software that accesses services made available by a server as part of a client-server model of a computer network. As an example, a client may be a smartphone or a tablet. As used herein, the term "server" may refer to a computing-capable device, in which case the client accesses services over a network. The server may be a physical computing device or a virtual computing device.

[0061] As shown in FIG. 12, at 1202, a conversion between a media file of media and a media presentation of media is performed. A media file is a collection of data establishing a bounded or unbounded presentation of media content within the context of a file format, such as ISOBMFF. A media presentation is a collection of data establishing a bounded or unbounded presentation of media content within the context of a streaming format, such as DASH. A media presentation includes a Main Stream Representation (MSR) and an External Stream Representation (ESR) associated with the MSR, where the concatenation of a first segment in the ESR, a second segment in the MSR corresponding to the first segment, and subsequent segments following the second segment in the MSR generates a file conforming to a file format standard. For example, the file format standard may be the International Organization for Standardization Based Media File Format (ISOBMFF). It should be understood that the file format standard may be any other suitable file format standard. The scope of the present disclosure is not limited in this respect.

[0062] Considering the above, a file obtained by concatenating a segment in the ESR, its corresponding segment in the MSR, and all subsequent segments in the MSR may conform to a file format standard. Compared to conventional methods that do not require such files to conform to a file format standard, the proposed method has the advantage of being able to more efficiently support EDRAP-based random access technology.

[0063] In some embodiments, an MSR may be associated with an ESR through a set of representation-level attributes in the MSR. In one example, the set of representation-level attributes may include an associationId attribute and an associationType attribute. As an example, the @id of the associated ESR is referenced by the value contained in the attribute @associationId, and the corresponding value in the attribute @associationType is equal to "aest". It should be understood that the above example is provided for illustrative purposes only. The scope of the present disclosure is not limited in this respect.

[0064] In some embodiments, a first descriptor in a dataset within a media presentation is determined during the conversion at 1202. The presence of the first descriptor indicates that a representation in the dataset may be an MSR. The conversion may be performed based on the first descriptor. As an example, the first descriptor may be defined as a data structure having an attribute equal to a Uniform Resource Name (URN) string. As an example, the attribute may be a schemeIdUri attribute. And the URN string may be "urn:mpeg:dash:msr:2022". Further, the media presentation may include a Media Presentation Description (MPD). And the data structure may be an EssentialProperty within the MPD. In an additional or alternative example, the dataset may be an adaptation set. It should be understood that the above examples are provided for illustrative purposes only. The scope of the present disclosure is not limited thereto.

[0065] Additionally or alternatively, a second descriptor in the dataset within the media presentation is determined during the conversion at 1202. The presence of the second descriptor indicates that the representation in the dataset may be an ESR. The conversion may be performed based on the second descriptor. As an example, the second descriptor may be defined as a data structure having an attribute equal to a URN string. As an example, the attribute may be a schemeIdUri attribute. And the URN string may be "urn:mpeg:dash:esr:2022". Further, the media presentation may include an MPD, and the data structure may be an EssentialProperty within the MPD. In an additional or alternative example, the dataset may be an adaptation set. It should be understood that the above examples are provided for illustrative purposes only. The scope of the present disclosure is not limited thereto.

[0066] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium is proposed. A media file of media is stored in the non-transitory computer-readable storage medium. The media file of the media can be generated by a method executed by a media processing device. The method performs conversion between the media file and a media presentation of the media. The media presentation includes an MSR and an ESR associated with the MSR. A file conforming to a file format standard is generated by concatenating a first segment in the ESR, a second segment in the MSR corresponding to the first segment, and a subsequent segment following the second segment in the MSR.

[0067] According to an embodiment of the present disclosure, a method for storing a media presentation of media is proposed. The method includes converting between a media file of the media and a media presentation of the media, and storing the media file in a non-transitory computer-readable storage medium. The media presentation includes an MSR and an ESR associated with the MSR. A file conforming to a file format standard is generated by concatenating a first segment in the ESR, a second segment in the MSR corresponding to the first segment, and a subsequent segment following the second segment in the MSR.

[0068] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium is proposed. A media presentation of media is stored in the non-transitory computer-readable storage medium. The media presentation of the media can be generated by a method executed by a media processing device. The method converts between a media file of the media and the media presentation. The media presentation includes an MSR and an ESR associated with the MSR. A file conforming to a file format standard is generated by concatenating a first segment in the ESR, a second segment in the MSR corresponding to the first segment, and a subsequent segment following the second segment in the MSR.

[0069] According to an embodiment of the present disclosure, a method for storing a media presentation of media is proposed. The method includes converting between a media file of the media and a media presentation of the media, and storing the media presentation in a non-transitory computer-readable storage medium. The media presentation includes an MSR and an ESR associated with the MSR. A file conforming to a file format standard is generated by concatenating a first segment in the ESR, a second segment in the MSR corresponding to the first segment, and a subsequent segment following the second segment in the MSR.

[0070] Embodiments of the present disclosure can be described in view of the following clauses, the features of which can be combined in any reasonable manner.

[0071] Clause 1. A media processing method, comprising performing a conversion between a media file of media and a media presentation of said media, said media presentation comprising a Main Stream Representation (MSR) and an External Stream Representation (ESR) associated with said MSR, wherein a file conforming to a file format standard is generated by concatenation of a first segment in said ESR, a second segment in said MSR that corresponds to said first segment, and a subsequent segment that follows the second segment in said MSR.

[0072] Clause 2. The method of clause 1, wherein the file format standard is the International Organization for Standardization Based Media File Format (ISOBMFF).

[0073] Clause 3. The method of clause 1 or 2, wherein the MSR is associated with the ESR through a set of representation-level attributes within the MSR.

[0074] Clause 4. The method of any one of clauses 1 to 3, wherein the set of representation-level attributes includes an associationId attribute and an associationType attribute.

[0075] Clause 5. The method of any one of clauses 1 to 4, wherein the step of performing the transformation comprises the step of determining a first descriptor in a dataset within the media presentation, the presence of which indicates that a representation within the dataset is an MSR; and performing the transformation based on the first descriptor.

[0076] Clause 6. The method of clause 5, wherein the first descriptor is defined as a data structure having an attribute equal to a uniform resource name (URN) string.

[0077] Clause 7. The method of clause 6, wherein the media presentation includes a Media Presentation Description (MPD) and the data structure is an EssentialProperty within the MPD.

[0078] Clause 8. The method of clause 7, wherein the attribute is a schemeIdUri attribute and the URN string is "urn:mpeg:darqdsh:msr:2022".

[0079] Clause 9. The method of any one of clauses 1 to 4, wherein the step of performing the transformation comprises the step of determining a second descriptor in a dataset within the media presentation, the presence of the second descriptor indicating that the representation in the dataset is an ESR; and performing the transformation based on the second descriptor.

[0080] Clause 10. The method of clause 9, wherein the second descriptor is defined as a data structure having attributes equal to a URN string.

[0081] Clause 11. The method of clause 10, wherein the media presentation includes an MPD and the data structure is an EssentialProperty within the MPD.

[0082] Clause 12. The method of clause 11, wherein the attribute is a schemeIdUri attribute and the URN string is "urn:mpeg:dash:esr:2022".

[0083] Clause 13. The method of any one of clauses 1 to 12, wherein performing said conversion comprises packing said media files within said media presentation.

[0084] Clause 14. The method of any one of clauses 1 to 12, wherein the step of performing the conversion includes the step of unpacking the media file from the media presentation.

[0085] Clause 15. An apparatus for processing media data, comprising a processor and a non-transitory memory comprising instructions that, when executed by the processor, cause the processor to perform a method according to any one of clauses 1 to 14.

[0086] Clause 16. A non-transitory computer-readable storage medium storing instructions that cause a processor to perform the method described in clauses 1-14.

[0087] Clause 17. A non-transitory computer-readable storage medium storing a media file of media generated by a method performed by a media processing device, the method comprising performing a conversion between the media file and a media presentation of the media, the media presentation comprising a Main Stream Representation (MSR) and an External Stream Representation (ESR) associated with the MSR, wherein concatenation of a first segment in the ESR, a second segment in the MSR that corresponds to the first segment, and a subsequent segment that follows the second segment in the MSR generates a file conforming to a file format standard.

[0088] Clause 18. A method of storing a media file of media, comprising: performing a conversion between the media file and a media presentation of the media; and storing the media file in a non-transitory computer-readable storage medium, wherein the media presentation includes a Main Stream Representation (MSR) and an External Stream Representation (ESR) associated with the MSR, and wherein concatenation of a first segment in the ESR, a second segment in the MSR that corresponds to the first segment, and a subsequent segment that follows the second segment in the MSR generates a file that conforms to a file format standard.

[0089] Clause 19. A non-transitory computer-readable storage medium storing a media presentation of media generated by a method performed by a media processing device, the method comprising performing a conversion between a media file of the media and the media presentation, the media presentation comprising a Main Stream Representation (MSR) and an External Stream Representation (ESR) associated with the MSR, wherein concatenation of a first segment in the ESR, a second segment in the MSR that corresponds to the first segment, and a subsequent segment that follows the second segment in the MSR generates a file that conforms to a file format standard.

[0090] Clause 20. A method of storing a media presentation of media, comprising: performing a conversion between a media file of the media and the media presentation; and storing the media presentation in a non-transitory computer-readable storage medium, wherein the media presentation comprises a Main Stream Representation (MSR) and an External Stream Representation (ESR) associated with the MSR, and wherein concatenation of a first segment in the ESR, a second segment in the MSR that corresponds to the first segment, and a subsequent segment that follows the second segment in the MSR generates a file that conforms to a file format standard.

[0091] Exemplary Devices 13 shows a block diagram of a computing device 1300 capable of implementing various embodiments of the present disclosure. The computing device 1300 may be implemented as or included in the source device 110 (or the video encoder 114 or 200) or the destination device 120 (or the video decoder 124 or 300).

[0092] It will be understood that the computing device 1300 shown in FIG. 13 is for illustrative purposes only and is not intended to limit in any way the functionality and scope of the embodiments of the present disclosure.

[0093] 13, the computing device 1300 may include a general-purpose computing device 1300. The computing device 1300 may include at least one or more processors or processing units 1310, memory 1320, a storage unit 1330, one or more communication units 1340, one or more input devices 1350, and one or more output devices 1360.

[0094] In some embodiments, computing device 1300 may be implemented as any user terminal or server terminal having computing capabilities. The server terminal may be a server provided by a service provider, a large-scale computing device, etc. The user terminal may be any type of mobile, fixed, or portable terminal, including, for example, a mobile phone, station, unit, device, multimedia computer, multimedia tablet, Internet node, communicator, desktop computer, laptop computer, notebook computer, netbook computer, tablet computer, personal communication system (PCS) device, personal navigation device, personal digital assistant (PDA), audio / video player, digital camera / camcorder, positioning device, television receiver, radio receiver, electronic book device, gaming device, or any combination thereof (including accessories and peripherals of these devices, or any combination thereof). It is contemplated that computing device 1300 may support any type of interface to a user (e.g., “wearable” circuitry, etc.).

[0095] The processing unit 1310 may be a physical or virtual processor and may perform various processes based on programs stored in the memory 1320. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to increase the parallel processing capabilities of the computing device 1300. The processing unit 1310 may be referred to as a central processing unit (CPU), a microprocessor, a controller, or a microcontroller.

[0096] Computing device 1300 typically includes a variety of computer storage media. Such media may be any media accessible by computing device 1300, including, but not limited to, volatile and nonvolatile media, or removable and non-removable media. Memory 1320 may be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or any combination thereof. Storage unit 1330 may be any removable or non-removable medium, including a machine-readable medium such as a memory, flash memory drive, magnetic disk, or other medium, that can be used to store information and / or data and that can be accessed by computing device 1300.

[0097] The computing device 1300 may further include additional removable / non-removable, volatile / non-volatile memory media. Although not shown in FIG. 13, a magnetic disk drive that reads from and writes to a removable non-volatile magnetic disk and an optical disk drive that reads from and writes to a removable non-volatile optical disk may be provided. In such cases, each drive may be connected to a bus (not shown) via one or more data media interfaces.

[0098] The communications unit 1340 communicates with additional computing devices via a communications medium. Furthermore, the functionality of the components within the computing device 1300 may be performed by a single computing cluster or multiple computing machines that can communicate via communications connections. Thus, the computing device 1300 may operate in a networked environment using logical connections with one or more other servers, networked personal computers (PCs), or additional general network nodes.

[0099] The input device(s) 1350 may be one or more of a variety of input devices, such as a mouse, a keyboard, a tracking ball, a voice input device, etc. The output device(s) 1360 may be one or more of a variety of output devices, such as a display, a speaker, a printer, etc. The communication unit 1340 may further enable the computing device 1300 to communicate with one or more external devices (not shown), such as a storage device and a display device, one or more devices that allow a user to interact with the computing device 1300, or any device (such as a network card, a modem, etc.) that allows the computing device 1300 to communicate with one or more other computing devices, as needed. Such communication may be performed via an input / output (I / O) interface (not shown).

[0100] In some embodiments, instead of being integrated into a single device, some or all of the components of computing device 1300 may be located in a cloud computing architecture. In a cloud computing architecture, components may be provided remotely and work together to perform the functions described in this disclosure. In some embodiments, cloud computing provides computing, software, data access, and storage services without requiring end users to be aware of the physical location or configuration of the systems or hardware providing these services. In various embodiments, cloud computing provides services over a wide area network (such as the Internet) using appropriate protocols. For example, a cloud computing provider may provide applications over a wide area network that can be accessed through a web browser or other computing component. Software or components of a cloud computing architecture and corresponding data may be stored on servers in remote locations. Computing resources in a cloud computing environment may be consolidated or distributed at remote data center locations. A cloud computing infrastructure may act as a single point of access for users but provide services through a shared data center. Thus, a cloud computing architecture may be used to provide the components and functions described herein from a service provider in a remote location. Alternatively, they may be provided from a conventional server or installed directly or otherwise on the client device.

[0101] The computing device 1300 may be used to perform video coding / decoding in embodiments of the present disclosure. The memory 1320 may include one or more video coding modules 1325 having one or more program instructions. These modules are accessible and executable by the processing unit 1310 to perform the functions of various embodiments described herein.

[0102] In example embodiments that perform video coding, input device 1350 may receive video data to be encoded as input 1370. The video data may be processed, for example, by video coding module 1325 to generate an encoded bitstream. The encoded bitstream may be provided as output 1380 via output device 1360.

[0103] In example embodiments that perform video decoding, input device 1350 may receive an encoded bitstream as input 1370. The encoded bitstream may be processed, for example, by video coding module 1325 to generate decoded video data. The decoded video data may be provided as output 1380 via output device 1360.

[0104] While the present disclosure has been particularly shown and described with reference to preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the present application as defined by the appended claims. Such variations are intended to be within the scope of the present application. Accordingly, the foregoing description of the embodiments of the present application is not intended to be limiting.

Claims

1. 1. A media processing method, comprising: performing a conversion between a media file of the media and a media presentation of said media; the media presentation includes a Main Stream Representation (MSR) and an Extra Stream Representation (ESR) associated with the MSR; a file conforming to a file format standard is generated by concatenating a first segment in the ESR, a second segment in the MSR corresponding to the first segment, and a subsequent segment following the second segment in the MSR; The step of performing the conversion comprises: determining a first descriptor in a dataset within the media presentation, the presence of the first descriptor indicating that a representation within the dataset is an MSR; performing the transformation based on the first descriptor. method.

2. the file format standard is the International Organization for Standardization Based Media File Format (ISOBMFF); The method of claim 1.

3. The MSR is associated with the ESR through a set of expression-level attributes within the MSR. The method of claim 1.

4. The set of representation-level attributes includes an associationId attribute and an associationType attribute. The method of claim 3.

5. the first descriptor is defined as a data structure having attributes equal to a uniform resource name (URN) string; The method of claim 1.

6. the media presentation comprises a Media Presentation Description (MPD); The data structure is an EssentialProperty in the MPD. The method of claim 5.

7. The attribute is a schemeIdUri attribute, The URN string is "urn:mpeg:darqdsh:msr:2022" The method of claim 6.

8. The step of performing the conversion comprises: determining a second descriptor in a dataset within the media presentation, the presence of the second descriptor indicating that a representation within the dataset is an ESR; performing the transformation based on the second descriptor. The method of claim 1.

9. the second descriptor is defined as a data structure with attributes equal to the URN string; The method of claim 8.

10. the media presentation comprises an MPD; The data structure is an EssentialProperty in the MPD.

10. The method of claim 9.

11. The attribute is a schemeIdUri attribute, The URN string is "urn:mpeg:dash:esr:2022", The method of claim 10.

12. The step of performing the conversion comprises: packing the media files into the media presentation; The method of claim 1.

13. performing the conversion includes unpacking the media file from the media presentation; The method of claim 1.

14. 1. An apparatus for processing media data, comprising a processor and a non-transitory memory comprising instructions, The instructions, when executed by the processor, cause the processor to perform the method of any one of claims 1 to 13. Device.

15. storing instructions for causing a processor to carry out the method of any one of claims 1 to 13; A non-transitory computer-readable storage medium.

16. 1. A method of storing a media presentation of media, comprising: performing a conversion between a media file of the media and the media presentation; storing the media presentation in a non-transitory computer-readable storage medium; the media presentation includes a Main Stream Representation (MSR) and an Extra Stream Representation (ESR) associated with the MSR; a file conforming to a file format standard is generated by concatenating a first segment in the ESR, a second segment in the MSR corresponding to the first segment, and a subsequent segment following the second segment in the MSR; The step of performing the conversion comprises: determining a first descriptor in a dataset within the media presentation, the presence of the first descriptor indicating that a representation within the dataset is an MSR; performing the transformation based on the first descriptor. method.

Images