Signaling for picture-in-picture in media container files and streaming manifests

Abstract

The present disclosure relates to signaling picture-in-picture (PiP) in media container files and streaming media manifests. In one aspect, signaling of a PiP experience in a container file is disclosed, where the PiP experience consists of two or more media tracks. A media track group is used to define the picture-in-picture experience, where one or more main media tracks and alternative / overlay media tracks are defined. The PiP experience is identified in a role-based manner. The media tracks in a PiP can be decoded independently, or, if the coding scheme supports it, the alternative coded stream can replace the corresponding region in the main picture stream in merge mode, for which the order of importance of the alternative pictures and the specific region of the main picture are also signaled. In another aspect, the picture-in-picture experience may be signaled in a streaming manifest. Such signaling may include identifying an adaptation set using a picture-in-picture descriptor, as well as using a role descriptor to identify a PiP experience, and further annotating sub-pictures in the manifest where one or more sub-pictures are identified as one or more regions for replacement.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to U.S. Non-provisional Application No. 18 / 341,471, filed June 26, 2023, which claims the benefit of priority to U.S. Provisional Application No. 63 / 388,555, filed July 12, 2022, and U.S. Provisional Application No. 63 / 417,103, filed October 18, 2022, which are incorporated by reference in their entireties herein.

[0002] This disclosure relates to signaling picture-in-picture (PiP) in media container files and streaming media manifests. [Background technology]

[0003] Media content may be organized in a container file having a predefined format. Such media content may include visual information such as images or videos. Different sets of visual information may be displayed simultaneously, for example in picture-in-picture (PiP) mode. Media container files may be downloaded for local playback or streamed from a server. Both downloaded media and streaming media may invoke the PiP mode. Information and configuration about media content for PiP use may need to be signaled in the media container file or bitstream, as well as in the media manifest used by the streaming application to perform adaptive streaming. Summary of the Invention [Means for solving the problem]

[0004] This disclosure relates to signaling picture-in-picture (PiP) in media container files and streaming media manifests.

[0005] In one aspect, signaling of a PiP experience in a container file is disclosed, where the PiP experience consists of two or more media tracks. A media track group is used to define a picture-in-picture experience, where one or more main media tracks and alternative / overlay media tracks are defined. The PiP experience is identified in a role-based manner. The media tracks in a PiP can be decoded independently, or, if the coding scheme supports it, the alternative coded stream can replace the corresponding region in the main picture stream in merge mode, for which the order of importance of the alternative pictures and the specific regions of the main picture are also signaled.

[0006] In other aspects, a picture-in-picture experience may be signaled in the streaming manifest, which may include identifying an adaptation set with a picture-in-picture descriptor, as well as using a role descriptor to identify the PiP experience, and further annotating one or more sub-pictures in the manifest where they are identified as one or more regions for substitution.

[0007] In some example implementations, a method for obtaining (Picture-in-Picture) PiP information is disclosed. The method may include the steps of: obtaining a media container file constructed in an ISO Base Media File Format (ISOBMFF), the media container file including a main media track and at least one sub-media track represented in a PiP mode; parsing the media container file and identifying a media track group for a PiP experience via a Pre-Selection Track Group Entry (Prse) box syntax element for a media track group definition; parsing the media container file and identifying the main media track and the at least one sub-media track belonging to the media track group via a Pre-Selection Group (Pres) box syntax element in a plurality of media track definitions in the media container file; parsing the media container file and determining a merging mode indicating whether a coded data unit of the at least one sub-media track is merged with a coded data unit of the main media track before decoding according to a presence or value of a sample flag merge syntax element in a component of the Pres syntax box element of the main media track or the at least one sub-media track; and decoding the main media track and the at least one sub-media track in the PiP mode according to the merging mode.

[0008] In the above example implementation, the Prse box syntax element belongs to a first predefined type of syntax element in the media container file configured to define a media track group.

[0009] In any one of the above example implementations, the syntax elements of the first predefined type each include a media track group type tag element, which indicates an intended experience of the corresponding media track group, including a PiP experience, using a set of predefined purpose keywords.

[0010] In any one of the above example implementations, the syntax elements of the first predefined type each include a media track group type descriptor, and the media track group type descriptor is configured to specify an intended experience of the corresponding media track group, including a PiP experience, using a role scheme having predefined roles.

[0011] In any one of the above example implementations, each of the syntax elements of the first predefined type in the media container file includes a number of tracks of a corresponding media track group.

[0012] In any one of the above example implementations, the Pres box syntax element belongs to a second predefined type of syntax element within the media track definition configured to specify an association of the media track with a media track group using a corresponding media track group identifier.

[0013] In any one of the above example implementations, each of the syntax elements of the second predefined type includes a media track group processing descriptor (prsp) for specifying PiP processing of a media track for a media track group.

[0014] In any one of the above example implementations, the media track group processing descriptor includes a priority parameter to indicate a priority of the media track relative to other media tracks in the media track group.

[0015] In any one of the above example implementations, the priority parameter of the main media track of the PiP experience includes the highest priority value for PiP processing.

[0016] In any one of the above example implementations, the priority parameter of at least one sub-media track of the PiP experience includes the lowest priority value for PIP processing.

[0017] In any one of the above example implementations, the sample flag merge syntax element belongs to a third predefined type of syntax element that is correspondingly included in a media track group processing descriptor associated with the main media track.

[0018] In any one of the above example implementations, a syntax element of a third predefined type associated with the main media track having a predefined value indicates that the main media track can be merged with the secondary media track before being decoded.

[0019] In any one of the above example implementations, when a syntax element of a third predefined type associated with the main media track is a predefined value, decoding the main media track and the at least one sub-media track in PiP mode includes merging the coded data units of the main media track and the coded data units of the at least one sub-media track for a single decoding.

[0020] In any one of the above example implementations, the media track group processing descriptor includes a region identification parameter for indicating a list of regions in the coded stream of the corresponding media track to be merged with the coded streams of other media tracks before decoding.

[0021] In any one of the above example implementations, when the syntax element of the third predefined type associated with the main media track is a predefined value, the region identification parameter associated with the main media track is non-NULL.

[0022] In any one of the above example implementations, when a media track is not the highest priority, the region identification parameters for the media track are ignored, even if present.

[0023] In any one of the above example implementations, the region identification parameter for the media track is non-NULL, and the priority of the media track is correspondingly highest.

[0024] In any one of the above example implementations, when the syntax element of the third predefined type associated with the main media track is not a predefined value, decoding the main media track and the at least one sub-media track in PiP mode includes processing the main media track and the at least one sub-media track by separate and independent decoding.

[0025] In any one of the above exemplary implementations, any of the main media track and the at least one sub-media track belongs to another media track group as indicated by another Prse box syntax element in the media container file.

[0026] In some other example implementations, a method for obtaining picture-in-picture (PiP) signaling information from a streaming media manifest is disclosed. The method may include: obtaining a streaming media manifest from a streaming server, parsing the streaming media manifest and identifying a set of PiP signaling information items associated with a set of streaming media content, constructing an adaptive request for the set of streaming media content according to the PiP signaling information items, receiving the set of streaming media content, and decoding and displaying the set of streaming media content according to the PiP signaling information items.

[0027] In the above example implementation, PiP signaling information items may be extracted from multiple adaptation sets in a streaming media manifest.

[0028] In any one of the above example implementations, the multiple adaptation sets form a PiP group.

[0029] In any one of the above example implementations, each of the multiple adaptation sets includes a supplemental descriptor for specifying a role of the corresponding adaptation set.

[0030] In any one of the above example implementations, the supplemental descriptor includes a role value and an identifier.

[0031] In any one of the above example implementations, the role value is among a set of role values ​​defined in a predefined role scheme.

[0032] In any one of the above example implementations, the predefined role schemes are represented in the multiple adaptation sets using a Universal Resource Name (URN) of the predefined role scheme.

[0033] In any one of the above example implementations, the role value indicates that the corresponding adaptation set is for PiP, and the identifier is the same for multiple adaptation sets of the PiP group and is assigned to distinguish the PiP group from other adaptation groups, including other PiP adaptation groups.

[0034] In any one of the above example implementations, the role value further indicates the role of the corresponding adaptation set within the PiP group, or includes one of a main adaptation or a sub-adaptation within the PiP group.

[0035] In any one of the above example implementations, at most one of the multiple adaptation sets is associated with the role of main adaptation set in the PiP group.

[0036] In any one of the above exemplary implementations, when an adaptation set of the multiple adaptation sets is a main adaptation set and is mergeable with a sub-adaptation set in a PiP group, the content component descriptor of the main adaptation set further includes a list of coded sub-components of the main adaptation set that are suitable for replacing the coded sub-adaptation set of the PIP group before being decoded.

[0037] In any one of the above example implementations, at least one of the multiple adaptation sets in the PiP group includes a second supplemental descriptor indicating that one of the multiple adaptation sets also belongs to a second PiP group.

[0038] Aspects of the present disclosure also provide a media streaming device or apparatus that includes circuitry configured to perform any one of the above method implementations.

[0039] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions configured, when executed by a media streaming device, to cause the media streaming device to perform any one of the method implementations described above.

[0040] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Brief description of the drawings]

[0041] [Figure 1] 1 illustrates a content distribution system according to one embodiment of the present disclosure. [Diagram 2] 1 illustrates a Dynamic Adaptive Streaming over HTTP (DASH) system according to one embodiment of the present disclosure. [Diagram 3] 1 illustrates a DASH client architecture according to one embodiment of the present disclosure. [Figure 4] 1 illustrates an exemplary picture-in-picture application. [Diagram 5] 1 illustrates an exemplary signaling scheme in a media container file. [Figure 6] 6 illustrates an exemplary data and logic flow for the signaling scheme of FIG. 5. [Figure 7] 1 illustrates an exemplary data and logic flow. [Figure 8] 1 shows a schematic diagram of a computer system according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0042] Streaming via HyperText Transfer Protocol (HTTP) FIG. 1 illustrates an exemplary content delivery system 100 in which a remote information processing device 120 is configured to request content from one or more centralized or distributed content servers 110 via a communication network 130. Specifically, the information processing device 120 may include dedicated hardware components, software components operating on general-purpose hardware, or a combination thereof, functioning as a content consumption application. The content consumption application may generate one or more requests that specify the content being requested and characteristics of the requested content. Each request may be built based on a stack of network protocols and communicated to the content server 110 via the communication network 130. In response, the content server may generate a bitstream according to the request, package the bitstream using a stack of network protocols, and communicate the bitstream package to the content consumption application.

[0043] In some example implementations, content may be requested all at once. In other words, the entire media content may be requested, received, and stored locally by the content consumption application. The locally stored content may be processed and consumed (e.g., extracted, decoded, and played) as needed, for example, by a media player that is part of or separate from the content consumption application. Such a process may be referred to as downloading.

[0044] In some other implementations, content may be streamed as it is consumed, rather than downloaded for later consumption. In such implementations, the entire requested content may not need to be stored in the content consumption application. Rather, a limited amount of content is continuously received from the content server 110 on a rolling basis and managed by an input / output local buffer for content processing and playback. Such implementations may be referred to as streaming. Although some media playback functions, such as rewinding, fast forwarding, and seeking, may involve complex media bitstream control and buffering, media streaming is typically more general-purpose and better suited to delivery of content that includes timed sequences of media that are not consumed repeatedly.

[0045] In the following disclosure, the terms "content" and "media" may be used interchangeably. The requested content may include various information items required for its consumption, including but not limited to the content itself and various metadata. The content itself may further include various media components, such as different tracks, including but not limited to video components / tracks, audio components / tracks, subtitles, etc. Metadata for describing the media content or for providing additional processing information may be treated as one or more separate tracks. Such content with its metadata may be generated by the content server 120 as a bitstream that can be parsed and decoded according to a protocol or set of rules known to the content consumption application. The singular term "content server" is used to represent a single server or multiple servers located in a central location or distributed at various geographic locations. Such content servers may be implemented as dedicated computing machines or may be constructed as virtual machines and / or virtually housed in a cloud computing environment. Further in the following disclosure, the terms "information processing device" (see 120 in FIG. 1) and "content consumption application" may be used interchangeably. Alternatively, these terms may also be referred to as "client", "client device / apparatus", "playback device / apparatus / client", etc. Although only a single information processing device 120 is shown in FIG. 1, there may be multiple independent information processing devices. In other words, the set of content servers 110 may be configured to simultaneously and independently provide streaming services to multiple content consumption applications.

[0046] In some example implementations, content generated for delivery by the content server 110 may be segmented to facilitate their streaming. For example, a timed sequence of media content, such as a movie, may be subdivided into time segments, each including several media frames. Each media segment may be self-contained such that its processing, including, for example, parsing, decoding, and playing, does not require knowledge of other media segments. Media content may be pre-segmented. Thus, media content may be stored and managed by the content server 120 on a segment-by-segment basis. Alternatively, media segments may be generated in real-time from proximally stored media content as they are required during the streaming process. In some further implementations, media segmentation may be hierarchical, including multiple levels of segmentation.

[0047] In some particular implementations for streaming, the decision as to which media segment, or which portion of a media segment, to request from the content server 110 may be determined by the content consumption application in real time, as controlled by a user's playback instructions through a user application interface. In this manner, the content server may be configured to respond to requests, generate and obtain segments or portions of segments of content with their metadata in accordance with the requests, and deliver the segments or portions of segments to the requesting content consumption application over the network 130.

[0048] In some exemplary implementations, the same media track of the media content may be prepared as different versions. For example, the same movie track may be prepared at different resolutions and / or frame rates. As another example, the same movie track may be prepared at different bit rates. As another example, the same audio movie may be prepared at different audio qualities and / or different audio channels (e.g., 5-channel audio, or 7-channel audio). Thus, the content consumption application may determine which version of the media track to stream and include such a selection in its request for the media content. Such a determination by the content consumption application may be made based on one or more of several exemplary factors, including, but not limited to, the playback capabilities of the information processing device 120 (e.g., display resolution, decoding speed, processing capabilities, buffer size, etc.), network bandwidth and throughput, etc. Thus, the streaming session may be adapted between different media consumption applications according to their device capabilities. A streaming architecture configured in this manner may be referred to as adaptive streaming. The streaming process may further be adaptive within each media consumption application in that different versions of a media track may be selected and requested at different times during a streaming session, for example according to real-time network conditions (e.g., bandwidth and throughput, and bitrate supported by the network bandwidth). A streaming architecture configured in this way may further be referred to as dynamic adaptive streaming. In particular, a streaming architecture configured to adapt to the bitrate of the media content may be referred to as dynamic adaptive bitrate streaming.

[0049] In some example implementations, requests for segments or portions of segments of a particular version of media content by a content consumption application in dynamic adaptive streaming may be built based on a media manifest as a streaming session progresses. The term "manifest" may be used to represent any collection of information items that describe media content, including segmentation, version, network location, and any other information that may be needed for any content consumption application to determine what and how to request at different points during a streaming session. The manifest may be generally referred to as a "media representation description" (MPD).

[0050] Such a manifest may be prepared at the content server side when a particular media content is created or generated. Such a manifest may be requested by a content consumption application and received from the content server at the start of a streaming session. The content consumption application may further request any updates to the manifest during the streaming session. Such a manifest may be used by the content consumption device as a blueprint for building subsequent requests for segments or portions of segments of a particular version of the media content during the streaming session.

[0051] In some example implementations, the media server may be configured to function similarly to a web server from the perspective of an external application. Thus, requests for media manifests and / or media segments or portions of media segments by a content consumption application may be made, for example, based on the HyperText Transfer Protocol (HTTP). Thus, requests may be structured as URLs, and the requested content may be delivered as a response to the HTTP request from the content server.

[0052] The details of how the manifest is specified, the content is segmented, organized, and versioned, and the HTTP is requested may depend on the particular adaptive streaming protocol, such as Dynamic Adaptive Streaming over HTTP (DASH), HTTP Live Streaming (HLS), Smooth Streaming Transport Protocol (SSTP), etc. Various additional example implementations below may be described in the context of DASH. However, the underlying principles are applicable to any type of adaptive streaming over HTTP. Additionally, the underlying principles are applicable to media content request mechanisms based on network protocols other than HTTP.

[0053] Dynamic Adaptive Streaming over HTTP (DASH) One exemplary protocol for implementing adaptive media streaming includes Dynamic Adaptive Streaming over Hypertext Transfer Protocol (DASH). As mentioned above, DASH represents one of the adaptive bitrate streaming implementations that enables streaming of media content using a content delivery network (CDN) based on a hypertext transfer protocol (HTTP) infrastructure, including content servers configured as web servers with various proxies and caches. Such a content server may be referred to as a DASH server. Thus, the above-mentioned content consumption application may be referred to as a DASH client.

[0054] DASH supports live streaming from DASH servers to DASH clients and allows DASH clients to control streaming sessions so that DASH servers do not have to deal with the additional burden of stream adaptation management in large-scale deployments. As mentioned above, DASH also allows DASH clients to select streaming from different DASH servers, thereby achieving further network load balancing for the benefit of DASH clients. DASH further provides dynamic switching between different media versions of a media track, for example by changing the bit rate to suit the network conditions and processing capabilities of the DASH client.

[0055] In DASH, the above-mentioned media manifest may be specifically referred to as an MPD (although the term MPD is used generally to refer to any type of manifest in adaptive streaming systems other than those based on DASH). For example, an MPD in DASH may be structured as a file that is fully or partially downloadable by a DASH client and provides items of information used by the DASH client to stream media content by selectively and adaptively requesting streaming media segments from a DASH server.

[0056] The MPD may be constructed in various formats. For example, the MPD may be constructed in the form of an Extensible Markup Language (XML) document or file. The MPD file may be requested and delivered to the DASH client. The MPD file may be requested by HTTP, for example, via an HTTP GET request. The MPD file may be delivered completely at the start of a streaming session. Alternatively, the MPD file can be fragmented and delivered partially. Thus, parts of the MPD file may be requested and delivered before the start of streaming, and other parts of the MPD file may be requested and delivered later to reduce session startup delays (so that streaming can start with an earlier media segment without having to wait for information items about later segments of media). The MPD file can also be updated during a streaming session (e.g., with segment information that is needed but not yet obtained).

[0057] In some example implementations, an MPD file describes the segmentation of media content, the organization of the segments, and the available versions of the segments. MPD supports the representation of content accessibility features, ratings, camera views, metadata, etc. DASH may also support the delivery of multi-view and scalable coded content.

[0058] In some example implementations, an MPD file may include a series of descriptions spanning one or more periods along a media consumption timeline (e.g., playback time of video content). Each of the one or more periods may be defined, for example, by a "Period" information element tag in the MPD file. Media content may be represented by an MPD file organized into multiple consecutive periods. The MPD file may identify a start time for each of the periods in the playback timeline. The start time may be defined as an absolute start time from the beginning of the media content or as a relative offset from another reference point in the playback timeline.

[0059] In some example implementations, for each media period, the MPD file may further specify one or more adaptation sets. Different adaptation sets may be specified to capture one or more different combinations (or subsets) of media components. For example, video and audio may be different adaptation sets. Different versions of audio (stereo or multi-channel audio) may be different adaptation sets. Different languages ​​of audio may be different adaptation sets. In one particular example, the MPD file may specify that each period includes one video adaptation set and multiple audio adaptation sets, one for each supported language. The adaptation sets may also include subtitles or any metadata.

[0060] In some example implementations, the adaptation sets of a particular period may be assigned to a group indicated by any attribute in the MPD file. Adaptation sets in the same group are generally considered as alternatives to each other. For example, each adaptation set of video data of a particular period may be assigned to the same group, so that any adaptation set may be selected for the video data of the multimedia content of the corresponding period. Media content in one period may be from either one adaptation set, or a combination of adaptation sets, and each group contributes to at most one adaptation set.

[0061] In some example implementations, each adaptation set may be specified by an MPD file as including one or more representations of the same media component of a corresponding time period. A representation may be, for example, one of several alternative encoded versions of audio or video data. Representations may vary by encoded type, for example, by bitrate, resolution, and / or codec of video data and bitrate and / or codec of audio data. The term representation may be used to refer to a section of encoded media data that corresponds to a particular time period of multimedia content and that has been encoded in a particular way to achieve a particular range of average bitrates. In some example implementations, for each representation in an adaptation set, the MPD file may specify attributes of the representation, including, but not limited to, video / audio type, video / audio codec, video frame width in pixels, video frame height in pixels, video / audio frame rate, and bandwidth (representing the average encoded bitrate).

[0062] Each representation of an adaptation set may also include one or more media components, depending on the combination of media components included in the adaptation set. Each media component in a representation may correspond to an encoded version of one individual media type, such as audio, video, or timed text (e.g., for closed captioning). Media components may be temporally contiguous across boundaries of consecutive media segments within one representation.

[0063] In some example implementations, a representation may include one or more segments. Each representation may include an initialization segment, or each segment of a representation may be self-initialized. When present, the initialization segment may include initialization information for accessing the representation. In some cases, the initialization segment does not include media data. Segments that include media data may represent time-segmented content. Segments between different representations may be time-aligned. For each media segment, the MPD file may include a unique identifier. Such an identifier, when combined with a base URL, base URN, or base Uniform Resource Identifier (URI), may form a unique URL, URN, or URI that represents the network location of the media segment, which may be included in an HTTP request for this media segment and may be used by the content server to locate the requested segment for delivery.

[0064] For example, the URL for requesting a media segment must be in the format of "http" or "https". <absolute-uri>and possibly further complemented by a byte range if a range attribute is provided with the URL. A byte range can be expressed to identify a contiguous range of bytes within a segment.

[0065] In some further example implementations, a sub-representation may be specified or described in an MPD file as being embedded (or included) in a regular representation, for example using a sub-representation element / indicator. A sub-representation element may be used to describe characteristics of one or several media content components embedded in a representation. For example, a sub-representation element may be used to describe characteristics of an embedded audio component (e.g., codec, sampling rate, etc.), embedded subtitles (e.g., codec), or a sub-representation element may be used to describe some embedded lower quality video layer (e.g., some lower frame rate, etc.). Sub-representations and representation elements may share some common attributes and elements.

[0066] In some example implementations, a DASH client may be configured to access, download, and request all or part of an MPD file from a DASH server. That is, the DASH client may obtain an MPD file for use in starting a live streaming session. Based on the MPD file and the representation selection, the DASH client can make several further decisions, including determining what the latest segment is available on the server, determining the segment availability start time of the next segment and possibly future segments, determining when to start playing a segment, and determining when to obtain / fetch / request a new MPD file.

[0067] In some example implementations, the MPD may further include information about DASH events to signal non-periodic information to a DASH client or DASH application. An event may be timed starting at a particular media presentation time having a duration. Additionally or alternatively, the event information may include a control message for a media player associated with a particular time during the playback of a media presentation, such as an ad insertion queue. Media that may be inserted during streaming may be provided from a separate server, such as an ad server. In addition to signaling events by the MPD separately from the media presentations, events may also be multiplexed in-band in only selected media presentations or in all presentations in one or several selected adaptation sets.

[0068] An exemplary DASH system 200 is shown in Figure 2. The DASH system 200 may include one or more centralized or distributed content servers 210 and information processing devices 230 connected by a network 250. The DASH system (200) may also include one or more supplemental content servers, such as one or more advertisement servers 220.

[0069] The content server 210 may provide the main content (e.g., a main program) and an MPD for the content to the information processing device 230. The manifest file may be generated by the MPD generator 214. The main content and the manifest file may be provided from the same server or different servers.

[0070] The information processing device 230 may include a DASH client 232 that communicates directly with the content server 210. The DASH client 232, controlled by a DASH application 234 of the information processing device 230, may request and / or receive an MPD and may request and obtain main content from an HTTP server 212 of the content server 210 based on the MPD. The MPD may be processed by the DASH client 232. In addition, the DASH client 232 may obtain advertising content from the advertising server 220 or other content (e.g., interactive content) from one or more supplemental content servers in response to DASH events. The main content and advertising content may be processed by the DASH client 232 and the DASH application 234 and output for display on a display device 236 of the information processing device 230. The display device 236 may be integrated into the information processing device 230 or may be external. In addition, the DASH client 232 may extract other event information from one or more timed metadata tracks and send the extracted event information to the DASH application 234 for further processing. DASH application 234 may be configured to display supplemental content based on event information, for example.

[0071] An example for the DASH client 232 is shown in FIG. 3. As shown in FIG. 3, the example DASH client 232 may include a DASH access engine 304, a selection logic 302, and media engines 306 and 308. The DASH access engine 302 may be configured to communicate with a content server, for example, to obtain a part or the entire MPD of the streaming media, to request and obtain segment data of the dynamically requested streaming media, and to request supplemental media (advertisements) according to the MPD DASH events. The selection logic 304 may be configured to determine the next one or more segments to request, including the selection of an adaptation set and a representation. Such a determination may be determined, for example, by a user instruction, as well as by other real-time information, such as network bandwidth and throughput. The media engine 306 may be configured to process the segment data received by the DASH access engine 302 according to the format (e.g., MPEG) of the media segments and the timing of the media segments to generate a main media output. The media engine 308 may be configured to process media content associated with timed DASH events from the DASH access engine 302, for example to generate supplemental media output (e.g., advertisements) that may be inserted into the main media output.

[0072] Media Container Files Media content may be stored in files having a variety of predefined formats. Media storage files may be used to define a general structure for files containing time-based multimedia data, such as video, audio, and other data associated with video and audio. The ISO Base Media File Format (ISOBMFF) is one exemplary multimedia container file. It was designed as a flexible and extensible format that facilitates the exchange, management, editing, and presentation of multimedia. In the following disclosure, the term "ISOBMFF" refers to a specific container file format, but is also used below to represent any media container file format to which the following underlying principles apply.

[0073] An ISOBMFF may contain timing, structure, and media information for a timed sequence of media data, such as an audiovisual presentation. The file structure may be designed to be object-oriented. An ISOBMFF may, for example, be decomposed in a simple manner into basic objects. The structure of the objects may be implied and derived from their defined types.

[0074] An ISOBMFF compliant file may be formed as a series of objects called "boxes". All data may be stored within the boxes. Boxes may be hierarchically cascaded and there may be no other data within the file. The presentation (e.g., a motion sequence) may be stored within several files. All timing and framing information may be included in the ISOBMFF file, and the auxiliary files may use essentially any format.

[0075] For example, a file type box ("ftyp") may be placed at the beginning of an ISOBMFF file. Within the file type box, general information may be specified, including but not limited to the type of encoding used, how the data for each encoding is stored, any constraints and extensions that apply to the file, compatibility and / or intended use of the file. As another example, the file type box may be followed by a movie box that contains cascading boxes that define various tracks of content.

[0076] ISOBMFF may support streaming of media data over a network as well as local playback. ISOBMFF files that support streaming may contain information about the data units to be streamed (e.g., how the basic streaming data in the file should be provided via the streaming protocol).

[0077] In some example implementations, a container file may be used to describe the media content that corresponds to a particular representation in the streaming manifest, in which each representation described in the manifest may be associated with a media container file.

[0078] Picture in Picture In some example implementations, visual media content may be overlaid in a picture-in-picture (PiP) mode. A picture-in-picture use case is illustrated in FIG. 4. As illustrated in FIG. 4, a PiP view includes a main picture and a PiP. The main picture captures the entire screen, while the overlay picture captures a portion of the screen and covers a corresponding area of ​​the main picture. The coordinates of the PiP are indicated by x, y, height, and width, and these parameters correspondingly define the position (e.g., top-left corner pixel coordinate) and size of the PiP relative to the main picture coordinates.

[0079] In the case of streaming media, the main video and the PiP video may be delivered as two separate streams. If they are independent streams, they may be decoded by separate decoders and then composed together for rendering. In some exemplary implementations, if the video codec supports stream merging, the PiP video stream may be combined with the main video stream, possibly replacing the streaming representing the coverage area of ​​the main video with the PiP video, and then a single stream may be sent to the decoder for decoding and subsequent rendering.

[0080] Therefore, in order to provide efficient PiP processing, various signaling information may be included in the media container file to specify the possible roles and relationships of various media contents in a PiP experience. Such signaling is then interpreted by the playback device for the intended PiP experience. Similarly, such signaling may also be included in the manifest for streaming (e.g., DASH MPD) to indicate various PiP possibilities to the streaming client. The streaming client can then parse the manifest and determine when and how to use and provide the PiP experience to the user. Such PiP signaling in the streaming manifest may be derived, for example, from PiP signaling in the underlying media container file associated with the streaming media.

[0081] A general PiP signaling solution may, for example, indicate how two or more tracks may be used for a PiP experience, i.e., the decoded pictures of a track may be overlaid on top of regions of the decoded pictures of the other track(s). Exemplary information that may be directly signaled or at least derived from the signaling may include, but is not limited to, the following: · The possible combinations of media content (e.g. tracks) that can create a PiP experience. Identifying the main and overlay content or tracks in a PiP experience. · Indicating the position of overlay content or tracks in a PiP experience.

[0082] The following example implementations provide modifications / extensions of several existing ISOBMFF and DASH manifest frameworks to support PiP signaling that enables a comprehensive and flexible PiP experience.

[0083] For example, the disclosed scheme uses the concept of track groups to define and signal media tracks as a group for providing a PiP experience. The exemplary scheme allows for multiple different PiP media track groupings that are uniquely identified. The media tracks within a PiP experience track group may be independently decodable or may be merged together for decoding. The exemplary scheme correspondingly includes signaling indicating such merging capabilities in each of the PiP experience track groups.

[0084] As another example, a picture-in-picture experience may be signaled in a streaming manifest. Such signaling may include identifying an adaptation set with a picture-in-picture descriptor, as well as using a role descriptor to identify the PiP experience, and further annotating one or more sub-pictures in the manifest where they are identified as one or more regions for substitution.

[0085] Exemplary PiP signaling in a media container file In some exemplary implementations, a track pre-selection group may be used to specify a PiP experience. As shown in 501 of FIG. 5, a media container file, for example of an ISOBMFF, may include a metadata box, referred to as "moov" in 502 of FIG. 5. An exemplary "moov" box 5020 may include general information for the media content included in the media container file. The "moov" box may include a movie header box, referred to as "mvhd" in 504 of FIG. 5, that includes information such as the creation and modification times of the presentation associated with the "moov" box.

[0086] The "moov" box 502 may additionally include a track group description box, referred to as "tkgd" at 506 in FIG. 5. The "tkgd" box may specify one or more descriptions of track groups for various media experiences thereunder. The specification or description of each of the track groups may be included in a preselected track group entry box, referred to as "prse" at 508 in FIG. 5. Each of these preselected groups may be identified as a collection of media tracks that together achieve a particular media experience. In the example of FIG. 5, two track groups are specified that correspond to two separate "prse" boxes.

[0087] Each of the preselection track group entry "prse" boxes may include a "track_group_id" element to specify a unique identification of the corresponding preselection group of tracks. The two example track groups specified in FIG. 5 correspond to track_group_id=1 and 2. Each of the preselection track group entry "prse" boxes also includes a "num_tracks" element to specify a number of tracks in the preselection group of tracks. Each of the "prse" boxes may optionally include one or more "preselection_tags", which may be codec-specific values ​​that the playback system can provide to the decoder to uniquely identify one out of several preselections in the media with respect to the nature of the preselection experience. In the example of FIG. 5, preselection track group 1 is identified as being a preselection for a PiP experience by its "preselection_tag". The identification of the "preselection tag" element using a "pip" value to signal a PiP is merely optional. As described below, the PiP experience may be signaled using other alternative elements in the "prse" box (e.g., in the "Type" box described below). For example, in the example of FIG. 5, the preselection group with "track_group_id"=2 is also a PIP preselection group. Here, an indication of the PIP may not be provided in the "preselection tag", but rather in the "Type" box. In some example implementations, both signaling options may be provided, and each of the "prse" boxes (or each preselection group) may use a different option to signal the PiP experience.

[0088] Each of the preselected track group entries "prse" may further include a "Type" box, which may alternatively be used to specify a role for the corresponding preselected track group. For example, a predefined role scheme may be defined for track group preselection. The predefined role scheme may specify a set of roles for track group preselection. The "Type" box for a particular preselected group specified by "prse" may include a data item indicating one of the predefined roles. For example, the "Type" box may signal a picture-in-picture experience as one of the predefined roles in which one or more areas of the main picture may be overlaid with one or more alternative pictures. The predefined role scheme may be specified, for example, by a DASH Role schemeIdURI as described in more detail below, and one of the predefined role values ​​may be "pip" to indicate a PiP experience. Other alternative role schemes may be used.

[0089] Separately within the "moov" box, the various media tracks may each be described by a track box called "trak" at 510 in Figure 5. As shown in Figure 5, multiple media tracks may be described / defined within a media container file. Correspondingly, there may be multiple "trak" boxes 510 included in the container file. Figure 5 shows three such examples, identified by their unique identifiers trak_id=1, 2, and 3.

[0090] Each of the "trak" boxes provides basic information of the corresponding media track. For example, a media track identifier may be specified ("trak_id"). Each of the "trak" boxes may further include an optional track group box, referred to as "trgr" at 512 in FIG. 5. The track group "trgr" box of each of the media tracks specifies one or more pre-selection track groups to which the media track belongs. Each of the one or more track groups to which the media track belongs may be specified by one pre-selection box, referred to as "pres" at 514 in FIG. 5. Thus, the track group "trgr" box of a media track may include one or more pre-selection boxes "pres", depending on how many pre-selection groups (as defined in the "prse" box 508 of the "tkgd" box 506 above) this particular media track is associated with.

[0091] Each of the one or more preselection "pres" boxes 514 may identify a corresponding track group by a preselection group ID, as shown in 516 of FIG. 5, and may further optionally specify how this particular media track is used relative to other media tracks in the preselection track group for the corresponding experience via, as an example, a preselection processing box referred to as "prsp" in 518 of FIG. 5. The "prsp" box may be optional for a particular group of tracks preselection and may be included as needed. For example, in FIG. 5, each of the media tracks having trak_id=1, 2, and 3 may be associated with or belong to one or both of the two exemplary preselection track groups defined by the two "prse" boxes 508 in the "tkgd" box 506 above, including the "prsp" box.

[0092] The example pre-selection process "prsp" box 518 for a media track with trak_id=1 for a particular pre-selection group with track_group_id may specify a track order parameter referred to as "track_order" at 520 or 521 in FIG. 5. The track order parameter may be used to specify a priority among the media tracks in the pre-selection group. For example, a lower value of the track order parameter may indicate a higher priority. In particular, in the example PiP pre-selection group, a track that is part of the main picture of the PiP experience may be set to track_order=0, as shown at 520 for the track with track_id=1. Any other media tracks of the PiP pre-selection group for a picture-in-picture experience that are intended to be used as an overlay or replacement for one or more regions of the main picture may be set to a track_order value greater than 0, as shown at 521 for tracks with track_id=1 and 2. Within the alternative tracks of the pre-selection group, a lower track_order value indicates the priority of the alternative picture of the media track relative to the other media tracks.

[0093] The example pre-selection processing "prsp" box 518 may further optionally specify a sample merge flag, referred to as "sample_merge_flag" in 522 and 523 of FIG. 5. The sample merge flag may be specified as a binary value. For example, a "sample_merge_flag" with a binary value of 1 may indicate that this track belongs to a merge track group and may be mergeable with one or more other tracks in this track group for merge decoding (merging of main and subpicture bitstreams before decoding). The media tracks in this merge track group may be sorted according to their "track_order" values, as specified with respect to the pre-selection group (520 and 521 of FIG. 5) and described above. A merge group may, for example, contain one media track with sample_merge_flag=1, followed by one or more media tracks with sample_merge_flag=0 or no sample_merge_flag (default 0 in this case). For example, all tracks in a merge group may be of the same media type and all samples may be time-aligned. In a PiP pre-selection group, the track with sample_merge_flag=1 may be the main picture with which other tracks may be merged and then decoded as merged to form a PiP experience, and media tracks with sample_merge_flag=0 in a group with a main track with sample_merge_flag=1 may be alternate pictures.

[0094] In the example of Figure 5, the PiP group with track_group Id=2 has a mergeable main track (which has track_id=1 and has its sample_merge_flag=1, as shown at 523) and a sub-picture track with track_id=3 that can be merged into the main picture (track_id=1) in this PiP group, but is not by itself mergeable for inclusion into other tracks by not including sample_merge_flag in its 'prsp' box. In this example, if sample_merge_flag is included for track 3 for group 2, it is included as having a 0 value.

[0095] Further in the example of FIG. 5, the PiP group with track_group Id=1 has a non-mergeable main track (with track_id=1 and its sample_merge_flag=0, as shown at 522). Therefore, this PiP should not be merged before decoding. Rather, each track is decoded separately and then merged according to the PiP configuration to form a combined picture after decoding. This PiP group includes track 2 as a sub-picture. Its sample_merge_flag, whether present or not, is not significant for this PiP group.

[0096] A media track participating in a PiP pre-selection group may further optionally include, for example in its "prsp" box corresponding to the PiP pre-selection group, a list of identifiers of regions, referred to as "region_ids" in 524 and 525 of FIG. 5. The list of region IDs may identify a list of marginal regions of coded video data units in this media track that other media tracks in the PiP pre-selection group may overlay. This field indicates the location of regions in the pre-decoding video bitstream of mergeable tracks where merging may take place, so that the decoder can properly assemble the merged bitstream for decoding. A NULL string or the absence of "region_ids" may indicate that the absence of this region of this corresponding media track is replaceable or substitutable in the coded bitstream. Thus, 'sample_merge_flat' should be set to 1 (mergeable before decoding) when 'region_ids' is not a NULL string, and thus the combination of sample_merge_flag=1 with non-NULL 'region_ids' may indicate that the coded data units represented by the 'region_ids' of this corresponding track may be replaced by other tracks with a track_order value greater than 0. Furthermore, in some example implementations, media tracks with non-NULL 'region_ids' should also have a 'track_order' of 0, indicating the media track as the main picture in a PiP experience.

[0097] With the above exemplary configuration scheme, an exemplary method for signaling PiP in a media container file may be constructed. In such an exemplary method, A pre-selected track group entry box in the container file, e.g., the "prse" box, may be used to define a track group configured to provide a PiP experience. A track group identifier, for example "track_group_id", may be used in the Pre-Select Track Groups entry box to specify an identifier for an example media track group associated with the example PiP experience. An integer element, for example "Num_tracks", may be included in the pre-selected track group entry box in the track group definition to indicate the total number of component tracks in the PiP pre-selected group. · An information box, for example a "Type" box, may be included within the pre-selected track group entry box to specify the role of the pre-selected group within a pre-defined set of roles that includes PiP. A pre-selection box within a track box, e.g. a "pres" box, to specify a media track group and indicate that the track is part of a group that may be configured to provide a PiP experience (a PiP pre-selection group). · A track group preselection identifier in the preselection box ("pres") of a media track to identify the corresponding track group, which in combination with the track group definition in the "prse" box above determines whether the track group is for a PiP experience or not. · A pre-selection processing box within a media track's pre-selection box ("pres") to define how the media track is processed for a PiP experience selection group, e.g. the "prsp" box. A priority value, e.g. "track_order", as part of the media track's preselection processing box ("prsp") associated with its PiP preselection group to indicate whether this media track is the main picture of the PiP experience (e.g. track_order=0) or an alternative picture of the PiP experience (e.g. track_order=1). In case the main PiP consists of multiple tracks, these multiple tracks contain "prsp" boxes under a "pres" box with the same track_group_id with track_order=0. A merge directive, e.g., “sample_merge_flag”, as part of the preselection processing box (“prsp”) of a media track associated with a PiP preselection group of media tracks to indicate whether a coded media track in a PiP experience is mergeable with other coded tracks in the group prior to coding, e.g., sample_merge_flag=1 may indicate that the track is mergeable. A region list, e.g., "region_ids", as part of a media track's preselection processing box ("prsp") associated with the media track's PiP preselection group, to specify a list of identifiers of interchangeable regions of the media track for the PiP experience before decoding.

[0098] Again, merely by way of example and as an application of the signaling principles described above, the container file of Figure 5 includes two track pre-selection groups, group 1 and group 2, and three media tracks, track 1, track 2, and track 3. Pre-selection track group 1 includes media tracks 1 and track 2, and pre-selection track group 2 includes media tracks 1 and track 3. Both pre-selection track groups provide a PiP experience, either by a "preselection_tag" having a value "pip", as specified in the "Type" box (the contents of the "Type" box are not shown in Figure 5 but are described above).

[0099] For pre-selection track group 2 in FIG. 5, a PiP experience is defined with the possibility of replacing the sub-picture stream of the main PiP picture with an alternative picture stream for decoding. Of the two media tracks (media track 1 and track 3) in this exemplary PiP experience selection group (track_group_id=2), media track 1 is provided with a "prsp" box to indicate how track 1 is used in the PiP experience with respect to the other media track, media track 3. Specifically, the media track with trak_id=1 forms the main picture (as indicated by its "track_order" of 0). Media track 1 as the main picture can be merged with content from other media tracks (e.g., media track 3) as indicated by its "sample_merge_flag" set to "1" for media track 1. Furthermore, a region with a region ID value of "1" in media track 1 can be replaced with the content of media track 3 before single decoding.

[0100] For pre-selected track group 1 in FIG. 5, the PiP experience is defined with two independent decodings (rather than a single decoding of the merged stream) because the sample_merge_flag associated with main picture track 1 is set to 0.

[0101] In the following it is further explained how the syntax for elements in the PiP signalling scheme within the above mentioned media container file can be specified.

[0102] Preselection Processing Box Definition: Box type: 'prsp' Container:PreselectedGroupBox ("pres") Required: No (optional) Quantity: Zero or One (Yes or No)

[0103] As mentioned above, and by way of example, this box is found in the "moov" box, in the "trak" box of a media track, in the "trgr" box, in the "pres" box. It may contain information about how the tracks contributing to the pre-selection may be processed. In some example implementations, a media type specific box may be used to describe further processing in the "prsp". This box is either present or absent in the pre-selection box in the track.

[0104] Preselection Processing Box Syntax aligned(8)class PreselectionProcessingBox extends FullBox('prsp',version=0,flags){ unsigned int(8) track_order; unsigned int(1)sample_merge_flag; unsigned int(7) reserved; utf8string region_ids; / / Further attributes and boxes to define additional processing: / / Tracks that contribute to preselection }

[0105] The semantics of the various syntax elements of the preselection processing box are explained further below.

[0106] Preselection Processing Box Semantics · "track_order" defines the ordering of this track relative to other tracks in the preselection group, as described below. · "sample_merge_flag" equal to 1 indicates that this track can be merged with other tracks before decoding. "region_ids" specifies a list of margin IDs for coded video data units whose corresponding alternative streams can replace other coded tracks in this preselection before decoding. A NULL string means that there are no replaceable regions. For example, this field can only have a non-NULL value if track_order=0. The use of this field for the PiP experience is described above and below, and in further detail below.

[0107] For example, in a picture-in-picture application, all tracks that are part of the main picture will have their track_order = 0. Any track in a picture-in-picture application that is intended to be used as an overlay or replacement for one or more regions of the main picture (known as an alternate picture) will have its track_order set with a value greater than 0. A lower track_order value indicates priority for the alternate picture.

[0108] Sample entry specific specifications may require that tracks for preselection be provided to each decoder instance in a particular order. Since other means such as track_id are unreliable for this purpose, track_order may be used to order the tracks in the preselection relative to each other. A lower number indicates that at a given time, samples of the corresponding track will be provided to the decoder before samples of the track with a higher track order number. If two tracks in a preselection have their track_order set to the same value, or if the preselection processing box is not present for at least one of the tracks, then the order of these tracks is irrelevant to the preselection and samples can be provided to the decoder in any order.

[0109] A merge group may be defined as a group of tracks sorted according to track_order, with one track with sample_merge_flag set to 1 followed by a group of consecutive tracks with sample_merge_flag set to 0. All tracks in a merge group shall be of the same media type and all samples shall be time aligned.

[0110] If the sample entry type is associated with a codec-specific process for merging preselected samples, this process is used.

[0111] The combination of sample_merge_flag=1 and non-NULL region_ids may indicate that the coded data units represented by the ids in region_ids may be replaced by other tracks with track_order values ​​greater than 0. The specific semantics of region IDs need to be explicitly specified for a particular codec.

[0112] In some example implementations, if the tracks in a merge group are all of the sample entry type "mhm2" (MPEG-H 3D Audio), the merge process may be defined, for example, in Section 14.6 of ISO / IEC 23008-3:2019.

[0113] In some example implementations, tracks within a merge group may have different sample entry types.

[0114] In some example implementations, if the sample entry type is not associated with a codec-specific process for merging pre-selected samples, and when region_ids is NULL, the following process shall be used: Merging within a merge group may proceed by forming tuples of track samples with the same timestamp across contributing tracks. The order of samples within the tuple may be determined by track_order. These tuples may be formed by byte-wise concatenation of the samples, resulting in a single sample assigned a respective timestamp. If the generation of a new track is the goal, each merge group may result in a separate output track conforming to a media type derived from the media type of the merged tracks. For tracks that are not part of a merge group, the merging process is not specifically limited by this disclosure.

[0115] Preselection Track Group Entry Box Definitions Box type:'prse' Container: TrackGroupDescriptionBox Required: No Quantity: zero or more

[0116] As mentioned above, the "prse" box may be located within the track preselection group box "tkgd" which is located within the "moov" box. This may contain information regarding the definition of the various preselection track groups.

[0117] Pre-selection can be authorized by language, type, or media-specific attributes, such as, for example, audio rendering instructions, audio interactivity, or channel layout. Attributes signaled in the pre-selection track group entry box may take precedence over attributes signaled in the contributing tracks.

[0118] The preselected track groups entry box may be configured to describe only track groups identified by track_group_type equal to "prse".

[0119] In some example implementations, all pre-selections that have at least one contributing track with the track_in_movie flag set to 1 may be acknowledged by a pre-selection track group entry box. Otherwise, the presence of the pre-selection track group entry box may be optional.

[0120] In some example implementations, all attributes that uniquely qualify a pre-selection shall be present within the pre-selection track group entry box of the pre-selection.

[0121] Preselection Track Group Entry Box Syntax aligned(8)class PreselectionTrackGroupEntryBox extends TrackGroupEntryBox('prse',version=0,flags) { unsigned int(8)num_tracks; utf8string preselection_tag; if(flags&1) { unsigned int(8)selection_priority; } if(flags&2) { unsigned int(8)segment_order; } / / A box describing the preselection }

[0122] The semantics of the various syntax elements of the preselection processing box are explained further below.

[0123] Preselection Track Group Entry Box Semantics The pre-selection track group entry box "prse" may contain information about which experiences are available when the corresponding pre-selection group is selected. Boxes suitable for describing pre-selections include, but are not limited to, the following list of boxes defined herein: A box that defines an audio element: AudioElementBox A box that provides audio element selection: AudioElementSelectionBox Extended Language Box:ExtendedLanguageBox - Box for specifying other user data: UserDataBox Box for specifying preselection type and information: KindBox A box that provides labeling information: LabelBox Audio rendering indication box: AudioRenderingIndicationBox A box for specifying the channel layout: ChannelLayout

[0124] In some example implementations, if a UserDataBox is included in a preselected track group entry box, it does not carry any of the above boxes.

[0125] In some example implementations, num_tracks specifies the number of non-alternate tracks grouped by this preselected track group.

[0126] In some example implementations, tracks grouped by a preselection track group may be tracks that have a "pres" track group with track_group_id equal to the ID of this preselection.

[0127] In some example implementations, the number of non-alternate tracks grouped by this pre-selected track group may be the sum of the following: the number of tracks grouped by this preselected track group that have alternate_group equal to 0, The number of unique non-zero alternate_group values ​​in all tracks grouped by this preselection track group.

[0128] In some example implementations, the value of num_tracks may be greater than or equal to the number of non-alternate tracks grouped by this preselected track group in this file. A value equal to 0 may indicate that the number of tracks grouped by this track group is unknown or not critical to processing the track group.

[0129] In some example implementations, the value of num_tracks may be greater than the number of non-alternate tracks that contain preselection group boxes ("pres") with the same track_group_id within this life when the preselection is split into multiple files.

[0130] In some example implementations, when a player has access to fewer non-alternate tracks grouped by this pre-selected track group than indicated by num_tracks, the player may need to omit tracks grouped by this pre-selected track group.

[0131] In some example implementations, the preselection_tag may be a codec-specific value that a playback system can provide to a decoder to uniquely identify one out of several preselections in the media.

[0132] In some example implementations, selection_priority may be an integer that declares the priority of pre-selection when other distinctions such as by media language are not possible. A lower number indicates a higher priority.

[0133] In some example implementations, segment_order, if present, specifies the segment ordering rule that is proposed to be followed to order the received preselected segments. The following values ​​are specified, by way of example, with semantics according to clause 5.3.11.5 of ISO / IEC 23009-1:2022: 0: undefined 1: Time Series 2: Total ordering

[0134] In some example implementations, other values ​​may be reserved. If segment_order is not present, its value is inferred to be equal to 0.

[0135] In some example implementations, not all tracks that contribute to the playback of a pre-selection are delivered in the same file.

[0136] In some example implementations, as described above, the type box may utilize the role scheme defined in clause 5.8.5.5 of ISO / IEC 23009-1:2022, since it provides a commonly used scheme for describing the characteristics of a pre-selection.

[0137] In some example implementations, the pre-selection track group entry box may carry information about the initial experience of the pre-selection in the reference tracks. The pre-selection experience may change during the playback of these tracks, for example, the audio language may change during playback. These changes are not affected by the information displayed in the pre-selection track group entry box.

[0138] In some example implementations, additional media type specific boxes may be used to describe the characteristics of the pre-selection. Readers may ignore and skip over unrecognized boxes.

[0139] In some example implementations, as described above, the Kind Box may be used to signal a picture-in-picture experience, where one or more areas of the main picture may be overlaid with one or more alternate pictures using the DASH Role schemeIdURI and value "pip".

[0140] Preselection Track Group Entry Box Design: Advantages This above design of the pre-selection track group entry box may provide several advantages. · Use the existing pre-selected track group box for PiP signaling. Allows multiple groups of PiPs to share a track. Allows for more than two alternative pictures in a PiP experience. Allows a main picture consisting of several tracks. Allows replacement of main picture subpictures / regions with alternative streams for single decoding of all tracks in a PiP.

[0141] Exemplary PiP Signaling in a Streaming Manifest In addition to including various signaling information in a media container file to specify possible roles and relationships of various media contents in a PiP experience, as described above, such signaling may also be included in a manifest for a streaming application (e.g., DASH MPD) to indicate various PiP possibilities to a streaming client. The streaming client can then determine when and how to use and provide a PiP experience by parsing the manifest and adaptively constructing a media request to a content server. Such PiP signaling in a streaming manifest may be derived, for example, from PiP signaling in an underlying media container file associated with the streaming media, as described above. In general, manifests associated with media files and media, such as container files, may be reconciled at generation time.

[0142] In some example implementations, the PiP experience may be provided as an additional role via the existing role scheme. Such a role may be signaled as a streaming manifest at various levels. For example, such a role may be signaled at the adaptation set level.

[0143] In one example implementation, example values ​​for the DASH Role scheme specified in urn:mpeg:dash:role:2011 may include the following values ​​to support PiP signaling in the manifest:

[0144] [Table 1A] [Table 1B]

[0145] The values ​​"PIP-main" and "PIP-sub" are specifically included for signaling PiP related information in a streaming manifest (e.g., DASH MPD). Thus, at the adaptation level, a Role@value specified as "PIP-main" for an adaptation set in the MPD signals to a streaming client that the corresponding adaptation set may be part of a PiP experience in providing a main picture of the PiP experience, whereas a Role@value specified as "PIP-sub" for an adaptation set in the MPD signals to a streaming client that the corresponding adaptation set may be part of a PiP experience in providing an alternative picture to the main picture in the PiP experience.

[0146] For example, such role values ​​in the role scheme may be included in a picture picture descriptor in a streaming manifest for signaling PIP contents and their configuration. Such PiP descriptors may be specified at various levels. For example, such PiP descriptors may be specified at an adaptation set level in a streaming manifest.

[0147] For example, a SupplementalProperty element in an adaptation set may include a @schemeIdUri attribute equal to a predefined urn that specifies the role scheme above that includes a PiP value.

[0148] At the exemplary adaptation set level, adaptation sets signaled with the same @id for the same descriptor and SupplementalProperty descriptor are considered to be intended for use in one PiP experience. As mentioned above, a PiP experience may consist of one or more main presentations. Any adaptation set of each main presentation may be annotated with a role descriptor with @value='pip-main'. A PiP experience may also include one or more alternative presentations, one or more of which may be overlaid on top of one of the main presentations. Any adaptation set of each alternative presentation may be annotated with a role descriptor with @value='pip-sub' in the SupplementalProperty descriptor.

[0149] In some example implementations, an adaptation set may include two or more PiP descriptors with different @ids to represent two or more PiP experiences. In other words, an adaptation set may be part of multiple different PiP experiences identified by different PiP descriptors in the adaptation set with different @id values. The same adaptation set may be a main picture in some of the multiple PiP experiences (and thus have an @value of "pip-main" for the corresponding @id value's descriptor), but a sub-picture in some others of the multiple PiP experiences (and thus have an @value of "pip-sub").

[0150] Specifically, an exemplary manifest main includes the following: AdaptationSet{ AdaptationSet-id=1 SupplementalProperty @id=1 @value="pip-main" ... } SupplementalProperty @id=2 @value="pip-main" ... } SupplementalProperty{ @id=3 @value="pip-sub" ... } } AdaptationSet{ AdaptationSet-id=2 SupplementalProperty{ @id=1 @value="pip-sub" ... } SupplementalProperty{ @id=3 @value="pip-main" ... } } AdaptationSet{ AdaptationSet-id=3 SupplementalProperty{ @id=1 @value="pip-sub" ... } SupplementalProperty{ @id=2 @value="pip-sub" ... } }

[0151] In this example, there are three SupplementalProperty IDs: 1, 2, and 3, each representing a PiP experience. In the first PiP experience, with a SupplementalProperty ID of 1, Adaptation Set 1 is the main picture, while Adaptation Sets 2 and 3 are subpictures. In the second PiP experience, with a SupplementalProperty ID of 2, Adaptation Set 1 is the main picture, while Adaptation Set 3 is a subpicture. In the third PiP experience, with a SupplementalProperty ID of 3, Adaptation Set 2 is the main picture, while Adaptation Set 1 is a subpicture.

[0152] The above descriptors may be used at levels other than the adaptation set level. These descriptors provide a flexible way to signal any combination of different adaptation sets within a PiP experience. An adaptation set can be used in multiple PIP experiences, which may be main picture or sub-picture. Each PiP experience may have a main adaptation set and one or more sub-adaptation sets. A PiP experience is identified by the @id in the SupplementalProperty descriptor.

[0153] In some further example implementations, a "ContentComponent" element may be defined and used to describe the characteristics of various sub-pictures of a coded stream to replace portions of the main picture.

[0154] Specifically, an adaptation set of a PiP main presentation may use one ContentComponent element to identify content component portions of the coded main presentation that are intended to be replaced by one or more PIP alternative presentations. Thus, the ContentComponent@tag may contain information for the decoder for the replacement process. For example, the ContentComponent may indicate a coded video data unit representing a target PiP region in the main video that will replace a corresponding coded video data unit of the PiP video before sending to the video decoder. In this way, the coded stream of the PiP can be merged before a single decoding and separate decoding of the main video and the PiP video can be avoided. For a particular picture in the main video, the corresponding video data unit of the PiP video may be all the coded video data units within the decoding time-synchronized sample in the supplemental video representation. In some example implementations, the format and semantics of the ContentComponent@tag may be defined by the corresponding coded video specification.

[0155] In a particular exemplary Coded Video specification, a sub-picture for a PiP may be identified by a sub-picture id. The following exemplary syntax for the ContentComponent@tag may be used: subpic1 subpic2 …. where Subpic 1, subpic 2... are whitespace separated subpicture ids of regions of the coded video bitstream, each defining one subpicture, and the group defines the overall region that can be used for picture-in-picture overlay.

[0156] In some example implementations, ContentComponent@id may be used to identify the region id for this content component and may be set equal to the region id identified in other ways, such as in the container file format as described above.

[0157] Various example fields of a ContentComponent are shown in Table 2 below.

[0158] [Table 2]

[0159] The above example ContentComponent element can be specified in an adaptation set or representation to annotate its subpictures. A streaming client (e.g., a DASH client) can provide the annotations to a bitstream manipulator to replace the desired subpicture stream with a picture-in-picture video stream before decoding and then feed the manipulated bitstream to an appropriate decoder.

[0160] FIG. 6 shows an example data and logic flow 600 for PiP signaling in a media container file. In step 602, a media container file constructed in ISO Base Media File Format (ISOBMFF) is received. The media container file includes a main media track and at least one sub-media track represented in a PiP mode. In step 604, the media container file is parsed to identify media track groups for a picture-in-picture (PiP) experience via a Pre-Selected Track Group Entry (Prse) box syntax element for a media track group definition. In step 606, the media container file is further parsed to identify a main media track and at least one sub-media track belonging to a media track group via a Pre-Selected Group (Pres) box syntax element in a plurality of media track definitions in the media container file. In step 608, the media container file is further parsed to determine a merge mode indicating whether a coded data unit of the sub-media track is merged with a coded data unit of the main media track before being decoded according to the presence or value of a syntax element of a sample flag merge syntax element in a component of a Pres box element of the main media track or at least one sub-media track. In step 610, the main media track and the at least one sub-media track are decoded in PiP mode according to a merging mode.

[0161] 7 shows another exemplary data and logic flow 700 for PiP signaling in a streaming media manifest. In step 702, a streaming media manifest is received from a streaming server. In step 704, the streaming media manifest is parsed to identify a set of PiP signaling information items associated with a set of streaming media content. In step 706, an adaptive request is constructed for the set of streaming media content according to the PiP signaling information items. In step 708, the set of streaming media content is received. In step 710, the set of streaming media content is decoded and displayed according to the PiP signaling information items.

[0162] The techniques described above can be implemented as computer software using computer readable instructions and physically stored on one or more computer readable media. For example, Figure 8 illustrates a computer system (800) suitable for implementing certain embodiments of the disclosed subject matter.

[0163] Computer software can be coded using any suitable machine code or computer language that is amenable to mechanisms such as assembly, compilation, linking, etc., to create code that includes instructions that can be executed by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., directly, or via interpretation, microcode execution, etc.

[0164] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, and the like.

[0165] 8 for computer system (800) are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing the embodiments of the present disclosure, nor should the arrangement of components be interpreted as having a dependency or requirement relating to any one or combination of components illustrated in the exemplary embodiment of computer system (800).

[0166] The computer system (800) may include certain human interface input devices. Such human interface input devices may respond to input by one or more human users, for example, via tactile input (keystrokes, swipes, data glove movements, etc.), audio input (voice, clapping, etc.), visual input (gestures, etc.), and olfactory input (not shown). The human interface devices may also be used to capture certain media not necessarily directly associated with conscious human input, such as audio (e.g., voice, music, ambient sounds), images (e.g., scanned images, photographic images obtained from still image cameras), and video (2D video, 3D video including stereoscopic video, etc.).

[0167] The input human interface devices may include one or more (only one of each is shown) of a keyboard (801), a mouse (802), a trackpad (803), a touch screen (810), a data glove (not shown), a joystick (805), a microphone (806), a scanner (807), and a camera (808).

[0168] The computer system (800) may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the human user's senses, for example, through haptic output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen (810), data gloves (not shown), or joystick (805), although there may also be haptic feedback devices that do not function as input devices), audio output devices (e.g., speakers (809), headphones (not shown)), visual output devices (e.g., screens (810), including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capability, each with or without haptic feedback capability, some of which may be capable of outputting two-dimensional visual output or three-dimensional or higher output via means such as stereo output), virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0169] The computer system (800) may also include human accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW (820), including media such as CDs / DVDs (821), thumb drives (822), removable hard drives or solid state drives (823), traditional magnetic media such as tape and floppy disks (not shown), dedicated ROM / ASIC / PLD based devices such as security dongles (not shown), and the like.

[0170] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the subject matter of this disclosure does not encompass transmission media, carrier waves or other transitory signals.

[0171] The computer system (800) may also include an interface (854) to one or more communication networks (855). The networks may be, for example, wireless, wired, or optical. The networks may further be local, wide area, metropolitan, vehicular, and industrial, real-time, delay tolerant, and the like. Examples of networks include local area networks such as Ethernet, cellular networks including wireless LAN, GSM, 3G, 4G, 5G, LTE, and the like, television wired or wireless wide area digital networks including cable, satellite, and terrestrial television, vehicular and industrial including CAN bus, and the like. Certain networks generally require an external network interface adapter attached to a particular general-purpose data port or peripheral bus (849) (e.g., a USB port of the computer system (800) or the like), while others are generally integrated into the core of the computer system (800) by attachment to a system bus as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (800) may communicate with other entities. For example, to other computer systems using local or wide area digital networks, such communications may be unidirectional, receive only (e.g., television broadcast), unidirectional transmit only (e.g., a CANbus to a particular CANbus device), or bidirectional. Specific protocols and protocol stacks may be used in each of these networks and network interfaces, as described above.

[0172] The aforementioned human interface devices, human accessible storage devices, and network interfaces may be attached to the core (840) of the computer system (800).

[0173] The cores (840) may include one or more central processing units (CPUs) (841), graphics processing units (GPUs) (842), specialized programmable processing units in the form of field programmable gate areas (FPGAs) (843), hardware accelerators for specific tasks (844), graphics adapters (850), etc. These devices may be connected via a system bus (848), along with read only memory (ROM) (845), random access memory (846), internal mass storage (847) such as internal hard drives or SSDs that are not user accessible, etc. In some computer systems, the system bus (848) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus (848) or via a peripheral bus (849). In one example, a screen (810) may be connected to the graphics adapter (850). Architectures for peripheral buses include PCI, USB, etc.

[0174] The CPU (841), GPU (842), FPGA (843) and accelerator (844) can execute certain instructions that may combine to constitute the aforementioned computer code. The computer code can be stored in ROM (845) or RAM (846). Temporary data can also be stored in RAM (846), while permanent data can be stored, for example, in internal mass storage (847). The use of cache memory, which may be closely associated with one or more of the CPU (841), GPU (842), mass storage (847), ROM (845), RAM (846), etc., can enable fast storage and retrieval in any of the memory devices.

[0175] The computer-readable medium can bear computer code for performing various computer-implemented operations. The medium and computer code can be those specially designed and constructed for the purposes of the present disclosure, or they can be of the available kind well known to those skilled in the computer software arts.

[0176] As a non-limiting example, a computer system (800) having an architecture, specifically a core (840), can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media may be user-accessible mass storage as described above, as well as media associated with specific storage of the core (840) of a non-transitory nature, such as the core internal mass storage (847) or ROM (845). Software implementing various embodiments of the present disclosure may be stored in such devices and executed by the core (840). Depending on the particular needs, the computer-readable media may include one or more memory devices or chips. The software may cause the core (840), and specifically the processor therein (including a CPU, GPU, FPGA, etc.) to perform certain processes or certain portions of certain processes described herein, including defining data structures stored in RAM (846) and modifying such data structures according to processes defined by the software. Additionally or alternatively, the computer system may provide functionality as a result of hardwired or otherwise embodied logic in circuitry (e.g., accelerator (844)) that may operate in place of or together with software to perform certain processes or certain portions of certain processes described herein. References to software may encompass logic, and vice versa, where appropriate. References to computer-readable media may encompass circuitry (such as integrated circuits (ICs)) that store software for execution, circuitry that embodies logic for execution, or both, where appropriate. The present disclosure encompasses any suitable combination of hardware and software.

[0177] While this disclosure describes several exemplary embodiments, there are alterations, substitutions, and various alternative equivalents that fall within the scope of this disclosure. Thus, it will be appreciated that those skilled in the art will be able to devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within the spirit and scope of the present disclosure. [Explanation of symbols]

[0178] 100 Content Delivery System 110 Content Server 120 Remote information processing device 130 Communication Network 200 DASH System 210 Content Server 212 HTTP Server 214 MPD generator 220 Ad Server 230 Information processing equipment 232 DASH client 234 DASH Applications 236 Display Devices 250 Network 302 Selection Logic 304 DASH Access Engine 306 Media Engine 308 Media Engine 800 Computer Systems 801 Keyboard 802 Mouse 803 Trackpad 805 Joystick 806 Microphone 807 Scanner 808 Camera 809 Speaker 810 Screen 820 CD / DVD ROM / RW 821 CD / DVD and other media 822 Thumb Drive 823 Removable Hard Drive or Solid State Drive 840 cores 841 Central Processing Unit (CPU) 842 Graphics Processing Unit (GPU) 843 Field Programmable Gate Area (FPGA) 844 Hardware Accelerator 845 Read-Only Memory (ROM) 846 Random Access Memory 847 Internal Mass Storage 848 System Bus 849 Surrounding Bus 850 Graphics Adapter 854 Interface 855 Communication Network

Claims

1. 1. A method for obtaining (Picture in Picture) PiP information, comprising: obtaining a media container file constructed in ISO Base Media File Format (ISOBMFF), the media container file including a main media track and at least one sub-media track represented in PiP mode; parsing the media container file and identifying media track groups for a PiP experience via a Preselected Track Group Entry (Prse) box syntax element for media track group definition; parsing the media container file and identifying the main media track and the at least one sub-media track that belong to the media track group via a Pre-Selection Group (Pres) box syntax element in a plurality of media track definitions in the media container file; analyzing the media container file to determine a merge mode indicating whether the coded data units of the at least one sub-media track are merged with the coded data units of the main media track before being decoded according to the presence or value of a sample flag merge syntax element in a component of the Pres box syntax element of the main media track or the at least one sub-media track; decoding the main media track and the at least one sub-media track in the PiP mode according to the merging mode; A method comprising:

2. The method of claim 1 , wherein the Prse Box syntax element belongs to a first predefined type of syntax element in the media container file configured to define a media track group.

3. the syntax elements of the first predefined type each include a media track group type tag element; the media track group type tag element indicates the intended experience of the corresponding media track group, including the PiP experience, using a set of predefined purpose keywords; The method of claim 2.

4. the syntax elements of the first predefined type each include a media track group type descriptor; the media track group type descriptor is configured to specify an intended experience of the corresponding media track group, including the PiP experience, using a role scheme having predefined roles. The method of claim 2.

5. The method of claim 2 , wherein each of the syntax elements of the first predefined type in the media container file comprises several tracks of the corresponding media track group.

6. The method according to any one of claims 1 to 5, wherein the Pres box syntax element belongs to a second predefined type of syntax element within the media track definition configured to specify an association of a media track with a media track group using a corresponding media track group identifier.

7. The method of claim 6 , wherein each of the syntax elements of the second predefined type includes a media track group processing descriptor (prsp) for specifying PiP processing of the media tracks for the media track group.

8. The method of claim 7 , wherein the media track group processing descriptor includes a priority parameter for indicating a priority of the media track relative to other media tracks in the media track group.

9. The method of claim 8 , wherein the priority parameter of the main media track of the PiP experience comprises a highest priority value for PiP processing.

10. The method of claim 9 , wherein the priority parameter of the at least one sub-media track of the PiP experience comprises a lowest priority value for PIP processing.

11. The method of claim 8 , wherein the sample flag merge syntax element belongs to a third predefined type of syntax element correspondingly included in the media track group processing descriptor associated with a main media track.

12. 12. The method of claim 11, wherein a syntax element of the third predefined type associated with the main media track having a predefined value indicates that the main media track can be merged with the secondary media track before being decoded.

13. 13. The method of claim 12, wherein when a syntax element of the third predefined type associated with the main media track is the predefined value, decoding the main media track and the at least one sub-media track in the PiP mode comprises merging the coded data units of the main media track and the coded data units of the at least one sub-media track for a single decoding.

14. 13. The method of claim 12, wherein the media track group processing descriptor includes region identification parameters for indicating a list of regions in a coded stream of a corresponding media track that are to be merged with coded streams of other media tracks before decoding.

15. 15. The method of claim 14, wherein the region identification parameter associated with the main media track is non-NULL when the syntax element of the third predefined type associated with the main media track is the predefined value.

16. The method of claim 14 , wherein the region identification parameters for the media track, if present, are ignored when the media track is not the highest priority.

17. The method of claim 14, wherein when the region identification parameter for the media track is non-NULL, the priority of the media track is correspondingly highest.

18. 13. The method of claim 12, wherein when a syntax element of the third predefined type associated with the main media track is not the predefined value, decoding the main media track and the at least one sub-media track in the PiP mode comprises processing the main media track and the at least one sub-media track by separate and independent decoding.

19. The method according to any one of claims 1 to 5, wherein any of the main media track and the at least one sub-media track belong to another media track group, as indicated by another Prse Box syntax element in the media container file.

20. A memory for storing instructions; Obtain a media container file constructed in ISO Base Media File Format (ISOBMFF), the media container file including a main media track and at least one sub-media track represented in PiP mode; Parsing the media container file and identifying media track groups for a picture-in-picture (PiP) experience via a Pre-Selected Track Group Entry (Prse) box syntax element for a media track group definition; Parsing the media container file and identifying the main media track and at least one sub-media track that belong to the media track group via a pre-selection group (Pres) box syntax element in a plurality of media track definitions in the media container file; analyzing the media container file to determine a merge mode indicating whether the coded data units of the at least one sub-media track are merged with the coded data units of the main media track before being decoded according to the presence or value of a sample flag merge syntax element in a component of the Pres box syntax element of the main media track or the at least one sub-media track; a processor for executing the instructions, said processor decoding the main media track and the at least one sub-media track in the PiP mode according to the merging mode; A media processing device comprising:

Images