METHOD FOR DECODING MEDIA DATA, AND DEVICE FOR DECODING MEDIA DATA

Description

1 / 113 “VIDEO DECODING MECHANISM FOR PARALLEL DECODING OF MULTIPLE INPUT VIDEO STREAMS

[0001] This application claims priority over U.S. Patent Application No. 18 / 638,091, filed April 17, 2024, and U.S. Provisional Patent Application No. 63 / 496,913, filed April 18, 2023, the entire content of which is incorporated herein by reference. U.S. Patent Application No. 18 / 638,091, filed April 17, 2024, claims the benefit of U.S. Provisional Patent Application No. 63 / 496,913, filed April 18, 2023. TECHNICAL FIELD

[0002] This disclosure refers to the decoding of video data. BACKGROUND

[0003] Digital video capabilities can be incorporated into a wide range of devices, including digital televisions, digital direct broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, digital cameras, digital recording devices, digital media players, video game consoles, cellular or satellite radiotelephones, video conferencing devices, and the like. Digital video devices implement video compression techniques, such as those described in the standards defined by MPEG-2, MPEG-4, ITU-T H.263 or ITU-T H.264 / MPEG4 Part 10, Advanced Video Coding (AVC), ITU-T H.265 (also called High Efficiency Video Coding (HEVC)), and Petition 870250090858, dated 06 / 10 / 2025, pp. 123 / 273 2 / 113 extensions of these standards, to transmit and receive digital video information more efficiently.

[0004] Video compression techniques perform spatial and / or temporal prediction to reduce or remove the inherent redundancy in video sequences. For block-based video coding, a video frame or slice can be partitioned into macroblocks. Each macroblock can be further partitioned. Macroblocks in an intracoded frame or slice (I) are coded using spatial prediction relative to neighboring macroblocks. Macroblocks in an intercoded frame or slice (P or B) can use spatial prediction relative to neighboring macroblocks in the same frame or slice or temporal prediction relative to other reference frames.

[0005] After the video data has been encoded, the video data can be packaged (packetized) for transmission or storage. The video data can be assembled into a video file conforming to any of several standards, such as the International Organization for Standardization (ISO) base media file format and extensions thereof, such as AVC. SUMMARY

[0006] In general, this disclosure describes techniques related to decoding multiple media streams for immersive media. This disclosure describes interfaces of a video decoding engine and operations related to elementary streams and metadata that can be performed by the video decoding engine. For Petition 870250090858, dated 06 / 10 / 2025, pp. 124 / 273 3 / 113 To support these operations, this disclosure also describes supplemental enhancement information (SEI) messages that can be used for various video codecs. In particular, this disclosure describes a decoding system in which multiple video decoding instances can be instantiated to support the parallel decoding of multiple video bitstreams. The video bitstreams can be selected according to the decoding and rendering characteristics, which can be signaled using profile / tier / level and hypothetical reference decoder (HRD) parameters. Additionally, bitstreams selected for parallel decoding can jointly satisfy the aggregate decoding and rendering characteristics.In this way, video bitstreams can be aggregated for parallel decoding, which can improve decoding speed and thus reduce the latency associated with displaying decoded video data.

[0007] In one example, a method for decoding media data includes: instantiating a first number of video decoder instances to be executed by video decoding hardware implemented in a circuit set; determining properties of a plurality of video media streams, the properties indicating that each of the plurality of video media streams is available for streaming selection; selecting a second number of input video media streams from the plurality of video media streams according to the determined properties of the second number of input video media streams; decoding, by the decoder instances Petition 870250090858, dated 06 / 10 / 2025, pp. 125 / 273 4 / 113 of video, the second input video media streams to form the second decoded video media streams; and output data from the second decoded video media streams.

[0008] In another example, a device for decoding video data includes a memory configured to store video data; and a processing system comprising one or more processors implemented in the circuit set, the processing system being configured to: instantiate a first number of video decoder instances to be executed by the processing system; determine properties of a plurality of video media streams, the properties indicating that each of the plurality of video media streams is available for streaming selection; select a second number of input video media streams from among the plurality of video media streams according to the determined properties of the second number of input video media streams;Execute the video decoder instances to decode the second set of incoming video media streams to form the second set of decoded video media streams; and output data from the second set of decoded video media streams.

[0009] Details of one or more examples are presented in the attached drawings and in the description below. Other attributes, objectives, and advantages will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a block diagram illustrating an example system that implements techniques for Petition 870250090858, dated 06 / 10 / 2025, pp. 126 / 273 5 / 113 streaming media data over a network, according to the techniques of the present disclosure.

[0011] Figure 2 is a block diagram illustrating elements of a sample video file.

[0012] Figure 3 is a block diagram illustrating an example video decoding mechanism, according to the techniques of the present disclosure.

[0013] Figure 4 is a block diagram illustrating an example relationship between video decoder instances and a video decoder hardware mechanism, according to the techniques of the present disclosure.

[0014] Figure 5 is a conceptual diagram illustrating an example of instantiation of multiple decoder instances of a video decoding mechanism using a video decoder interface, according to the techniques of the present disclosure.

[0015] Figure 6 is a block diagram illustrating an example video decoder interface system decoder model, according to the techniques of the present disclosure.

[0016] Figure 7 is a conceptual diagram illustrating an example input received by a filter and filtered output data, according to the techniques of the present disclosure.

[0017] Figures 8A to 8C are conceptual diagrams illustrating an example equal-width input video received by an insert function and combined output video, according to the techniques of the present disclosure. Petition 870250090858, dated 06 / 10 / 2025, pp. 127 / 273 6 / 113

[0018] Figures 9A to 9C are conceptual diagrams illustrating an example equal-height input video received by an insert function and combined output video, according to the techniques of the present disclosure.

[0019] Figures 10A and 10B are conceptual diagrams illustrating examples of input videos received by an append function and corresponding output video data, according to the techniques of the present disclosure.

[0020] Figures 11A and 11B are conceptual diagrams illustrating examples of input videos received by a stacking function and corresponding output video data, according to the techniques of the present disclosure.

[0021] Figure 12 is a conceptual diagram illustrating example locations of video data layer boundaries, according to the techniques of the present disclosure.

[0022] Figure 13 is a conceptual diagram illustrating an example of attaching two layers of video data as a connection map, according to the techniques of this disclosure.

[0023] Figure 14 is a conceptual diagram illustrating an example of attaching three layers of video data as a connection map, according to the techniques of this disclosure.

[0024] Figure 15 is a conceptual diagram illustrating an example of a video decoder mechanism employing a decoder mechanism control. Petition 870250090858, dated 06 / 10 / 2025, pp. 128 / 273 7 / 113 of video based on application for 2x2 mosaic video objects, according to the techniques of the present disclosure.

[0025] Figure 16 is a conceptual diagram illustrating an example of a video decoder mechanism executing four decoder mechanisms for 2x2 mosaic video objects, according to the techniques of the present disclosure.

[0026] Figure 17 is a conceptual diagram illustrating an example of a video decoder implementing a decoder mechanism for 2x2 mosaic video objects, according to the techniques of the present disclosure.

[0027] Figure 18 is a conceptual diagram illustrating an example implementation for formatting four media streams with different steps, according to the techniques of this disclosure.

[0028] Figure 19 is a conceptual diagram illustrating an example implementation for formatting four media streams with two steps, according to the techniques of the present disclosure.

[0029] Figure 20 is a conceptual diagram illustrating an example of a video decoder mechanism using supplementary enhancement information (SEI)-based control information for 2x2 mosaic video objects, according to the techniques of the present disclosure.

[0030] Figure 21 is a conceptual diagram illustrating an example of connected components and buffer usage, according to the techniques of this disclosure. Petition 870250090858, dated 06 / 10 / 2025, pp. 129 / 273 8 / 113

[0031] Figure 22 is a conceptual diagram illustrating example port configurations for various buffers, according to the techniques of the present disclosure.

[0032] Figure 23 is a conceptual diagram illustrating an example of media source extension (MSE) media interfaces, in accordance with the techniques of this disclosure.

[0033] Figure 24 is a block diagram illustrating a hypothetical example video decoding reference decoder.

[0034] Figure 25 is a block diagram illustrating an example model of a video decoding system that can implement techniques of the present disclosure to decode multiple video streams in parallel using multiple video decoding instances.

[0035] Figure 26 is a block diagram illustrating another example model of a video decoding system that can implement techniques of the present disclosure to decode multiple video streams in parallel using multiple video decoding instances.

[0036] Figure 27 is a block diagram illustrating another example model of a video decoding system that can implement techniques of the present disclosure to decode multiple video streams in parallel using multiple video decoding instances.

[0037] Figure 28 is a flowchart illustrating an example method of decoding video data according to Petition 870250090858, dated 06 / 10 / 2025, pp. 130 / 273 9 / 113 using the techniques of this disclosure. DETAILED DESCRIPTION

[0038] In general, this disclosure describes a video decoding mechanism that can combine multiple input video streams to form a single output video stream. In some examples, the video decoding mechanism may instantiate multiple video decoding instances to decode multiple input video streams and then combine the decoded video streams. In some examples, the video decoding mechanism may use a single video decoding instance and use an input formatting unit to combine multiple input video streams and then use the single video decoding instance to decode the combined multiple input video streams. Various combinations of these examples are also possible, as discussed below.

[0039] In this way, media data playback from multiple input video streams can be achieved. Such playback can be synchronized between the various video streams, so that the frames from the various input video streams are played back at appropriate playback times. These techniques can be used for two-dimensional video playback or other playback environments, such as extended reality (XR), augmented reality (AR), mixed reality (MR), or virtual reality (VR).

[0040] Figure 1 is a block diagram illustrating an example system 10 that implements techniques for streaming media data over a network. In this example, system 10 includes the preparation device for Petition 870250090858, dated 06 / 10 / 2025, pp. 131 / 273 10 / 113 content 20, server device 60, and client device 40. Client device 40 and server device 60 are communicatively coupled by network 74, which may comprise the Internet. In some examples, content preparation device 20 and server device 60 may also be coupled by network 74, or by another network, or they may be directly and communicatively coupled. In some examples, content preparation device 20 and server device 60 may comprise the same device.

[0041] The content preparation device 20, in the example of Figure 1, comprises an audio source 22 and a video source 24. The audio source 22 may comprise, for example, a microphone that produces electrical signals representative of the captured audio data to be encoded by the audio encoder 26. Alternatively, the audio source 22 may comprise a storage medium that stores previously recorded audio data, an audio data generator, such as a computerized synthesizer, or any other audio data source. The video source 24 may comprise a video camera that produces video data to be encoded by the video encoder 28, a storage medium encoded with previously recorded video data, a video data generation unit, such as a computer graphics source, or any other video data source.The content preparation device 20 is not necessarily communicatively coupled to the server device 60 in all examples, but it may store multimedia content on a separate medium that is read by the server device 60.

[0042] Raw audio and video data can Petition 870250090858, dated 06 / 10 / 2025, pp. 132 / 273 11 / 113 Understanding analog or digital data. Analog data can be digitized before being encoded by audio encoder 26 and / or video encoder 28. Audio source 22 can obtain audio data from a speaking participant while the speaking participant is speaking, and video source 24 can simultaneously obtain video data from the speaking participant. In other examples, audio source 22 can comprise a computer-readable storage medium comprising stored audio data, and video source 24 can comprise a computer-readable storage medium comprising stored video data. In this way, the techniques described in this disclosure can be applied to live, streaming, real-time audio and video data or to archived and pre-recorded audio and video data.

[0043] Audio frames that correspond to video frames are, in general, audio frames that contain audio data that was captured (or generated) by audio source 22 contemporaneously with video data captured (or generated) by video source 24 that is contained within the video frames. For example, although a speaking participant generally produces audio data per speech, audio source 22 captures the audio data and video source 24 captures the video data of the speaking participant at the same time, that is, while audio source 22 is capturing the audio data. In this way, an audio frame can temporally correspond to one or more particular video frames. Consequently, an audio frame corresponding to a video frame generally corresponds to a situation in which audio data and video data were Petition 870250090858, dated 06 / 10 / 2025, pp. 133 / 273 12 / 113 captured at the same time, where an audio frame and a video frame comprise, respectively, the audio data and the video data that were captured simultaneously.

[0044] In some examples, the audio encoder 26 may encode a timestamp in each encoded audio frame that represents a time at which the audio data for the encoded audio frame was recorded, and similarly, the video encoder 28 may encode a timestamp in each encoded video frame that represents a time at which the video data for an encoded video frame was recorded. In such examples, an audio frame corresponding to a video frame may comprise an audio frame comprising a timestamp and a video frame comprising the same timestamp. The content preparation device 20 may include an internal clock from which the audio encoder 26 and / or the video encoder 28 may generate the timestamps, or which the audio source 22 and the video source 24 may use to associate audio and video data, respectively, with a timestamp.

[0045] In some examples, audio source 22 may send data to audio encoder 26 that corresponds to a time when the audio data was recorded, and video source 24 may send data to video encoder 28 that corresponds to a time when the video data was recorded. In some examples, audio encoder 26 may encode a sequence identifier in encoded audio data to indicate a relative temporal order of encoded audio data, but without necessarily indicating an absolute time when the Petition 870250090858, dated 06 / 10 / 2025, pp. 134 / 273 13 / 113 audio data was recorded, and similarly, the video encoder 28 can also use sequence identifiers to indicate a relative temporal order of encoded video data. Similarly, in some examples, a sequence identifier can be mapped or otherwise correlated to a timestamp.

[0046] Audio encoder 26 generally produces a stream of encoded audio data, while video encoder 28 produces a stream of encoded video data. Each individual data stream (whether audio or video) can be called an elementary stream. An elementary stream is an example of a media stream. However, a media stream may include one or more elementary streams and / or other data, so a media stream is not necessarily the same as a single elementary stream. A media stream may also contain metadata, such as non-VCL NAL units. An elementary video data stream may include a series of images (also referred to as frames). This disclosure refers to a subframe as an independently decodable unit smaller than a frame to which post-decoding processing may have been applied by a video decoder.This disclosure refers to a video object as an independently decodable substream of an elementary video stream. This disclosure refers to a video object identifier as an integer or other value that identifies a video object.

[0047] An elementary stream is a single, digitally encoded (possibly compressed) component of a media presentation. For example, the encoded video or audio portion of a media presentation could be an elementary stream. Petition 870250090858, dated 06 / 10 / 2025, pp. 135 / 273 14 / 113 An elementary stream can be converted into a packetized elementary stream (PES) before being encapsulated in a video file. In the same media presentation, a stream ID can be used to distinguish the PES packets belonging to an elementary stream from one another. The basic data unit of an elementary stream is a packetized elementary stream (PES) packet. Thus, encoded video data generally corresponds to elementary video streams. Similarly, audio data corresponds to one or more of the respective elementary streams.

[0048] In the example in Figure 1, the encapsulation unit 30 of the content preparation device 20 receives elementary streams comprising encoded video data from the video encoder 28 and elementary streams comprising encoded audio data from the audio encoder 26. In some examples, each of the video encoder 28 and the audio encoder 26 may include packers to form PES packets from encoded data. In other examples, each of the video encoder 28 and the audio encoder 26 may interface with their respective packers to form PES packets from encoded data. In still other examples, the encapsulation unit 30 may include packers to form PES packets from both encoded audio and video data.

[0049] The 28 video encoder can encode video data from multimedia content in various ways to produce different representations of the multimedia content at various bitrates and with various characteristics such as pixel resolutions, frame rates, compliance with various encoding standards, compliance Petition 870250090858, dated 06 / 10 / 2025, pp. 136 / 273 15 / 113 with multiple profiles and / or profile levels for various encoding standards, representations that have one or multiple views (e.g., for two-dimensional or three-dimensional playback), or other features. A representation, as used in this disclosure, may comprise one of audio data, video data, text data (e.g., for subtitles), or other data. The representation may include an elementary stream, such as an elementary audio stream or an elementary video stream. Each PES package may include a stream_id that identifies the elementary stream to which the PES package belongs. The encapsulation unit 30 is responsible for assembling elementary streams into streamable media data.

[0050] The encapsulation unit 30 receives PES packets for elementary streams of a media presentation from the audio encoder 26 and the video encoder 28 and forms corresponding Network Abstraction Layer (NAL) units from the PES packets. The encoded video segments can be organized into NAL units, which provide network-friendly video representation addressing applications such as video telephony, storage, broadcast, or streaming. NAL units can be categorized into Video Coding Layer (VCL) NAL units and non-VCL NAL units. VCL units may contain the core compression engine and may include block, macroblock, and / or slice-level data. Other NAL units may be non-VCL NAL units.In some examples, an image encoded in a time instance, typically presented as a primary encoded image, may be contained within an access unit, which may... Petition 870250090858, dated 06 / 10 / 2025, pp. 137 / 273 16 / 113 include one or more NAL units.

[0051] Non-VCL NAL units may include parameter set NAL units and SEI NAL units, among others. Parameter sets may contain sequence-level header information (in sequence parameter sets (SPS)) and infrequently changed image-level header information (in picture parameter sets (PPS)). With parameter sets (e.g., PPS and SPS), rarely changed information does not need to be repeated for each sequence or image; thus, encoding efficiency can be improved. Furthermore, the use of parameter sets can enable out-of-band transmission of important header information, avoiding the need for redundant transmissions for error resilience.In out-of-band transmission examples, parameter set NAL units can be transmitted on a different channel than other NAL units, such as SEI NAL units.

[0052] Supplementary Enhancement Information (SEI) may contain information that is not necessary for decoding encoded image samples from VCL NAL units, but may assist in processes related to decoding, display, error resilience, and other purposes. SEI messages may be contained in non-VCL NAL units. SEI messages are the normative part of some standard specifications and are therefore not always mandatory for standard-compliant decoder implementation. SEI messages may be sequence-level SEI messages or sequence-level SEI messages. Petition 870250090858, dated 06 / 10 / 2025, pp. 138 / 273 17 / 113 image. Some sequence-level information may be contained in SEI messages, such as resizability information SEI messages in the SVC example and view resizability information SEI messages in multi-view video (MVC). These example SEI messages may carry information about, for example, the extraction of operation points and the characteristics of the operation points.

[0053] Server device 60 includes a Real-time Transport Protocol (RTP) transmission unit 70 and a network interface 72. In some instances, server device 60 may include a plurality of network interfaces. In addition, any or all of the attributes of server device 60 may be implemented in other devices in a content delivery network, such as routers, bridges, proxy devices, switches, or other devices. In some instances, the intermediate devices in a content delivery network may cache multimedia content data 64 and include components that substantially adapt to those of server device 60. Generally, network interface 72 is configured to send and receive data via network 74.

[0054] The RTP transmission unit 70 is configured to deliver media data to client device 40 via network 74 in accordance with RTP, which is standardized in Request for Comment (RFC) 3550 by the Internet Engineering Task Force (IETF). The RTP transmission unit 70 may also implement protocols related to RTP, such as Petition 870250090858, dated 06 / 10 / 2025, pp. 139 / 273 18 / 113 the RTP Control Protocol (RTCP), the Real-time Streaming Protocol (RTSP), the Session Initiation Protocol (SIP), and / or the Session Description Protocol (SDP). The RTP transmission unit 70 can send media data via network interface 72, which can implement the Uniform Datagram Protocol (UDP) and / or the Internet Protocol (IP). Thus, in some examples, the server device 60 can send media data via RTP and RTSP through UDP using network 74.

[0055] RTP transmission unit 70 can receive an RTSP description request, for example, from client device 40. The RTSP description request can include data indicating which data types are supported by client device 40. RTP transmission unit 70 can respond to client device 40 with data indicating media streams, such as media content 64, that can be sent to client device 40, along with a corresponding network location identifier, such as a Uniform Resource Locator (URL) or a Uniform Resource Name (URN).

[0056] The RTP transmission unit 70 can then receive an RTSP configuration request from client device 40. The RTSP configuration request can generally indicate how a media stream should be transported. The RTSP configuration request may contain the network location identifier for the requested media data (by Petition 870250090858, dated 06 / 10 / 2025, pp. 140 / 273 19 / 113 example, media content 64) and a transport specifier, such as local ports to receive RTP data and control data (e.g., RTCP data) on client device 40. The RTP transmission unit 70 can respond to the RTSP configuration request with an acknowledgment and data representing ports on the server device 60 through which the RTP data and control data will be sent. The RTP transmission unit 70 can then receive an RTSP playback request, to cause the media stream to be played, i.e., sent to client device 40 via network 74. The RTP transmission unit 70 can also receive an RTSP teardown request to terminate the streaming session, in response to which the RTP transmission unit 70 can stop sending media data to client device 40 for the corresponding session.

[0057] The RTP receiving unit 52 can similarly initiate a media stream by initially sending an RTSP description request to the server device 60. The RTSP description request can indicate the data types supported by the client device 40. The RTP receiving unit 52 can then receive a response from the server device 60 specifying available media streams, such as media content 64, that can be sent to the client device 40, along with a corresponding network location identifier, such as a Uniform Resource Locator (URL) or a Uniform Resource Name (URN).

[0058] The RTP receiving unit 52 can then generate an RTSP configuration request and send it to Petition 870250090858, dated 06 / 10 / 2025, pp. 141 / 273 20 / 113 RTSP configuration request for server device 60. As mentioned above, the RTSP configuration request may contain the network location identifier for the requested media data (e.g., media content 64) and a transport specifier, such as local ports to receive RTP data and control data (e.g., RTCP data) on client device 40. In response, the RTP receiving unit 52 may receive an acknowledgment from server device 60, including the ports on server device 60 that server device 60 will use to send media data and control data.

[0059] After establishing a media streaming session between server device 60 and client device 40, the RTP transmission unit 70 of server device 60 can send media data (e.g., media data packets) to client device 40 according to the media streaming session. Server device 60 and client device 40 can exchange control data (e.g., RTCP data) indicating, for example, reception statistics by client device 40, so that server device 60 can perform congestion control or otherwise diagnose and resolve transmission faults.

[0060] Network interface 54 can receive and provide media from a selected media presentation to the RTP receiving unit 52, which can, in turn, provide the media data to the decapsulation unit 50. The decapsulation unit 50 can decapsulate elements of a video file into constituent PES streams, unpack the PES streams to recover encoded data, and send the data. Petition 870250090858, dated 06 / 10 / 2025, pp. 142 / 273 21 / 113 encoded data is sent either to audio decoder 46 or to video decoder 48, depending on whether the encoded data is part of an audio or video stream, for example, as indicated by the stream's PES packet headers. Audio decoder 46 decodes encoded audio data and sends the decoded audio data to audio output 42, while video decoder 48 decodes encoded video data and sends the decoded video data, which may include a plurality of views of a stream, to video output 44.

[0061] According to the techniques of this disclosure, video decoder 48 may correspond to a video decoding mechanism. The video decoding mechanism, as explained in more detail below, may include an input interface for receiving one or more media data streams and one or more metadata streams. The video decoding mechanism may include hardware components, such as memory and processing circuitry, which may instantiate one or more instances of video decoder. Each instance of video decoder may be exposed to an application layer as a respective video decoder with its own interface through which it sends encoded video data to be decoded.The Video Decoder 48 can run each of several instances of the Video Decoder to decode media data streams in parallel and can format the resulting output so that the output data is synchronized in time. In some instances, the Video Decoder 48 can concatenate, append, stack, or otherwise combine or prune video data for presentation purposes. Petition 870250090858, dated 06 / 10 / 2025, pp. 143 / 273 22 / 113 via video output 44, as discussed in more detail below.

[0062] Each of the video encoder 28, the video decoder 48, the audio encoder 26, the audio decoder 46, the encapsulation unit 30, the RTP receiving unit 52 and the decapsulation unit 50 may be implemented as any one of several suitable sets of processing circuits, as applicable, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic circuit sets, software, hardware, firmware or any combination thereof.Each of the video encoder 28 and video decoder 48 may be included in one or more encoders or decoders, each of which may be integrated as part of a combined video encoder / decoder (CODEC encoder / decoder). Similarly, each of the audio encoder 26 and audio decoder 46 may be included in one or more encoders or decoders, wherein each may be integrated as part of a combined CODEC. An apparatus comprising the video encoder 28, the video decoder 48, the audio encoder 26, the audio decoder 46, the encapsulation unit 30, the RTP receiving unit 52 and / or the decapsulation unit 50 may comprise an integrated circuit, a microprocessor and / or a wireless communication device, such as a mobile phone. Petition 870250090858, dated 06 / 10 / 2025, pp. 144 / 273 23 / 113

[0063] Client device 40, server device 60, and / or content preparation device 20 may be configured to operate in accordance with the techniques in this disclosure. For example purposes, this disclosure describes these techniques with respect to client device 40 and server device 60. However, it should be understood that content preparation device 20 may be configured to perform these techniques instead of (or in addition to) server device 60.

[0064] The encapsulation unit 30 can form NAL units comprising a header that identifies a program to which the NAL unit belongs, as well as a payload, for example, audio data, video data, or data describing the transport or program stream to which the NAL unit corresponds. For example, in H.264 / AVC, a NAL unit includes a 1-byte header and a variable-size payload. A NAL unit that includes video data in its payload can comprise various levels of video data granularity. For example, a NAL unit can comprise a video data block, a plurality of blocks, a video data slice, or an entire video data image. The encapsulation unit 30 can receive encoded video data from the video encoder 28 in the form of PES packets of elementary streams. The encapsulation unit 30 can associate each elementary stream with a corresponding program.

[0065] The encapsulation unit 30 can also assemble access units from a plurality of NAL units. In general, an access unit can comprise one or more NAL units to represent a Petition 870250090858, dated 06 / 10 / 2025, pp. 145 / 273 24 / 113 video data frame, as well as audio data corresponding to the frame when such audio data is available. An access unit generally includes all NAL units for an output time instance, for example, all audio and video data for a time instance. For example, if each view has a frame rate of 20 frames per second (fps), then each time instance can correspond to a time interval of 0.05 seconds. During this time interval, the specific frames for all views of the same access unit (the same time instance) can be simultaneously rendered. In one example, an access unit might comprise an image encoded in a time instance, which can be presented as a primary encoded image.

[0066] Consequently, an access unit can comprise all audio and video frames of a common temporal instance, for example, all views corresponding to time X. This disclosure also refers to an encoded image of a particular view, as a view component. That is, a view component can comprise an image (or frame) encoded for a particular view at a particular time. Consequently, an access unit can be defined by comprising all view components of a common temporal instance. The decoding order of access units does not necessarily have to be the same as the output or display order.

[0067] After the encapsulation unit 30 has NAL units and / or access units mounted on a video file based on the received data, the unit of Petition 870250090858, dated 06 / 10 / 2025, pp. 146 / 273 25 / 113 Encapsulation 30 passes the video file to the output interface 32 for output. In some examples, the encapsulation unit 30 may store the video file locally or send the video file to a remote server via output interface 32, instead of sending the video file directly to the client device 40. Output interface 32 may comprise, for example, a transmitter, a transceiver, a device for recording data on a computer-readable medium, such as an optical drive, a magnetic media drive (e.g., floppy disk drive), a universal serial bus (USB) port, a network interface, or another output interface. Output interface 32 outputs the video file to a computer-readable medium, such as a broadcast signal, a magnetic medium, an optical medium, memory, a flash drive, or another computer-readable medium.

[0068] Network interface 54 can receive a NAL unit or a network access unit 74 and provide the NAL unit or the access unit to the decapsulation unit 50, via the RTP receiving unit 52. The decapsulation unit 50 can decapsulate elements of a video file into constituent PES streams, unpack the PES streams to recover encoded data, and send the encoded data either to the audio decoder 46 or to the video decoder 48, depending on whether the encoded data is part of an audio or video stream, for example, as indicated by the PES packet headers of the stream. The audio decoder 46 decodes encoded audio data and sends the decoded audio data to the audio output 42, while the video decoder 48 Petition 870250090858, dated 06 / 10 / 2025, pp. 147 / 273 26 / 113 decodes encoded video data and sends the decoded video data, which may include multiple views of a stream, to video output 44.

[0069] Figure 2 is a block diagram illustrating elements of an example video file 150. As described above, video files according to the ISO base media file format, and its extensions, store data in a series of objects, called boxes. In the example in Figure 2, the video file 150 includes the file type box (FTYP) 152, the movie box (MOOV) 154, the segment index boxes (sidx) 162, the movie fragment boxes (MOOF movie fragment) 164, and the fragment random access box (MFRA) 166.Although Figure 2 represents an example of a video file, it should be understood that other media files may include other types of media data (e.g., audio data, timed text data, or the like) that are structured similarly to the video file data 150, according to the ISO base media file format and its extensions.

[0070] File type box (FTYP) 152 generally describes a file type for video file 150. File type box 152 may include data identifying a specification describing a best use for video file 150. File type box 152 may alternatively be placed before MOOV box 154, film fragment boxes 164, and / or MFRA box 166.

[0071] The MOOV 154 box, in the example in Figure 2, includes the movie header box (MVHD). Petition 870250090858, dated 06 / 10 / 2025, pp. 148 / 273 27 / 113 156, the track box 158 and one or more movie extends boxes 160. In general, the MVHD box 156 may describe general characteristics of the video file 150. For example, the MVHD box 156 may include data describing when the video file 150 was originally created, when the video file 150 was last modified, a timescale for the video file 150, a playback duration for the video file 150, or other data that generally describes the video file 150.

[0072] The TRAK box 158 may include data for a video file track 150. The TRAK box 158 may include a track header box (TKHD) that describes characteristics of the track corresponding to the TRAK box 158. In some examples, the TRAK box 158 may include video-encoded images, while in other examples, the video-encoded images of the track may be included in the movie fragments 164, which may be referenced by TRAK box 158 data and / or sidx boxes 162.

[0073] In some examples, video file 150 may include more than one track. Consequently, MOOV box 154 may include a number of TRAK boxes equal to the number of tracks in video file 150. TRAK box 158 may describe characteristics of a corresponding track from video file 150. For example, TRAK box 158 may describe temporal and / or spatial information for the corresponding track. A TRAK box similar to TRAK box 158 in MOOV box 154 may describe characteristics of a parameter set track, when the unit of Petition 870250090858, dated 06 / 10 / 2025, pp. 149 / 273 28 / 113 encapsulation 30 (Figure 1) includes a parameter set track in a video file, such as video file 150. The encapsulation unit 30 can signal the presence of sequence-level SEI messages in the parameter set track in the TRAK box that describes the parameter set track.

[0074] MVEX 160 boxes can describe characteristics of the corresponding film fragments 164, for example, to signal that video file 150 includes film fragments 164, in addition to the video data included in MOOV box 154, if any. In the context of streaming video data, encoded video images can be included in film fragments 164, rather than in MOOV box 154. Consequently, all encoded video samples can be included in film fragments 164, rather than in MOOV box 154.

[0075] The MOOV 154 box may contain a number of MVEX 160 boxes equal to the number of movie fragments 164 in the video file 150. Each of the MVEX 160 boxes may describe characteristics of a corresponding fragment of the movie fragments 164. For example, each MVEX box may include a movie extends header box (MEHD) that describes a temporal duration for the corresponding fragment of the movie fragments 164.

[0076] As mentioned above, encapsulation unit 30 can store a sequence data set in a video sample that does not include actual encoded video data. A video sample can generally correspond to an access unit, which is a representation of an encoded image at a specific time instance. Petition 870250090858, dated 06 / 10 / 2025, pages 150 / 273 29 / 113 In the context of AVC, the encoded image includes one or more VCL NAL units, which contain the information to construct all the pixels of the access unit and other associated non-VCL NAL units, such as SEI messages. Consequently, encapsulation unit 30 may include a sequence data set, which may include sequence-level SEI messages, in one of the film fragments 164. Encapsulation unit 30 may additionally signal the presence of a sequence data set and / or sequence-level SEI messages as present in one of the film fragments 164 in one of the MVEX boxes 160 corresponding to one of the film fragments 164.

[0077] SIDX 162 boxes are optional elements of the 150 video file. That is, video files that conform to the 3GPP file format, or other file formats, do not necessarily include SIDX 162 boxes. Following the example of the 3GPP file format, a SIDX box can be used to identify a subsegment of a segment (e.g., a segment contained in the 150 video file). The 3GPP file format defines a subsegment as a self-contained set of one or more consecutive movie fragment boxes with corresponding media data box(es), and a media data box containing data referenced by a movie fragment box must follow the movie fragment box and precede the subsequent movie fragment box containing information about the same track.The 3GPP file format also indicates that a SIDX box contains a sequence of references to subsegments of the (sub)segment documented by the box. The referenced subsegments are contiguous in time. Petition 870250090858, dated 06 / 10 / 2025, pp. 151 / 273 30 / 113 presentation. Similarly, the bytes referenced by a segment index box are always contiguous within the segment. The referenced size provides the count of the number of bytes in the referenced material.

[0078] The SIDX boxes 162 generally provide representative information for one or more subsegments of a segment included in the video file 150. For example, such information may include playback times at which the subsegments begin and / or end, byte offsets for the subsegments, whether the subsegments include (e.g., begin with) a stream access point (SAP), a type for the SAP (e.g., whether the SAP is an instantaneous decoder refresh (IDR) image, a clean random access (CRA) image, a broken link access (BLA) image, or similar), a position of the SAP (in terms of playback time and / or byte offset) in the subsegment, and similar.

[0079] Film fragments 164 may include one or more encoded video images. In some examples, film fragments 164 may include one or more groups of pictures (GOPs), where each of these may include multiple encoded video images, for example, frames or images. Additionally, as described above, film fragments 164 may include sequence datasets in some examples. Each film fragment 164 may include a Film Fragment Header Box (MFHD, not shown in Figure 2). The MFHD box may describe characteristics of the corresponding film fragment, such as a sequence number for the film fragment. The fragments Petition 870250090858, dated 06 / 10 / 2025, pages 152 / 273 31 / 113 of film 164 can be included in sequence number order in video file 150.

[0080] The MFRA 166 box can describe random access points in the 164 movie fragments of the 150 video file. This can help perform trick modes, such as searching for particular temporal locations (i.e., playback times) in a segment encapsulated by the 150 video file. The MFRA 166 box is generally optional and does not need to be included in video files in some instances. Similarly, a client device, such as client device 40, does not necessarily need a reference to the MFRA 166 box to correctly decode and display the video data from the 150 video file. The MFRA 166 box can include a number of track fragment random access (TFRA) boxes (not shown) equal to the number of tracks in the 150 video file or, in some instances, equal to the number of media tracks (e.g., tracks without hints) in the 150 video file.

[0081] In some examples, film fragments 164 may include one or more stream access points (SAPs), such as IDR images. Similarly, MFRA box 166 may provide indications of the locations of SAPs in video file 150. Consequently, a temporal subsequence of video file 150 may be formed from SAPs of video file 150. The temporal subsequence may also include other images, such as P-frames and / or B-frames that depend on SAPs. The frames and / or slices of the temporal subsequence may be arranged in the segments so that frames / slices of the temporal subsequence that depend on other frames / slices of the subsequence may be Petition 870250090858, dated 06 / 10 / 2025, pages 153 / 273 32 / 113 properly decoded. For example, in the hierarchical arrangement of data, the data used to predict other data can also be included in the temporal subsequence.

[0082] Figure 3 is a block diagram illustrating an example video decoding mechanism 200, according to the techniques of the present disclosure. The video decoding mechanism 200 may correspond to the video decoder 48 of Figure 1. In this example, the video decoding mechanism 200 includes input video decoding interface 202, input formatting unit 204, hardware video decoding mechanism 212, time locking unit 206, output formatting unit 208, and output video decoding interface 210. The hardware video decoding mechanism 212 may run multiple instances of video decoder 214A to 214N (video decoding instances 214).

[0083] According to the techniques of this disclosure, the VDE video decoding engine 200 enables the decoding, synchronization, and formatting of media streams. Media streams include one or more aggregated elementary streams or parts thereof. Media streams are fed via the VDE 200 input video decoding interface (IVDI) 202 and provided to subsequent elements of the rendering pipeline via the output video decoding interface (OVDI) 210 in their decoded form.

[0084] Between IVDI 202 and OVDI 210, the VDE 200 input formatting unit 204 extracts and merges independently decodable regions from a set of streams of Petition 870250090858, dated 06 / 10 / 2025, pp. 154 / 273 33 / 113 input media and generates a set of elementary streams fed to the video decoder instances 214, which are running within the VDE 200. The VDE 200 can perform a merge operation or a pull operation on the input media streams, so the number of running video decoder instances 214 is different from the number of input media streams required by the application.

[0085] For example, the VDE 200 may not be able to decode a single 4K input media stream with one decoder instance, but the VDE 200 may be able to decode some of the independently decodable regions present in a lower-resolution 4K input video stream. In this case, the VDE 200 may check for the availability of sufficient resources to run 214 video decoder instances in parallel. In some instances, multiple elementary streams that are output by the 204 input formatting unit may be fed to a single 214 video decoder instance.

[0086] The VDE 200 accepts media streams and metadata streams via IVDI 202. There is at least one media stream as input, but there is no restriction on the number of metadata streams relative to the number of media streams being consumed simultaneously by the VDE 200. Thus, the VDE 200 input includes N media streams, where N is at least 1, and M metadata streams, where M is zero or more.

[0087] The VDE 200 outputs decoded video sequences and metadata streams via OVDI 210. There is at least one decoded video sequence as output, but there is no restriction on the number of metadata streams with respect to Petition 870250090858, dated 06 / 10 / 2025, pages 155 / 273 34 / 113 number of decoded video sequences being simultaneously output by the VDE 200. These two types of output streams can be provided in the form of multiplexed output buffers, including both the decoded media data and its associated metadata. Thus, the VDE 200 output includes Q decoded sequences, where Q is at least 1, and P metadata streams, where P is zero or more.

[0088] Figure 4 is a block diagram illustrating an example relationship between the video decoder instances 214 and the hardware video decoding engine 212, according to the techniques of the present disclosure. In this example, one or more video decoder instances 214 are run by the same hardware video decoding engine 212. The hardware video decoding engine 212 exposes each of the video decoder instances 214 to the application layer as multiple decoder instances, each with its own input and output interfaces.

[0089] Figure 5 is a conceptual diagram illustrating an example of instantiation of multiple instances of video decoder 214 of a hardware video decoding mechanism 212 using a video decoder interface, according to the techniques of the present disclosure. The example in Figure 5 further depicts the clock component 222, the aggregate buffer 224, and the application configuration information 220. The hardware video decoding mechanism 212 can receive application configuration information 220 from a multimedia application, such as a media player, extended reality (XR) player, or the like. A video decoding interface can Petition 870250090858, dated 06 / 10 / 2025, pages 156 / 273 35 / 113 Include functions as defined using IDL syntax, as specified in ISO / IEC 19516, Information technology — Object management group — Interface definition language (IDL), version 4.2.

[0090] In the example in Figure 5, the video decoder instances 214 can use some of the functionalities of a video decoding interface. The video decoder instances 214, in this example, have identifiers 1 to 3 and belong to a group with identifier 4. Through this grouping mechanism, the three instances write decoded sequences to the aggregate buffer 224, and the decoding operations through these instances are performed in a coordinated manner (with reference to a clock signal from the clock component 222) so that no instance runs ahead of or behind the others.

[0091] As declarações de IDL da função queryCurrentAggregateCapabilities() juntamente com as estruturas AggregateCapabilities e PerformancePoint e as flags de capacidades podem ser definidas da seguinte forma: const unsigned long CAP_INSTANCES_FLAG = 0x1; const unsigned long CAP_BUFFER_MEMORY_FLAG = 0x2; const unsigned long CAP_BITRATE_FLAG = 0x4; const unsigned long CAP_MAX_SAMPLES_SECOND_FLAG = 0x8; const unsigned long CAP_MAX_PERFORMANCE_POINT_FLAG = 0xA; struct PerformancePoint { float picture_rate; unsigned long width; unsigned long height; Petição 870250090858, de 06 / 10 / 2025, pág. 157 / 273 36 / 113 unsigned long bit_depth; }; struct Aggregatecapabilities { unsigned long flags; unsigned long max_instances; unsigned long buffer_memory; unsigned long bitrate; unsigned long max_samples_second; PerformancePoint max_performance_point; }; AggregateCapabilities querycurrentAggregateCapabilities ( in string component_name, in unsigned long flags );

[0092] The queryCurrentAggregateCapabilities() function can be used by the application to query the instant aggregation capabilities of a decoder platform for a specific codec component. Capability flags can be set separately or in a single function call to query one or more parameters.

[0093] The component_name can provide the name of the decoding platform component to which the query applies. The name All can be used to indicate that the query is not for a particular component, but for all components of the decoding platform. Components are hardware or software functionalities exposed by the Video 200 decoding engine, such as Petition 870250090858, dated 06 / 10 / 2025, pp. 158 / 273 37 / 113 decoders.

[0094] CAP_INSTANCES_FLAG queries the max_instances parameter, which indicates the maximum number of decoder instances that can be instantiated at this time for the provided decoder component.

[0095] CAP_BUFFER_MEMORY_FLAG queries the buffer_memory parameter, which indicates the maximum instantaneous global buffer size available in bytes that can be allocated independently of any components at this time on the decoder platform for buffer swapping. Memory allocation can be done by the application or by the VDE 200 itself, depending on the VDE instantiation.

[0096] CAP_BITRATE_FLAG queries the bit rate parameter, which indicates the maximum instantaneous bit rate encoded in bits per second that the queried component is capable of processing.

[0097] CAP_MAX_SAMPLES_SECOND_FLAG queries the max_samples_second parameter, which indicates the instantaneous maximum number of combined luma and chroma samples per second that the queried component is capable of processing.

[0098] CAP_MAX_PERFORMANCE_POINT_FLAG queries the max_performance_point parameter, which indicates the maximum performance point of a bitstream that can be decoded by the component indicated in a new instance of that decoder component.

[0099] A performance point can contain the following parameters: • picture_rate indicating the instantaneous image rate of the maximum performance point in frames per second. Petition 870250090858, dated 06 / 10 / 2025, pp. 159 / 273 38 / 113 • height indicating the height in luma samples of the maximum performance point. • width indicating the width in luma samples of the maximum performance point. • bit_depth indicating the bit depth of the luma samples at the maximum performance point.

[00100] Each maximum performance point parameter does not necessarily represent the maximum in that dimension. The parameters may be the combination of all dimensions that constitute the maximum performance point.

[00101] The IDL declarations of the getInstance() function and the associated ErrorAllocation exception can be defined as follows: exception ErrorAllocation { string reason; }; unsigned long getInstance( in string component_name, in unsigned long group_id / / optional, default value = -1 ) raises(ErrorAllocation);

[00102] The result of a successful call to the getInstance() function call may provide an instance identifier and group_id that is assigned or created for that new instance, if a new instance was requested. The default behavior is that the decoder instance does not belong to any already established group, but is assigned to a newly created group.

[00103] Multiple instances of decoder Petition 870250090858, dated 06 / 10 / 2025, pp. 160 / 273 39 / 113 belonging to the same group means that VDE 200 treats these instances collectively, so that the decoding status of these instances progresses in sync and not competing with each other. As a consequence, VDE 200 can guarantee synchronized output recording operation, possibly in aggregate buffer 224. There are no conditions for two video decoder instances to be in the same group.

[00104] The IDL declarations of the setConfig() function, the associated ErrorConfig exception, the ConfigDataParameters structure, and the ConfigParameters enumeration can be defined as follows: enum ConfigParameters { CONFIG_OUTPUT_BUFFER}; struct ConfigDataParameters { SampleFormat sample_format; SampleType sample_type; sample_stride unsigned long; unsigned long line stride; unsigned long buffer_offset; }; exception ErrorConfig { string reason; }; boolean setConfig (in unsigned long instance_id, in ConfigParameters config_parameters, Petition 870250090858, dated 06 / 10 / 2025, pp. 161 / 273 40 / 113 in ConfigDataParameters config_data_parameters ) raises(ErrorConfig);

[00105] The setConfig() function can be called with the CONFIG_OUTPUT_BUFFER parameter to provide the output buffer format. The buffer format can contain the following parameters: • sample_format indicating the format of each sample, which can be a scalar, a 2D vector, a 3D vector, or a 4D vector. • sample_type indicating the type of each component in the sample. • sample_stride indicating the number of bytes between 2 consecutive samples of this output. • line_stride indicating the number of bytes between the first byte of one line and the first byte of the next line in this output. • buffer_offset indicating the offset to the output buffer, from which the output frame should be written.

[00106] The IDL declarations for the getParameter() and setParameter() functions, as well as the associated ErrorParameter exception and the ExtParameters enumeration, can be defined as follows: enum ExtParameters { PARAM_PARTIAL_OUTPUT, PARAM_SUBFRAME_OUTPUT, PARAM_METADATA_CALLBACK, PARAM_OUTPUT_CROP, PARAM_OUTPUT_CROP_WINDOW, Petition 870250090858, dated 06 / 10 / 2025, pp. 162 / 273 41 / 113 PARAM_MAX_OFFTIME_JITTER}; struct CropWindow { unsigned long x; unsigned long y; unsigned long width; unsigned long height; }; exception ErrorParameter { string reason; }; any getParameter ( in unsigned long instance_id, in ExtParameters ext_parameters, out any parameter ); boolean setParameter ( in unsigned long instance_id, in ExtParameters ext_parameters, in any parameter ) raises(ErrorParameter);

[00107] The getParameter() and setParameter() functions can receive extended parameters, as discussed below.

[00108] PARAM_PARTIAL_OUTPUT can indicate whether subframe output is required, desired, or not allowed. Petition 870250090858, dated 06 / 10 / 2025, pages 163 / 273 42 / 113

[00109] PARAM_SUBFRAME_OUTPUT can indicate one or more subframes to be output by the decoder.

[00110] PARAM_METADATA_CALLBACK can define a callback function for a specific metadata type. The list of supported metadata types can be codec-dependent and can be defined for each codec independently.

[00111] PARAM_OUTPUT_CROP may indicate that only part of the decoded frame is desired in the output. The decoder instance can use this information to intelligently reduce its decoding processing by discarding units that do not fall within the cropped output region whenever possible.

[00112] PARAM_OUTPUT_CROP_WINDOW can indicate the portion of the decoded frame to be cropped and output.

[00113] PARAM_MAX_OFFTIME_JITTER can indicate the maximum amount of time in microseconds between consecutive executions of the decoder instance. This parameter may be relevant whenever the underlying hardware component is shared between multiple decoder instances, which requires context switching between the different decoder instances.

[00114] Vulkan® Video (VK) is an extension of the Vulkan API that defines functions exposed per graphics processing unit (GPU). This extension provides interfaces to an application to leverage the hardware decoding and encoding capabilities present in GPUs. A VK video session includes a single decoding session in a single layer. As a result, a single VK video session corresponds to a single instance of a video decoder. Petition 870250090858, dated 06 / 10 / 2025, pp. 164 / 273 43 / 113 as depicted in Figure 3. The mapping of VDI functions to VK is summarized in Table 1 below: TABLE 1 VDI Functions Mapping of VK queryCurrent AggregateCapabilities New function vkGetPhysicalDeviceCurrentVideoCapabilities MPEG() getInstance (grouping) Extending VkVideoSessionCreateInfoKHR with a group identifier passed in the new structure VkVideoSessionCreateInfoGroupingMPEG. Calling existing vkCreateVideoSessionKHR(). setConfig (buffer configuration) Mapping existing structures VkVideoSessionCreateInfoKHR and VkVideoPictureResourceKHR. getParameter and setParameter New structure VkVideoSessionOutputParameterMPEG

[00115] The VK Video API provides a function for querying capabilities for a single VK video profile, called vkGetPhysicalDeviceVideoCapabilitiesKHR(). Similar to this function, the VK VDI mapping defines the vkGetPhysicalDeviceCurrentVideoCapabilitiesMPEG() function. In contrast to the vkGetPhysicalDeviceVideoCapabilitiesKHR() function, the vkGetPhysicalDeviceCurrentVideoCapabilitiesMPEG() function allows querying the aggregate capabilities of the physical device. When called with a certain profile, the aggregate capabilities belong to that specific profile.

[00116] The vkGetPhysicalDeviceCurrentVideoCapabilitiesMPEG() function can be declared as follows: Petition 870250090858, dated 06 / 10 / 2025, pages 165 / 273 44 / 113 VkResult vkGetPhysicalDeviceCurrentVideoCapabilitiesMPEG( VkPhysicalDevice physicalDevice, VkVideoProfileKHR* pVideoProfile, VkCurrentVideoCapabilitiesMPEG* pCapabilities);

[00117] In the above statement, physicalDevice represents the physical device whose video decoding or encoding capabilities are to be consulted; pVideoProfile is a pointer to a VkVideoProfileKHR structure with a codec-specific operation video profile structure linked; and pCapabilities is a pointer to a VkCurrentVideoCapabilitiesMPEG structure in which the capabilities are returned.

[00118] The structure VkCurrentVideoCapabilitiesMPEG contains the information returned by a call to the vkGetPhysicalDeviceCurrentVideoCapabilitiesMPEG() function. above. The VkCurrentVideoCapabilitiesMPEG structure can be declared as follows: typedef struct VkCurrentVideoCapabilitiesMPEG { VkStructureType sType; void* pNext; uint32_t maxInstances; uint32_t bufferMemory; uint32_t bitrate; uint32_t maxSamplesSecond; VkPerformancePointMPEG*maxPerformancePoint;} VkCurrentVideoCapabilitiesMPEG;

[00119] In the declaration above, sType is the type of this structure; pNext is NULL or a pointer to a structure that Petition 870250090858, dated 06 / 10 / 2025, pp. 166 / 273 45 / 113 extends this structure; pictureRate, height, width, and bitDepth can be the same as defined above.

[00120] The structure VkVideoSessionCreateInfoGroupingMPEG allows you to attach a group identifier to a video decoding instance created via the VK Video API. This framework extends the VkVideoSessionCreateInfoKHR framework defined in the VK Video API. The VkVideoSessionCreateInfoGroupingMPEG framework can be declared as follows: typedef struct VkVideoSessionCreateInfoGroupingMPEG { VkStructureType sType; const void* pNext; uint32_t groupId; } VkVi deoSessi onCreateInfoGroupingMPEG;

[00121] In the above statement, sType is the type of this structure; pNext is NULL or a pointer to a structure that extends this structure; and groupId can be the same as defined above.

[00122] The structure VkVideoSessionOutputParameterMPEG contains parameters that configure the output properties of the VK video session. VkVideoSessionOutputParameterMPEG can be declared as follows: typedef struct VkVideoSessionOutputParameterMPEG { VkStructureType sType; const void* pNext; VkFlag partialOutput; uint32_t* subframeCount; uint32_t* pSubframeOutput; VkFlag outputCrop; Petition 870250090858, dated 06 / 10 / 2025, pp. 167 / 273 46 / 113 VkExtent2D* pOutputCropWindow; uint32_t maxOfftimeJitter; void* pMetadataCallback; } VkVi deoSessi onOutputParameterMPEG;

[00123] In the above statement, sType is the type of this structure; pNext is NULL or a pointer to a structure that extends this structure; partialOutput, subframeCount, pSubframeOutput, outputCrop, pOutputCropWindow, maxOfftimeJitter, and pMetadataCallback can have the same semantics discussed above.

[00124] Figure 6 is a block diagram illustrating an example video decoder interface (VDI) system decoder model according to the techniques of the present disclosure. In this example, the VDI system decoder model includes media stream delivery interface 202 (corresponding to IVDI 202), input formatting unit 204, decoder buffers 230, hardware video decoding engine 212 (which runs multiple instances of instantiated video decoder 214A to 214N), composition memory 232, and compositor unit 234.

[00125] The VDI system decoder model in Figure 6 extends the system decoder (SDM) model defined in ISO / IEC 14496-1. Compared to SDM, VDI SDM introduces a new interface in addition to the elementary stream interface called the media stream delivery interface 202. This interface is the input to the input formatting unit 204, also called the input formatting function, which takes as input the so-called media streams. The output of the input formatting unit 204 is one or more Petition 870250090858, dated 06 / 10 / 2025, pp. 168 / 273 47 / 113 elementary streams that can be additionally passed to video decoder instances 214.

[00126] The concepts for specification include formatting, timing and buffering models.

[00127] The media stream delivery interface 202 is a concept that models the exchange of media stream data between the delivery interface and the input formatting function (input formatting unit 204).

[00128] The input formatter (input formatter unit 204) receives one or more media streams as input and generates one or more elementary streams as output. A single input formatter can be linked to multiple decoding buffers, such as the buffers of decoding buffer 230, when input formatter unit 204 produces individual elementary streams or multilayer elementary streams.

[00129] Each of the video decoder instances 214 can output decoded video data to the respective portions of the composition memory 232. The compositor unit 234 can retrieve decoded video data from the respective portions of the composition memory 232 and compose an output video stream from the decoded video data.

[00130] The hardware video decoding mechanism 212 can generate one or more instances of video decoder 214. The number of running instances can be an optimization choice for the platform when considering available resources such as computational load, power consumption, memory, etc. However, the number of input media streams fed Petition 870250090858, dated 06 / 10 / 2025, pp. 169 / 273 48 / 113 via IVDI 202 is dictated by the application's needs to properly render the media experience. Therefore, one or more input media streams can be fed to the same video decoding instance 214 via input formatting unit 204.

[00131] Input formatting unit 204 performs various operations on media streams and video objects. Input formatting unit 204 produces results in one or more elementary streams conforming to the profile, tier, level, or any other performance constraints of the corresponding video decoder instances 214 that are expected to consume the elementary streams, including the fullness of the hypothetical reference decoder model buffer. Examples of such operations are defined below in an atomic manner, so that more complex operations can be achieved by combining them, provided the final output includes valid elementary streams. The actual implementation of these combined operations is outside the scope of this document and may be subject to optimization by the implementers, as explained in more detail below.

[00132] A media stream can include one or more video objects, and a video object can be contained within an elementary stream. Each video object in an elementary stream provides information to enable defined operations, such as a means to determine the location and size of the video object in the image, the number of luma and chroma samples in the video object, the bit depth of the video object's encoded image, and so on.

[00133] Figure 7 is a conceptual diagram illustrating an example input received by a filter and Petition 870250090858, dated 06 / 10 / 2025, pages 170 / 273 49 / 113 filtered output data, according to the techniques of this disclosure. Input formatting unit 204 performs the filtering function to extract a video object from an input media stream and returns an elementary stream as output that includes the selected video object.

[00134] Input formatting unit 204 can filter by video object identifier and / or by media data types, such as a media stream type, an elementary stream type, an access unit type, a video object identifier type, and / or a video object sample type. A filter can be defined in the form: f: MDS x I -^ ES, which indicates that the input includes a media stream (MDS) with at least one video object and an identifier (I) of a selected video object to be extracted, and that the output includes an elementary stream (ES) with a video object corresponding to the video object selected by the input. A signature for this filtering operation can be ElementaryStream output_stream filtering(MediaStream input_stream, VideoObjectIdentifier id).

[00135] Thus, to perform filtering, for each i-th access unit in the input media stream, input formatting unit 204 can make a copy of the access unit. Then, input formatting unit 204 can list the video object samples present in the copied access unit. If a video object sample does not match the video object identifier passed as input, the video object sample can be removed from the copied access unit. Finally, input formatting unit 204 can attach the unit to Petition 870250090858, dated 06 / 10 / 2025, pp. 171 / 273 50 / 113 access copied to the elementary output stream as a new access unit. That is, the input formatting unit 204 can implement a filtering process based on a selected object identifier, i.e., the original access units are first copied and then removed from unwanted objects. In this way, the operation does not need to create and initialize an empty access unit, and the properties of the input access units are passed to the access units of the output stream.

[00136] If the video object is a slice, the filtering function extracts that slice from each encoded image in the input media stream and passes the extracted slice into the output elementary stream. In the example in Figure 7, input formatting unit 204 receives a media data stream including images, such as image 240. Image 240 includes slice 242. Input formatting unit 204 can extract slice 242 and output slice 242 as output 244, as shown in Figure 7. During this operation, the SPS, PPS, and slice header may need to be updated as required by the corresponding video encoding specification to correctly signal the video size of the output elementary stream, information about the slice and tile layout, and the video object identifier, for example, the slice address.

[00137] Figures 8A to 8C are conceptual diagrams illustrating an example equal-width input video received by an insertion function and combined output video, according to the techniques of the present disclosure. Input formatting unit 204 can also perform an insertion operation. The operation of Petition 870250090858, dated 06 / 10 / 2025, pp. 172 / 273 51 / 113 insertion can be defined as f: MDS x MDS -^ MDS. The input includes two media streams containing at least one video object each, and the output can be a media stream with as many video objects as the sum of the video objects in both media streams. The signature for the insertion operation can be MediaStream output_stream inserting(MediaStream input_stream_1, MediaStream input_stream_2).

[00138] To perform the insertion, for each i-th access unit in the first and second input media streams, the input formatting unit 204 can make a copy of the i-th access unit from the second input media stream. Then, the input formatting unit 204 can list the video object samples present in the i-th access unit from the first input media stream. The input formatting unit 204 can add each video object sample to the copied access unit. Finally, the input formatting unit 204 can attach the copied access unit to the output media stream as a new access unit. The insertion can stop as soon as one of the two input media streams finishes. The insertion operation can be defined as the insertion of video objects from the first input media stream into the second input media stream.In this way, the insertion operation does not need to create and initialize an empty access unit, but the properties of the access units of the second input media stream are passed to the access units of the output media stream.

[00139] The insertion function moves video objects from a first input media stream to a second Petition 870250090858, dated 06 / 10 / 2025, pages 173 / 273 52 / 113 input media stream and outputs the resulting output media stream, which includes the video objects from the first and second input media streams.

[00140] If the video objects are slices, such as slice 250 in Figure 8A and slice 252 in Figure 8B, the width or height of the encoded images of the input media streams are equal in order to maintain the rectangular format of the output media stream video. If the widths of the two input videos are equal, as shown in Figures 8A and 8B, the two videos are joined vertically to form image 254 in Figure 8C. During this operation, the SPS, PPS, and slice header may need to be updated to correctly signal the size of the output media stream video, information about the slice and tile layout, and video object identifiers, for example, slice addresses.

[00141] Figures 9A to 9C are conceptual diagrams illustrating an example input video with equal height received by an insertion function and combined output video, according to the techniques of the present disclosure. If the heights of two videos are equal, as shown for slice 260 of Figure 9A and slice 262 of Figure 9B, then the two videos can be joined horizontally to form image 264 of Figure 9C. During this operation, the SPS, PPS, and slice header may also need to be updated to correctly signal the video size of the output media stream, information about the slice and tile layout, and video object identifiers, for example, the Petition 870250090858, dated 06 / 10 / 2025, pp. 174 / 273 53 / 113 slice addresses.

[00142] Figures 10A and 10B are conceptual diagrams illustrating examples of input videos received by an appending function and corresponding output video data, according to the techniques of the present disclosure. The input formatting unit 204 can perform an appending operation, which can be defined as f: MDS ^ MDS. The appending operation can take as input a media stream with at least two video objects and output a media stream with two video objects that are left and right spatial neighbors. A signature for the appending operation can be MediaStream output_stream appending(MediaStream input_stream, VideoObjectIdentifier object_id_1, VideoObjectIdentifier object_id_2).

[00143] To perform the attachment operation, input formatting unit 204 can, for each i-th access unit in the input media stream, make a copy of that i-th access unit. Then, input formatting unit 204 can set the position of the video object samples belonging to the video object identified by the second video object identifier to the right of the video object samples belonging to the object identified by the first video object identifier in that copied access unit. This positioning is done in such a way that the upper boundaries of both video object samples are aligned. Finally, input formatting unit 204 can attach the copied access unit to the output media stream as a new access unit.

[00144] The attachment function positions a first video object to the right of a second video object in Petition 870250090858, dated 06 / 10 / 2025, pages 175 / 273 54 / 113 decoded images from the output media stream, which contains these two video objects. The output media stream is media containing at least the first and second video objects positioned as side-by-side neighbors.

[00145] If the video object is a slice, the slices of both video objects in the input media stream have the same height, as shown in the example of slices 272A, 272B of images 270A, 270B in Figures 10A and 10B. Slice 272B, on the right side, is moved next to slice 272A, on the left side, in image 270B of the output media stream. Slice 274, which was between slices 27A, 272B in Figure 270A, is moved to the right of slices 272A, 272B, as shown in figure 270B of Figure 10B. During this operation, the slice headers may need to be updated to correctly signal changes to the video object identifiers, for example, the slice addresses.

[00146] Figures 11A and 11B are conceptual diagrams illustrating examples of input videos received by a stacking function and corresponding output video data, according to the techniques of the present disclosure. The input formatting unit 204 can perform a stacking function, defined as f: MDS -^ MDS. The stacking function can take a media stream with at least two video objects as input and produce a media stream with two video objects that are superior and inferior spatial neighbors as output. A signature for the stacking function can be MediaStream output_stream stacking(MediaStream input_stream, VideoObjectIdentifier object_id_1, VideoObjectIdentifier object_id_2).

[00147] To perform the stacking function, the Petition 870250090858, dated 06 / 10 / 2025, pp. 176 / 273 55 / 113 Input Formatting Unit 204 can, for each i-th access unit in the input media stream, make a copy of that i-th access unit. Then, Input Formatting Unit 204 can set the position of the video object samples belonging to the video object identified by the second video object identifier below the video object samples belonging to the object identified by the first video object identifier in that copied access unit. This positioning is done in such a way that the left boundaries of both video object samples are aligned. Finally, Input Formatting Unit 204 can attach the copied access unit to the output media stream as a new access unit.

[00148] The stacking function positions a first video object on top of a second video object in the decoded images of the media stream that contains these two video objects. The output media stream contains at least the first and second video objects positioned as top and bottom neighbors.

[00149] If the video object is a slice, the slices of both video objects in the input media stream have the same height, as shown in the example of slices 282A, 282B of images 280A, 280B of Figures 11A and 11B. Slice 282B on the right side is moved below slice 282A on the left side, as shown in image 280B of Figure 11B in the output media stream. Slices 284 and 286, below slice 282A in image 280A of Figure 11A, are moved sequentially to the right of slice 282B, as shown in image 280B of Figure 11B. During this operation, the headers of Petition 870250090858, dated 06 / 10 / 2025, pp. 177 / 273 Slices 56 / 113 may need to be updated to correctly signal changes to video object identifiers, for example, slice addresses.

[00150] The techniques in this disclosure can be used in conjunction with an instance of video decoder compliant with ITU-T H.266 / Versatile Video Coding (VVC). VVC is published under ISO / IEC 23090-3. Table 2 below provides links between VDI concepts and the concepts defined in VVC: TABLE 2 Concept Definitions VVC ElementaryStream bitstream AccessUnit access unit VideoObject Identifier nuh_layer_id VideoObjectSample image unit

[00151] An elementary VVC stream can be a video stream compliant with ISO / IEC 23090-3, and the independent layer information SEI message can be defined according to the definition below.

[00152] A VVC media stream used as an instantiation of the media stream may obey the following rules: • There must be at least one VPS in the media stream, and the parameters on each VPS must be as follows: The `vps_all_independent_layers_flag` flag should be set to 1. • The value of sh_picture_header_in_slice_header_flag must be equal to 0 for all encoded slices. Petition 870250090858, dated 06 / 10 / 2025, pages 178 / 273 57 / 113 • When present, the value of vps_num_output_layer_sets_minus2 must be equal to 0.

[00153] An incoming VVC media stream passed as an argument to the filtering function may conform to the following rules, in addition to those discussed above: • There must be VCL NAL units with at least two different nuh_layer_id values. • At least one of the two different nuh_layer_id values ​​must be equal to the object identifier passed as an argument to the filtering function.

[00154] An elementary VVC stream generated as output from the filtering function may conform to the following rules, in addition to those discussed above: • The number of access units in the elementary outgoing stream must be equal to the number of access units in the elementary incoming stream. • The number of VCL NAL units in the elementary output stream is equal to the number of VCL NAL units with nuh_layer_id equal to the object identifier passed as an argument to the function. • For each VCL NAL unit in the elementary output stream, there must exist a VCL NAL unit in the elementary input stream that is exactly identical to the bit. • All NAL units in the elementary output stream must have the same nuh_layer_id value, and this value Petition 870250090858, dated 06 / 10 / 2025, pp. 179 / 273 58 / 113 nuh_layer_id must be equal to the object identifier passed as an argument to the function.

[00155] Two VVC input media streams passed as arguments to the insert function may conform to the following rules, in addition to those discussed above: • The nuh_layer_id value of each NAL unit in the first input media stream must be different from any nuh_layer_id value present in the second input media stream. • If an SPS or PPS in the first input media stream has the same identifier as an SPS or PPS in the second input media stream, then those two SPSs or two PPSs must have the same payload.

[00156] A VVC media stream generated as output from the insert function may conform to the following rules, in addition to those discussed above: • The number of VCL NAL units in the output media stream is equal to the sum of the number of VCL NAL units in both input media streams. • For each VCL NAL unit in the output media stream, there must exist a VCL NAL unit in one of the two input media streams that is exactly identical to the bit.

[00157] A VVC input media stream passed as an argument to the append function may conform to the following rules, in addition to those discussed above: Petition 870250090858, dated 06 / 10 / 2025, pp. 180 / 273 59 / 113 • There must be VCL NAL units with at least two different nuh_layer_id values. • At least two of the two different nuh_layer_id values ​​must be the same as the two object identifiers passed as arguments to the attachment function.

[00158] A VVC media stream generated as output from the attachment function may conform to the following rules, in addition to those discussed above: • The number of VCL NAL units in the output media stream is equal to the number of VCL NAL units in the input media stream. • For each VCL NAL unit in the output media stream, there must exist a VCL NAL unit in the input media stream that is exactly identical to the bit. • There must be a standalone layer information SEI message where nuh_layer_id is equal to the first video object identifier. • There must be a standalone layer information SEI message whose nuh_layer_id is equal to the second video object identifier. • The independent layer information SEI message whose nuh_layer_id is equal to the first video object identifier must have its boundary_identifier_east value equal to the boundary_identifier_west value of the independent layer information SEI message whose nuh_layer_id is equal to the second identifier. Petition 870250090858, dated 06 / 10 / 2025, pp. 181 / 273 60 / 113 video object.

[00159] A VVC input media stream passed as an argument to the stacking function may conform to the following rules, in addition to those discussed above: • There must be VCL NAL units with at least two different nuh_layer_id values. • At least two of the two different nuh_layer_id values ​​must be the same as the two object identifiers passed as arguments to the attachment function.

[00160] A VVC media stream generated as output from the stacking function may conform to the following rules, in addition to those discussed above: • The number of VCL NAL units in the output media stream is equal to the number of VCL NAL units in the input media stream. • For each VCL NAL unit in the output media stream, there must exist a VCL NAL unit in the input media stream that is exactly identical to the bit. • There must be a standalone layer information SEI message where nuh_layer_id is equal to the first video object identifier. • There must be a standalone layer information SEI message whose nuh_layer_id is equal to the second video object identifier. • The SEI message contains independent layer information where nuh_layer_id is equal to the first Petition 870250090858, dated 06 / 10 / 2025, pages 182 / 273 61 / 113 The video object identifier must have its boundary_identifier_south value equal to the boundary_identifier_north value of the independent layer information SEI message whose nuh_layer_id is equal to the second video object identifier.

[00161] Alternatively, the techniques in this disclosure can be used with a video decoder instance compatible with Essential Video Coding (EVC). EVC is published under ISO / IEC 23094-1. Table 3 below provides an example of linking VDI concepts from this disclosure with the concepts defined in the EVC descriptive report. TABLE 3 Concept Definitions EVC ElementaryStream bitstream AccessUnit access unit VideoObjectIdentifier the smallest ID value of the tiles in a slice VideoObjectSample slice

[00162] An EVC media stream used as an instantiation of the media stream, as discussed above, may conform to the following rules: • There must be at least two independently decodable slices, where the lowest tile ID value in each slice is different.

[00163] An EVC input media stream passed as an argument to the filtering function may conform to the following rules: Petition 870250090858, dated 06 / 10 / 2025, pages 183 / 273 62 / 113 • One of the lowest tile ID values ​​in each slice must be equal to the object identifier passed as an argument to the filtering function.

[00164] An elementary EVC stream generated as output from the filtering function may conform to the following rules: • The number of access units in the elementary output stream must be equal to the number of access units in the input media stream. • The number of VCL NAL units in the elementary output stream is equal to the number of VCL NAL units with the lowest tile ID value in the slice equal to the object identifier passed as an argument to the function. • For each VCL NAL unit in the elementary output stream, there must be a VCL NAL unit in the input media stream that is exactly identical. • All NAL units in the elementary output stream must have the same lowest tile ID value in the slice value, and this value must be equal to the identifier of the object passed as an argument to the function.

[00165] Two EVC input media streams passed as arguments to the insert function may conform to the following rules: • At least one of the pic_width_in_luma_samples or pic_height_in_luma_samples values ​​from the two media streams must be identical. Petition 870250090858, dated 06 / 10 / 2025, pp. 184 / 273 63 / 113 • If the values ​​of pic_width_in_luma_samples are identical, then the values ​​of num_tile_columns_minus1 must be identical. • If the values ​​of pic_height_in_luma_samples are identical, then the values ​​of num_tiles_row_minus1 must be identical. • If an SPS or PPS in the first input media stream has the same identifier as an SPS or PPS in the second input media stream, then those two SPSs or two PPSs must have the same payload.

[00166] An EVC media stream generated as output from the insert function may conform to the following rules: • The number of VCL NAL units in the output media stream is equal to the sum of the number of VCL NAL units in both input media streams. • For each VCL NAL unit in the output media stream, there must exist a VCL NAL unit in one of the two input media streams that is exactly identical to the bit.

[00167] An EVC input media stream passed as an argument to the append function may conform to the following rules: • At least two of the lowest tile ID values ​​in each slice must be equal to the two object identifiers passed as arguments to the attachment function. • The height of the slices, the number of tile rows Petition 870250090858, dated 06 / 10 / 2025, pages 185 / 273 64 / 113 of the tiles included in the slices when uniform tile spacing is used, whose lowest tile ID values ​​in each slice are identical since the arguments of the attachment function are identical.

[00168] An EVC media stream generated as output from the attachment function may conform to the following rules: • The number of VCL NAL units in the output media stream is equal to the number of VCL NAL units in the input media stream. • For each VCL NAL unit in the output media stream, there must exist a VCL NAL unit in the input media stream that is exactly identical to the bit.

[00169] An EVC input media stream passed as an argument to the stacking function may conform to the following rules: • At least two of the lowest tile ID values ​​in each slice must be equal to the two object identifiers passed as arguments to the attachment function. • The width of the slices, the number of tile columns of tiles included in the slices when uniform tile spacing is used, where the lowest tile ID values ​​in each slice are identical since the arguments of the attachment function are identical. [0017 0] An EVC media stream generated as output Petition 870250090858, dated 06 / 10 / 2025, pages 186 / 273 The 65 / 113 stacking function may comply with the following rules: • The number of VCL NAL units in the output media stream is equal to the number of VCL NAL units in the input media stream. • For each VCL NAL unit in the output media stream, there must exist a VCL NAL unit in the input media stream that is exactly identical to the bit.

[00171] A control interface for the video decoding mechanism of this disclosure can be specified using the IDL syntax specified in ISO / IEC 29516. Alternatively, a control interface for the video decoding mechanism can be defined for the OpenMax IL interface.

[00172] Figure 12 is a conceptual diagram illustrating example locations of video data layer boundaries, according to the techniques of the present disclosure. In the example in Figure 12, a supplemental enhancement information (SEI) message may specify, among other information, data representing, for a given layer i 290, whether layer i 290 couples to another layer at a north boundary 292, an east boundary 294, a south boundary 296, and / or a west boundary 298.

[00173] An example of a generic envelope for carrying SEI messages, as defined in this document, is defined below. Some VDI SEI messages may only apply to certain video encoding specifications, for example, HEVC, VVC, or EVC. The VDI SEI envelope may be Petition 870250090858, dated 06 / 10 / 2025, pages 187 / 273 66 / 113 registered as a SEI payload in the ISO / IEC 230903 standard.

[00174] Table 4 below defines the example syntax for the SEI VDI envelope: TABLE 4 Syntax Size Type vdi_sei_envelope(payloadSize) { vdi_sub_type 8 unsigned integer if(vdi_sub_type == 0) independent_layer_info(payloadSize - 1) otherwise reserved_message(payloadSize - 1)}

[00175] The semantics for vdi_sub_type can be as follows: vdi_sub_type indicates the type of payload carried in the SEI VDI envelope.

[00176] Table 5 below defines the example syntax for the independent layer information SEI message: TABLE 5 Syntax Size Type independent_layer_info(payloadSize) { boundary_identifier_north_present_flag 1 bit if(boundary_identifier_north_present_flag) boundary_identifier_north 16 unsigned integer boundary_identifier_east_present_flag 1 bit Petition 870250090858, dated 06 / 10 / 2025, pp. 188 / 273 67 / 113 if(boundary_identifier_east_present_flag) boundary_identifier_east 16 unsigned integer boundary_identifier_south_present_flag 1 bit if(boundary_identifier_south_present_flag) boundary_identifier_south 16 unsigned integer boundary_identifier_west_present_flag 1 bit if(boundary_identifier_west_present_flag) boundary_identifier_west 16 unsigned integer}

[00177] A semântica para as semânticas de mensagem de SEI das informações (info) de camada independente pode ser conforme definido abaixo: The independent layer information (SEI) message provides the spatial alignment of the different independent layers present in a bitstream, expressing the relative positioning between these layers using corresponding boundary identifiers. In a multilayer bitstream, there must be at most two occurrences of a given boundary identifier, creating a pair of corresponding boundary identifiers. In other words, a layer has at most one neighbor per boundary. This SEI message can be extracted by VDE and used for the output formatting function to place Petition 870250090858, dated 06 / 10 / 2025, pp. 189 / 273 68 / 113 correctly decodes the images from each layer into the final output image. In this case, the decoder instance can ignore this SEI message, if present. Alternatively, the decoder instance may be able to decode a multilayer bitstream as well as parse the SEI message, and in this case, no output formatting function is needed. boundary_identifier_north_present_flag, boundary_identifier_east_present_flag, boundary_identifier_south_present_flag, and boundary_identifier_west_present_flag equal to 1 specify that the SEI message contains a boundary identifier for the north, east, south, and / or west border, respectively. boundary_identifier_north, boundary_identifier_east, boundary_identifier_south, and boundary_identifier_west specify the boundary identifier, respectively, on the north, east, south, and west edges of the decoded image of the associated layer; the associated layer being the layer whose nuh_layer_id is equal to the nuh_layer_id of the SEI message. If not present, the boundary identifiers, respectively, on the north, east, south, and west edges of the decoded image of the associated layer are not defined.

[00178] As noted above, in the example in Figure 12, the image of layer i 2 90 represents an image of layer i. An independent layer information SEI message can specify whether the north edge 292 should be attached to an edge of another layer, or whether the east edge 294 should Petition 870250090858, dated 06 / 10 / 2025, pp. 190 / 273 69 / 113 to be attached to an edge of another layer, if the south edge 296 must be attached to an edge of another layer and / or if the west edge 298 must be attached to an edge of another layer.

[00179] For two layers, the i-th and j-th layers, when the pair of boundary_identifier_north values ​​of the i-th layer and boundary_identifier_south values ​​of the j-th layer are equal, then the decoded image of the i-th layer and the decoded image of the j-th layer must be placed adjacent in the composite output image and share a common north / south boundary. For the i-th and j-th layers, when the pair of boundary_identifier_east values ​​of the i-th layer and boundary_identifier_west values ​​of the j-th layer are equal, then the decoded image of the i-th layer and the decoded image of the j-th layer are adjacent in the composite output image and share a common east / west boundary.

[00180] Two decoded images adjacent to the north / south boundary can be aligned at their western boundary in the final output image. Two decoded images adjacent to the east / west boundary can be aligned at their northern boundary in the final output image.

[00181] SEI messages of independent layer information present in the layers of an output layer set (OLS) can collectively describe a 4-connection graph, and each layer of the OLS can be connected to the graph.

[00182] An example of a process to generate the final output image is informative. The expected operations performed to generate the final output image are based on the images. Petition 870250090858, dated 06 / 10 / 2025, pp. 191 / 273 70 / 113 decoded values ​​from each layer of a selected OLS are described below: • For each access unit: If the VPS is present, analyze the VPS and store the list of layers in the bitstream. For each PPS present, determine the luma sample size of the corresponding layer. For each independent layer information SEI message, analyze the payload and store the boundary identifiers for the corresponding layer. If any SEI message containing VPS, PPS, or independent layer information is present on the current access unit, calculate the horizontal (XPos) and vertical (YPos) positions of the upper-left corner of each layer-cropped decoded image in the final output image. An example sequence of steps to calculate XPos and YPos is as follows: For each layer: • Analyze the list of cropped image sizes and the border identifier. • Identify the neighboring layers: north, east, south, and west. Place each layer in a grid, with each grid cell corresponding to a north, east, south, west value for each layer that corresponds to the corresponding neighboring value, respectively south, west, north, east. Petition 870250090858, dated 06 / 10 / 2025, pp. 192 / 273 71 / 113 The XPos value for each layer corresponds to the sum of the widths of each layer in the same grid row, to the left of the current layer. The YPos value for each layer corresponds to the sum of the heights of each layer in the same grid column, above the current layer. Initialize an image buffer of size FinalWidth (width) and FinalHeight (height) for the final output image, where FinalWidth and FinalHeight are the width and height of the final output image when all layers are concatenated according to the defined graph. For each image unit: Decode the encoded image. If the images are ready for output: For each layer in the selected OLS. • Apply the window clipping to the decoded image of the current layer. • Retrieve XPos and YPos, the positions of the current layer in the final output image in the luma sample. • Copy the cropped decoded image into the final output image buffer at the position (XPos, YPos) corresponding to the upper left corner of the cropped decoded image.

[00183] If, at the end of this process, the combination of all decoded images does not provide decoded sample values ​​for all samples of the final output image, the implementation can determine the values ​​to be used for these unused samples.

[00184] Figure 13 is a conceptual diagram. Petition 870250090858, dated 06 / 10 / 2025, pages 193 / 273 Figure 72 / 113 illustrates an example of appending two layers of video data according to a connection map, in accordance with the techniques of this disclosure. In particular, Layer 0 image 306 and Layer 1 image 308 can be combined according to connection map 300 to form output image 310. As shown in this example, connection map 300 includes left image 302, corresponding to Layer 0, and right image 304, corresponding to Layer 1. Left image 302 includes a value of 1 on the eastern boundary and right image 304 includes a value of 1 on the western boundary. Thus, VDE 200 can determine that Layer 0 images should be appended, on the right side, to the left side of Layer 1 images.

[00185] Table 6 represents an example of Layer 0 and Layer 1 properties: TABLE 6 Sequence Layer Resolution Edge identifiers (N, E, S, W) Left image 302 0 416x240 0 1 0 0 Right image 304 1 416x240 0 0 0 1

[00186] Table 7 represents an example of final output image properties: TABLE 7 Resolution 832 x 240 Output sequence Image on the left 302 | Image on the right 304

[00187] Figure 14 is a conceptual diagram. Petition 870250090858, dated 06 / 10 / 2025, pp. 194 / 273 Figure 73 / 113 illustrates an example of attaching three layers of video data as a connection map, according to the techniques of this disclosure. In this example, connection map 320 includes layer 0 image 322, layer 1 image 324, and layer 2 image 326. Layer 0 image 322 includes a value of 1 on an eastern boundary and a value of 2 on a southern boundary, while layer 1 image 324 includes a value of 1 on a western boundary and layer 2 image 326 includes a value of 2 on a northern boundary.Thus, for input image 328 of layer 0, input image 330 of layer 1, and input image 332 of layer 2, by connection map 320, output image 334 is formed by attaching input image 330 to the right side of input image 328 and input image 332 below input image 328, so that input images 328 and 330 share a top boundary and input images 328 and 332 share a left boundary.

[00188] Table 8 represents an example of the properties of Layers 0 to 2: TABLE 8 Sequence Layer Resolution Edge identifiers (N, E, S, W) Input image 328 0 416x240 0 1 2 0 Input image 330 1 416x240 0 0 0 1 Input image 332 2 832x480 2 0 0 0 Petition 870250090858, dated 06 / 10 / 2025, pp. 195 / 273 74 / 113

[00189] Based on this configuration, Table 9 presents the properties of the final output images: TABLE 9 Resolution 832 x 720 Output sequence Input Image 328 | Input Image 330 Input image 332

[00190] Figure 15 is a conceptual diagram illustrating an example of a video decoding mechanism 340 employing an application-based video decoding mechanism control for 2x2 mosaic video objects, according to the techniques of the present disclosure. In particular, the video decoding mechanism (VDE) 340 receives media data from media streams 342A to 342D via a media stream interface. The VDE 340 receives configuration data 344 from a media application. According to the configuration data 344, the VDE 340 extracts video objects from the respective images of media streams 342A to 342D to form the output image 346 in the form of a 2x2 mosaic.

[00191] The various operations, as discussed above, and the associated input and output constraints provide structural units for various implementations of the input formatting function. The way a given implementation converts media streams into elementary streams based on the requested decoded sequence configuration is informative and left to optimization by the implementer, provided that the output elementary streams meet the Petition 870250090858, dated 06 / 10 / 2025, pp. 196 / 273 75 / 113 Elementary Flow Interface Requirements.

[00192] In the example in Figure 15, the objective is to take four video objects as input in four different media streams 342A to 342D and obtain as output from VDE 340 a decoded sequence containing the four decoded video objects arranged in a 2x2 grid in each image of the decoded sequence. It is assumed that the media application is able to communicate with VDE 340 via an external communication channel to express how the media stream should be arranged in the decoded sequences. For the purposes of the illustration, it is assumed that the four video objects have the same video resolution. Figure 15 depicts the input situation at the media stream interface, input to VDE 340, and the intended situation at the output of VDE 340.

[00193] The output of the input formatting function can be an elementary stream interface. That is, the input formatting function (e.g., input formatting unit 204) can output data streams that are elementary streams. Whether it should be one elementary stream or multiple elementary streams depends on the implementation and the platform. For this example, two examples are described. The first example is having one instance of a video decoder per media stream (as in Figure 16), the second example is having a single instance of a decoder for the four media streams (as in Figure 17).

[00194] Figure 16 is a conceptual diagram illustrating an example of a video decoding mechanism executing four instances of a decoder mechanism for 2x2 mosaic video objects, according to techniques of the present disclosure. In this example, the Petition 870250090858, dated 06 / 10 / 2025, pp. 197 / 273 76 / 113 VDE 350 includes video decoder instances 352A to 352D and output formatting unit 356. Each of the video decoder instances 352A to 352D receives video objects as input from the respective media streams 358A to 358D and decodes the video objects to form the respective decoded video objects 354A to 354D. Output formatting unit 356 assembles decoded video objects 354A to 354D into the output image 360, representing a 2x2 mosaic of the decoded video objects 354A to 354D.

[00195] In the case of a media stream video decoder, as in the example in Figure 16, the input formatting function is effectively an identity operation, that is, each media stream is passed to a decoder instance. This also means that each media stream is, in this case, also an elementary stream, which is permitted by the definition of a media stream. Then, the VDE 350 executes these four decoder instances in parallel, collects each output image, and assembles the output images in a 2 by 2 arrangement for each temporally placed set of decoded images, as shown in Figure 16.

[00196] Figure 17 is a conceptual diagram illustrating an example of a video decoder executing a decoder engine instance for 2x2 mosaic video objects, according to the techniques of the present disclosure. In this example, VDE 370 includes input formatting unit 372 and a single decoder instance 374. Input formatting unit 372 receives video objects from each of the media streams 376A to 376D and assembles them. Petition 870250090858, dated 06 / 10 / 2025, pp. 198 / 273 77 / 113 video objects are received in a single encoded input frame using the various operations discussed above. This encoded input frame is part of an elementary stream. Decoder instance 374 then decodes the encoded frame to form the decoded image 378, including video objects from each of the media streams 376A to 376D.

[00197] In the case of a video decoder for the four media streams, the input formatting unit 372 creates a single elementary stream from the four input media streams. After that, the VDE 370 executes a conventional pipeline with a single decoder instance 374 and outputs the decoded image 378 from the decoder instance without the need for additional processing before being output by the VDE 370.

[00198] Figure 18 is a conceptual diagram illustrating an example implementation for formatting four media streams with different steps, according to the techniques of this disclosure. Regarding the actual implementation of the input formatting function for the example, as discussed in relation to Figure 17, there may be different approaches. Figure 18 depicts one such example. In this example, the operations include insertion, appending, and stacking operations to form the input video data. In this example, the implementation would create a cascade of these operations in a graph, as depicted in Figure 18.

[00199] In particular, in this example, media streams 380A to 380D provide video objects. An input formatting unit, such as input formatting unit 372, can perform the 382A insertion operation on objects of Petition 870250090858, dated 06 / 10 / 2025, pp. 199 / 273 78 / 113 video from media streams 380A and 380B to form a first intermediate video object. Input formatting unit 372 can then perform insertion operation 382B on the first intermediate video object and on a video object from media stream 380C to form a second intermediate video object. Input formatting unit 372 can then perform insertion operation 382C on the second intermediate video object and on a video object from media stream 380D to form a third intermediate video object.

[00200] The third intermediate video object would contain each of the various video objects, but not necessarily assembled correctly. Thus, the input formatting unit 372 can first perform an append operation 384 (1, 2) to properly combine the video objects from media streams 380A and 380B, as shown in the intermediate video object 390. The input formatting unit 372 can then perform a stacking operation 386 (1, 3) to properly add the video object from media stream 380C, as shown in the intermediate video object 392. Finally, the input formatting unit 372 can perform the append operation 388 (3, 4) to properly add the video object from media stream 380D, as shown in the final video object 394.

[00201] Figure 19 is a conceptual diagram illustrating an example implementation for formatting four media streams with two steps, according to the techniques of this disclosure. Figure 19 depicts an alternative example to that shown in Figure 18 to achieve similar results to the example in Figure 17. In this example, a Petition 870250090858, dated 06 / 10 / 2025, pages 200 / 273 79 / 113 implementation can support an insertion function that receives multiple media streams as input and generates an elementary stream in a single function call. Additionally, this implementation can also have a proprietary function, called arrangement, so that the arrangement can be set on the elementary stream in a single function call.

[00202] In particular, in the example in Figure 19, the input formatting unit 372 can receive video objects from each of the media streams 400A to 400D. The input formatting unit 372 can initially perform the insertion operation 402 to insert the video objects into the combined video object 406. The input formatting unit 372 can then perform the arrangement operation 404 to properly arrange the video objects, forming the final video object 408.

[00203] In yet another example, there may be only one function combining the media streams, disposing the video objects from the media streams and outputting the disposed video objects as part of an elementary stream.

[00204] Figure 20 is a conceptual diagram illustrating an example of a video decoder mechanism using control information based on supplementary enhancement information (SEI) for 2x2 mosaic video objects, according to the techniques of the present disclosure. In this example, VDE 412 receives video objects from media streams 410A to 410D, which also include SEI messages. The SEI messages may include data representing, for example, a series of video encoding instances to be instantiated and / or arrangement information indicating how to arrange the video objects. Petition 870250090858, dated 06 / 10 / 2025, pages 201 / 273 80 / 113 from media streams 410A to 410D. VDE 412 can decode video objects and arrange video objects according to SEI messages to finally form the output image 414.

[00205] This example shows the same intended output result as discussed above. The difference in Figure 20 is that there is no assumed communication between the VDE 412 and the media application about how to arrange the video objects together in the decoded sequence. Instead, this arrangement configuration is given by the SEI messages of independent layer information contained in each media stream. This information can be used to perform the append operations.

[00206] Figure 21 is a conceptual diagram illustrating an example of connected components and buffer usage, according to the techniques of the present disclosure. An OpenMAX IL client can call OMX_Init() as a first call to OpenMAX IL. OMX_Init() initializes the OMX core engine before any use of it. After this is done, the engine can be released by calling OMX_Deinit().

[00207] OMX defines a naming convention for component names with the following format: OMX.<vendor_name> .<vendor_specified_convention> Once the instance is no longer needed, OMX_FreeHandle() is called to release all related resources.

[00208] The function can be called multiple times with the same component name to create multiple instances of the component.

[00209] OMX_GetHandle() is used to locate a Petition 870250090858, dated 06 / 10 / 2025, pages 202 / 273 81 / 113 requested component via provided name. If the requested component is available, the OMX core engine will invoke the component methods to populate the component identifier and configure callbacks. OpenMAX AL is the interface that will be used by the application to perform media playback and processing. However, the OpenMAX IL interface is the interface that provides direct access to the video decoder components and their capabilities. Therefore, this disclosure describes the OpenMAX IL interface for the purpose of providing additional attributes that can enable a flexible multi-video decoder platform and its interface for six degrees of freedom (6DoF) applications.

[00210] A tunnel can be used to connect the input and output ports of two connected components. OMX_SetupTunnel() is used to establish a tunnel connecting an output port of one component to the input port of another component. When creating the tunnel, the components can negotiate a compatible input / output format for the connected ports. When it is no longer needed, the application calls OMX_TeardownTunnel() to bring down the tunnel.

[00211] In the example in Figure 21, component 424 includes an output port, component 426 includes an input port and an output port, and component 428 includes an input port. The use of buffer 420 corresponds to the output ports of components 424 and 426, while buffer port 422 corresponds to the input ports of components 426 and 428.

[00212] The components communicate with each other and with the application through buffer exchange. For this purpose, OMX_AllocateBuffer(), OMX_UseBuffer(), OMX_FillThisBuffer(), Petition 870250090858, dated 06 / 10 / 2025, pages 203 / 273 82 / 113 OMX_EmptyThisBuffer() and OMX_FreeBuffer() are defined. These function calls are non-blocking. A component asks a preceding component to fill an input buffer by calling the OMX_FillThisBuffer() method and asks a subsequent component to retrieve the contents of an output port buffer by calling the OMX_EmptyThisBuffer() function. Only one buffer per tunnel can be used, and one of the two components acts as the provider of that buffer.

[00213] OMX_SetConfig() is used to configure a component from the application. The application passes a structure containing the configuration parameters for the component. The configuration parameters are published by each component and are specific to the component.

[00214] Figure 22 is a conceptual diagram illustrating example port configurations for various buffers, according to the techniques of the present disclosure. Figure 22 depicts example buffers 440A to 440N. The port configuration is used to define the format of the data to be transferred on a component port. The buffer header contains a reference to the buffer pBuffer, an offset within that buffer nOffset, and the length of that buffer nFilledLen. Multiple buffers can be used to pass data, allowing for more flexibility in communication between components, i.e., more than one frame can be exchanged at a time.

[00215] There is no frame alignment requirement to start the buffer. The application or preceding components provide frame alignment information as part of the buffer header using the OMX_BUFFERFLAG_ENDOFFRAME flag. It is also possible to flag the Petition 870250090858, dated 06 / 10 / 2025, pages 204 / 273 83 / 113 subframe boundaries to identify the boundaries of the NAL unit using OMX_BUFFERFLAG_ENDOFSUBFRAME.

[00216] A timestamp is also provided by the buffer header for each buffer. The nTimestamp corresponds to the presentation timestamp of the first media sample that begins in the current buffer. If multiple samples are included in the current buffer, the initial timestamp of the following samples is inferred from the sample duration nTimestamp. This information can then be propagated through the pipeline and passed to the application via the output buffer.

[00217] The buffer header structure can be: typedef struct OMX_BUFFERHEADERTYPE { OMX_U32 nSize; OMX_VERSIONTYPE nVersion; OMX_U8* pBuffer; OMX_U32 nAllocLen; OMX_U32 nFilledLen; OMX_U32 nOffset; OMX_PTR pAppPrivate; OMX_PTR pPlatformPrivate; OMX_PTR pInputPortPrivate; OMX_PTR pOutputPortPrivate; OMX_HANDLETYPE hMarkTargetComponent; OMX_PTR pMarkData; OMX_U32 nTickCount; OMX_TICKS nTimeStamp; OMX_U32 nFlags; Petição 870250090858, de 06 / 10 / 2025, pág. 205 / 273 84 / 113 OMX_U32 nOutputPortIndex; OMX_U32 nInputPortIndex; } OMX_BUFFERHEADERTYPE;

[00218] A lista de flags de buffer pode ser: #define OMX_BUFFERFLAG_EOS 0x00000001 #define OMX_BUFFERFLAG_STARTTIME 0x00000002 #define OMX_BUFFERFLAG_DECODEONLY 0x00000004 #define OMX_BUFFERFLAG_DATACORRUPT 0x00000008 #define OMX_BUFFERFLAG_ENDOFFRAME 0x00000010 #define OMX_BUFFERFLAG_SYNCFRAME 0x00000020 #define OMX_BUFFERFLAG_EXTRADATA 0x00000040 #define OMX_BUFFERFLAG_CODECCONFIG 0x00000080 #define OMX_BUFFERFLAG_TIMESTAMPINVALID 0x00000100 #define OMX_BUFFERFLAG_READONLY 0x00000200 #define OMX_BUFFERFLAG_ENDOFSUBFRAME 0x00000400 #define OMX_BUFFERFLAG_SKIPFRAME 0x00000800

[00219] OpenMAX IL introduces the possibility of using an EGL image as an output buffer. An EGL image is designed to share data between rendering-based EGL interfaces, such as OpenGL, and OpenMAX components. It is up to the component to implement OMX_UseEGLImage() to bind the output to an EGL image instead of a traditional buffer.

[00220] Figure 23 is a conceptual diagram illustrating an example of Media Source Extension (MSE) media interfaces, according to the techniques of the present disclosure. MSE is a set of extensions to the media source attributes of HTML5 video and audio elements. It allows flexible control of media streams through JavaScript code using the definition of MediaSource objects. Petition 870250090858, dated 06 / 10 / 2025, pages 206 / 273 85 / 113 A MediaSource object can have one or more SourceBuffer objects. Applications attach media segments to SourceBuffer objects. A SourceBuffer can have multiple tracks, which are decoded and played back separately. Figure 23 depicts an example MSE configuration including the interface between the MSE API and the HTML5 media element.

[00221] The example in Figure 23 shows the media source (MS) 450 including the source buffers (SBs) 452A, 452B, and 452C. The source buffer 452A provides data for the track buffers (TBs) 454A, which include video and audio data. The video data is decoded by the video decoder (VDC) 456A, while the two sets of audio data are decoded by audio decoders (ADCs) 458A and 458B. The source buffer 452B provides data for the track buffer 454B, which includes video data decoded by the video decoder 456B. The 452C source buffer provides data for the 454C track buffer, which includes audio data decoded by the 458C audio decoder.Audio data selector 460 selects and / or combines decoded audio data to output audio data to the audio device (ADV) 464, while video data selector 462 selects and / or combines decoded video data to send video tags and decoded video data to the display region 466.

[00222] Table 10 represents a possible mapping of VDI functions in the MSE API: TABLE 10 VDI Functionality: MSE Mapping Petition 870250090858, dated 06 / 10 / 2025, pages 207 / 273 86 / 113 queryCurrentAggregate Capabilities / ) MediaSource.queryCurrentAggregate Capabilities()a getInstance() with grouping MediaSource.addSourceBuffer()b setConfig() CONFIG_OUTPUT_BUFFER VideoTrack.setConfig()c getParameter() and setParameter() VideoTrack and AudioTrack, getParameter() and setParameter()d NOTE The rationale and description are provided below. a A new method of the MediaSource object is used to query the current decoding capabilities. b Tracks of the same type, for example, VideoTracks, that belong to the same SourceBuffer are considered alternatives and only one is decoded and presented. When creating a new SourceBuffer, a group identifier for each track type can be provided. This grouping applies to all currently instantiated MediaSource objects. This allows the grouping of multiple decoder instances that belong to multiple HTML5 media elements. c New method of the HTML5 VideoTrack object.d New methods for HTML5 VideoTrack and AudioTrack objects.

[00223] In addition, an extension to the HTML5 video element can be used to allow data to be emitted to buffers, for example, WebGL buffers that are created via gl.createBuffer() functions. An extension to the input byte stream format can also be used to add support for raw media data, for example, raw AVC media streams.

[00224] Figure 24 is a block diagram illustrating a hypothetical example video decoding reference decoder. In this example, the hypothetical reference decoder includes a hypothetical 500 stream scheduler, a coded image buffer (CPB). Petition 870250090858, dated 06 / 10 / 2025, pages 208 / 273 87 / 113 coded picture buffer) 502, a decoding unit 504, a decoded picture buffer (DPB) 506, and an output trimming unit 508. A video decoding interface may conform to the hypothetical example video decoding reference decoder in Figure 24. That is, a video decoding interface may include components similar to those shown and described in relation to Figure 24.

[00225] A video decoding interface may have the following parameters that describe characteristics (e.g., decoding capabilities) of the video decoding interface: a profile (e.g., subsampling and bit depth support, such as 4:2:0 color format and 10 bits for bit depth); aggregate level capabilities (e.g., megabits per second (MB / s) bit rate, such as level 6.1); a number of instances (e.g., 16 instances); and supported codecs (e.g., ITU-T H.264 / Advanced Video Coding (AVC) and ITU-T H.265 / High Efficiency Video Coding (HEVC)). Therefore, the content may include maximum data signaling decoding / rendering capabilities required for each bitstream (e.g., using profile / level / tier signaling). A group of encoded bitstreams may be associated with a group with common maximum capabilities.

[00226] The video decoding interface can perform various functions. These functions may include decoding video bitstreams in accordance with the standard and storing encoded video data in a common encoded image buffer for a group of decoders. Associated access units via provided bitstreams Petition 870250090858, dated 06 / 10 / 2025, pages 209 / 273 88 / 113 simultaneously with the encoded image buffer can be provided synchronously to a group of decoders. Supplemental Enhancement Information (SEI) messages associated with the access units can be provided synchronously at the decoder output.

[00227] An access unit can be called access unit n, where n is a number that uniquely identifies the access unit. N can be incremented by 1 for each subsequent access unit in decoding order. Each decoding unit can be called decoding unit m, where m is a number that uniquely identifies the decoding unit. The first decoding unit in the first ordinal access unit (access unit 0) can be called decoding unit 0. The value of m can be incremented by 1 for each subsequent decoding unit in decoding order. Figure n can refer to the encoded image or the decoded image of access unit n. Timing information related to a specific decoding unit can arrive before the CPB removal time of that decoding unit.

[00228] The hypothetical reference decoder (HRD) in Figure 24 may include initialization parameters. For example, HRD_parameters() and sub_layer_hrd_parameters() may include encoded image buffer size (CPB) and decoded image buffer size (DPB), which may be signaled in an SEI message, a sequence parameter set (SPS), an image parameter set (PPS), or inferred from other values. For example, the CPB size may be inferred as CpbVclFactor * MaxCPB. Additionally, Petition 870250090858, dated 06 / 10 / 2025, pp. 210 / 273 89 / 113 a bit rate value can represent the maximum input bit rate (at access unit (AU) and decoding unit (DU) levels). Additionally, a subimage HRD flag can indicate whether an operating mode is AU mode or DU mode.

[00229] In the example in Figure 24, the hypothetical reference decoder can be initialized at decoding unit 0, with CPB 502 and DPB 506 being set to empty. The hypothetical stream scheduler 500 can generally schedule the retrieval of video data (e.g., access units) from multiple bitstreams and store the retrieved video data for CPB 502. That is, the hypothetical stream scheduler 500 can deliver data associated with decoding units flowing to CPB 502 according to a specific arrival schedule. The multiple bitstreams can be a group of bitstreams with maximum aggregate decoding capabilities that can be satisfied by the corresponding decoding device. Decoding unit 504 can retrieve decoding units from CPB 502 corresponding to the retrieved video data and decode the decoding units (e.g., blocks, slices, frames / images, etc.).In the case of a hypothetical reference decoder, it is assumed that decoding unit 504 decodes data associated with each decoding unit instantaneously at the time of CPB removal from the corresponding decoding unit. Decoding unit 504 can store the decoded data for DPB 506. Output cropping unit 508 can retrieve decoded images from DPB 506 and crop the images before output to a display device. A decoded image can be removed from DPB 506 when the image is no longer needed. Petition 870250090858, dated 06 / 10 / 2025, pages 211 / 273 90 / 113 is needed for interpretation reference and is no longer needed for output.

[00230] The CPB 502 can operate as follows. Initially, the decoding unit m can arrive at an initial arrival time. In some examples, the initial arrival time for the decoding unit m (i.e., initArrivalTime[m]) can be set equal to the final arrival time of the access unit of a previous access unit (m-1) for CBR mode. Otherwise, for non-CBR mode, the initial arrival time for the decoding unit m can be set equal to the maximum of the arrival time of the previous access unit (AuFinalArrivalTime[m1]) and the initial arrival time of the oldest decoding unit of the current access unit (initArrivalEarliestTime[m]).

[00231] The hypothetical reference decoder can calculate initArrivalEarliestTime[m] as equal to Au / DuNominalRemovalTime[m] (InitCpbRemovalDelay[SchedSelIdx] + InitCpbRemovalDelayOffset[SchedSelIdx]) / 90000. The hypothetical reference decoder can calculate AuFinalArrivalTime [m-1] as equal to initArrivalTime[m] + sizeInBits[m] / BitRate[SchedSelIdx]. The hypothetical reference decoder can calculate a time for the removal of the decoding unit according to AuNominalRemovalTime [0] = InitCpbRemovalDelay [SchedSelIdx] / 90000.

[00232] DPB 506 can contain multiple image storage buffers. Each of the image storage buffers can contain one or more decoded images, which Petition 870250090858, dated 06 / 10 / 2025, pages 212 / 273 91 / 113 can be marked as used for reference or retained for future output. The hypothetical reference decoder can calculate DpbOutputTime[n], a time at which to remove image n from DPB 506, as being equal to AuCpbRemovalTime[n] + ClockSubTick * picSptDpbOutputDuDelay. The 504 decoding unit can store a currently decoded image in DPB 506 in an empty image storage buffer. The fullness of DPB 506 can then be incremented by one. When TwoVersionsOfCurrDecPicFlag is equal to 0 and pps_curr_pic_ref_enabled_flag is equal to 1, the corresponding image can be marked as used for long-term reference. After all slices of the current image have been decoded, the image can be marked as used for short-term reference.

[00233] The techniques of this disclosure can be applied in situations where there are N video streams that must be decoded simultaneously. Each of the N video streams may have corresponding profile / tier / level requirements and HRD requirements, as specified in the parameters included in the bitstream. A video decoding system according to the techniques of this disclosure may correspond to one of the following models, shown in Figures 25 to 27, respectively, below.

[00234] Figure 25 is a block diagram illustrating an example model of a video decoding system that can implement techniques of the present disclosure to decode multiple video streams in parallel using multiple video decoding instances. In this example, encoded picture buffers (CPBs) and decoded picture buffers (DPBs) are implemented. Petition 870250090858, dated 06 / 10 / 2025, pages 213 / 273 92 / 113 independently for each of the N decoders.

[00235] In particular, the example in Figure 25 depicts the video decoding interface (VDI) of the CPB 530, the video decoding instances 540, and the VDI of the decoded image buffer 550. To decode N streams in parallel, N decoding instances can be instantiated. Thus, the video decoding instances 540 include decoders 542A to 542N (542 decoders). Each of the 542 decoders is associated with a respective CPB 532A to 543N (532 CPBs) and a respective DPB 552A to 552N (552 DPBs).

[00236] The data associated with the decoding units flowing to one of the CPBs 532 of each stream according to a specified arrival schedule can be delivered by a common hypothetical stream scheduler (HSS) 520 that schedules the N bitstreams for decoding by the respective decoders 542. For each access unit, all data associated with the access unit can be removed and decoded by one of the corresponding decoders 542 at the time of removal of the CPB from the access unit. One of the decoders 542 can then place the decoded image in one of the corresponding DPBs 552 for reference during the stream decoding process, as well as for output and clipping. A decoded image can be removed from one of the DPBs 552 after the image has been emitted when the image is no longer needed for interprediction reference.

[00237] At any given time, each of the individual bitstreams may conform to the profile / tier / level Petition 870250090858, dated 06 / 10 / 2025, pages 214 / 273 93 / 113 signaled and the HRD parameters of the individual stream. At any time, the sum of the CPB size can conform to the signaling profile / tier / level. At any time, the processing speed of the aggregate decoder (samples per second) can conform to the signaling profile / tier / level. And at any time, the sum of the DPB size can conform to the signaling profile / tier / level. Common HRD parameters for initial delay can also be specified.

[00238] Figure 26 is a block diagram illustrating another example model of a video decoding system that can implement techniques of the present disclosure to decode multiple video streams in parallel using multiple video decoding instances. In this example, CPBs are run independently for each of the N decoders, but a combined DPB is used to ensure that the output of the N decoders can be provided in a timely and synchronized manner for output or other subsequent processing.

[00239] In particular, the example in Figure 26 depicts CPB VDI 570, video decoding instances 580, and DPB VDI 590. To decode N streams in parallel, N decoding instances can be instantiated. Thus, the video decoding instances 580 include decoders 582A to 582N ​​(582 decoders). Each of the 582 decoders is associated with a respective CPB 572A to 572N (572 CPBs). In this example, DPB VDI 590 includes DPBs 592A to 592N, as well as a common DPB 594 that is shared among all decoding instances. Petition 870250090858, dated 06 / 10 / 2025, pages 215 / 273 94 / 113

[00240] The data associated with the decoding units flowing to one of the 572 CPBs of each stream according to a specified arrival schedule can be delivered by a common hypothetical stream scheduler (HSS) 560 that schedules the N bitstreams for decoding by the respective 582 decoders. For each access unit, all data associated with the access unit can be removed and decoded by one of the corresponding 582 decoders at the time of removal of the CPB from the access unit. One of the 582 decoders can then place the decoded image in one of the corresponding 592 DPBs for reference during the stream decoding process, as well as for output and clipping.A decoded image can be removed from one of the 552 DPBs after the image has been emitted when the image is no longer needed for interprediction reference and when the decoded image has an output time that is the longest among all remaining decoded images for the 582 decoder group.

[00241] At any given time, each of the individual bitstreams can conform to the signaled profile / tier / level and HRD parameters of the individual stream. At any given time, the sum of the CPB size can conform to the signaling profile / tier / level. At any given time, the processing speed of the aggregate decoder (samples per second) can conform to the signaling profile / tier / level. At any given time, the sum of the DPB size can conform to the signaling profile / tier / level. And at any given time, the common DPB size can conform to the signaling profile / tier / level. Common HRD parameters for initial delay. Petition 870250090858, dated 06 / 10 / 2025, pages 216 / 273 95 / 113 can also be specified.

[00242] Figure 27 is a block diagram illustrating another example model of a video decoding system that can implement techniques of the present disclosure to decode multiple video streams in parallel using multiple video decoding instances. In this example, there is a common CPB and a common DPB for each of the N decoders.

[00243] In particular, the example in Figure 27 depicts CPB VDI 610, video decoding instances 620, and DPB VDI 630. To decode N streams in parallel, N decoding instances can be instantiated. Thus, the video decoding instances 620 include decoders 622A to 622N (622 decoders). Each of the 622 decoders is associated with a respective CPB 614A to 614N (614 CPBs). Additionally, in this example, CPB VDI 610 includes a common CPB 612. In this example, DPB VDI 630 includes DPBs 632A to 632N, as well as a common DPB 634 that is shared among all decoding instances.

[00244] The data associated with the decoding units flowing to one of the 572 CPBs of each stream according to a specified arrival schedule can be delivered by a common hypothetical stream scheduler (HSS) 600 that schedules the N bitstreams for decoding by the respective decoders 622. The addition of each decoding unit can be done according to a common HSS. For each access unit, all data associated with the access unit can be removed and decoded by one of the corresponding decoders 622 at the time of removal of the CPB from the access unit. One of the decoders 622 can then place the image Petition 870250090858, dated 06 / 10 / 2025, pp. 217 / 273 96 / 113 decoded in one of the corresponding 632 DPBs for reference during the stream decoding process, as well as for output and clipping. A decoded image can be removed from one of the 632 DPBs after the image has been output when the image is no longer needed for interprediction reference and when the decoded image has an output time that is the longest among all remaining decoded images for the 622 decoder group.

[00245] At any given time, each of the individual bitstreams can conform to the signaled profile / tier / level and HRD parameters of the individual stream. At any given time, the sum of the CPB size can conform to the signaling profile / tier / level. At any given time, the processing speed of the aggregate decoder (samples per second) can conform to the signaling profile / tier / level. At any given time, the sum of the DPB size can conform to the signaling profile / tier / level. At any given time, the common DPB size can conform to the signaling profile / tier / level. And at any given time, the common CPB can conform to the common signaling profile / tier / level. Common HRD parameters for initial delay can also be specified.

[00246] Bitstreams can be generated in several ways. In one example, bitstreams can be generated together. For example, bitstreams can be encoded with VDI-based decoding in mind. General HRD parameters can be defined and, in general, a set of encoders can be controlled to ensure that common HRD parameters are maintained. Petition 870250090858, dated 06 / 10 / 2025, pp. 218 / 273 97 / 113

[00247] In another example, bitstreams can be generated individually. That is, streams can be encoded independently. Each stream can be annotated with an individual profile / tier / level and HRD parameters. Additional information can be provided for each bitstream to support joint decoding (e.g., decoded images). A common HRD operation can be derived by the decoder / client device.

[00248] In some examples, a flow scheduler can select flows based on annotation parameters.

[00249] In some examples, each bitstream may include encoding metadata so that HRD parameters can be derived in real time on the decoder / client device side.

[00250] Figure 28 is a flowchart illustrating an example method of decoding video data according to the techniques of this disclosure. The method in Figure 28 can be performed by any of the various decoding devices of this disclosure. For example and explanation purposes, the method in Figure 28 is described in relation to video decoding mechanism 200 in Figure 3. However, other devices can perform this or similar techniques, such as video decoder 48 in Figure 1, hardware video decoding mechanism 212 in Figures 4 to 6, video decoder mechanism 340 in Figure 15, video decoder mechanism 350 in Figure 16, video decoder mechanism 370 in Figure 17, video decoder mechanism 412 in Figure 20, decoding unit 504 in Figure 24, video decoding instances 540 in Figure 25, instances of Petition 870250090858, dated 06 / 10 / 2025, pp. 219 / 273 98 / 113 video decoding 580 from Figure 26 or the video decoding 620 instances from Figure 27.

[00251] Initially, a stream scheduling unit (e.g., RTP 52 receiving unit of Figure 1) can initially determine properties of a variety of video bitstreams (700). The properties can be, for example, profile / tier / level information, HRD parameters, or the like. The bitstreams can be bitstreams that are intended to be combined for parallel retrieval, decoding, and presentation, or they can be separate bitstreams that are subsequently combined. The RTP 52 receiving unit can select several of the bitstreams for retrieval using the properties (702). For example, the RTP 52 receiving unit can determine decoding capabilities of, for example, the video decoding engine 200 of Figure 3.

[00252] The video decoding mechanism 200 can then instantiate a number of video decoder instances 214 (704), for example, a number of video decoder instances equal to the number of video bitstreams to be recovered. The video decoder instances 214 can then decode the selected bitstreams (706). The output formatting unit 208 can additionally format the decoded images (708), for example, as shown and described in relation to any of Figures 8 to 20. The video decoding mechanism 200 can then output the formatted images (710).

[00253] In this way, the method in Figure 28 represents an example of a data decoding method. Petition 870250090858, dated 06 / 10 / 2025, pages 220 / 273 99 / 113 of media, including: instantiating a first number of video decoder instances to be executed by video decoding hardware implemented in a circuit set; determining properties of a plurality of video media streams, the properties indicating that each of the plurality of video media streams is available for streaming selection; selecting a second number of input video media streams from the plurality of video media streams according to the determined properties of the second number of input video media streams; decoding, by the video decoder instances, the second number of input video media streams to form the second number of decoded video media streams; and outputting data from the second number of decoded video media streams.

[00254] Several examples of the techniques of this disclosure are summarized in the following clauses:

[00255] Clause 1: A method for decoding media data, the method comprising: instantiating a first number of video decoder instances to be executed by video decoding hardware; decoding, by the video decoder instances, a second number of input video media streams to form one or more decoded video media streams; and outputting data from the one or more decoded video media streams.

[00256] Clause 2: The method of clause 1, which additionally comprises receiving configuration data indicating the first number.

[00257] Clause 3: The method of clause 2, where receiving configuration data includes receiving the data Petition 870250090858, dated 06 / 10 / 2025, pages 221 / 273 100 / 113 configuration of a media player application.

[00258] Clause 4: The method of clause 2, wherein receiving configuration data comprises receiving one or more supplementary enhancement information (SEI) messages with the incoming video media streams.

[00259] Clause 5: The method of any of clauses 1 to 4 which further comprises: receiving a third number of received video media streams; and formatting the received video media streams to form the second number of input video media streams.

[00260] Clause 6: The method of clause 5, wherein the formatting of the received video media stream comprises inserting a first video object from a first received video media stream onto one side of a second video object from a second video media stream.

[00261] Clause 7: The method of either of clauses 5 and 6, wherein the formatting of received video media streams comprises inserting a first video object from a first received video media stream above or below a second video object from a second video media stream.

[00262] Clause 8: The method of any of clauses 5 to 7, wherein the formatting of received video media streams comprises attaching a first video object from a first received video media stream to a second video object from the first received video media stream.

[00263] Clause 9: The method of any of clauses 5 to 8, wherein the formatting of received video media streams comprises stacking a first video object from a first received video media stream on top Petition 870250090858, dated 06 / 10 / 2025, pages 222 / 273 101 / 113 a second video object from the first received video media stream.

[00264] Clause 10: The method of any of clauses 1 to 9, wherein the output of data from one or more decoded video media streams comprises appending a first decoded image from a first video media stream above, below, to the left or to the right of a second decoded image from a second video media stream.

[00265] Clause 11: The method of any of clauses 5 to 10, which additionally comprises receiving configuration data from a media application indicating how the input video media streams should be combined to form a single output video media stream.

[00266] Clause 12: The method of any of clauses 5 to 10, which additionally comprises receiving information from a supplementary enhancement information message (SEI) indicating how the input video media streams should be combined to form a single output video media stream.

[00267] Clause 13: A device for decoding media data, wherein the device comprises one or more means for carrying out the method of any of clauses 1 to 12.

[00268] Clause 14: The system of clause 13, in which the one or more media comprise one or more processors and a memory configured to store media data.

[00269] Clause 15: The device of clause 13, wherein the device comprises at least one of: an integrated circuit; a microprocessor; and a device of Petition 870250090858, dated 06 / 10 / 2025, pages 223 / 273 102 / 113 wireless communication.

[00270] Clause 16: A device for retrieving media data, the device comprising: means for instantiating a first number of video decoder instances to be executed by video decoding hardware; means for executing the video decoder instances to decode a second number of input video media streams to form one or more decoded video media streams; and means for outputting data from the one or more decoded video media streams.

[00271] Clause 17: A method for decoding media data, the method comprising: instantiating a first number of video decoder instances to be executed by video decoding hardware implemented in a circuit set; determining properties of a plurality of video media streams, the properties indicating that each of the plurality of video media streams is available for streaming selection; selecting a second number of input video media streams from the plurality of video media streams according to the determined properties of the second number of input video media streams; decoding, by the video decoder instances, the second number of input video media streams to form the second number of decoded video media streams; and outputting data from the second number of decoded video media streams.

[00272] Clause 18: The method of clause 17, which additionally comprises receiving configuration data indicating the first number.

[00273] Clause 19: The method of clause 18, in Petition 870250090858, dated 06 / 10 / 2025, pp. 224 / 273 103 / 113 which receives configuration data includes receiving configuration data from a media player application.

[00274] Clause 20: The method of clause 18, wherein receiving configuration data comprises receiving one or more supplementary enhancement information (SEI) messages with the incoming video media streams.

[00275] Clause 21: The method of clause 17 which further comprises: receiving a third number of received video media streams; and formatting the received video media streams to form the second number of input video media streams.

[00276] Clause 22: The method of clause 21, wherein the formatting of the received video media stream comprises inserting a first video object from a first received video media stream into one side of a second video object from a second video media stream.

[00277] Clause 23: The method of clause 21, wherein the formatting of received video media streams comprises inserting a first video object from a first received video media stream above or below a second video object from a second video media stream.

[00278] Clause 24: The method of clause 21, wherein the formatting of received video media streams comprises attaching a first video object from a first received video media stream to a second video object from the first received video media stream.

[00279] Clause 25: The method of clause 21, wherein the formatting of received video media streams comprises stacking a first video object from a first received video media stream on top of a second Petition 870250090858, dated 06 / 10 / 2025, pages 225 / 273 104 / 113 video object of the first received video media stream.

[00280] Clause 26: The method of clause 21, which further comprises receiving configuration data from a media application indicating how the incoming video media streams should be combined to form a single outgoing video media stream.

[00281] Clause 27: The method of clause 21, which additionally comprises receiving information from a supplementary enhancement information (SEI) message indicating how the incoming video media streams should be combined to form a single outgoing video media stream.

[00282] Clause 28: The method of clause 17, wherein the output of data from one or more decoded video media streams comprises appending a first decoded image from a first video media stream above, below, to the left or to the right of a second decoded image from a second video media stream.

[00283] Clause 29: The method of clause 17, in which the properties of the plurality of video media streams comprise profile, tier and level requirements, and hypothetical reference decoder (HRD) requirements.

[00284] Clause 30: The method of clause 17, wherein each of the first number of video decoder instances includes a respective encoded image buffer (CPB) and a respective decoded image buffer (DPB).

[00285] Clause 31: The method of clause 17, wherein each of the first number of video decoder instances includes a respective encoded image buffer (CPB) and wherein the first number of decoder instances of Petition 870250090858, dated 06 / 10 / 2025, pp. 226 / 273 105 / 113 video shares a common decoded image buffer (DPB).

[00286] Clause 32: The method of clause 17, wherein the first instance number of video decoders includes a shared common encoded image buffer (CPB) and a shared common decoded image buffer (DPB).

[00287] Clause 33: The method of clause 17, wherein the selection of the second number of input video media streams comprises: determining the decoding capabilities of the first number of video decoder instances; determining rendering capabilities to render the second number of decoded video media streams; and selecting the second number of input video media streams that can be decoded according to the decoding capabilities and rendered according to the rendering capabilities.

[00288] Clause 34: A device for decoding media data, wherein the device comprises: a memory configured to store video data; and a processing system comprising one or more processors implemented in the circuit set, the processing system being configured to: instantiate a first number of video decoder instances to be executed by the processing system; determine properties of a plurality of video media streams, the properties indicating that each of the plurality of video media streams is available for streaming selection; select a second number of input video media streams from among the plurality of video media streams according to the determined properties of the second Petition 870250090858, dated 06 / 10 / 2025, pages 227 / 273 106 / 113 input video media streams; execute video decoder instances to decode the second input video media stream to form the second decoded video media stream; and output data from the second decoded video media stream.

[00289] Clause 35: The device of clause 34, wherein the memory comprises a common decoded image buffer (DPB) for storing decoded images for each of the video decoder instances.

[00290] Clause 36: The device of clause 34, wherein the memory comprises a common encoded image buffer (CPB) for storing encoded images for each of the video decoder instances.

[00291] Clause 37: A method for decoding media data, the method comprising: instantiating a first number of video decoder instances to be executed by video decoding hardware implemented in a circuit set; determining properties of a plurality of video media streams, the properties indicating that each of the plurality of video media streams is available for streaming selection; selecting a second number of input video media streams from the plurality of video media streams according to the determined properties of the second number of input video media streams; decoding, by the video decoder instances, the second number of input video media streams to form the second number of decoded video media streams; and outputting data from the second number Petition 870250090858, dated 06 / 10 / 2025, pages 228 / 273 107 / 113 of decoded video media streams.

[00292] Clause 38: The method of aspect 37, which additionally comprises receiving configuration data indicating the first number.

[00293] Clause 39: The method of clause 38, wherein receiving configuration data comprises receiving configuration data from a media player application.

[00294] Clause 40: The method of either of clauses 38 and 39, wherein receiving configuration data comprises receiving one or more supplementary enhancement information (SEI) messages with the incoming video media streams.

[00295] Clause 41: The method of any of clauses 37 to 40 which further comprises: receiving a third number of received video media streams; and formatting the received video media streams to form the second number of incoming video media streams.

[00296] Clause 42: The method of clause 41, wherein the formatting of the received video media stream comprises inserting a first video object from a first received video media stream onto one side of a second video object from a second video media stream.

[00297] Clause 43: The method of either of clauses 41 and 42, wherein the formatting of received video media streams comprises inserting a first video object from a first received video media stream above or below a second video object from a second video media stream.

[00298] Clause 44: The method of any of clauses 41 to 43, in which the formatting of media streams of Petition 870250090858, dated 06 / 10 / 2025, pages 229 / 273 108 / 113 received video involves attaching a first video object from a first received video media stream to a second video object from the first received video media stream.

[00299] Clause 45: The method of any of clauses 41 to 44, wherein the formatting of received video media streams comprises stacking a first video object from a first received video media stream on top of a second video object from the first received video media stream.

[00300] Clause 46: The method of any of clauses 41 to 45, which additionally comprises receiving configuration data from a media application indicating how the input video media streams should be combined to form a single output video media stream.

[00301] Clause 47: The method of any of clauses 41 to 46, which additionally comprises receiving information from a supplementary enhancement information message (SEI) indicating how the input video media streams should be combined to form a single output video media stream.

[00302] Clause 48: The method of any of clauses 37 to 47, wherein the output of data from one or more decoded video media streams comprises appending a first decoded image from a first video media stream above, below, to the left or to the right of a second decoded image from a second video media stream.

[00303] Clause 49: The method of any of clauses 37 to 48, wherein the properties of the plurality of video media streams comprise profile, tier and level requirements, and hypothetical reference decoder requirements. Petition 870250090858, dated 06 / 10 / 2025, pages 230 / 273 109 / 113 (HRD).

[00304] Clause 50: The method of any of clauses 37 to 49, wherein each of the first number of video decoder instances includes a respective encoded image buffer (CPB) and a respective decoded image buffer (DPB).

[00305] Clause 51: The method of any of clauses 37 to 50, wherein each of the first number of video decoder instances includes a respective encoded image buffer (CPB) and wherein the first number of video decoder instances share a common decoded image buffer (DPB).

[00306] Clause 52: The method of any of clauses 37 to 51, wherein the first video decoder instance number includes a shared common encoded image buffer (CPB) and a shared common decoded image buffer (DPB).

[00307] Clause 53: The method of any of clauses 37 to 52, wherein the selection of the second number of input video media streams comprises: determining the decoding capabilities of the first number of video decoder instances; determining rendering capabilities to render the second number of decoded video media streams; and selecting the second number of input video media streams that can be decoded according to the decoding capabilities and rendered according to the rendering capabilities.

[00308] Clause 54: A device for decoding media data, wherein the device comprises: a memory configured to store video data; and Petition 870250090858, dated 06 / 10 / 2025, pages 231 / 273 110 / 113 a processing system comprising one or more processors implemented in the circuit set, the processing system being configured to: instantiate a first number of video decoder instances to be executed by the processing system; determine properties of a plurality of video media streams, the properties indicating that each of the plurality of video media streams is available for streaming selection; select a second number of input video media streams from among the plurality of video media streams according to the determined properties of the second number of input video media streams; execute the video decoder instances to decode the second number of input video media streams to form the second number of decoded video media streams; and output data from the second number of decoded video media streams.

[00309] Clause 55: The device of clause 54, wherein the memory comprises a common decoded image buffer (DPB) for storing decoded images for each of the video decoder instances.

[00310] Clause 56: The device of either clause 54 or 55, wherein the memory comprises a common encoded image buffer (CPB) for storing encoded images for each of the video decoder instances.

[00311] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented Petition 870250090858, dated 06 / 10 / 2025, pages 232 / 273 111 / 113 In software, functions can be stored or transmitted as one or more instructions or code in a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media can include computer-readable storage media, which correspond to a tangible medium, such as data storage media, or communication media, including any medium that facilitates the transfer of a computer program from one place to another, for example, according to a communication protocol. In this way, computer-readable media can generally correspond to (1) tangible computer-readable storage media that are non-transient or (2) a communication medium, such as a signal or a carrier wave.Data storage media can be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in this disclosure. A computer program product may include a computer-readable medium.

[00312] By way of example, and not limitation, such computer-readable storage media may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory or any other media that can be used to store code. Petition 870250090858, dated 06 / 10 / 2025, pages 233 / 273112 / 113 of a desired program in the form of instructions or data structures that can be accessed by a computer. Furthermore, any connection is properly termed a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave will be included in the definition of medium. It should be understood, however, that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but instead refer to tangible non-transient storage media.As used in the present invention, disks (disk and disc) include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, where disks generally reproduce data magnetically, while discs reproduce data optically by means of lasers. Combinations of the above items should also be included within the scope of computer-readable media.

[00313] Instructions can be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other sets of discrete or integrated logic circuits. Petition 870250090858, dated 06 / 10 / 2025, pages 234 / 273 113 / 113 equivalents. Consequently, the term processor, as used in the present invention, may refer to any of the aforementioned structures or any other structure suitable for implementing the techniques described in the present invention. Furthermore, in some respects, the functionality described in the present invention may be provided in dedicated hardware and / or software modules configured for encoding and decoding, or may be incorporated into a combined codec. Additionally, the techniques may be fully implemented in one or more circuits or logic elements.

[00314] The techniques in this disclosure can be implemented in a wide variety of devices or appliances, including a wireless handset, an integrated circuit (IC), or a set of ICs (e.g., a chipset). Various components, modules, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require implementation by different hardware units. Instead, as described above, multiple units can be combined into a codec hardware unit or provided by a collection of interoperable hardware units, including one or more processors, as described above, in conjunction with suitable software and / or firmware.

[00315] Several examples have been described. These and other examples are within the scope of the following claims. Petition 870250090858, dated 06 / 10 / 2025, pages 235 / 273

Claims

1 / 5 CLAIMS 1. A method for decoding media data characterized by comprising: instantiating a first number of video decoder instances to be executed by video decoding hardware implemented in a circuit set; determining properties of a plurality of video media streams, the properties indicating that each of the plurality of video media streams is available for streaming selection; selecting a second number of input video media streams from the plurality of video media streams according to the determined properties of the second number of input video media streams; decoding, by the video decoder instances, the second number of input video media streams to form the second number of decoded video media streams; and outputting data from the second number of decoded video media streams.

2. Method, according to claim 1, characterized by additionally comprising receiving configuration data indicating the first number.

3. Method, according to claim 2, characterized by receiving configuration data from a media player application.

4. Method, according to claim 2, characterized by receiving the configuration data, including Petition 870250090858, dated 06 / 10 / 2025, page 236 / 273 2 / 5, and receiving one or more supplementary enhancement information (SEI) messages with the incoming video media streams.

5. A method according to claim 1, characterized by further comprising: receiving a third number of incoming video media streams; and formatting the incoming video media streams to form the second number of incoming video media streams.

6. Method, according to claim 5, characterized in that the formatting of the received video media stream comprises inserting a first video object from a first received video media stream onto one side of a second video object from a second video media stream.

7. A method according to claim 5, characterized in that the formatting of the received video media streams comprises inserting a first video object from a first received video media stream above or below a second video object from a second video media stream.

8. Method according to claim 5, characterized in that the formatting of received video media streams comprises attaching a first video object from a first received video media stream to a second video object from the first received video media stream.

9. A method according to claim 5, characterized in that the formatting of the received video media streams comprises stacking a first video object from a first received video media stream onto a second video object from the first received video media stream.

10. Method according to claim 5, Petition 870250090858, dated 10 / 06 / 2025, pp. 237 / 273 3 / 5 characterized by further comprising receiving configuration data from a media application indicating how the input video media streams should be combined to form a single output video media stream.

11. A method according to claim 5, characterized by additionally comprising receiving information from a supplementary enhancement information (SEI) message indicating how the input video media streams should be combined to form a single output video media stream.

12. Method, according to claim 1, characterized in that the output data from one or more decoded video media streams comprises attaching a first decoded image from a first video media stream above, below, to the left or to the right of a second decoded image from a second video media stream.

13. Method, according to claim 1, characterized in that the properties of the plurality of video media streams comprise profile, tier and level requirements, and hypothetical reference decoder (HRD) requirements.

14. Method, according to claim 1, characterized in that each of the first number of video decoder instances includes a respective encoded image buffer (CPB) and a respective decoded image buffer (DPB).

15. Method, according to claim 1, characterized in that each of the first number of video decoder instances includes a respective encoded image buffer (CPB), and wherein the first Petition 870250090858, dated 06 / 10 / 2025, pp. 238 / 273 4 / 5 number of video decoder instances shares a common decoded image buffer (DPB).

16. Method according to claim 1, characterized in that the first number of video decoder instances include a shared common encoded image buffer (CPB) and a shared common decoded image buffer (DPB).

17. A method according to claim 1, characterized in that the selection of the second number of input video media streams comprises: determining the decoding capabilities of the first number of video decoder instances; determining the rendering capabilities to render the second number of decoded video media streams; and selecting the second number of input video media streams that can be decoded according to the decoding capabilities and rendered according to the rendering capabilities.

18. Device for decoding media data characterized by comprising: a memory configured to store video data; and a processing system comprising one or more processors implemented in a circuit array, wherein the processing system is configured to: instantiate an initial number of video decoder instances to be executed by the processing system; determine properties of a plurality of Petition 870250090858, dated 06 / 10 / 2025, page.239 / 273 5 / 5 video media streams, the properties indicating that each of the plurality of video media streams is available for streaming selection; select a second input video media stream from among the plurality of video media streams according to the properties determined of the second input video media stream; execute the video decoder instances to decode the second input video media stream to form the second decoded video media stream; and output data from the second decoded video media stream.

19. Device according to claim 18, characterized in that the memory comprises a common decoded image buffer (DPB) for storing decoded images for each of the video decoder instances.

20. Device according to claim 18, characterized in that the memory comprises a common encoded image buffer (CPB) for storing encoded images for each of the video decoder instances. Petition 870250090858, dated 06 / 10 / 2025, pp. 240 / 273

Images