Video decoding engine for parallel decoding of multiple input video streams

Abstract

An example apparatus for decoding media data includes: a memory configured to store video data; and a processing system comprising one or more processors implemented in a circuit, the processing system configured to instantiate a first number of video decoder instances to be executed by the processing system; determining an attribute of the plurality of video media streams, the attribute indicating that each of the plurality of video media streams is available for stream selection; selecting the second number of input video media streams from the plurality of video media streams according to the determined attributes of the second number of input video media streams; executing the video decoder instance to decode the second number of input video media streams to form a second number of decoded video media streams; and outputting data of the second number of decoded video media streams.

Description

[0001] This application claims priority to U.S. Patent Application No. 18 / 638,091, filed April 17, 2024, and U.S. Provisional Application No. 63 / 496,913, filed April 18, 2023, the entire contents of which are incorporated herein by reference. U.S. Patent Application No. 18 / 638,091, filed April 17, 2024, claims the benefit of U.S. Provisional Application No. 63 / 496,913, filed April 18, 2023. Technical Field

[0002] This disclosure relates to the decoding of video data. Background Technology

[0003] Digital video capabilities can be integrated into a wide range of devices, including digital televisions, digital direct broadcasting systems, wireless broadcasting systems, personal digital assistants (PDAs), portable or desktop computers, digital cameras, digital recording devices, digital media players, video game devices, video game consoles, cellular or satellite radio phones, and video conferencing equipment. Digital video devices implement video compression technologies, such as those described in standards defined by MPEG-2, MPEG-4, ITU-T H.263 or ITU-T H.264 / MPEG-4 Part 10, Advanced Video Codec (AVC), ITU-T H.265 (also known as High Efficiency Video Codec (HEVC)), and extensions to such standards, to more efficiently send and receive digital video information.

[0004] Video compression techniques perform spatial and / or temporal prediction to reduce or eliminate inherent redundancy in video sequences. For block-based video coding and decoding, video frames or slices can be divided into macroblocks. Each macroblock can be further subdivided. Macroblocks in intra-frame (I) frames or slices are encoded using spatial prediction about adjacent macroblocks. Macroblocks in inter-frame (P) frames or slices can use spatial prediction about adjacent macroblocks in the same frame or slice, or temporal prediction about other reference frames.

[0005] After video data is encoded, it can be grouped for transmission or storage. Video data can be assembled into video files conforming to any of various standards, such as the International Organization for Standardization (ISO) Basic Media File Format and its extensions (such as AVC). Summary of the Invention

[0006] Generally, this disclosure describes techniques related to the decoding of multiple media streams for immersive media. This disclosure describes an interface to a video decoding engine and operations related to the basic stream and metadata that can be performed by the video decoding engine. To support these operations, this disclosure also describes Supplemental Enhancement Information (SEI) messages available for various video codecs. In particular, this disclosure describes a decoding system in which multiple video decoding instances can be instantiated to support parallel decoding of multiple video bitstreams. The video bitstreams can be selected based on decoding and rendering characteristics, which can be signaled using profile / layer / level and hypothetical reference decoder (HRD) parameters. Furthermore, the selected bitstreams for parallel decoding can collectively satisfy aggregated decoding and rendering characteristics. In this way, video bitstreams can be aggregated for parallel decoding, which can improve decoding speed and thus reduce the latency associated with displaying decoded video data.

[0007] In one example, a method for decoding media data includes: instantiating a first number of video decoder instances to be executed by video decoding hardware implemented in a circuit; determining attributes of a plurality of video media streams, the attributes indicating that each of the plurality of video media streams is available for stream selection; selecting a second number of input video media streams from the plurality of video media streams based on the determined attributes of the second number of input video media streams; decoding the second number of input video media streams by the video decoder instances to form a second number of decoded video media streams; and outputting data of the second number of decoded video media streams.

[0008] In another example, an apparatus for decoding video data includes: a memory configured to store the video data; and a processing system including one or more processors implemented in a circuit, the processing system being configured to: instantiate a first number of video decoder instances to be executed by the processing system; determine attributes of a plurality of video media streams, the attributes indicating that each of the plurality of video media streams is available for stream selection; select a second number of input video media streams from the plurality of video media streams based on the determined attributes of the second number of input video media streams; execute the video decoder instances to decode the second number of input video media streams to form a second number of decoded video media streams; and output data of the second number of decoded video media streams.

[0009] Details of one or more examples are set forth in the accompanying drawings and the following description. Other features, objects, and advantages will become apparent from the specification, drawings, and claims. Attached Figure Description

[0010] Figure 1 This is a block diagram illustrating an example system for implementing techniques for streaming media data over a network according to the present disclosure.

[0011] Figure 2 This is a block diagram showing the elements of the example video file.

[0012] Figure 3 This is a block diagram illustrating an example video decoding engine according to the technology disclosed herein.

[0013] Figure 4 This is a block diagram illustrating an example relationship between a video decoder instance and a video decoder hardware engine according to the technology of this disclosure.

[0014] Figure 5 This is a conceptual diagram illustrating an example of using a video decoder interface to instantiate multiple decoder instances of a video decoding engine according to the technology of this disclosure.

[0015] Figure 6 This is a block diagram illustrating an example video decoder interface system decoder model according to the present disclosure.

[0016] Figure 7 This is a conceptual diagram illustrating an example input and filtered output data received by a filter according to the technology of this disclosure.

[0017] Figures 8A to 8C This is a conceptual diagram illustrating an example input video of equal width and a combined output video received by an insertion function according to the technology of this disclosure.

[0018] Figures 9A to 9C This is a conceptual diagram illustrating an example input video of equal height and a combined output video received by an insertion function according to the technology of this disclosure.

[0019] Figure 10A and Figure 10B This is a conceptual diagram illustrating an example of input video and corresponding output video data received by additional functions according to the technology of this disclosure.

[0020] Figure 11A and Figure 11B This is a conceptual diagram illustrating an example of input video and corresponding output video data received by a stacking function according to the technology of this disclosure.

[0021] Figure 12 This is a conceptual diagram illustrating an example location of the boundary of a video data layer according to the technology of this disclosure.

[0022] Figure 13 This is a conceptual diagram illustrating an example of two layers of video data attached to each connection diagram according to the technology of this disclosure.

[0023] Figure 14 This is a conceptual diagram illustrating an example of three layers of video data attached to each connection diagram according to the technology disclosed herein.

[0024] Figure 15 This is a conceptual diagram illustrating an example of how a video decoder engine, according to the technology of this disclosure, uses application-based video decoder engine control to mosaic 2×2 video objects.

[0025] Figure 16 This is a conceptual diagram illustrating an example of a video decoder engine executing four decoder engines to mosaic a 2×2 video object according to the technology of this disclosure.

[0026] Figure 17 This is a conceptual diagram illustrating an example of a video decoder executing a decoder engine to mosaic 2×2 video objects according to the technology of this disclosure.

[0027] Figure 18 This is a conceptual diagram illustrating an example implementation of the technology according to the present disclosure for formatting four media streams in different steps.

[0028] Figure 19 This is a conceptual diagram illustrating an example implementation of the technology according to this disclosure for formatting four media streams in two steps.

[0029] Figure 20 This is a conceptual diagram illustrating an example of a video decoder engine according to the technology of this disclosure using control information based on supplementary enhancement information (SEI) to mosaic 2×2 video objects.

[0030] Figure 21 This is a conceptual diagram illustrating an example of the use of connection components and buffers according to the technology disclosed herein.

[0031] Figure 22 This is a conceptual diagram illustrating example port configurations of various buffers according to the technology disclosed herein.

[0032] Figure 23 This is a conceptual diagram illustrating an example of a Media Source Extension (MSE) media interface according to the technology disclosed herein.

[0033] Figure 24 This is a block diagram illustrating a hypothetical reference decoder for example video decoding.

[0034] Figure 25 This is a block diagram illustrating an example model of a video decoding system that can perform the techniques of this disclosure to decode multiple video streams in parallel using multiple video decoding instances.

[0035] Figure 26 This is a block diagram illustrating another example model of a video decoding system that can perform the techniques of this disclosure to decode multiple video streams in parallel using multiple video decoding instances.

[0036] Figure 27 This is a block diagram illustrating another example model of a video decoding system that can perform the techniques of this disclosure to decode multiple video streams in parallel using multiple video decoding instances.

[0037] Figure 28 This is a flowchart illustrating an example method for decoding video data according to the technology disclosed herein. Detailed Implementation

[0038] Generally, this disclosure describes a video decoding engine that can combine multiple input video streams to form a single output video stream. In some examples, the video decoding engine can instantiate multiple video decoding instances to decode multiple input video streams and then combine the decoded video streams. In some examples, the video decoding engine can use a single video decoding instance and an input formatting unit to combine multiple input video streams, and then use the single video decoding instance to decode the combined multiple input video streams. Various combinations of these examples are also possible, as discussed below.

[0039] In this way, playback of media data from multiple input video streams can be achieved. This playback can be synchronized across various video streams, allowing frames from each input video stream to be played back at the appropriate playback time. These techniques can be used for 2D video playback or other playback environments such as extended reality (XR), augmented reality (AR), mixed reality (MR), or virtual reality (VR).

[0040] Figure 1 This is a block diagram illustrating an example system 10 for implementing techniques for streaming media data over a network. In this example, system 10 includes a content preparation device 20, a server device 60, and a client device 40. Client device 40 and server device 60 are communicatively coupled via a network 74, which may include the Internet. In some examples, content preparation device 20 and server device 60 may also be coupled via network 74 or another network, or they may be directly communicatively coupled. In some examples, content preparation device 20 and server device 60 may include the same device.

[0041] exist Figure 1In the example, content preparation device 20 includes an audio source 22 and a video source 24. Audio source 22 may include, for example, a microphone that generates electrical signals representing captured audio data to be encoded by audio encoder 26. Alternatively, audio source 22 may include a storage medium storing previously recorded audio data, an audio data generator such as a computerized synthesizer, or any other audio data source. Video source 24 may include a video camera generating video data to be encoded by video encoder 28, a storage medium encoded with previously recorded video data, a video data generation unit such as a computer graphics source, or any other video data source. In all examples, content preparation device 20 is not necessarily communicatively coupled to server device 60, but multimedia content may be stored on a separate medium that is read by server device 60.

[0042] The raw audio and video data may include analog or digital data. Analog data may be digitized before being encoded by audio encoder 26 and / or video encoder 28. Audio source 22 may acquire audio data from a speaker while the speaker is speaking, and video source 24 may acquire video data of the speaker simultaneously. In other examples, audio source 22 may include a computer-readable storage medium containing stored audio data, and video source 24 may include a computer-readable storage medium containing stored video data. In this way, the techniques described in this disclosure can be applied to live, streaming, real-time audio and video data, or to archived, pre-recorded audio and video data.

[0043] An audio frame corresponding to a video frame is typically an audio frame containing audio data captured (or generated) by audio source 22, which occurs simultaneously with video data captured (or generated) by video source 24 and contained within the video frame. For example, when a speaker typically generates audio data by speaking, audio source 22 captures the audio data, and video source 24 simultaneously (i.e., when audio source 22 captures the audio data) captures the speaker's video data. Therefore, an audio frame may temporally correspond to one or more specific video frames. Thus, an audio frame corresponding to a video frame typically corresponds to the case where audio and video data are captured simultaneously, and the audio frame and video frame respectively include the simultaneously captured audio and video data.

[0044] In some examples, audio encoder 26 may encode a timestamp representing the time when audio data is recorded in encoded audio frames into each encoded audio frame, and similarly, video encoder 28 may encode a timestamp representing the time when video data is recorded in encoded video frames into each encoded video frame. In such examples, audio frames corresponding to video frames may include audio frames containing timestamps and video frames containing the same timestamps. Content preparation device 20 may include an internal clock from which audio encoder 26 and / or video encoder 28 may generate timestamps, or audio source 22 and video source 24 may use the internal clock to associate audio and video data with timestamps respectively.

[0045] In some examples, audio source 22 may send data corresponding to the time when audio data is recorded to audio encoder 26, and video source 24 may send data corresponding to the time when video data is recorded to video encoder 28. In some examples, audio encoder 26 may encode sequence identifiers into the encoded audio data to indicate the relative temporal order of the encoded audio data, but not necessarily the absolute time of recording the audio data; similarly, video encoder 28 may use sequence identifiers to indicate the relative temporal order of the encoded video data. Similarly, in some examples, sequence identifiers may be mapped or otherwise associated with timestamps.

[0046] Audio encoder 26 typically produces an encoded audio data stream, while video encoder 28 produces an encoded video data stream. Each individual data stream (whether audio or video) may be referred to as a primary stream. A primary stream is an example of a media stream. However, a media stream may include one or more primary streams and / or other data, such that a media stream is not necessarily the same as a single primary stream. A media stream may also contain metadata, such as non-VCL NAL units. A primary stream of video data may include a series of pictures (also called frames). This disclosure refers to a subframe as a smaller, independently decodable unit than a frame to which a video decoder may have applied post-decoding processing. This disclosure refers to a video object as an independently decodable substream of a video primary stream. This disclosure refers to a video object identifier as an integer or other value that identifies a video object.

[0047] A primary stream is a single digital codec (potentially compressed) component of a media presentation. For example, the codec video or audio portion of a media presentation can be a primary stream. A primary stream can be converted into a Blocked Primary Stream (PES) before being encapsulated into a video file. Within the same media presentation, a stream ID is used to distinguish PES packets belonging to one primary stream from those belonging to another. The basic data unit of a primary stream is the Blocked Primary Stream (PES) packet. Therefore, codec video data typically corresponds to a primary video stream. Similarly, audio data corresponds to one or more corresponding primary streams.

[0048] exist Figure 1 In one example, the encapsulation unit 30 of the content preparation device 20 receives a base stream comprising encoded and decoded video data from a video encoder 28 and a base stream comprising encoded and decoded audio data from an audio encoder 26. In some examples, the video encoder 28 and the audio encoder 26 may each include a packer for forming PES packets from the encoded data. In other examples, the video encoder 28 and the audio encoder 26 may each be connected to a corresponding packer interface for forming PES packets from the encoded data. In other examples, the encapsulation unit 30 may include a packer for forming PES packets from the encoded audio data and the encoded video data.

[0049] Video encoder 28 can encode video data of multimedia content in various ways to produce different representations of the multimedia content at various bit rates and with various characteristics, such as pixel resolution, frame rate, conformance to various codec standards, various profiles and / or profile levels conformance to various codec standards, representations with one or more views (e.g., for two-dimensional or three-dimensional playback), or other such characteristics. The representation used in this disclosure may include audio data, video data, text data (e.g., for closed captions), or other such data. The representation may include a primary stream, such as an audio primary stream or a video primary stream. Each PES packet may include a stream_id identifying the primary stream to which the PES packet belongs. Encapsulation unit 30 is responsible for assembling the primary streams into streamable media data.

[0050] Encapsulation unit 30 receives PES packets from audio encoder 26 and video encoder 28 for media presentation and forms corresponding Network Abstraction Layer (NAL) units from the PES packets. Codec video segments can be organized into NAL units, providing a “network-friendly” video representation for applications such as video telephony, storage, broadcasting, or streaming. NAL units can be categorized as Video Codec Layer (VCL) NAL units and non-VCL NAL units. VCL units may contain the core compression engine and may include block, macroblock, and / or slice-level data. Other NAL units may be non-VCL NAL units. In some examples, a codec frame in a time instance, typically presented as the main codec frame, may be included in an access unit, which may include one or more NAL units.

[0051] Non-VCL NAL units can include parameter set NAL units and SEI NAL units, etc. Parameter sets can contain sequence-level header information (in the Sequence Parameter Set (SPS)) and infrequently changing picture-level header information (in the Picture Parameter Set (PPS)). For parameter sets (e.g., PPS and SPS), infrequently changing information does not need to be repeated for each sequence or picture; therefore, encoding / decoding efficiency can be improved. Furthermore, the use of parameter sets enables out-of-band transmission of important header information, avoiding the need for redundant transmissions for error recovery. In out-of-band transmission examples, parameter set NAL units can be transmitted on channels different from other NAL units (such as SEI NAL units).

[0052] Supplemental Enhancement Information (SEI) can contain information that is not essential for decoding the encoded / decoded picture samples from VCL NAL units, but can assist in processes related to decoding, display, error recovery, and other purposes. SEI messages can be included in non-VCL NAL units. SEI messages are a normative part of some standards specifications and are therefore not always mandatory for standards-compliant decoder implementations. SEI messages can be sequence-level SEI messages or picture-level SEI messages. Some sequence-level information can be included in SEI messages, such as the scalability information SEI message in the SVC example and the view scalability information SEI message in Multi-View Video (MVC). These example SEI messages can convey information such as the extraction and characteristics of operation points.

[0053] Server device 60 includes a Real-Time Transport Protocol (RTP) sending unit 70 and a network interface 72. In some examples, server device 60 may include multiple network interfaces. Furthermore, any or all features of server device 60 may be implemented on other devices in the content delivery network, such as routers, bridges, proxy devices, switches, or other devices. In some examples, intermediate devices in the content delivery network may cache data for multimedia content 64 and include components that substantially conform to those of server device 60. Typically, network interface 72 is configured to send and receive data via network 74.

[0054] RTP sending unit 70 is configured to distribute media data to client device 40 via network 74 according to RTP, which was standardized by the Internet Engineering Task Force (IETF) in Request for Comments (RFC) 3550. RTP sending unit 70 may also implement RTP-related protocols such as RTP Control Protocol (RTCP), Real-time Streaming Protocol (RTSP), Session Initiation Protocol (SIP), and / or Session Description Protocol (SDP). RTP sending unit 70 may send media data via network interface 72, which may implement Unified Datagram Protocol (UDP) and / or Internet Protocol (IP). Therefore, in some examples, server device 60 may use network 74 to send media data via RTP and RTSP over UDP.

[0055] RTP sending unit 70 may receive RTSP description requests from, for example, client device 40. The RTSP description request may include data indicating what types of data the client device 40 supports. RTP sending unit 70 may respond to client device 40 with data indicating media streams (such as media content 64) that can be sent to client device 40, along with corresponding network location identifiers (such as Uniform Resource Locators (URLs) or Uniform Resource Names (URNs)).

[0056] RTP sending unit 70 can then receive an RTSP setup request from client device 40. The RTSP setup request typically instructs how the media stream should be transmitted. The RTSP setup request may include a network location identifier and transmission specifier for the requested media data (e.g., media content 64), such as the local port used to receive RTP data and control data (e.g., RTCP data) on client device 40. RTP sending unit 70 can respond to the RTSP setup request with data acknowledging and indicating the port through which server device 60 will transmit RTP data and control data. RTP sending unit 70 can then receive an RTSP playback request to allow the media stream to be "played," i.e., transmitted to client device 40 via network 74. RTP sending unit 70 can also receive an RTSP teardown request to terminate the streaming session; in response to this request, RTP sending unit 70 can stop transmitting media data for the corresponding session to client device 40.

[0057] Similarly, RTP receiving unit 52 can initiate a media stream by first sending an RTSP description request to server device 60. The RTSP description request can indicate the data types supported by client device 40. RTP receiving unit 52 can then receive a reply from server device 60 specifying the available media streams (such as media content 64) that can be sent to client device 40, and the corresponding network location identifiers (such as Uniform Resource Locator (URL) or Uniform Resource Name (URN)).

[0058] RTP receiving unit 52 can then generate an RTSP setup request and send it to server device 60. As described above, the RTSP setup request may include a network location identifier and transport specifier for the requested media data (e.g., media content 64), such as the local port used to receive RTP data and control data (e.g., RTCP data) on client device 40. In response, RTP receiving unit 52 can receive an acknowledgment from server device 60, which includes the port of server device 60 that server device 60 will use to send media data and control data.

[0059] After a media stream session is established between server device 60 and client device 40, the RTP sending unit 70 of server device 60 can send media data (e.g., media data packets) to client device 40 according to the media stream session. Server device 60 and client device 40 can exchange control data (e.g., RTCP data) indicating, for example, the reception statistics of client device 40, so that server device 60 can perform congestion control or otherwise diagnose and resolve transmission failures.

[0060] Network interface 54 can receive media presented by the selected media and provide it to RTP receiving unit 52, which in turn can provide the media data to decapsulation unit 50. Decapsulation unit 50 may, depending on whether the encoded data is part of an audio or video stream (e.g., as indicated by the PES packet header of the stream), decapsulate elements of the video file into a component PES stream, unpack the PES stream to retrieve the encoded data, and send the encoded data to audio decoder 46 or video decoder 48. Audio decoder 46 decodes the encoded audio data and sends the decoded audio data to audio output 42, while video decoder 48 decodes the encoded video data and sends the decoded video data to video output 44, which may include multiple views of the stream.

[0061] According to the technology disclosed herein, video decoder 48 may correspond to a video decoding engine. As explained in more detail below, the video decoding engine may include an input interface for receiving one or more media data streams and one or more metadata streams. The video decoding engine may include hardware components such as memory and processing circuitry that can instantiate one or more video decoder instances. Each video decoder instance may be exposed to the application layer as a corresponding video decoder with its own interface through which encoded video data to be decoded is sent. Video decoder 48 may execute each of the various video decoder instances to decode the media data streams in parallel and may format the resulting output so that the output data is synchronized in time. In some examples, as discussed in more detail below, video decoder 48 may concatenate, append, stack, or otherwise combine or trim video data for the purpose of presentation via video output 44.

[0062] Video encoder 28, video decoder 48, audio encoder 26, audio decoder 46, encapsulation unit 30, RTP receiver unit 52, and decapsulation unit 50 can each be implemented as any of a variety of suitable processing circuits, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic circuits, software, hardware, firmware, or any combination thereof. Each of video encoder 28 and video decoder 48 can be included in one or more encoders or decoders, either of which can be integrated as part of a combined video encoder / decoder (CODEC). Similarly, each of audio encoder 26 and audio decoder 46 can be included in one or more encoders or decoders, either of which can be integrated as part of a combined CODEC. The apparatus including video encoder 28, video decoder 48, audio encoder 26, audio decoder 46, encapsulation unit 30, RTP receiver unit 52, and / or decapsulation unit 50 can include integrated circuits, microprocessors, and / or wireless communication devices (such as cellular phones).

[0063] Client device 40, server device 60, and / or content preparation device 20 may be configured to operate according to the techniques of this disclosure. For illustrative purposes, this disclosure describes these techniques with respect to client device 40 and server device 60. However, it should be understood that content preparation device 20 may be configured to perform these techniques in place of (or in addition to) server device 60.

[0064] Encapsulation unit 30 can form NAL units, which include a header identifying the program to which the NAL unit belongs, and a payload, such as audio data, video data, or data describing the transmission or program stream corresponding to the NAL unit. For example, in H.264 / AVC, a NAL unit includes a 1-byte header and payloads of varying sizes. NAL units that include video data in their payloads can include video data at various granularities. For example, a NAL unit can include video data blocks, multiple blocks, video data slices, or entire frames of video data. Encapsulation unit 30 can receive encoded video data from video encoder 28 in the form of PES packets of the elementary stream. Encapsulation unit 30 can associate each elementary stream with its corresponding program.

[0065] The encapsulation unit 30 can also assemble access units from multiple NAL units. Typically, an access unit may include one or more NAL units representing frames of video data, and, when such audio data is available, the corresponding audio data for that frame. An access unit typically includes all NAL units of an output time instance, such as all audio and video data for that time instance. For example, if each view has a frame rate of 20 frames per second (fps), each time instance may correspond to a time interval of 0.05 seconds. During this time interval, specific frames from all views of the same access unit (same time instance) can be rendered simultaneously. In one example, an access unit may include a codec screen within a time instance, which can be presented as the main codec screen.

[0066] Therefore, an access unit can include all audio and video frames of a common time instance, for example, all views corresponding to time X. This disclosure also refers to the encoded screen of a specific view as a "view component." That is, a view component can include the encoded screen (or frame) of a specific view at a specific time. Therefore, an access unit can be defined as all view components including a common time instance. The decoding order of the access units need not be the same as the output or display order.

[0067] After the encapsulation unit 30 assembles the NAL units and / or access units into a video file based on the received data, the encapsulation unit 30 passes the video file to the output interface 32 for output. In some examples, the encapsulation unit 30 may store the video file locally or send the video file to a remote server via the output interface 32, instead of sending the video file directly to the client device 40. The output interface 32 may include, for example, a transmitter, a transceiver, or a device for writing data to a computer-readable medium (e.g., an optical drive, a magnetic media drive (e.g., a floppy disk drive), a Universal Serial Bus (USB) port, a network interface, or other output interface). The output interface 32 outputs the video file to a computer-readable medium, such as a transmission signal, magnetic media, optical media, memory, flash memory, or other computer-readable media.

[0068] Network interface 54 can receive NAL units or access units via network 74 and provide NAL units or access units to decapsulation unit 50 via RTP receiving unit 52. Decapsulation unit 50 may, depending on whether the encoded data is part of an audio or video stream (e.g., as indicated by the stream's PES packet header), decapsulate elements of the video file into a component PES stream, unpack the PES stream to retrieve the encoded data, and send the encoded data to audio decoder 46 or video decoder 48. Audio decoder 46 decodes the encoded audio data and sends the decoded audio data to audio output 42, while video decoder 48 decodes the encoded video data and sends the decoded video data to video output 44, which may include multiple views of the stream.

[0069] Figure 2 This is a block diagram illustrating the elements of example video file 150. As described above, video files, according to the ISO Basic Media File Format and its extensions, store data in a series of objects called "boxes". Figure 2 In the example, video file 150 includes a file type (FTYP) box 152, a movie (MOOV) box 154, a segment index (sidx) box 162, a movie clip (MOOF) box 164, and a movie clip random access (MFRA) box 166. Although Figure 2 This represents an example of a video file, but it should be understood that, based on the ISO Basic Media File Format and its extensions, other media files may include other types of media data (e.g., audio data, timed text data, etc.) constructed similarly to the data in video file 150.

[0070] The File Type (FTYP) box 152 generally describes the file type of the video file 150. The File Type box 152 may include specification data that identifies the best use of the video file 150. The File Type box 152 may alternatively be placed before the MOOV box 154, the Movie Clip box 164, and / or the MFRA box 166.

[0071] exist Figure 2 In the example, MOOV box 154 includes a Movie Header (MVHD) box 156, a Track (TRAK) box 158, and one or more Movie Extension (MVEX) boxes 160. Typically, the MVHD box 156 describes general characteristics of the video file 150. For example, the MVHD box 156 may include data describing when the video file 150 was initially created, when the video file 150 was last modified, the timestamp of the video file 150, the playback duration of the video file 150, or other data that generally describes the video file 150.

[0072] TRAK box 158 may include data from a track of video file 150. TRAK box 158 may include a Track Header (TKHD) box that describes the characteristics of the track corresponding to TRAK box 158. In some examples, TRAK box 158 may include encoded video frames, while in other examples, encoded video frames of a track may be included in a movie clip 164, which may be referenced by data from TRAK box 158 and / or sidx box 162.

[0073] In some examples, video file 150 may include more than one track. Therefore, MOOV box 154 may include TRAK boxes in a number equal to the number of tracks in video file 150. TRAK boxes 158 may describe the characteristics of the corresponding tracks in video file 150. For example, TRAK boxes 158 may describe the temporal and / or spatial information of the corresponding tracks. When encapsulation unit 30 ( Figure 1 When a parameter set track is included in a video file (such as video file 150), a TRAK box, similar to MOOV box 154 and TRAK box 158, can describe the characteristics of the parameter set track. Encapsulation unit 30 can signal the presence of a sequence level SEI message in the parameter set track within the TRAK box describing the parameter set track.

[0074] The MVEX box 160 can describe the characteristics of the corresponding movie clip 164, for example, by signaling to the video file 150 that, in addition to the video data (if any) included in the MOOV box 154, the movie clip 164 is also included. In the context of streaming video data, encoded video frames may be included in the movie clip 164 instead of the MOOV box 154. Therefore, all encoded video samples may be included in the movie clip 164 instead of the MOOV box 154.

[0075] MOOV boxes 154 may include an equal number of MVEX boxes 160 as the number of movie segments 164 in the video file 150. Each MVEX box in the MVEX boxes 160 may describe the characteristics of a corresponding movie segment in the movie segment 164. For example, each MVEX box may include a Movie Extended Header Box (MEHD) box, which describes the time duration of a corresponding movie segment in the movie segment 164.

[0076] As described above, encapsulation unit 30 may store a sequence dataset in a video sample that does not contain actual encoded video data. The video sample may typically correspond to an access unit, which is a representation of an encoded frame at a specific time instance. In the context of AVC, an encoded frame includes one or more VCL NAL units (which contain information such as SEI messages for all pixels used to construct the access unit and other associated non-VCL NAL units). Therefore, encapsulation unit 30 may include a sequence dataset in a movie clip of movie clip 164, which may include a sequence-level SEI message. Encapsulation unit 30 may further signal the presence of the sequence dataset and / or the sequence-level SEI message as present in a movie clip of movie clip 164 within an MVEX box of MVEX box 160 corresponding to a movie clip of movie clip 164.

[0077] SIDX box 162 is an optional element of video file 150. That is, video files conforming to 3GPP file formats or other such file formats do not necessarily need to include SIDX box 162. According to the example of the 3GPP file format, SIDX boxes can be used to identify sub-segments of a segment (e.g., a segment contained within video file 150). The 3GPP file format defines a sub-segment as "a self-contained set of one or more consecutive movie clip boxes having a corresponding media data box and a media data box containing data referenced by a movie clip box that must follow that movie clip box and precede the next movie clip box containing information about the same track." The 3GPP file format also instructs that the SIDX box "contains a series of references to sub-segments of the (sub)segment recorded by that box. The referenced sub-segments are consecutive in presentation time. Similarly, bytes referenced by the segment index box are always consecutive within the segment. The size of the reference gives a count of the number of bytes in the referenced material."

[0078] SIDX box 162 typically provides information representing one or more sub-segments of a segment included in video file 150. For example, such information may include the playback time of the start and / or end of the sub-segment, the byte offset of the sub-segment, whether the sub-segment includes (e.g., begins at) a Stream Access Point (SAP), the type of SAP (e.g., whether the SAP is an Instant Decoder Refresh (IDR) screen, a Clean Random Access (CRA) screen, a Broken Link Access (BLA) screen, etc.), the location of the SAP within the sub-segment (in terms of playback time and / or byte offset), etc.

[0079] Movie clip 164 may include one or more encoded video frames. In some examples, movie clip 164 may include one or more groups of frames (GOPs), where each GOP may include several encoded video frames, such as frames or pictures. Additionally, as mentioned above, in some examples, movie clip 164 may include a sequence dataset. Each movie clip in movie clip 164 may include a movie clip header box (MFHD). Figure 2 (Not shown in the image). The MFHD box can describe the characteristics of the corresponding movie clip, such as the movie clip's sequence number. Movie clips 164 can be included in video file 150 in the order of their sequence numbers.

[0080] MFRA box 166 can describe random access points within movie segments 164 of video file 150. This can help perform special effects modes, such as searching for specific time positions (i.e., playback times) within segments encapsulated by video file 150. In some examples, MFRA box 166 is typically optional and does not need to be included in the video file. Similarly, client devices (such as client device 40) do not necessarily need to refer to MFRA box 166 to correctly decode and display the video data of video file 150. MFRA box 166 can include a number equal to the number of tracks in video file 150, or in some examples, a number equal to the number of media tracks (e.g., non-cue tracks) in video file 150 (TFRA) boxes (not shown).

[0081] In some examples, movie clip 164 may include one or more streaming access points (SAPs), such as IDR frames. Similarly, MFRA box 166 can provide an indication of the position of the SAP within video file 150. Therefore, a temporal subsequence of video file 150 can be formed from the SAP of video file 150. The temporal subsequence may also include other frames, such as SAP-dependent P-frames and / or B-frames. Frames and / or slices of the temporal subsequence can be arranged within segments such that frames / slices of the temporal subsequence that depend on other frames / slices of the subsequence can be correctly decoded. For example, in a hierarchical arrangement of data, data used for prediction of other data may also be included in the temporal subsequence.

[0082] Figure 3 This is a block diagram illustrating an example video decoding engine 200 according to the technology disclosed herein. The video decoding engine 200 may correspond to... Figure 1 The video decoder 48. In this example, the video decoding engine 200 includes an input video decoding interface 202, an input formatting unit 204, a hardware video decoding engine 212, a time-locking unit 206, an output formatting unit 208, and an output video decoding interface 210. The hardware video decoding engine 212 can execute various video decoder instances 214A to 214N (video decoding instance 214).

[0083] According to the technology disclosed herein, the video decoding engine (VDE) 200 is capable of decoding, synchronizing, and formatting media streams. The media stream includes one or more aggregated base streams or portions thereof. The media stream is fed through the input video decoding interface (IVDI) 202 of the VDE 200 and provided in its decoded form to subsequent elements of the rendering pipeline via the output video decoding interface (OVDI) 210.

[0084] Between IVDI 202 and OVDI 210, the input formatting unit 204 of VDE 200 extracts and merges individual decodeable regions from a set of input media streams, generating a set of basic streams that are fed to video decoder instances 214 running inside VDE 200. VDE 200 can perform merging or extraction operations on the input media streams such that the number of running video decoder instances 214 differs from the number of input media streams required by the application.

[0085] For example, VDE 200 might not be able to decode a single 4K input media stream using a single decoder instance, but it might be able to decode several independent decodeable regions within the 4K input video stream at a lower resolution. In this case, VDE 200 can verify the availability of sufficient resources to run video decoder instance 214 in parallel. In some examples, multiple elementary streams output by input formatting unit 204 can be fed into a single instance within video decoder instance 214.

[0086] The VDE 200 accepts media streams and metadata streams via IVDI 202. There is at least one media stream as input, but there is no limit to the number of metadata streams relative to the number of media streams consumed simultaneously by the VDE 200. Therefore, the input to the VDE 200 consists of N media streams and M metadata streams, where N is at least 1 and M is zero or greater.

[0087] The VDE 200 outputs decoded video sequences and metadata streams via the OVDI 210. At least one decoded video sequence is output, but there is no limit to the number of metadata streams relative to the number of decoded video sequences simultaneously output by the VDE 200. Both output stream types can be provided as multiplexed output buffers, including decoded media data and its associated metadata. Therefore, the VDE 200 output includes Q decoded sequences and P metadata streams, where Q is at least 1 and P is zero or greater.

[0088] Figure 4 This is a block diagram illustrating an example relationship between a video decoder instance 214 and a hardware video decoding engine 212 according to the technology of this disclosure. In this example, one or more video decoder instances 214 are executed by the same hardware video decoding engine 212. The hardware video decoding engine 212 exposes each of the video decoder instances 214 to the application layer as several decoder instances, each decoder instance having its own input and output interfaces.

[0089] Figure 5 This is a conceptual diagram illustrating an example of a plurality of video decoder instances 214 instantiated using a video decoder interface of a hardware video decoding engine 212 according to the present disclosure. Figure 5 The example further depicts clock component 222, aggregation buffer 224, and application configuration information 220. Hardware video decoding engine 212 can receive application configuration information 220 from multimedia applications such as media players and extended reality (XR) players. The video decoding interface may include functionality defined using the IDL syntax specified in ISO / IEC 19516 "Information technology—Object management group—Interface definition language (IDL)" version 4.2.

[0090] exist Figure 5 In the example, video decoder instance 214 can use some functions of the video decoding interface. In this example, video decoder instance 214 has identifiers 1 to 3 and belongs to a group with identifier 4. Through this grouping mechanism, the three instances write the decoding sequence to the aggregation buffer 224, and the decoding operations across these instances are performed in a coordinated manner (referencing the clock signal from clock component 222) so that no instance runs before or after the other instances.

[0091] The IDL declaration of the queryCurrentAggregateCapabilities() function, as well as the AggregateCapabilities and PerformancePoint structures and capability flags, can be defined as follows:

[0092] const unsigned long CAP_INSTANCES_FLAG = 0x1;

[0093] const unsigned long CAP_BUFFER_MEMORY_FLAG = 0x2;

[0094] const unsigned long CAP_BITRATE_FLAG = 0x4;

[0095] const unsigned long CAP_MAX_SAMPLES_SECOND_FLAG = 0x8;

[0096] const unsigned long CAP_MAX_PERFORMANCE_POINT_FLAG = 0xA;

[0097] struct PerformancePoint {

[0098] float picture_rate;

[0099] unsigned long width;

[0100] unsigned long height;

[0101] unsigned long bit_depth;

[0102] };

[0103] struct AggregateCapabilities {

[0104] unsigned long flags;

[0105] unsigned long max_instances;

[0106] unsigned long buffer_memory;

[0107] unsigned long bitrate;

[0108] unsigned long max_samples_second;

[0109] PerformancePoint max_performance_point;

[0110] };

[0111] AggregateCapabilities queryCurrentAggregateCapabilities (

[0112] in string component_name,

[0113] in unsigned long flags );

[0115] Applications can use the queryCurrentAggregateCapabilities() function to query the transient aggregation capabilities of a specific codec component's decoder platform. Capability flags can be set individually or within a single function call to query one or more parameters.

[0116] `component_name` can provide the name of a component of the decoding platform for which the application queries. The name "All" can be used to indicate that the query is not targeting a specific component, but rather all components of the decoding platform. A component is a hardware or software function exposed by the video decoding engine 200 (such as a decoder).

[0117] The CAP_INSTANCES_FLAG query returns the max_instances parameter, which indicates the maximum number of decoder instances that can be instantiated for the provided decoder component at this time.

[0118] The CAP_BUFFER_MEMORY_FLAG query query for the buffer_memory parameter, which indicates the instantaneous global maximum available buffer size in bytes. This buffer size can be allocated independently of any component on the decoder platform at this time for buffer swapping. Memory allocation can be done by the application or by the VDE 200 itself, depending on the VDE instantiation.

[0119] The CAP_BITRATE_FLAG query parameter is the bitrate parameter, which indicates the maximum instantaneous encoding / decoding bitrate that the queried component can handle, in bits per second.

[0120] The CAP_MAX_SAMPLES_SECOND_FLAG query retrieves the max_samples_second parameter, which indicates the maximum instantaneous number of combined luminance and chrominance samples that the queried component can process per second.

[0121] The CAP_MAX_PERFORMER_POINT_FLAG query retrieves the max_performance_point parameter, which indicates the maximum performance point of the bitstream that can be decoded by the component indicated in the new instance of this decoder component.

[0122] Performance points can include the following parameters:

[0123] • picture_rate indicates the instantaneous frame rate at the point of maximum performance, measured in frames per second.

[0124] •height represents the height of the brightness sample at the point of maximum performance.

[0125] • width represents the brightness sample width at the maximum performance point.

[0126] • bit_depth represents the bit depth of the luminance sample at the highest performance point.

[0127] Each parameter at the maximum performance point does not necessarily represent the maximum value in that dimension. Parameters can be a combination of all dimensions that make up the maximum performance point.

[0128] The IDL declaration for the getInstance() function and the associated ErrorAllocation exception can be defined as follows:

[0129] exception ErrorAllocation {

[0130] string reason;

[0131] };

[0132] unsigned long getInstance(

[0133] in string component_name,

[0134] in unsigned long group_id / / optional, default value = -1

[0135] raises(ErrorAllocation);

[0136] If a new instance is requested, a successful call to the getInstance() function may provide the instance's identifier and the group_id assigned to or created for this new instance. The default behavior is that the decoder instance does not belong to any existing group, but is instead assigned to a newly created group.

[0137] Having several decoder instances belonging to the same group means that the VDE 200 processes these instances collectively, ensuring that their decoding states are synchronized rather than competing with each other. As a result, the VDE 200 can guarantee synchronized output write operations, possibly to the aggregation buffer 224. There is no condition where two video decoder instances are in the same group.

[0138] The IDL declaration of the setConfig() function, the associated ErrorConfig exception, the ConfigDataParameters structure, and the ConfigParameters enumeration can be defined as follows:

[0139] enum ConfigParameters {

[0140] CONFIG_OUTPUT_BUFFER

[0141] };

[0142] struct ConfigDataParameters {

[0143] SampleFormat sample_format;

[0144] SampleType sample_type;

[0145] unsigned long sample_stride;

[0146] unsigned long line_stride;

[0147] unsigned long buffer_offset;

[0148] };

[0149] exception ErrorConfig {

[0150] string reason;

[0151] };

[0152] boolean setConfig (

[0153] in unsigned long instance_id,

[0154] in ConfigParameters config_parameters,

[0155] in ConfigDataParameters config_data_parameters

[0156] raises(ErrorConfig);

[0157] The `setConfig()` function can be called with the parameter `CONFIG_OUTPUT_BUFFER` to specify the format of the output buffer. The buffer format can include the following parameters:

[0158] • sample_format indicates the format of each sample, which can be a scalar, 2D vector, 3D vector, or 4D vector.

[0159] • sample_type indicates the type of each component of the sample.

[0160] • sample_stride indicates the number of bytes between two consecutive samples in the output.

[0161] • line_stride indicates the number of bytes between the first byte of the first line of the output and the first byte of the next line.

[0162] • buffer_offset indicates the offset within the output buffer from which the output frame should begin writing.

[0163] The IDL declarations for the getParameter() and setParameter() functions, along with the associated ErrorParameter exception and ExtParameters enumeration, can be defined as follows:

[0164] enum ExtParameters {

[0165] PARAM_PARTIAL_OUTPUT,

[0166] PARAM_SUBFRAME_OUTPUT,

[0167] PARAM_METADATA_CALLBACK,

[0168] PARAM_OUTPUT_CROP,

[0169] PARAM_OUTPUT_CROP_WINDOW,

[0170] PARAM_MAX_OFFTIME_JITTER

[0171] };

[0172] struct CropWindow {

[0173] unsigned long x;

[0174] unsigned long y;

[0175] unsigned long width;

[0176] unsigned long height;

[0177] };

[0178] exception ErrorParameter {

[0179] string reason;

[0180] };

[0181] any getParameter (

[0182] in unsigned long instance_id,

[0183] in ExtParameters ext_parameters,

[0184] out any parameter );

[0186] boolean setParameter (

[0187] in unsigned long instance_id,

[0188] in ExtParameters ext_parameters,

[0189] in any parameter

[0190] raises(ErrorParameter);

[0191] The getParameter() and setParameter() functions can receive extended parameters as described below.

[0192] PARAM_PARTIAL_OUTPUT can indicate whether the output of a subframe is required, desired, or not allowed.

[0193] PARAM_SUBFRAME_OUTPUT can indicate one or more subframes output by the decoder.

[0194] PARAM_METADATA_CALLBACK allows you to set a callback function for a specific metadata type. The list of supported metadata types can depend on the codec and can be defined independently for each codec.

[0195] PARAM_OUTPUT_CROP indicates that only a portion of the frame needs to be decoded at the output. Decoder instances can use this information to intelligently reduce their decoding process by discarding as many units as possible that do not fall within the clipped output region.

[0196] PARAM_OUTPUT_CROP_WINDOW can indicate the parts of the decoded frame to be clipped and output.

[0197] PARAM_MAX_OFFTIME_JITTER indicates the maximum amount of time between consecutive executions of a decoder instance, in microseconds. This parameter can be relevant whenever the underlying hardware components are shared across multiple decoder instances, requiring context switching between different decoder instances.

[0198] Vulkan® Video (VK) is an extension of the Vulkan API that defines the capabilities exposed by the Graphics Processing Unit (GPU). This extension provides an interface for applications to leverage the hardware decoding and encoding capabilities present on the GPU. A VK video session consists of a single decoding session on a single layer. The result is... Figure 3 As shown, a single VK video session corresponds to a single video decoder instance. Table 1 below summarizes the mapping of VDI functions on VK:

[0199] Table 1

[0200]

[0201] The VK video API provides a function to query the capabilities of a single VK video profile, called `vkGetPhysicalDeviceVideoCapabilitiesKHR()`. Similar to this function, the VDI VK mapping defines the function `vkGetPhysicalDeviceCurrentVideoCapabilitiesMPEG()`. Unlike `vkgetphysicaldevicevideocapabilitieskhr()`, `vkGetPhysicalDeviceCurrentVideoCapabilitiesMPEG()` allows querying aggregate capabilities of a physical device. When it is called with a specific profile, the aggregate capabilities belong to that given profile.

[0202] The vkGetPhysicalDeviceCurrentVideoCapabilitiesMPEG() function can be declared as follows:

[0203] VkResult vkGetPhysicalDeviceCurrentVideoCapabilitiesMPEG(

[0204] VkPhysicalDevice physicalDevice,

[0205] VkVideoProfileKHR pVideoProfile,

[0206] VkCurrentVideoCapabilitiesMPEG pCapabilities);

[0207] In the above statement, physicalDevice refers to the physical device whose video decoding or encoding capabilities are to be queried; pVideoProfile is a pointer to a VkVideoProfileKHR structure, which has a chained video profile structure specific to codec operations; and pCapabilities is a pointer to a VkCurrentVideoCapabilitiesMPEG structure, which returns the capabilities.

[0208] The VkCurrentVideoCapabilitiesMPEG structure stores the information returned by calling the vkGetPhysicalDeviceCurrentVideoCapabilitiesMPEG() function defined above. The VkCurrentVideoCapabilitiesMPEG structure can be declared as follows:

[0209] typedef struct VkCurrentVideoCapabilitiesMPEG {

[0210] VkStructureTypesType;

[0211] void pNext;

[0212] uint32_tmaxInstances;

[0213] uint32_tbufferMemory;

[0214] uint32_tbitrate;

[0215] uint32_tmaxSamplesSecond;

[0216] VkPerformancePointMPEG maxPerformancePoint;

[0217] } VkCurrentVideoCapabilitiesMPEG;

[0218] In the above declaration, sType is the type of this structure; pNext is NULL or a pointer to a structure that extends this structure; pictureRate, height, width, and bitDepth can be the same as defined above.

[0219] The VkVideoSessionCreateInfoGroupingMPEG structure allows group identifiers to be appended to video decoding instances created via the VK Video API. This structure extends the VkVideoSessionCreateInfoKHR structure defined in the VK Video API. The VkVideoSessionCreateInfoGroupingMPEG structure can be declared as follows:

[0220] typedef struct VkVideoSessionCreateInfoGroupingMPEG {

[0221] VkStructureType sType;

[0222] const void pNext;

[0223] uint32_t groupId;

[0224] } VkVideoSessionCreateInfoGroupingMPEG;

[0225] In the above declaration, sType is the type of this structure; pNext is NULL or a pointer to a structure that extends this structure; and groupId can be the same as defined above.

[0226] The VkVideoSessionOutputParameterMPEG structure contains parameters that configure the output properties of a VK video session. VkVideoSessionOutputParameterMPEG can be declared as follows:

[0227] typedef struct VkVideoSessionOutputParameterMPEG {

[0228] VkStructureType sType;

[0229] const void pNext;

[0230] VkFlag partialOutput;

[0231] uint32_t subframeCount;

[0232] uint32_t pSubframeOutput;

[0233] VkFlag outputCrop;

[0234] VkExtent2D pOutputCropWindow;

[0235] uint32_t maxOfftimeJitter;

[0236] void pMetadataCallback;

[0237] } VkVideoSessionOutputParameterMPEG;

[0238] In the above declaration, sType is the type of this structure; pNext is NULL or a pointer to a structure that extends this structure; partialOutput, subframeCount, pSubframeOutput, outputCrop, pOutputCropWindow, maxOfftimeJitter, and pMetadataCallback can have the same semantics as discussed above.

[0239] Figure 6 This is a block diagram illustrating an example video decoder interface (VDI) system decoder model according to the present disclosure. In this example, the VDI system decoder model includes a media stream distribution interface 202 (corresponding to IVDI 202), an input formatting unit 204, a decoder buffer 230, a hardware video decoding engine 212 (which executes various instantiated video decoder instances 214A to 214N), a synthesis memory 232, and a synthesizer unit 234.

[0240] Figure 6The VDI system decoder model is an extension of the System Decoder Model (SDM) defined in ISO / IEC 14496-1. Compared to the SDM, the VDI SDM introduces a new interface in addition to the basic stream interface called Media Stream Distribution Interface 202. This interface is the input to Input Formatting Unit 204, also known as the input formatting function, which takes the so-called media stream as input. The output of Input Formatting Unit 204 is one or more basic streams that can be further passed to Video Decoder Instance 214.

[0241] The concepts of the specification include formatting, timing, and buffering models.

[0242] The media stream distribution interface 202 is a concept that models the exchange of media stream data between the distribution interface and the input formatting function (input formatting unit 204).

[0243] An input formatter (input formatting unit 204) takes one or more media streams as input and generates one or more elementary streams as output. When input formatting unit 204 generates a single elementary stream or multiple elementary streams, a single input formatter can be attached to several decoding buffers, such as the buffers of decoding buffer 230.

[0244] Each video decoder instance 214 can output decoded video data to a corresponding portion of the synthesis memory 232. The synthesizer unit 234 can retrieve the decoded video data from the corresponding portion of the synthesis memory 232 and synthesize an output video stream from the decoded video data.

[0245] The hardware video decoding engine 212 can generate one or more video decoder instances 214. The number of running instances may be an optimized choice for the platform when considering available resources such as computational load, energy consumption, and memory. However, the number of input media streams fed through IVDI 202 is determined by the application requirements for appropriately presenting the media experience. Therefore, one or more input media streams can be fed to the same video decoding instance within video decoding instance 214 via input formatting unit 204.

[0246] Input formatting unit 204 performs several operations on media streams and video objects. Input formatting unit 204 produces results in one or more elementary streams that conform to the profile, layer, level, or any other performance constraints (including the buffer fullness of a hypothetical reference decoder model) of the corresponding video decoder instance 214 that are expected to consume that elementary stream. Examples of such operations are defined atomically below, allowing for the combination of them to implement more complex operations, provided the final output includes a valid elementary stream. The actual implementation of these combined operations is beyond the scope of this document and can be optimized by implementers, as explained in more detail below.

[0247] A media stream may include one or more video objects, and video objects may be contained within a basic stream. Each video object in the basic stream provides information for enabling defined operations, such as means for determining the position and size of the video object in the frame, the number of luminance and chrominance samples in the video object, and the bit depth of the encoded / decoded frame of the video object.

[0248] Figure 7 This is a conceptual diagram illustrating example input and filtered output data received by a filter according to the technology of this disclosure. The input formatting unit 204 performs a filtering function to extract a video object from the input media stream and returns a basic stream including the selected video object as output.

[0249] The input formatting unit 204 can filter based on the video object identifier and / or the type of media data, such as media stream type, elementary stream type, access unit type, video object identifier type, and / or video object sample type. The filter can be based on… The filtering operation is defined in the form of “ElementaryStream output_stream filtering(MediaStream input_stream, VideoObjectIdentifier id)”, which indicates that the input includes a media stream (MDS) having at least one video object and an identifier (I) of the selected video object to be extracted, and the output includes a basic stream (ES) having a video object corresponding to each input. The signature of this filtering operation can be “ElementaryStream output_stream filtering(MediaStream input_stream, VideoObjectIdentifier id)”.

[0250] Therefore, to perform filtering, for each i-th access unit in the input media stream, the input formatting unit 204 can create a copy of that access unit. Then, the input formatting unit 204 can list the video object samples present in the copied access units. If a video object sample does not correspond to a video object identifier passed as input, it can be removed from the copied access unit. Finally, the input formatting unit 204 can append the copied access unit as a new access unit to the output base stream. In other words, the input formatting unit 204 can implement the filtering process based on the selected object identifier, i.e., first copying the original access unit and then removing the original access unit from unwanted objects. This way, the operation does not require creating and initializing empty access units, and the attributes of the input access units are passed to the access units of the output stream.

[0251] When the video object is a slice, the filtering function extracts the slice from each codec frame of the input media stream and passes the extracted slice in the output elementary stream. Figure 7In the example, input formatting unit 204 receives a media data stream including a frame such as frame 240. Frame 240 includes slice 242. Figure 7 As shown, the input formatting unit 204 can extract slice 242 and output slice 242 in the form of output 244. During this operation, the SPS, PPS, and slice header may need to be updated according to the requirements of the corresponding video codec specification to correctly signal the size of the output primary stream video, information about the slice and tile layout, and video object identifiers (e.g., slice addresses).

[0252] Figures 8A to 8C This is a conceptual diagram illustrating an example input video of equal width and a combined output video received by an insertion function according to the technology of this disclosure. The input formatting unit 204 can also perform an insertion operation. The insertion operation can be defined as " The input consists of two media streams, each containing at least one video object, and the output can be a single media stream with as many video objects as the sum of the video objects in the two media streams. The signature of the insert operation can be "MediaStream output_stream inserting(MediaStreaminput_stream_1, MediaStream input_stream_2)".

[0253] To perform the insertion, for each i-th access unit in the first and second input media streams, the input formatting unit 204 can copy the i-th access unit of the second input media stream. Then, the input formatting unit 204 can list the video object samples present in the i-th access unit from the first input media stream. The input formatting unit 204 can add each video object sample to the copied access unit. Finally, the input formatting unit 204 can append the copied access unit as a new access unit to the output media stream. The insertion can stop once one of the two input media streams ends. The insertion operation can be defined as inserting a video object from the first input media stream into the second input media stream. In this way, the insertion operation does not require creating and initializing empty access units, but the attributes of the access units of the second input media stream are passed to the access units of the output media stream.

[0254] The insertion function brings a video object from the first input media stream into the second input media stream and outputs the resulting output media stream, which includes video objects from both the first and second input media streams.

[0255] In the case where the video object is a slice, such as Figure 8A 250 slices and Figure 8BSlice 252, where the width or height of the input media stream's codec frame is equal, to maintain the rectangular shape of the output media stream's video. When the widths of the two input videos are equal, such as... Figure 8A and Figure 8B As shown, the two videos are vertically stitched together to form... Figure 8C Screen 254. During this operation, it may be necessary to update the SPS, PPS, and slice headers to correctly signal the video size of the output media stream, information about slice and tile layout, and video object identifiers (such as slice addresses).

[0256] Figures 9A to 9C This is a conceptual diagram illustrating an example input video and a combined output video of equal height received by an insertion function according to the technology of this disclosure. If the two videos have equal heights, such as... Figure 9A slice 260 and Figure 9B As shown in slice 262, these two videos can be horizontally stitched together to form... Figure 9C Screen 264. During this operation, it may be necessary to update the SPS, PPS, and slice headers to correctly signal the video size of the output media stream, information about slice and tile layouts, and video object identifiers (such as slice addresses).

[0257] Figure 10A and Figure 10B This is a conceptual diagram illustrating an example of input video and corresponding output video data received by an additional function according to the technology of this disclosure. The input formatting unit 204 can perform additional operations, which can be defined as " The append operation can take a media stream with at least two video objects as input and form a media stream with two video objects that are spatially adjacent to each other as output. The signature of the append operation can be "MediaStream output_stream appending(MediaStream input_stream, VideoObjectIdentifier object_id_1, VideoObjectIdentifier object_id_2)".

[0258] To perform the additional operation, the input formatting unit 204 can copy the i-th access unit for each i-th access unit in the input media stream. Then, the input formatting unit 204 can position the video object sample belonging to the video object identified by the second video object identifier within the copied access unit to the right of the video object sample belonging to the object identified by the first video object identifier. This positioning is accomplished by aligning the top boundaries of the two video object samples. Finally, the input formatting unit 204 can append the copied access unit as a new access unit to the output media stream.

[0259] The additional feature positions the first video object to the right of the second video object in the decoded frame of the output media stream, which contains both video objects. The output media stream is media that contains at least the first and second video objects positioned side-by-side.

[0260] When the video object is a slice, two video object slices in the input media stream have the same height, such as... Figure 10A and Figure 10B Examples of slices 272A and 272B in frames 270A and 270B are shown. In frame 270B of the output media stream, slice 272B on the right is moved next to slice 272A on the left. Slice 274, located between slices 27A and 272B in frame 270A, is moved to the right of slices 272A and 272B, as shown. Figure 10B As shown in frame 270B. During this operation, it may be necessary to update the slice header to properly signal changes to the video object identifier (e.g., slice address).

[0261] Figure 11A and Figure 11B This is a conceptual diagram illustrating an example of input video and corresponding output video data received by a stacking function according to the technology of this disclosure. The input formatting unit 204 can perform operations defined as " The stacking function takes a media stream with at least two video objects as input and outputs a media stream with two spatially adjacent video objects. The signature of the stacking function can be "MediaStream output_stream stacking(MediaStream input_stream, VideoObjectIdentifier object_id_1, VideoObjectIdentifier object_id_2)".

[0262] To perform the stacking function, the input formatting unit 204 can copy the i-th access unit for each i-th access unit in the input media stream. Then, the input formatting unit 204 can position the video object sample belonging to the video object identified by the second video object identifier below the video object sample belonging to the object identified by the first video object identifier within the copied access unit. This positioning is accomplished by aligning the left edges of the two video object samples. Finally, the input formatting unit 204 can append the copied access unit as a new access unit to the output media stream.

[0263] The stacking function positions the first video object above the second video object in the decoded frame of the media stream, which contains both the first and second video objects. The output media stream must contain at least the first and second video objects positioned vertically adjacent to each other.

[0264] When the video object is a slice, two video object slices in the input media stream have the same width, such as... Figure 11A and Figure 11B Examples of slices 282A and 282B from frames 280A and 280B are shown. (As in the output media stream) Figure 11B As shown in frame 280B, slice 282B on the right is moved below slice 282A on the left. Figure 11A In frame 280A, slices 284 and 286 below slice 282A are sequentially moved to the right of slice 282B, as shown below. Figure 11B As shown in frame 280B. During this operation, it may be necessary to update the slice header to properly signal changes to the video object identifier (e.g., slice address).

[0265] The techniques disclosed herein can be used in conjunction with video decoder instances conforming to ITU-T H.266 / Various Video Codec (VVC). VVC is disclosed according to ISO / IEC 23090-3. Table 2 below provides the binding between VDI concepts and the concepts defined in VVC:

[0266] Table 2

[0267]

[0268] The VVC primary stream can be a compatible video stream according to ISO / IEC 23090-3, and the Independent Layer Information (SEI) message can be defined according to the following definition.

[0269] VVC media streams used as instantiations of media streams can follow these rules:

[0270] • There should be at least one VPS in the media stream, and the parameters in each VPS should be as follows:

[0271] The o flag vps_all_independent_layers_flag should be set to 1.

[0272] • For all codec slices, the value of sh_picture_header_in_slice_header_flag should be equal to 0.

[0273] • If it exists, the value of vps_num_output_layer_sets_minus2 should be equal to 0.

[0274] In addition to the rules discussed above, the VVC input media stream passed as a parameter to the filter function can also follow these rules:

[0275] • There should be a VCL NAL unit with at least two different nuh_layer_id values.

[0276] • One of at least two distinct nuh_layer_id values ​​should be equal to the object identifier passed as an argument to the filter function.

[0277] In addition to the rules discussed above, the VVC elementary stream generated as the output of the filter function can also follow these rules:

[0278] • The number of accessed units in the output elementary stream should be equal to the number of accessed units in the input elementary stream.

[0279] • The number of VCL NAL cells in the output primitive stream is equal to the number of VCL NAL cells whose nuh_layer_id is equal to the object identifier passed as an argument to the function.

[0280] • For each VCL NAL cell in the output elementary stream, there should be a VCL NAL cell in the input elementary stream that is exactly the same bit.

[0281] • All NAL cells in the output primitive stream should have the same nuh_layer_id value, and this nuh_layer_id value should be equal to the object identifier passed as an argument to the function.

[0282] In addition to the rules discussed above, the two VVC input media streams passed as parameters to the insertion function can also follow these rules:

[0283] • The nuh_layer_id value of each NAL unit in the first input media stream should be different from any nuh_layer_id value present in the second input media stream.

[0284] • If an SPS or PPS in the first input media stream has the same identifier as an SPS or PPS in the second input media stream, then the two SPS or two PPS will have the same payload.

[0285] In addition to the rules discussed above, the VVC media streams generated as the output of the insertion function can also follow these rules:

[0286] • The number of VCL NAL units in the output media stream is equal to the sum of the number of VCL NAL units in the two input media streams.

[0287] • For each VCL NAL cell in the output media stream, there should be a VCL NAL cell with exactly the same bits in one of the two input media streams.

[0288] In addition to the rules discussed above, VVC input media streams passed as parameters to additional functions can also follow these rules:

[0289] • There should be a VCL NAL unit with at least two different nuh_layer_id values.

[0290] • Two of the at least two distinct nuh_layer_id values ​​should be equal to the two object identifiers passed as arguments to the additional function.

[0291] In addition to the rules discussed above, the VVC media streams generated as the output of additional functions can also follow these rules:

[0292] • The number of VCL NAL units in the output media stream is equal to the number of VCL NAL units in the input media stream.

[0293] • For each VCL NAL cell in the output media stream, there should be a VCL NAL cell in the input media stream that is exactly the same bit.

[0294] • There should be an independent layer information SEI message whose nuh_layer_id is equal to the first video object identifier.

[0295] • There should be an independent layer information SEI message whose nuh_layer_id is equal to the second video object identifier.

[0296] • The independent layer information SEI message whose nuh_layer_id is equal to the first video object identifier should have its boundary_identifier_east value equal to the boundary_identifier_west value of the independent layer information SEI message whose nuh_layer_id is equal to the second video object identifier.

[0297] In addition to the rules discussed above, the VVC input media streams passed as arguments to the stacking function can also follow these rules:

[0298] • There should be a VCL NAL unit with at least two different nuh_layer_id values.

[0299] • Two of the at least two distinct nuh_layer_id values ​​should be equal to the two object identifiers passed as arguments to the additional function.

[0300] In addition to the rules discussed above, the VVC media streams generated as the output of the stacking function can also follow these rules:

[0301] • The number of VCL NAL units in the output media stream is equal to the number of VCL NAL units in the input media stream.

[0302] • For each VCL NAL cell in the output media stream, there should be a VCL NAL cell in the input media stream that is exactly the same bit.

[0303] • There should be an independent layer information SEI message whose nuh_layer_id is equal to the first video object identifier.

[0304] • There should be an independent layer information SEI message whose nuh_layer_id is equal to the second video object identifier.

[0305] • The independent layer information SEI message whose nuh_layer_id is equal to the first video object identifier should have its boundary_identifier_south value equal to the boundary_identifier_north value of the independent layer information SEI message whose nuh_layer_id is equal to the second video object identifier.

[0306] Alternatively or concurrently, the techniques disclosed herein can be used with instances of Basic Video Codec (EVC) compatible video decoders. EVC is disclosed according to ISO / IEC 23094-1. Table 3 below provides examples of the binding of the VDI concepts of this disclosure with the concepts defined in the EVC specification.

[0307] Table 3

[0308]

[0309] The EVC media streams used as instantiations for media streams, as discussed above, can conform to the following rules:

[0310] • At least two independent, decodable slices should be defined, each containing a different minimum tile ID.

[0311] The EVC input media stream passed as a parameter to the filter function can conform to the following rules:

[0312] • One of the minimum values ​​of the tile IDs in each slice should be equal to the object identifier passed as a parameter to the filter function.

[0313] The EVC basic flow generated as the output of the filter function can conform to the following rules:

[0314] • The number of accessed units in the output primary stream should be equal to the number of accessed units in the input media stream.

[0315] • The number of VCL NAL cells in the output primitive stream is equal to the minimum value of the tile ID in the slice, which is equal to the number of VCL NAL cells of the object identifier passed as a function parameter.

[0316] • For each VCL NAL cell in the output elementary stream, there should be a VCL NAL cell in the input media stream that is exactly the same bit.

[0317] • All NAL cells in the output basic stream should have the same minimum value of the tile ID in the slice value, and that value should be equal to the object identifier passed as an argument to the function.

[0318] The two EVC input media streams passed as parameters to the insertion function can conform to the following rules:

[0319] • At least one of the values ​​of pic_width_in_luma_samples or pic_height_in_luma_samples for the two media streams should be the same.

[0320] • If the values ​​of pic_width_in_luma_samples are the same, then the values ​​of num_tile_columns_minus1 should be the same.

[0321] • If the values ​​of pic_height_in_luma_samples are the same, then the values ​​of num_tiles_row_minus1 should be the same.

[0322] • If an SPS or PPS in the first input media stream has the same identifier as an SPS or PPS in the second input media stream, then the two SPS or two PPS will have the same payload.

[0323] The EVC media stream generated as the output of the insertion function can conform to the following rules:

[0324] • The number of VCL NAL units in the output media stream is equal to the sum of the number of VCL NAL units in the two input media streams.

[0325] • For each VCL NAL cell in the output media stream, there should be a VCL NAL cell in one of the two input media streams that is exactly the same bit.

[0326] EVC input media streams passed as parameters to additional functions can conform to the following rules:

[0327] • At least two of the minimum IDs of the tiles in each slice should be equal to the two object identifiers passed as arguments to the additional function.

[0328] • Because the parameters of the additional functions are the same, the slice height, the number of tile rows included in the slice when using uniform tile spacing, and the minimum value of the tile ID in each slice are the same.

[0329] The EVC media stream generated as the output of the additional function can conform to the following rules:

[0330] • The number of VCL NAL units in the output media stream is equal to the number of VCL NAL units in the input media stream.

[0331] • For each VCL NAL cell in the output media stream, there should be a VCL NAL cell in the input media stream that is exactly the same bit.

[0332] The EVC input media stream passed as a parameter to the stacking function can conform to the following rules:

[0333] • At least two of the minimum IDs of the tiles in each slice should be equal to the two object identifiers passed as arguments to the additional function.

[0334] • Because the parameters of the additional functions are the same, the slice width, the number of tile columns included in the slice when using uniform tile spacing, and the minimum value of the tile ID in each slice are the same.

[0335] The EVC media stream generated as the output of the stacking function can conform to the following rules:

[0336] • The number of VCL NAL units in the output media stream is equal to the number of VCL NAL units in the input media stream.

[0337] • For each VCL NAL cell in the output media stream, there should be a VCL NAL cell in the input media stream that is exactly the same bit.

[0338] The control interface to the video decoding engine of this disclosure can be specified using the IDL syntax specified in ISO / IEC 29516. Alternatively, the control interface to the video decoding engine can be defined for the OpenMax IL interface.

[0339] Figure 12 This is a conceptual diagram illustrating an example location of the boundary of a video data layer according to the technology of this disclosure. Figure 12 In the example, among other information, the Supplemental Enhancement Information (SEI) message can specify data such as whether layer i 290 is coupled to another layer at the northern boundary 292, eastern boundary 294, southern boundary 296 and / or western boundary 298 for a given layer i 290.

[0340] The following defines an example of a generic packet used to carry SEI messages, as defined in this document. Some VDI SEI messages may only be applicable to certain video codec specifications, such as HEVC, VVC, or EVC. VDI SEI packets can be registered as SEI payloads in ISO / IEC 23090-3.

[0341] Table 4 below defines the example syntax for VDI SEI packets:

[0342] Table 4

[0343]

[0344] The semantics of vdi_sub_type can be expressed as follows: "vdi_sub_type indicates the type of payload carried in a VDI SEI packet".

[0345] Table 5 below defines the example syntax for the Layer Independent Information (SEI) message:

[0346] Table 5

[0347]

[0348] The semantics of the Independent Layer Information (SEI) message can be defined as follows:

[0349] Layer Independent Information (SEI) messages provide spatial alignment of different independent layers present in a bitstream by using matched boundary identifiers to express the relative positioning between these layers. In a multi-layer bitstream, a given boundary identifier should appear at most twice, thus creating a pair of matched boundary identifiers. In other words, each layer boundary has at most one neighbor.

[0350] The SEI message can be extracted by the VDE and used for output formatting to correctly place the decoded frames from each layer into the final output frame. In this case, the decoder instance can ignore the SEI message (if it exists). Alternatively, the decoder instance can be able to decode multi-layer bitstreams and parse the SEI message, in which case output formatting is not required.

[0351] The values ​​of boundary_identifier_north_present_flag, boundary_identifier_east_present_flag, boundary_identifier_south_present_flag, and boundary_identifier_west_present_flag being equal to 1 indicate that the SEI message contains boundary identifiers for the north, east, south, and / or west boundaries, respectively.

[0352] `boundary_identifier_north`, `boundary_identifier_east`, `boundary_identifier_south`, and `boundary_identifier_west` specify the boundary identifiers at the north, east, south, and west boundaries of the decoded screen of the associated layer, respectively. The associated layer is the layer whose `nuh_layer_id` is equal to the `nuh_layer_id` of the SEI message. If none of these exist, boundary identifiers for the north, east, south, and west boundaries of the decoded screen of the associated layer are not defined.

[0353] As mentioned above, in Figure 12 In the example, layer i screen 290 represents the screen of layer i. The Independent Layer Information (SEI) message can specify whether the northern boundary 292 should be attached to the boundary of another layer, whether the eastern boundary 294 should be attached to the boundary of another layer, whether the southern boundary 296 should be attached to the boundary of another layer, and / or whether the western boundary 298 should be attached to the boundary of another layer.

[0354] For layers i and j, if the pairwise values ​​of `boundary_identifier_north` for layer i and `boundary_identifier_south` for layer j are equal, then the decoded frames of layer i and j will be placed adjacent to each other in the synthesized output frame, sharing a common boundary at the north / south boundary. For layers i and j, if the pairwise values ​​of `boundary_identifier_east` for layer i and `boundary_identifier_west` for layer j are equal, then the decoded frames of layer i and j will be adjacent to each other in the synthesized output frame, sharing a common boundary at the east / west boundary.

[0355] Two decoded frames with adjacent north / south boundaries can be aligned on their western boundary in the final output frame. Two decoded frames with adjacent east / west boundaries can be aligned on their northern boundary in the final output frame.

[0356] The independent layer information SEI messages existing in the layers of the Output Layer Set (OLS) can collectively describe a 4-connected graph, and each layer in the OLS can be connected to this graph.

[0357] The example process for generating the final output screen is informative. The expected operations performed to generate the final output screen based on the decoded screens from each layer of the selected OLS are described below:

[0358] • For each access unit:

[0359] If the VPS exists, resolve the VPS and store the layer list in the bitstream.

[0360] For each existing PPS, determine the brightness sample size of the corresponding layer.

[0361] For each existing independent layer information (SEI) message, parse the payload and store the boundary identifier of the corresponding layer.

[0362] If any of the VPS, PPS, or Independent Layer Information (SEI) messages are present in the current access unit, calculate the horizontal position XPos and vertical position YPos of the top-left corner of each cropped decoded frame in the final output frame. The example steps for calculating XPos and YPos are as follows:

[0363] For each layer:

[0364] • Parse the list of cropped image sizes and boundary identifiers.

[0365] • Determine the adjacent layers in the north, east, south, and west.

[0366] Each layer is placed in a grid, and each grid cell corresponds to the north, east, south, and west values ​​of each layer. These north, east, south, and west values ​​are matched with their corresponding adjacent values, which are south, west, north, and east, respectively.

[0367] The value XPos for each layer corresponds to the sum of the widths of all layers in the same row of the grid to the left of the current layer. The value YPos for each layer corresponds to the sum of the heights of all layers in the same column of the grid above the current layer.

[0368] o initializes a screen buffer with a size of FinalWidth and a height of FinalHeight for the final output screen, where FinalWidth and FinalHeight are the width and height of the final output screen when all layers are connected according to the defined graphics.

[0369] o For each frame unit:

[0370] Decode the encoded / decoded video.

[0371] oIf the image is ready to be output:

[0372] For each layer in the selected OLS.

[0373] • Apply consistent window clipping to the decoded screen of the current layer.

[0374] • Retrieve XPos and YPos to determine the position of the current layer in the final output image of the luminance samples.

[0375] • Copy the cropped decoded screen from the final output screen buffer at the position (XPos, YPos) corresponding to the top left corner of the cropped decoded screen.

[0376] If, at the end of the process, the combination of all decoded images does not provide decoded sample values ​​for all samples of the final output image, the implementation can determine values ​​for these unused samples.

[0377] Figure 13This is a conceptual diagram illustrating an example of attaching two layers of video data to each connection diagram according to the technology of this disclosure. Specifically, layer 0 screen 306 and layer 1 screen 308 can be combined according to connection diagram 300 to form output screen 310. As shown in this example, connection diagram 300 includes a left screen 302 corresponding to layer 0 and a right screen 304 corresponding to layer 1. Left screen 302 includes a value "1" at its eastern boundary, and right screen 304 includes a value "1" at its western boundary. Therefore, VDE 200 can determine that the layer 0 screen will be attached to the left side of the layer 1 screen from the right.

[0378] Table 6 shows examples of attributes for layer 0 and layer 1:

[0379] Table 6

[0380]

[0381] Table 7 shows examples of the attributes of the final output image:

[0382] Table 7

[0383]

[0384] Figure 14 This is a conceptual diagram illustrating an example of attaching three layers of video data to each connection diagram according to the technology of this disclosure. In this example, connection diagram 320 includes layer 0 screen 322, layer 1 screen 324, and layer 2 screen 326. Layer 0 screen 322 includes a value "1" on the eastern boundary and a value "2" on the southern boundary, while layer 1 screen 324 includes a value "1" on the western boundary, and layer 2 screen 326 includes a value "2" on the northern boundary. Therefore, according to connection diagram 320 for input screen 328 of layer 0, input screen 330 of layer 1, and input screen 332 of layer 2, output screen 334 is formed by attaching input screen 330 to the right of input screen 328 and attaching input screen 332 below input screen 328, such that input screens 328 and 330 share a top boundary, and input screens 328 and 332 share a left boundary.

[0385] Table 8 shows examples of attributes for layers 0 to 2:

[0386] Table 8

[0387]

[0388] Based on this configuration, Table 9 presents the attributes of the final output screen:

[0389] Table 9

[0390]

[0391] Figure 15This is a conceptual diagram illustrating an example of a video decoding engine 340 employing application-based video decoder engine control to mosaic 2×2 video objects according to the technology of this disclosure. Specifically, the video decoding engine (VDE) 340 receives media data from media streams 342A to 342D via a media stream interface. The VDE 340 receives configuration data 344 from the media application. Based on the configuration data 344, the VDE 340 extracts video objects from the corresponding frames of the media streams 342A to 342D to form an output frame 346 in a 2×2 mosaic format.

[0392] The various operations discussed above, along with the associated input and output constraints, provide building blocks for various implementations of the input formatting functionality. The way an implementation converts a media stream into a basic stream based on the requested decoding sequence configuration is informative, and optimization is left to the implementer as long as the output basic stream meets the requirements of the basic stream interface.

[0393] exist Figure 15 In the example, the goal is to take four video objects as input to four different media streams 342A to 342D and obtain a decoded sequence as the output of the VDE 340, which contains four decoded video objects arranged in a 2×2 grid in each frame of the decoded sequence. It is assumed that the media application can communicate with the VDE 340 via an external communication channel to indicate how the media streams will be arranged in the decoded sequence. For illustrative purposes, it is assumed that the four video objects have the same video resolution. Figure 15 The input conditions at the media stream interface, the input conditions at the VDE 340, and the expected conditions at the output conditions of the VDE 340 are described.

[0394] The output of the input formatting function can be a basic stream interface. That is, the input formatting function (e.g., input formatting unit 204) can output a data stream as a basic stream. Whether it should be one basic stream or multiple basic streams depends on the implementation and platform. For this example, two examples are described. The first example is that each media stream has one video decoder instance (e.g., ... Figure 16 As shown), the second example is for four media streams with a single decoder instance (such as...). Figure 17 (As shown).

[0395] Figure 16This is a conceptual diagram illustrating an example of a video decoder engine executing four decoder engine instances to mosaic a 2×2 video object according to the technology of this disclosure. In this example, the VDE 350 includes video decoder instances 352A to 352D and an output formatting unit 356. Each of the video decoder instances 352A to 352D receives a video object as input from a corresponding media stream 358A to 358D and decodes the video object to form a corresponding decoded video object 354A to 354D. The output formatting unit 356 assembles the decoded video objects 354A to 354D into an output frame 360, representing a 2×2 mosaic of the decoded video objects 354A to 354D.

[0396] In the case where each media stream has a video decoder, such as in Figure 16 In the example, the input formatting function is actually a recognition operation; that is, each media stream is passed to a decoder instance. This also means that each media stream is also a basic stream in this case, which is allowed by the definition of a media stream. The VDE 350 then runs these four decoder instances in parallel, collecting each output frame and assembling the output frames into a format as shown. Figure 16 The diagram shows a 2×2 arrangement of decoded frames juxtaposed in each time frame.

[0397] Figure 17 This is a conceptual diagram illustrating an example of a video decoder executing a decoder engine instance to mosaic 2×2 video objects according to the technology of this disclosure. In this example, the VDE 370 includes an input formatting unit 372 and a single decoder instance 374. The input formatting unit 372 receives video objects from each of media streams 376A to 376D and assembles the received video objects into a single input encoded frame using the various operations discussed above. This input encoded frame constitutes part of a base stream. The decoder instance 374 then decodes the encoded frame to form a decoded picture 378, including video objects from each of media streams 376 to 376D.

[0398] In the case of a video decoder used for four media streams, the input formatting unit 372 creates a single elementary stream from the four input media streams. The VDE 370 then runs a regular pipeline with a single decoder instance 374, and outputs a decoded picture 378 from the decoder instance without further processing before being output by the VDE 370.

[0399] Figure 18 This is a conceptual diagram illustrating an example implementation of the technology according to this disclosure for formatting four media streams in different steps. Regarding... Figure 17 The actual implementation of the input formatting function in the discussed examples can take different forms. Figure 18An example of this type is depicted. In this example, the operations include insertion, appending, and stacking operations to form input video data. In this example, the implementation will be as follows: Figure 18 The diagram shows those operations cascading together.

[0400] Specifically, in this example, media streams 380A to 380D provide video objects. An input formatting unit, such as input formatting unit 372, can perform an insertion operation 382A on the video objects from media streams 380A and 380B to form a first intermediate video object. Input formatting unit 372 can then perform an insertion operation 382B on the first intermediate video object and a video object from media stream 380C to form a second intermediate video object. Input formatting unit 372 can then perform an insertion operation 382C on the second intermediate video object and a video object from media stream 380D to form a third intermediate video object.

[0401] The third intermediate video object will contain each of the various video objects, but not necessarily in a correctly assembled form. Therefore, the input formatting unit 372 can first perform additional (1,2) operations 384 to appropriately combine the video objects from media streams 380A and 380B, as shown in the intermediate video object 390. The input formatting unit 372 can then perform a stacking (1,3) operation 386 to appropriately add the video objects from media stream 380C, as shown in the intermediate video object 392. Finally, the input formatting unit 372 can perform additional (3,4) operations 388 to appropriately add the video objects from media stream 380D, as shown in the final video object 394.

[0402] Figure 19 This is a conceptual diagram illustrating an example implementation of the technology according to this disclosure for formatting four media streams in two steps. Figure 19 Depicting Figure 18 Alternative examples to the example shown, to achieve the same Figure 17 Similar results can be obtained from the example. In this example, the implementation could support an insertion function that receives several media streams as input and generates a base stream in a single function call. Furthermore, this implementation could also have a proprietary function called "arrangement" so that an arrangement can be built into the base stream in a single function call.

[0403] In particular, Figure 19In the example, input formatting unit 372 can receive video objects from each of media streams 400A to 400D. Input formatting unit 372 can first perform an insertion operation 402 to insert the video objects into the combined video object 406. Input formatting unit 372 can then perform an arrangement operation 404 to appropriately arrange the video objects to form the final video object 408.

[0404] In yet another example, there could be only one function to combine media streams, arrange video objects from the media streams, and output the arranged video objects as part of the base stream.

[0405] Figure 20 This is a conceptual diagram illustrating an example of a video decoding engine according to the present disclosure using control information based on Supplemental Enhancement Information (SEI) to mosaic 2×2 video objects. In this example, VDE 412 receives video objects from media streams 410A to 410D, and also includes SEI messages. The SEI messages may include data representing, for example, the number of video encoding instances to be instantiated and / or indicating how to arrange the video objects from media streams 410A to 410D. VDE 412 can decode the video objects and arrange them according to the SEI messages to ultimately form an output screen 414.

[0406] This example shows the same expected output as discussed above. Figure 20 The difference is that there is no hypothetical communication between VDE 412 and the media application regarding how video objects are arranged together in the decoding sequence. Instead, this arrangement configuration is given by a separate Layer Information (SEI) message contained in each media stream. This information can be used to perform additional operations.

[0407] Figure 21 This is a conceptual diagram illustrating an example of the use of connection components and buffers according to the technology disclosed herein. An OpenMAX IL client can call OMX_Init() as its first call to OpenMAX IL. OMX_Init() initializes the OMX core engine before any use of it. Once complete, the engine can be released by calling OMX_Deinit().

[0408] OMX uses the following format to define a naming convention for component names: OMX.<vendor_name> .<vendor_specified_convention> Once the instance is no longer needed, call OMX_FreeHandle() to release all associated resources.

[0409] This function can be called multiple times using the same component name to create multiple instances of the component.

[0410] OMX_GetHandle() is used to locate a requested component by its provided name. If the requested component is available, the OMX core engine calls the component's methods to populate the component handle and set a callback. OpenMAX AL is the interface used by applications to perform media playback and processing. However, the OpenMAX IL interface provides direct access to video decoder components and their capabilities. Therefore, this disclosure describes the OpenMAX IL interface with the aim of providing additional features that enable flexible multi-video decoder platforms and their interfaces to be used in six degrees of freedom (6DoF) applications.

[0411] Tunnels can be used to connect the input and output ports of two connected components. `OMX_SetupTunnel()` is used to create a tunnel connecting the output port of one component to the input port of another. When creating a tunnel, the components can negotiate compatible input / output formats for the connected ports. When no longer needed, the application calls `OMX_TeardownTunnel()` to tear down the tunnel.

[0412] exist Figure 21 In the example, component 424 includes an output port, component 426 includes both an input port and an output port, and component 428 includes an input port. Buffer port 420 corresponds to the output ports of components 424 and 426, while buffer port 422 corresponds to the input ports of components 426 and 428.

[0413] Components communicate with each other and with the application via buffer swapping. For this purpose, OMX_AllocateBuffer(), OMX_UseBuffer(), OMX_FillThisBuffer(), OMX_EmptyThisBuffer(), and OMX_FreeBuffer() are defined. These function calls are non-blocking. A component requests the preceding component to fill its input buffer by calling the OMX_FillThisBuffer() method, and requests the following component to retrieve the contents of its output port buffer by calling the OMX_EmptyThisBuffer() function. Each tunnel can use only one buffer, and one of the two components acts as the provider of that buffer.

[0414] OMX_SetConfig() is used to configure application components. The application passes a structure containing configuration parameters to the component. These configuration parameters are published by each component and are component-specific.

[0415] Figure 22 This is a conceptual diagram illustrating example port configurations of various buffers according to the technology disclosed herein. Figure 22 Example buffers 440A to 440N are shown. The port configuration defines the format of the data to be transmitted on the component port. The buffer header contains a reference to the buffer, pBuffer, the offset within the buffer, nOffset, and the length of the buffer, nFilledLen. Multiple buffers can be used to pass data, allowing for more flexible communication between components; that is, multiple frames can be exchanged at once.

[0416] There are no requirements for frame alignment at the start of the buffer. The application or preceding components provide frame alignment information as part of the buffer header using the OMX_BUFFERFLAG_ENDOFFRAME flag. OMX_BUFFERFLAG_ENDOFSUBFRAME can also be used to signal subframe boundaries to identify NAL cell boundaries.

[0417] The buffer header also provides a timestamp for each buffer. `nTimestamp` corresponds to the rendering timestamp of the first media sample starting from the current buffer. If the current buffer contains multiple samples, the start timestamp of the next sample is inferred from `nTimestamp` and the sample duration. This information can then be propagated through the pipeline and can be passed to the application via the output buffer.

[0418] The buffer header structure can be:

[0419] typedef struct OMX_BUFFERHEADERTYPE

[0420] {

[0421] OMX_U32 nSize;

[0422] OMX_VERSIONTYPE nVersion;

[0423] OMX_U8 pBuffer;

[0424] OMX_U32 nAllocLen;

[0425] OMX_U32 nFilledLen;

[0426] OMX_U32 nOffset;

[0427] OMX_PTR pAppPrivate;

[0428] OMX_PTR pPlatformPrivate;

[0429] OMX_PTR pInputPortPrivate;

[0430] OMX_PTR pOutputPortPrivate;

[0431] OMX_HANDLETYPE hMarkTargetComponent;

[0432] OMX_PTR pMarkData;

[0433] OMX_U32 nTickCount;

[0434] OMX_TICKS nTimeStamp;

[0435] OMX_U32 nFlags;

[0436] OMX_U32 nOutputPortIndex;

[0437] OMX_U32 nInputPortIndex;

[0438] } OMX_BUFFERHEADERTYPE;

[0439] The list of buffer flags can be:

[0440] #define OMX_BUFFERFLAG_EOS 0x00000001

[0441] #define OMX_BUFFERFLAG_STARTTIME 0x00000002

[0442] #define OMX_BUFFERFLAG_DECODEONLY 0x00000004

[0443] #define OMX_BUFFERFLAG_DATACORRUPT 0x00000008

[0444] #define OMX_BUFFERFLAG_ENDOFFRAME 0x00000010

[0445] #define OMX_BUFFERFLAG_SYNCFRAME 0x00000020

[0446] #define OMX_BUFFERFLAG_EXTRADATA 0x00000040

[0447] #define OMX_BUFFERFLAG_CODECCONFIG 0x00000080

[0448] #define OMX_BUFFERFLAG_TIMESTAMPINVALID 0x00000100

[0449] #define OMX_BUFFERFLAG_READONLY 0x00000200

[0450] #define OMX_BUFFERFLAG_ENDOFSUBFRAME 0x00000400

[0451] #define OMX_BUFFERFLAG_SKIPFRAME 0x00000800

[0452] OpenMAX IL introduces the possibility of using EGL images as output buffers. EGL images are designed for sharing data between rendering-based EGL interfaces such as OpenGL and OpenMAX components. Instead of traditional buffers, components implement OMX_UseEGLImage() to link output to EGL images.

[0453] Figure 23 This is a conceptual diagram illustrating an example of a Media Source Extension (MSE) media interface according to the technology disclosed herein. MSE is a set of extensions to the Media Source attributes of HTML5 video and audio elements. It enables flexible control of the media stream via JavaScript code using the definition of a MediaSource object. A MediaSource object can have one or more SourceBuffer objects. Applications append media segments to SourceBuffer objects. A SourceBuffer may have multiple tracks, which are decoded and played separately. Figure 23 This describes a sample setup of MSE, including the interface between the MSE API and HTML5 media elements.

[0454] Figure 23The example illustrates a media source (MS) 450 including source buffers (SB) 452A, 452B, and 452C. Source buffer 452A provides data to track buffer (TB) 454A, which includes both video and audio data. The video data is decoded by video decoder (VDC) 456A, while two sets of audio data are decoded by audio decoders (ADC) 458A and 458B. Source buffer 452B provides data to track buffer 454B, which includes video data decoded by video decoder 456B. Source buffer 452C provides data to track buffer 454C, which includes audio data decoded by audio decoder 458C. Audio data selector 460 selects and / or combines decoded audio data to output audio data to audio device (ADV) 464, while video data selector 462 selects and / or combines decoded video data to send video tags and decoded video data to display area 466.

[0455] Table 10 shows the possible mappings from VDI functions to the MSE API:

[0456] Table 10

[0457]

[0458] Additionally, extensions to HTML5 video elements can be used to allow data output to buffers, such as WebGL buffers created using the gl.createBuffer() function. Extensions to input byte stream formats can also be used to add support for raw media data, such as AVC raw media streams.

[0459] Figure 24 This is a block diagram illustrating an example video decoding hypothetical reference decoder. In this example, the hypothetical reference decoder includes a hypothetical stream scheduler 500, a codec picture buffer (CPB) 502, a decoding unit 504, a decoded picture buffer (DPB) 506, and an output cropping unit 508. The video decoding interface can conform to... Figure 24 This is an example hypothetical reference decoder for video decoding. In other words, the video decoding interface can include something similar to... Figure 24 The components shown and described.

[0460] A video decoding interface may have the following parameters describing its characteristics (e.g., decoding capabilities): profile (e.g., subsampling and bit depth support, such as 4:2:0 color format and 10 bits for bit depth); aggregation level capability (e.g., megabits per second (MB / s) bit rate, such as "Level 6.1"); number of instances (e.g., 16 instances); and supported codecs (e.g., ITU-T H.264 / Advanced Video Codec (AVC) and ITU-T H.265 / High-Efficiency Video Codec (HEVC)). Therefore, the content may include signaling data informing each bitstream of the maximum decoding / rendering capability required (e.g., using profile / level / layer signaling). Codec bitstream groups may be associated with groups that share a common maximum capability.

[0461] The video decoding interface can perform various functions. These functions may include decoding compliant video bitstreams and storing encoded video data in a common codec picture buffer for the decoder group. Associated access units across bitstreams provided to the codec picture buffer can be synchronously provided to the decoder group. Supplemental Enhancement Information (SEI) messages associated with the access units can be synchronously provided at the decoder output.

[0462] An access unit can be referred to as "access unit n", where n is a unique number identifying the access unit. For each subsequent access unit, N can be incremented by 1 in the decoding order. Each decoding unit can be referred to as "decoding unit m", where m is a unique number identifying the decoding unit. The first decoding unit in the first sequential access unit (access unit 0) can be referred to as "decoding unit 0". For each subsequent decoding unit, the value of m can be incremented by 1 in the decoding order. "Frame n" can refer to the encoding / decoding frame or the decoding frame of access unit n. Timing information associated with a specific decoding unit can arrive before the CPB removal time of that decoding unit.

[0463] Figure 24 The hypothetical reference decoder (HRD) can include initialization parameters. For example, hrd_parameters() and sub_layer_hrd_parameters() can include the codec picture buffer (CPB) size and the decode picture buffer (DPB) size, which can be signaled in SEI messages, Sequence Parameter Set (SPS), Picture Parameter Set (PPS), or inferred from other values. For example, the CPB size can be inferred as CpbVclFactor. MaxCPB. Additionally, the bit rate value can represent the maximum input bit rate (at the Access Unit (AU) and Decoding Unit (DU) levels). Furthermore, the sub-screen HRD flag can indicate whether the operating mode is AU or DU mode.

[0464] exist Figure 24 In the example, the hypothetical reference decoder can be initialized at decoding unit 0, and both CPB 502 and DPB 506 are set to empty. The hypothetical stream scheduler 500 can typically schedule the retrieval of video data (e.g., access units) from various bitstreams and store the retrieved video data in CPB 502. That is, the hypothetical stream scheduler 500 can distribute data associated with decoding units flowing into CPB 502 according to a specific arrival schedule. Various bitstreams can be a set of bitstreams with an aggregated maximum decoding capability that can be satisfied by the corresponding decoder device. Decoding unit 504 can retrieve decoding units corresponding to the retrieved video data from CPB 502 and decode the decoding units (e.g., blocks, slices, frames / pictures, etc.). In the case of the hypothetical reference decoder, it is assumed that decoding unit 504 decodes the data associated with each decoding unit instantaneously at the CPB removal time of the corresponding decoding unit. Decoding unit 504 can store the decoded data in DPB 506. Output cropping unit 508 can retrieve the decoded picture from DPB 506 and crop the picture before outputting it to the display device. When a decoded frame is no longer needed for inter-frame prediction reference and is no longer needed for output, the frame can be removed from DPB 506.

[0465] CPB 502 can operate as follows. First, the decoding unit m can arrive at its initial arrival time. In some examples, for CBR mode, the initial arrival time of the decoding unit m (i.e., initArrivalTime[m]) can be set to be equal to the final arrival time of the previously accessed unit (m-1). Otherwise, for non-CBR mode, the initial arrival time of the decoding unit m can be set to the maximum of the arrival time of the previously accessed unit (AuFinalArrivalTime[m-1]) and the initial arrival time of the earliest decoding unit of the currently accessed unit (initArrivalEarliestTime[m]).

[0466] The hypothetical reference decoder can calculate initArrivalEarliestTime[m] as equal to Au / DuNominalRemovalTime[m] – (InitCpbRemovalDelay[SchedSelIdx] + InitCpbRemovalDelayOffset[SchedSelIdx]) / 90000. The hypothetical reference decoder can calculate AuFinalArrivalTime[m-1] as equal to initArrivalTime[m] + sizeInBits[m] / BitRate[SchedSelIdx]. The hypothetical reference decoder can calculate the time for removing the decoding unit based on AuNominalRemovalTime[0] = InitCpbRemovalDelay[SchedSelIdx] / 90000.

[0467] The DPB 506 can contain various picture storage buffers. Each picture storage buffer can contain one or more decoded pictures, which can be marked "for reference" or saved for future output. The hypothetical reference decoder can calculate the time DpbOutputTime[n] for removing picture n from the DPB 506 as equal to AuCpbRemovalTime[n] + ClockSubTick. `picSptDpbOutputDuDelay`. Decoding unit 504 can store the currently decoded frame in DPB 506 in an empty frame storage buffer. Then, the fullness of DPB 506 can be increased by 1. When `TwoVersionsOfCurrDecPicFlag` equals 0 and `pps_curr_pic_ref_enabled_flag` equals 1, the corresponding frame can be marked as "for long-term reference". After all slices of the current frame have been decoded, the frame can be marked as "for short-term reference".

[0468] The technology disclosed herein can be applied to situations where there are N video streams to be decoded simultaneously. Each of the N video streams can have corresponding profile / layer / level requirements and HRD requirements, as specified in the parameters included in the bitstream. The video decoding system according to the technology disclosed herein can correspond to the following respectively: Figures 25 to 27 One of the models shown in the image.

[0469] Figure 25This is a block diagram illustrating an example model of a video decoding system that can perform the techniques of this disclosure to decode multiple video streams in parallel using multiple video decoding instances. In this example, the codec picture buffer (CPB) and the decoder picture buffer (DPB) operate independently for each of the N decoders.

[0470] In particular, Figure 25 The example depicts a CPB video decoding interface (VDI) 530, a video decoding instance 540, and a decoding picture buffer VDI 550. To decode N streams in parallel, N decoding instances can be instantiated. Therefore, the video decoding instance 540 includes decoders 542A to 542N (decoders 542). Each of the decoders 542 is associated with a corresponding one of the CPBs 532A to 543N (CPB532) and a corresponding one of the DPBs 552A to 552N (DPB 552).

[0471] Data associated with a decoding unit in one of the CPBs 532 that flows into each stream according to the specified arrival schedule can be distributed by a common hypothetical stream scheduler (HSS) 520, which schedules N bitstreams for decoding by the corresponding decoders 542. For each access unit, all data associated with the access unit can be removed and decoded by the corresponding decoder 542 at the CPB removal time of the access unit. Then, one of the decoders 542 can place the decoded frame in the corresponding DPB 552 for reference during the decoding process of the stream, as well as for output and trimming. When the decoded frame is no longer needed for inter-frame prediction reference, it can be removed from one of the DPBs 552 after it has been output.

[0472] At any given time, each bitstream can conform to the configuration file / layer / level and HRD parameters of the respective stream's signaling notification. At any given time, the sum of CPB sizes can conform to the configuration file / layer / level signaling. At any given time, the aggregation decoder processing speed (samples per second) can conform to the configuration file / layer / level signaling. And at any given time, the sum of DPB sizes can conform to the configuration file / layer / level signaling. A common HRD parameter for initial delay can also be specified.

[0473] Figure 26 This is a block diagram illustrating another example model of a video decoding system that can implement the techniques of this disclosure to decode multiple video streams in parallel using multiple video decoding instances. In this example, the CPB operates independently for each of the N decoders, but a combined DPB is used to ensure that the outputs of the N decoders are provided with appropriate timing in a synchronous manner for output or other subsequent processing.

[0474] In particular, Figure 26 The example depicts a CPB VDI 570, a video decoding instance 580, and a DPB VDI 590. To decode N streams in parallel, N decoding instances can be instantiated. Therefore, the video decoding instance 580 includes decoders 582A to 582N ​​(decoders 582). Each decoder 582 is associated with a corresponding one of the CPBs 572A to 572N (CPB 572). In this example, the DPB VDI 590 includes DPBs 592A to 592N and a common DPB 594 shared among all decoding instances.

[0475] Data associated with a decoding unit in one of the CPBs 572 that flows into each stream according to the specified arrival schedule can be distributed by a common hypothetical stream scheduler (HSS) 560, which schedules N bitstreams for decoding by the corresponding decoders 582. For each access unit, all data associated with the access unit can be removed and decoded by the corresponding decoder 582 at the CPB removal time of the access unit. Then, one of the decoders 582 can place the decoded frame in the corresponding DPB 592 for reference during the decoding process of the stream, as well as for output and trimming. After the decoded frame has been output, it can be removed from one of the DPBs 552 when the decoded frame is no longer needed for inter-frame prediction reference and when the decoded frame has the largest output time among all remaining decoded frames in the group of decoders 582.

[0476] At any given time, each bitstream can conform to the configuration file / layer / level and HRD parameters of the respective stream's signaling notification. At any given time, the sum of CPB sizes can conform to the configuration file / layer / level signaling. At any given time, the aggregation decoder processing speed (samples per second) can conform to the configuration file / layer / level signaling. At any given time, the sum of DPB sizes can conform to the configuration file / layer / level signaling. And at any given time, the common DPB size can conform to the configuration file / layer / level signaling. A common HRD parameter for initial delay can also be specified.

[0477] Figure 27 This is a block diagram illustrating another example model of a video decoding system that can perform the techniques of this disclosure to decode multiple video streams in parallel using multiple video decoding instances. In this example, each of the N decoders has both a common CPB and a common DPB.

[0478] In particular, Figure 27The example depicts a CPB VDI 610, a video decoding instance 620, and a DPB VDI 630. To decode N streams in parallel, N decoding instances can be instantiated. Therefore, the video decoding instance 620 includes decoders 622A to 622N (decoders 622). Each decoder 622 is associated with a corresponding one of CPBs 614A to 614N (CPB 614). Additionally, in this example, the CPB VDI 610 includes a common CPB 612. In this example, the DPB VDI 630 includes DPBs 632A to 632N, and a common DPB 634 shared among all decoding instances.

[0479] Data associated with a decoding unit in one of the CPB 572s flowing into each stream according to the specified arrival schedule can be distributed by a common hypothetical stream scheduler (HSS) 600, which schedules N bitstreams for decoding by the corresponding decoder 622. The addition of each decoding unit can be accomplished according to the common HSS. For each access unit, all data associated with the access unit can be removed and decoded by the corresponding decoder 622 at the CPB removal time of the access unit. Then, one of the decoders 622 can place the decoded frame in the corresponding DPB 632 for reference during the decoding process of the stream, as well as for output and trimming. After a decoded frame has been output, it can be removed from one of the DPB 632 when the decoded frame is no longer needed for inter-frame prediction reference and when the decoded frame has the largest output time among all remaining decoded frames in the group of decoders 622.

[0480] At any given time, each bitstream can conform to the configuration file / layer / level and HRD parameters of the respective stream's signaling notification. At any given time, the sum of CPB sizes can conform to the configuration file / layer / level signaling. At any given time, the aggregation decoder processing speed (samples per second) can conform to the configuration file / layer / level signaling. At any given time, the sum of DPB sizes can conform to the configuration file / layer / level signaling. At any given time, the common DPB size can conform to the common configuration file / layer / level signaling. And at any given time, the common CPB can conform to the common configuration file / layer / level signaling. The common HRD parameter for initial delay can also be specified.

[0481] Bitstreams can be generated in various ways. In one example, bitstreams can be generated jointly. For instance, VDI-based decoding can be used to encode the bitstream. A general HRD parameter can be defined, and a set of encoders can be controlled overall to ensure that common HRD parameters are maintained.

[0482] In another example, bitstreams can be generated separately. That is, the streams can be encoded independently. Each stream can be annotated with a separate configuration file / layer / level and HRD parameters. Additional information can be provided for each bitstream to support joint decoding (e.g., decoding frames). Common HRD operations can be derived by the decoder / client device.

[0483] In some examples, the stream scheduler can select a stream based on commented parameters.

[0484] In some examples, each bitstream may include encoded metadata, allowing HRD parameters to be dynamically exported at the decoder side / client device.

[0485] Figure 28 This is a flowchart illustrating an example method for decoding video data according to the technology disclosed herein. Figure 28 The method can be performed by any of the various decoding devices disclosed herein. For illustrative and explanatory purposes, regarding Figure 3 The video decoding engine 200 is used to describe Figure 28 However, other devices can perform this or similar techniques, such as... Figure 1 Video decoder 48 Figures 4 to 6 Hardware video decoding engine 212, Figure 15 Video decoder engine 340, Figure 16 Video decoder engine 350, Figure 17 Video decoder engine 370, Figure 20 Video decoder engine 412, Figure 24 Decoding unit 504, Figure 25 Video decoding example 540 Figure 26 Video decoding example 580 or Figure 27 Video decoding example 620.

[0486] First, the flow scheduling unit (e.g., Figure 1 The RTP receiving unit 52 can first determine the attributes (700) of various video bitstreams. Attributes may be, for example, profile / layer / level information, HRD parameters, etc. The bitstreams may be bitstreams intended to be combined for parallel retrieval, decoding, and presentation, or they may be separate bitstreams that are subsequently combined. The RTP receiving unit 52 can use attributes to select several bitstreams for retrieval (702). For example, the RTP receiving unit 52 can determine, for example... Figure 3 The video decoding engine has a decoding capability of 200.

[0487] The video decoding engine 200 can then instantiate multiple video decoder instances 214 (704), for example, a number of video decoder instances equal to the number of video bitstreams to be retrieved. The video decoder instances 214 can then decode the selected bitstreams (706). The output formatting unit 208 can further format the decoded images (708), for example, as per Figures 8 to 10. Figure 20 Any of the ones shown and described. The video decoding engine 200 can then output a formatted picture (710).

[0488] In this way, Figure 28 The method represents an example of a method for decoding media data, comprising: instantiating a first number of video decoder instances to be executed by video decoding hardware implemented in a circuit; determining attributes of a plurality of video media streams, the attributes indicating that each of the plurality of video media streams is available for stream selection; selecting a second number of input video media streams from the plurality of video media streams based on the determined attributes of the second number of input video media streams; decoding the second number of input video media streams by the video decoder instances to form a second number of decoded video media streams; and outputting data of the second number of decoded video media streams.

[0489] Various examples of the techniques disclosed herein are summarized in the following clauses:

[0490] Clause 1: A method for decoding media data, the method comprising: instantiating a first number of video decoder instances to be executed by video decoding hardware; using the video decoder instances to decode a second number of input video media streams to form one or more decoded video media streams; and outputting data of the one or more decoded video media streams.

[0491] Clause 2: The method according to Clause 1 also includes receiving an indication of a first quantity of configuration data.

[0492] Clause 3: The method according to Clause 2, wherein receiving configuration data includes receiving configuration data from a media player application.

[0493] Clause 4: The method according to Clause 2, wherein receiving configuration data includes receiving one or more Supplemental Enhancement Information (SEI) messages together with the input video media stream.

[0494] Clause 5: The method according to any one of Clauses 1 to 4 further includes: receiving a third number of received video media streams; and formatting the received video media streams to form a second number of input video media streams.

[0495] Clause 6: The method according to Clause 5, wherein formatting the received video media stream includes inserting a first video object from the first received video media stream onto one side of a second video object from the second video media stream.

[0496] Clause 7: The method according to any one of Clauses 5 and 6, wherein formatting the received video media stream includes inserting a first video object from the first received video media stream above or below a second video object from the second video media stream.

[0497] Clause 8: The method according to any one of Clauses 5 to 7, wherein formatting the received video media stream includes appending a first video object from the first received video media stream to a second video object from the first received video media stream.

[0498] Clause 9: The method according to any one of Clauses 5 to 8, wherein formatting the received video media stream includes stacking a first video object from the first received video media stream on top of a second video object from the first received video media stream.

[0499] Clause 10: The method according to any one of Clauses 1 to 9, wherein outputting data of one or more decoded video media streams includes appending a first decoded frame of a first video media stream to the top, bottom, left, or right of a second decoded frame of a second video media stream.

[0500] Clause 11: The method according to any one of Clauses 5 to 10 further includes receiving configuration data from the media application instructing how the input video media streams are combined to form a single output video media stream.

[0501] Clause 12: The method according to any one of Clauses 5 to 10 also includes receiving a Supplemental Enhancement Information (SEI) message indicating how the input video media streams are combined to form a single output video media stream.

[0502] Clause 13: An apparatus for decoding media data, the apparatus comprising one or more components for performing the methods of any one of Clauses 1 to 12.

[0503] Clause 14: A device pursuant to Clause 13, wherein one or more components include one or more processors and a memory configured to store media data.

[0504] Clause 15: Device pursuant to Clause 13, wherein the device includes at least one of the following: an integrated circuit; a microprocessor; and a wireless communication device.

[0505] Clause 16: An apparatus for retrieving media data, the apparatus comprising: means for instantiating a first number of video decoder instances to be executed by video decoding hardware; means for executing the video decoder instances to decode a second number of input video media streams to form one or more decoded video media streams; and means for outputting data of the one or more decoded video media streams.

[0506] Clause 17: A method for decoding media data, the method comprising: instantiating a first number of video decoder instances to be executed by video decoding hardware implemented in a circuit; determining attributes of a plurality of video media streams, the attributes indicating that each of the plurality of video media streams is available for stream selection; selecting a second number of input video media streams from the plurality of video media streams based on the determined attributes of a second number of input video media streams; decoding the second number of input video media streams by the video decoder instances to form a second number of decoded video media streams; and outputting data of the second number of decoded video media streams.

[0507] Clause 18: The method according to Clause 17 also includes receiving an indication of a first quantity of configuration data.

[0508] Clause 19: The method according to Clause 18, wherein receiving configuration data includes receiving configuration data from a media player application.

[0509] Clause 20: The method according to Clause 18, wherein receiving configuration data includes receiving one or more Supplemental Enhancement Information (SEI) messages together with the input video media stream.

[0510] Clause 21: The method pursuant to Clause 17 further includes: receiving a third number of received video media streams; and formatting the received video media streams to form a second number of input video media streams.

[0511] Clause 22: The method according to Clause 21, wherein formatting the received video media stream includes inserting a first video object from the first received video media stream onto one side of a second video object from the second video media stream.

[0512] Clause 23: The method according to Clause 21, wherein formatting the received video media stream includes inserting a first video object from the first received video media stream above or below a second video object from the second video media stream.

[0513] Clause 24: The method according to Clause 21, wherein formatting the received video media stream includes appending a first video object from the first received video media stream to a second video object from the first received video media stream.

[0514] Clause 25: The method according to Clause 21, wherein formatting the received video media stream includes stacking a first video object from the first received video media stream on top of a second video object from the first received video media stream.

[0515] Clause 26: The method according to Clause 21 also includes receiving configuration data from the media application instructing how the input video media streams are combined to form a single output video media stream.

[0516] Clause 27: The method according to Clause 21 also includes receiving information that indicates how the input video media streams are combined to form a single output video media stream.

[0517] Clause 28: The method according to Clause 17, wherein outputting data of one or more decoded video media streams includes attaching a first decoded frame of a first video media stream above, below, to the left, or to the right of a second decoded frame of a second video media stream.

[0518] Clause 29: According to the method of Clause 17, the attributes of multiple video media streams include profiles, layer and level requirements, and hypothetical reference decoder (HRD) requirements.

[0519] Clause 30: The method according to Clause 17, wherein each of the first number of video decoder instances includes a corresponding codec picture buffer (CPB) and a corresponding decoder picture buffer (DPB).

[0520] Clause 31: The method according to Clause 17, wherein each of the first number of video decoder instances includes a corresponding codec picture buffer (CPB), and wherein the first number of video decoder instances share a common decoded picture buffer (DPB).

[0521] Clause 32: The method according to Clause 17, wherein the first number of video decoder instances includes a shared common codec picture buffer (CPB) and a shared common decoder picture buffer (DPB).

[0522] Clause 33: The method according to Clause 17, wherein selecting a second number of input video media streams includes: determining the decoding capabilities of a first number of video decoder instances; determining the rendering capabilities of the second number of decoded video media streams; and selecting a second number of input video media streams capable of being decoded according to the decoding capabilities and rendered according to the rendering capabilities.

[0523] Clause 34: An apparatus for decoding media data, the apparatus comprising: a memory configured to store video data; and a processing system including one or more processors implemented in a circuit, the processing system being configured to: instantiate a first number of video decoder instances to be executed by the processing system; determine attributes of a plurality of video media streams, the attributes indicating that each of the plurality of video media streams is available for stream selection; select a second number of input video media streams from the plurality of video media streams based on the determined attributes of the second number of input video media streams; execute the video decoder instances to decode the second number of input video media streams to form a second number of decoded video media streams; and output data of the second number of decoded video media streams.

[0524] Clause 35: A device pursuant to Clause 34, wherein the memory includes a common decoded frame buffer (DPB) for storing the decoded frames of each of the video decoder instances.

[0525] Clause 36: A device pursuant to Clause 34, wherein the memory includes a common codec picture buffer (CPB) for storing the encoded picture of each of the video decoder instances.

[0526] Clause 37: A method for decoding media data, the method comprising: instantiating a first number of video decoder instances to be executed by video decoding hardware implemented in a circuit; determining attributes of a plurality of video media streams, the attributes indicating that each of the plurality of video media streams is available for stream selection; selecting a second number of input video media streams from the plurality of video media streams based on the determined attributes of a second number of input video media streams; decoding the second number of input video media streams by the video decoder instances to form a second number of decoded video media streams; and outputting data of the second number of decoded video media streams.

[0527] Clause 38: The method pursuant to Clause 37 also includes receiving an indication of a first quantity of configuration data.

[0528] Clause 39: The method according to Clause 38, wherein receiving configuration data includes receiving configuration data from a media player application.

[0529] Clause 40: The method according to any one of Clauses 38 and 39, wherein receiving configuration data includes receiving one or more Supplemental Enhancement Information (SEI) messages together with the input video media stream.

[0530] Clause 41: The method pursuant to any one of Clauses 37 to 40 further includes: receiving a third number of received video media streams; and formatting the received video media streams to form a second number of input video media streams.

[0531] Clause 42: The method according to Clause 41, wherein formatting the received video media stream includes inserting a first video object from the first received video media stream onto one side of a second video object from the second video media stream.

[0532] Clause 43: The method according to any one of Clauses 41 and 42, wherein formatting the received video media stream includes inserting a first video object from the first received video media stream above or below a second video object from the second video media stream.

[0533] Clause 44: The method according to any one of Clauses 41 to 43, wherein formatting the received video media stream includes appending a first video object from the first received video media stream to a second video object from the first received video media stream.

[0534] Clause 45: The method according to any one of Clauses 41 to 44, wherein formatting the received video media stream includes stacking a first video object from the first received video media stream on top of a second video object from the first received video media stream.

[0535] Clause 46: The method according to any one of Clauses 41 to 45 further includes receiving configuration data from the media application instructing how the input video media streams are combined to form a single output video media stream.

[0536] Clause 47: The method according to any one of Clauses 41 to 46 further includes receiving a Supplemental Enhancement Information (SEI) message indicating how the input video media streams are combined to form a single output video media stream.

[0537] Clause 48: The method according to any one of Clauses 37 to 47, wherein outputting data of one or more decoded video media streams includes appending a first decoded frame of a first video media stream to the top, bottom, left, or right of a second decoded frame of a second video media stream.

[0538] Clause 49: The method of any one of Clauses 37 to 48, wherein the attributes of multiple video media streams include profile, layer and level requirements and hypothetical reference decoder (HRD) requirements.

[0539] Clause 50: The method according to any one of Clauses 37 to 49, wherein each of the first number of video decoder instances includes a corresponding codec picture buffer (CPB) and a corresponding decoder picture buffer (DPB).

[0540] Clause 51: The method according to any one of Clauses 37 to 50, wherein each of the first number of video decoder instances includes a corresponding codec picture buffer (CPB), and wherein the first number of video decoder instances share a common decoded picture buffer (DPB).

[0541] Clause 52: The method of any one of Clauses 37 to 51, wherein the first number of video decoder instances includes a shared common codec picture buffer (CPB) and a shared common decoder picture buffer (DPB).

[0542] Clause 53: The method according to any one of Clauses 37 to 52, wherein selecting the second number of input video media streams comprises: determining the decoding capabilities of the first number of video decoder instances; determining the rendering capabilities for rendering the second number of decoded video media streams; and selecting the second number of input video media streams capable of decoding according to the decoding capabilities and rendering according to the rendering capabilities.

[0543] Clause 54: An apparatus for decoding media data, the apparatus comprising: a memory configured to store video data; and a processing system including one or more processors implemented in a circuit, the processing system being configured to: instantiate a first number of video decoder instances to be executed by the processing system; determine attributes of a plurality of video media streams, the attributes indicating that each of the plurality of video media streams is available for stream selection; select a second number of input video media streams from the plurality of video media streams based on the determined attributes of the second number of input video media streams; execute the video decoder instances to decode the second number of input video media streams to form a second number of decoded video media streams; and output data of the second number of decoded video media streams.

[0544] Clause 55: A device pursuant to Clause 54, wherein the memory includes a common decoded frame buffer (DPB) for storing the decoded frames of each of the video decoder instances.

[0545] Clause 56: A device pursuant to any one of Clauses 54 and 55, wherein the memory includes a common codec picture buffer (CPB) for storing the encoded picture of each of the video decoder instances.

[0546] In one or more examples, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality may be stored or transmitted as one or more instructions or code on or through a computer-readable medium and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium, which corresponds to a tangible medium such as a data storage medium, or a communication medium, including, for example, any medium that facilitates the transfer of a computer program from one place to another according to a communication protocol. In this way, a computer-readable medium may generally correspond to (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures to implement the techniques described in this disclosure. A computer program product may include a computer-readable medium.

[0547] By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage, disk storage or other magnetic storage devices, flash memory, or any other medium that can be used to store required program code in the form of instructions or data structures and that can be accessed by a computer. Moreover, any connection is appropriately referred to as a computer-readable medium. For example, the definition of medium includes coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies (such as infrared, radio, and microwave) if instructions are sent from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies (such as infrared, radio, and microwave). However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but rather refer to non-transient tangible storage media. As used herein, disks and optical discs include compact optical discs (CDs), laser optical discs, optical discs, digital versatile optical discs (DVDs), floppy disks, and Blu-ray discs, wherein disks typically reproduce data magnetically, while optical discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0548] Instructions can be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the term "processor" as used herein can refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein can be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into a combined codec. Similarly, the technique can be fully implemented in one or more circuit or logic elements.

[0549] The technologies disclosed herein can be implemented in various devices or apparatuses, including wireless mobile phones, integrated circuits (ICs), or a set of ICs (e.g., chipsets). Various components, modules, or units are described in this disclosure to emphasize functional aspects of a device configured to perform the disclosed technologies, but they do not necessarily need to be implemented by different hardware units. Rather, as described above, various units can be combined in a codec hardware unit or provided by a collection of interoperable hardware units, including one or more processors as described above, combined with suitable software and / or firmware.

[0550] Various examples have been described. These and other examples are within the scope of the appended claims.

Claims

1. A method of decoding media data, the method comprising: instantiating a first number of video decoder instances to be performed by video decoding hardware implemented in circuitry; determining properties of a plurality of video media streams, the properties indicating that each of the plurality of video media streams is available for stream selection; selecting a second number of input video media streams from the plurality of video media streams according to the determined properties of the second number of input video media streams; decoding, by the video decoder instances, the second number of input video media streams to form a second number of decoded video media streams; and outputting data of the second number of decoded video media streams.

2. The method of claim 1, further comprising receiving configuration data indicating the first number.

3. The method of claim 2, wherein receiving the configuration data comprises receiving the configuration data from a media player application.

4. The method of claim 2, wherein receiving the configuration data comprises receiving one or more supplemental enhancement information (SEI) messages with the input video media streams.

5. The method of claim 1, further comprising: receiving a third number of received video media streams; and formatting the received video media streams to form the second number of input video media streams.

6. The method of claim 5, wherein formatting the received video media streams comprises inserting a first video object from a first received video media stream to a side of a second video object from a second video media stream.

7. The method of claim 5, wherein formatting the received video media streams comprises inserting a first video object from a first received video media stream above or below a second video object from a second video media stream.

8. The method of claim 5, wherein formatting the received video media streams comprises appending a first video object from a first received video media stream to a second video object from the first received video media stream.

9. The method of claim 5, wherein formatting the received video media streams comprises stacking a first video object from a first received video media stream on top of a second video object from the first received video media stream.

10. The method of claim 5, further comprising receiving configuration data from a media application indicating how the input video media streams are combined to form a single output video media stream.

11. The method of claim 5, further comprising receiving supplemental enhancement information (SEI) messages indicating how the input video media streams are combined to form a single output video media stream.

12. The method of claim 1, wherein outputting data of the one or more decoded video media streams comprises appending a first decoded picture of a first video media stream above, below, to the left of, or to the right of a second decoded picture of a second video media stream. ​ ​ 13. The method of claim 1, wherein the properties of the plurality of video media streams comprise profile, level and tier requirements and hypothetical reference decoder (HRD) requirements.

14. The method of claim 1, wherein each of the first number of video decoder instances comprises a respective coded picture buffer (CPB) and a respective decoded picture buffer (DPB).

15. The method of claim 1, wherein each of the first number of video decoder instances comprises a respective coded picture buffer (CPB), and wherein the first number of video decoder instances share a common decoded picture buffer (DPB).

16. The method of claim 1, wherein the first number of video decoder instances comprises a shared common coded picture buffer (CPB) and a shared common decoded picture buffer (DPB).

17. The method of claim 1, wherein selecting the second number of input video media streams comprises: determining decoding capabilities of the first number of video decoder instances; determining rendering capabilities for rendering the second number of decoded video media streams; and selecting the second number of the input video media streams that are decodable according to the decoding capabilities and are renderable according to the rendering capabilities.

18. A device for decoding media data, the device comprising: a memory configured to store video data; and a processing system comprising one or more processors implemented in circuitry, the processing system configured to: instantiate a first number of video decoder instances to be executed by the processing system; determine properties of a plurality of video media streams, the properties indicating each of the plurality of video media streams is available for stream selection; select, from the plurality of video media streams, a second number of input video media streams according to the determined properties of the second number of input video media streams; execute the video decoder instances to decode the second number of input video media streams to form a second number of decoded video media streams; and output data of the second number of decoded video media streams.

19. The device of claim 18, wherein the memory comprises a common decoded picture buffer (DPB) for storing decoded pictures of each of the video decoder instances.

20. The device of claim 18, wherein the memory comprises a common coded picture buffer (CPB) for storing coded pictures of each of the video decoder instances. ​ ​ ​

Images