Processing multi-layer video stream
By generating joint basic streams and managing multi-layer video streams with nested identifiers, the problems of decoder adaptation and file format support in multi-layer video stream processing technology are solved, and efficient multi-layer video stream decoding and video quality adjustment are achieved. .
Patent Information
- Application Number
- CN202280088309.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-01-19
- Filing Date
- 2022-11-21
- Publication Date
- 2024-11-05
AI Technical Summary
The existing multi-layer video streaming technology is difficult to adapt to existing and new decoders, resulting in cumbersome application of multi-layer encoding schemes and difficult to support the flexible delivery and reuse of multi-layer videos, affecting the adjustment and decoding efficiency of video quality. .
By generating joint elementary streams, allowing video data from different layers to be transmitted and decoded as a single data stream, using nested identifiers and metadata to manage the decoding configurations of different layers, enabling flexible decoding and adaptation of multi-layer video streams to achieve flexible decoding and adaptation of existing decoding for multi-layer video streams. utensil.
It realizes efficient decoding of multi-layer video streams, supports flexible adjustment of video quality and adaptation of decoders, reduces the need for updates to decoding hardware and software, and improves the transmission and storage efficiency of multi-layer video streams.
Smart Images

Figure CN118901243A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to processing of multi-layer video streams. In particular, the present invention relates to one or more of encoding and decoding multi-layer video streams, for example, using different methods to transmit multi-layer streams to a decoding device and achieve efficient decoding. Background Art
[0002] Multi-layer video decoding schemes have been around for many years, but have encountered problems with widespread adoption. Most video content on the Internet is still encoded using H.264 or MPEG-4 Part 10 Advanced Video Coding (MPEG-4 AVC), with this format used for between 80% and 90% of online video content. The content is typically supplied to a decoding device as a single video stream that has a one-to-one relationship with an available hardware and / or software video decoder, e.g., a single stream is received, parsed, and decoded by a single video decoder to output a reconstructed video signal. Therefore, many video decoder implementations are developed based on this framework. In order to support different encodings, decoders are typically configured with a simple switching mechanism driven by metadata that identifies the stream format.
[0003] Existing multi-layer decoding schemes include the Scalable Video Coding (SVC) extension to H.264, the scalable extension to H.265 or MPEG-H Part 2 High Efficiency Video Coding (SHVC), and new standards such as MPEG-5 Part 2 Low Complexity Enhancement Video Coding (LCEVC). While H.265 is an evolution of the decoding framework used by H.264, LCEVC takes a different approach to scalable video. SVC and SHVC operate by creating different coding layers and feeding different spatial resolutions to each of these coding layers. Each layer encodes the input according to a normal AVC or HEVC encoder, possibly using information generated by a lower coding layer. On the other hand, LCEVC generates one or more layers of enhanced residuals compared to the base coding, where the base coding can have a lower spatial resolution.
[0004] One reason for the slow adoption of multi-layer decoding schemes is that it is difficult to adapt existing and new decoders to process multi-layer coded streams. As discussed above, a video stream is typically a single data stream paired one-to-one with a suitable decoder, whether the decoder is implemented in hardware or software or a combination of the two. Client devices and media players (including Internet browsers) are therefore constructed to receive data streams, determine what video encoding the stream uses, and then pass the stream to a suitable video decoder. Within this framework, multi-layer schemes (such as SVC and SHVC) are typically packaged as larger single video streams comprising multiple layers, wherein these streams can be detected as "SVC" or "SHVC", and multiple layers are extracted from a single stream and passed to SVC or SHVC decoders for reconstruction. However, this method often weakens some benefits of multi-layer coding. Therefore, the conclusion drawn by many developers and engineers is that the multi-layer decoding scheme is too cumbersome, and instead returns to the multicast of a single H.264 video stream.
[0005]
[0006] It would therefore be desirable to have an improved method and system for decoding multi-layer video data that overcomes some of the shortcomings discussed above and allows for more benefits of multi-layer coding schemes to be realized.
[0006] An overview of the SVC extension is provided in the paper "The Scalable Video Coding Extension of the H.264 / AVC Standard" by Heiko Schwarz and Mathias Wien, published in IEEE Signal Processing Magazine, Vol. 135, March 2008.
[0007] An overview of SHVC extensions is provided in the paper “Overview of SHVC: Scalable Extensions of the High Efficiency Video Coding Standard” by Jill Boyce, Yan Ye, Jianle Chen, and Adarsh K. Ramasubramonian, published in IEEE Transactions on Circuits and Systems for Video Technology, Vol. 26, No. 1, January 2016.
[0008] Decoding techniques for LCEVC are described in the draft text of ISO / IEC FDIS 23094-2 published at the 129th MPEG meeting in Brussels in January 2020, and in the final approved text and WO 2020 / 188273 A1. Figure 29B of WO 2020 / 188273 A1 describes a hypothetical reference decoder in which a demultiplexer provides a base bitstream to a base decoder and an enhancement bitstream to an enhancement decoder.
[0009] US2010 / 0272190 A1 describes a scalable transmission / reception device and method for improving the availability of broadcast services, which can allow a receiver to select the best video according to the degree of attenuation of a broadcast signal by scalably encoding video data and transmitting it through different transmission schemes for each layer. US2010 / 0272190 A1 encodes HD and SD video streams using an H.264 scalable video encoder (i.e., using SVC), and uses different packet streams to generate different layers of SVC encoding. At the decoding device, a DVB-S2 receiver / demodulator receives / demodulates a satellite broadcast signal from a transmitting satellite and recovers a first layer packet stream and a second layer packet stream. At the decoding device, a scalable combiner combines the recovered first layer packet stream and second layer packet stream in input order to generate a single transport stream. A subsequent demultiplexer demultiplexes and depacketizes the combined transport stream and splits it into first layer and second layer video streams, which are then passed to the H.264 scalable video decoder for decoding and generation of reconstruction of the original HD video stream.
[0010] WO 2017 / 141038 A1 describes a physical adapter configured to receive a data stream comprising data usable to derive a reproduction of a signal at a first quality level and reconstruction data produced by processing the reproduction of the signal at a second, higher quality level, and instructions for how to reconstruct the reproduction at the second quality level using the reproduction at the first quality level. WO 2017 / 141038 A1 describes how to synchronize different elementary streams, namely a first elementary stream with a first packet identifier (PID) and a second elementary stream with a second packet identifier (PID), using a presentation time stamp (PTS).
[0011] For multi-layer streams, there is also the general problem of stream management. The different layers of a multi-layer stream may be generated together or separately, and may be served together or separately. It is desirable to have improved methods and systems for transmitting and retransmitting multi-layer streams over a network. For example, it is desirable to allow a content distributor to easily and flexibly modify the video quality by adding additional layers in a multi-layer scheme. It is also desirable to be able to flexibly re-multiplex a multi-layer video stream without interrupting downstream multi-layer decoding.
[0012] There is also the problem of supplying multi-layer streams as static file formats. For example, a video stream can be read from a fixed or portable medium (such as a solid-state device or a portable disk), or downloaded and stored as a file for later viewing. Existing file formats are difficult to support the transportation of multi-layer videos because these file formats typically assume a one-to-one mapping with media content and decoding configurations, and multi-layer streams can use different decoding configurations for different layers. Changes in file formats often have no practical effect because they require updates to decoding hardware and software, and may affect the decoding of legacy formats.
[0013] All of the above publications set forth above are incorporated herein by reference. Summary of the invention
[0014] Various aspects of the invention are set out in the attached independent claims. Variations to these aspects are set out in the attached dependent claims. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 is a schematic diagram showing how encoded video data may be transmitted within various data streams.
[0016] Figure 2A is a schematic diagram illustrating an example system for receiving and decoding multiple encoded data streams.
[0017] Figure 2B and Figure 2C are schematic diagrams illustrating example components of a stream receiver and a video decoder, respectively.
[0018] Figure 3 is a schematic diagram illustrating an example system for efficiently decoding multi-layer encoded video data.
[0019] Figure 4 is a schematic diagram illustrating an example set of components for efficiently decoding multi-layer encoded video data.
[0020] Figure 5 is a schematic diagram showing an example of a joint packet flow.
[0021] Figure 6 is a flow chart illustrating an example method of decoding a multi-layer video stream.
[0022] Figure 7 and Figure 8 are schematic diagrams illustrating example multi-layer encoder and decoder configurations, respectively.
[0023] Fig. 9 is a schematic diagram illustrating certain data processing operations performed by an example multi-layer encoder.
[0024] Fig.10 is a schematic diagram illustrating a combined multi-layer stream as may be generated at an encoder for transmission to a decoder.
[0025] Fig.11 is a flow chart illustrating a method of combining different laminar flows to form a multi-layer flow.
[0026] Fig.12 is a flow chart illustrating a method of using metadata to link different layer streams. DETAILED DESCRIPTION
[0027] Certain examples described herein allow decoding devices to be easily adapted to handle multi-layer video coding schemes. Certain examples are described with reference to LCEVC multi-layer video streams, but the general concepts can be applied to other multi-layer video schemes, including SVC and SHVC, as well as multi-layer watermarking and content delivery schemes.
[0028] Different examples are presented. In one set of examples, a single or joint packet stream is generated for a multi-layer video stream. This can be a joint elementary packet stream. Despite containing data for multiple levels or layers of a multi-layer video stream, the single or joint packet stream can be processed by existing decoders in a one-to-one manner. For example, backward compatibility can be maintained by passing a single or joint data stream including encoded data for multiple layers to a first layer decoder, wherein the configuration of the single or joint data stream is such that data associated with layers other than the first layer is ignored. Other layer decoders, including layer decoders operating according to different decoding methods (e.g., based on different video coding standards), can receive a single or joint data stream or other layer data from the stream and provide enhancements to first layer decoding.
[0029] In the accompanying drawings, Figures 1 to 2C Involves comparing media decoding pipelines, Figures 3 to 6 A specific example of an improved decoding system and method is described. Figures 7 to 9 provides examples of specific multi-layer video schemes that can be used with any of the described examples, Fig.10 and Fig.11 shows an alternative example in which a joint elementary stream may be generated at an encoder, and Fig.12 Examples are shown in which aspects discussed herein may be adapted to label different layers within a single or multiple data streams.
[0030] In some examples, such as Figures 1 to 6In the example shown in , different layers of multi-layer video decoding are sent as separate packets within the transport stream. This allows different layers to be efficiently served separately and allows enhancement layers to be easily added to a pre-existing or pre-configured base layer. At the decoding device, the different packet substreams are received and parsed, for example based on a packet identifier (PID) within a packet header. This can be done by an existing decoding pipeline configured to parse different single-layer encodings (e.g., non-scalable H.264 or H.265 streams).
[0031] Several examples will now be described with reference to the accompanying drawings. Figures 1 to 2C involves comparing media decoding pipelines, while Figures 3 to 6 Specific examples of improved decoding systems and methods are described. Then, Figures 7 to 9 Provides a Figures 3 to 6 An example of a specific multi-layer video scenario used together with an example of
[0032] Figure 1 An example 100 of a transport stream (TS) 102 is shown, which can be used to send encoded video data to one or more decoding devices. The transport stream 102 includes a sequence of 188-byte TS packets 110 of fixed length. Each TS packet 110 has a header 112 and a payload 114, which can have a variable length. The header 112 includes one or more data fields. One of these data fields provides a packet identifier (PID) 116. The PID is used to distinguish different substreams within the transport stream. The PID can be a number of bits of a fixed length (e.g., 13 bits) storing a numeric or alphanumeric identifier (typically represented as a hexadecimal value). For example, the PID 116 can be used to identify different video streams that are multiplexed together into a single stream forming the transport stream 102. The example transport stream specification is set forth in MPEG-2 Part 1 and is defined as part of ISO / IEC Standard 13818-1 or ITU-T Rec. H.222.0.
[0033] Figure 1 Also shown is a so-called PID stream 104 that can be extracted from the transport stream 102. The PID stream 104 comprises a stream of consecutive packets 110 having a common (i.e., shared) PID value. The PID stream 104 can be created by demultiplexing the transport stream 102 based on the PID value. The PID stream 104 thus represents a substream of the transport stream 102.
[0034] In some cases, there may be special PID values reserved for index tables. In one case, a PID value may be reserved for a program association table (PAT), which contains a directory listing of a set of program map tables, and a program map table (PMT) includes a mapping between one or more PID values and a specific "program". Originally a "program" associated with a specific broadcast program, but with Internet streaming, the term is widely used to be associated with the content of a specific video stream. The PMT can provide additional metadata about the content sent as part of the PID stream 104 within the transport stream 102. The PMT may include program descriptors. These are groups of bytes (multiples of 8 bits) in which the length of the descriptor can be defined (for example, the length can indicate the N descriptors that follow, each of which is 8 bits). Descriptors can be provided for the entire MPEG-2 program or for individual elementary streams. They can be provided optionally so that some elementary streams do not carry descriptors. In some cases, descriptors can typically be provided as part of program specific information (PSI), which includes metadata for content supplied as part of the transport stream. In certain examples described later in this specification, descriptors may be used to pair different layers within a multi-layer stream without disrupting backward decoding capabilities for lower layers.
[0035] Figure 1 Also shown is a packetized elementary stream (PES) 106 constructed based on the payload data of the plurality of TS packets 110. The PES includes data from the payload of the PID stream carrying the media sample data. The media sample data may include video data as well as other modalities such as audio data, subtitle data, or volumetric data. Figure 1 In the PES, a PES is generated by combining the payloads 114 of multiple media TS packets 110 associated with a common (i.e., shared) PID value. The PES includes a packet stream, where each PES packet consists of a header (i.e., a PES header) 122 and a payload 124, which carries the combined data. The start of a new PES packet is indicated by a one-bit field from the TS header 112, which is called a payload unit start indicator (PUSI) 126. When the PUSI is set, the first byte of the TS packet payload 114 indicates the location where a new PES payload unit begins. This allows a decoding device that begins receiving data in the middle of a transmission to determine when to start extracting data. The PES header 122 contains a presentation timestamp (PTS) 128. This indicates the presentation time for the corresponding media segment encapsulated within the payload 124.
[0036] Figure 1Finally, the content of a PES payload 124 for a video stream is shown. In this case, the PES payload 124 comprises a sequence 108 (i.e., a NALU stream) of NAL units 130. These may form part of an access unit for a video stream, i.e., a group of NAL units associated with a particular output time, that are consecutive in decoding order and that contain coded pictures or frames. Figure 1 A NALU stream 108 is shown which may be provided to a suitable video decoder for decoding.
[0037] Figure 1 It is shown how to use transport streams to broadcast video streams (e.g., over the air or over a network). A variety of file formats are also defined for carrying encoded video streams. This includes formats based on the International Standards Organization (ISO) Base Media File Format (BMFF) and formats based on the Common Media Application Format (CMAF). CMAF is an extensible standard for encoding and packaging segmented media objects for delivery and decoding on end-user devices. CMAF allows different implementations, such as Hypertext Transfer Protocol (HTTP) Live Streaming (HLS) and MPEG DASH (Dynamic Adaptive Streaming over HTTP). However, although file formats such as BMFF and CMAF are designed to help manage the complexity of video encoding formats, they are defined based on a single-layer coding framework (or in the case of SVC and SHVC, an overall scalable structure). These file formats are not defined to explicitly support multi-layer video encoding, where different layers are generated using different encoding methods and decoded using different decoding methods. Therefore, it is a challenge to support multi-layer video with these known file formats.
[0038] Both BMFF and CMAF define a "container", which is part of the file structure that stores the encoded media data. Metadata for the container can define the encoding standard that has been used to generate the encoded media data. In one case, the file format can be defined in the file type field ("ftyp") located at the beginning of the file of the specified encoding format, for example, "AVC1" is used for AVC, or "HEVC" is used for HEVC. The file format can be parsed by the decoding device and used to activate the appropriate decoder for the file format. However, this form of file format identification requires a one-to-one mapping between the identified file format and the decoder implementation. Although this works for overall scalable technologies, such as SVC or SHVC, where a single decoder receives and decodes all layers within a multi-layer video encoding, it does not work when different decoders are used for different layers (such as in LCEVC). It also causes problems with backward compatibility. For example, a stream marked as "SVC1" will be passed to an SVC decoder, and if the SVC1 decoder does not exist, an error will be raised, although each layer within SVC is based on the AVC format.
[0039] In some examples described herein, a flexible method for decoding multi-layer video is provided. In these examples, a "nested" identifier is used to identify the encoded multi-layer video content within a file. The "nested" identifier operates as an effective identifier for at least one layer (e.g., for the base layer) of the multi-layer video content. Thus, even if the decoder for one or more other layers of the multi-layer video content is unavailable, at least one layer can be delivered for decoding using an available first decoder (e.g., a legacy decoder) by parsing the nested identifier. In this case, data for one or more other layers in the multi-layer video content can be ignored, and only data for at least one layer can be decoded using an available decoder. However, if decoders for one or more other layers are available, these decoders can be activated based on additional information derived from the nested identifier and at least data for one or more other layers can be delivered. In some cases, data for all layers is delivered to each decoder, and the decoder is configured to ignore data that is not related to its given layer (e.g., using the default functionality of the decoder, data marked with values that the decoder cannot recognize can be ignored). In one case, the nested identifier can be implemented using a descriptor field of a transport stream and / or an elementary stream.
[0040] Figure 2A It shows that it can be used to Figure 1 An example of a media playback pipeline 200 configured to decode a media stream and display it. The media playback pipeline 200 may form part of a client or decoding device such as a smartphone, laptop, smart TV or other media receiver and / or player.
[0041] exist Figure 2A In the example, stream receiver 210 receives stream data 205 from a source. Stream data 205 may be received over a communication network (e.g., as a received encoded stream) and / or may be received as a file system data stream (e.g., from a memory, a local storage device, or a computer-readable medium). The term "stream" is used herein to refer to a continuous portion of data being received or accessed. Stream receiver 210 in this example is configured to receive a stream of data in the form of a container (such as a stream similar to Figure 1 The stream receiver 210 may include a transport stream of the transport stream 102 in the form of a transport stream (e.g., a transport stream of a data packet) or a digital multimedia container (e.g., an MPEG-4 Part 14 (MP4) container). The stream receiver 210 is configured to detect the container type and then parse the stream data 205 based on the container type. For example, the stream receiver 210 may detect whether the stream data 205 is a transport stream or another MPEG container (e.g., an MP4 container) and then launch an appropriate parser (e.g., a TS parser or an MP4 parser) for the container type. The stream receiver 210 may parse the stream data 205 to locate both media sample data and metadata within the stream data 205. For example, metadata for a transport stream may include PAT data and / or PMT data that maps a PID with a data packet header to a program, and metadata for an MP4 container may include a list of "tracks" present within an MP4 file. Thus, the stream receiver 210 may conditionally parse the stream data 205 to identify metadata in the form of a "directory" to identify what packetized elementary streams (e.g., TS containers) or tracks (e.g., MP4 containers) are present.
[0042] After the stream data 205 is parsed by the stream receiver 210 , the stream receiver 210 is configured to instantiate one or more decoders to decode the various data streams contained within the stream data 205 . Figure 2A A plurality of video decoders 220-A to 220-C and an audio decoder 240 are shown. Other decoders, such as subtitle decoders, may also be provided. Figure 2A In the example of FIG. 1 , the stream receiver 210 may be configured to receive the stream data 205 based on metadata carried in the stream data 205 (such as metadata accompanying the stream data 205). Figure 1104 in the stream data 205 or metadata for a particular MP4 track). The stream receiver 210 can instantiate a specific decoder based on the identified stream type or MP4 type for a particular PID stream. For example, the stream receiver 210 can instantiate an H.264 video decoder for a PID stream identified as an H.264 video stream and an H.265 video decoder for a PID stream identified as an H.265 video stream. An audio stream accompanying a video stream (e.g., where audio and video streams are multiplexed) can be identified, and an audio decoder 240 can be instantiated based on the audio stream type. In one case, the stream receiver 210 may not directly instantiate a decoder, but may pass the demultiplexed data stream for a program or track to an existing decoder, for example, where the existing decoder is determined based on metadata carried in the stream data 205. Although Figure 2A Only one audio decoder 240 is shown, but multiple audio decoders may be provided in an implementation (e.g., for different audio formats similar to the case of the video decoder shown). Similarly, other decoders such as subtitle decoders have been omitted for clarity, but these decoders may also be instantiated in parallel with the video and audio decoders shown.
[0043] Figure 2A The video decoder 220 shown in FIG. 1 is configured to process a particular encoded video stream, such as from Figure 1 The PES 106 or corresponding MP4 track of the video is decoded and outputs decoded video data ready for display. The audio decoder 240 performs a similar function for the encoded audio stream to output the encoded audio data for output. Figure 2A In the embodiment of the present invention, the video decoder 220 provides their output to the display output synthesizer 230, which is configured to render the decoded video data on a display. The display can be an integrated display (such as a touch screen of a smart phone) or a separate display (such as a monitor, headphones, or a projector). The audio decoder 240 provides its output to the audio mixer 250 for outputting the decoded audio data on a suitable speaker device (such as a headset, an integrated speaker, or an external sound system). In general, the decoded data may include video data frames, audio samples, or subtitles for display through one or more frames, etc.
[0044] Figure 2B Two subcomponents of the stream receiver 210 are shown according to a specific example. Figure 2B, the stream receiver 210 includes a container detector 212 and a stream parser 214. The container detector 212 is configured to detect the container type of the received stream data 205. For example, the received stream data 205 may include a data stream received over a network using a specific communication protocol or a data stream received from an operating system file system function. The container detector 212 can at least detect the container type of the stream data 205, for example, whether the stream data 205 includes a transport stream or another MPEG media container. The stream parser 214 receives the stream data 205 and the detected container type from the container detector 212. The stream parser 214 uses the detected container type to parse the metadata carried within the stream data 205 (for example, locate and extract the PAT for the transport stream or the track list for the MP4 file), and then instantiate a suitable decoder as described in more detail below.
[0045] Figure 2C Three subcomponents of the video decoder 220 are shown according to a specific example. The audio decoder 240 and other decoder types may also have a similar form. Figure 2C The video decoder 220 in TS 100 includes an elementary stream (ES) parser 222, an access unit (AU) generator 224, and a renderer 226. The elementary stream parsed by the ES parser 222 may be a packetized elementary stream (PES) or a non-packetized elementary stream. The ES parser 222 is configured to receive the output 216 of the stream receiver 210 and extract appropriate coded media data from the output 216. In one case, the ES parser 222 receives stream data 205 and a PID or track of interest from the stream parser 214. The PID of interest may include a specific program (i.e., a specific video content) that the user wishes to watch. In this case, the ES parser 222 extracts or demultiplexes the identified program or track data stream, for example, based on the identified PID or MP4 track. The ES parser 222 can therefore extract the PID stream 104 from the TS stream 102. In this case, the output 216 of the stream receiver 210 may include stream data 205, which is delivered to each decoder as a complete stream. In other cases, stream parser 214 may extract or demultiplex before ES parser 222 so that the output for a particular video decoder 220 includes a single identified program or track data stream. Thus, the exact location of extraction or demultiplexing may vary depending on different implementations.
[0046] Once a particular PID stream or track is identified, the ES parser 222 is configured to process the data of that stream or track to provide encoded media data to the AU generator 224. This may include extracting data from the payloads of the plurality of TS packets 110 to form a Figure 1The PES 106 shown. The PES 106 may then be passed to an AU generator 224. For a container "track", similar elementary streams may be generated from the extracted track data. The AU generator 224 is then configured to parse the coded media data at the access unit level. This may include extracting actual sample data for passing to a renderer 226 for reconstruction of the original signal data (such as decoded video frames). This may also include additional coding scheme specific parsing of the PES 106 or track, such as extracting and parsing coding scheme specific metadata. The coding scheme specific metadata may include one or more of sequence parameter set (SPS) data, picture parameter set (PPS) data (e.g., for H.264 video), or supplemental enhancement information (SEI) data. The coding scheme specific metadata may be "in-band", i.e., interleaved with the (media) sample data. In one case, the ... Figure 1 The media sample data is extracted in the form of a sequence of NAL units 108 as shown. This sequence of NAL units 108 may form an input to the renderer 226.
[0047] The renderer 226 is configured to decode the media sample data received from the AU generator 224 to produce decoded media 228 (e.g., frames, subtitles, or audio samples) ready for output, for example, on a display. The renderer 226 may include a specific codec or decoder, such as an H.264 or H.265 decoder (or an MP3 / 4 or Advanced Audio Coding—AAC decoder for audio data).
[0048] exist FIG. 2A to FIG. 2C In the example of FIG. 1 , each video decoder 220 is implemented as a specific decoding pipeline that receives a single demultiplexed stream or multiplexed data at a transport level or a container level, the latter being processed to extract a single demultiplexed stream. Each video decoder 220 includes a coding scheme specific implementation, since at least the AU generator 224 and the renderer 226 implement coding scheme specific parsing and decoding. Thus, in the comparative example, the stream receiver 210 can parse the PMT of the transport stream to enable extraction of the PMT data such as Figure 1 The multiple PES shown. The stream receiver 210 can create or instantiate a specific video decoder 220 to act as a PES parser based on the stream type and the parsed elementary stream data (e.g., as determined by the container detector 212 and the stream parser 214). In this case, the instantiated video decoder 220 reads the payload of the PES, for example, according to a specific decoding scheme, and the AU generator 224 of the video decoder 220 outputs (e.g., creates) a set of access units by parsing the sequence of NAL units. The access unit can then be supplied as an input data source for a specific renderer. This process can form the basis of a media player implemented on a specific client device.
[0049] exist Figures 1 to 2C In the context of a comparative example, Figures 3 to 6 Examples of improved systems and methods for decoding multi-layer video streams are provided. Figures 1 to 2C The comparative examples of SVC and SHVC are suitable for decoding single-layer media streams, such as conventional H.264 or H.265 data streams, but it is not straightforward to adapt such a framework to handle multi-layer data streams, especially multi-layer data streams where each layer may be encoded using a different decoding scheme. For example, SVC and SHVC are typically provided in the same form as their parent H.264 or H.265 single-layer schemes and are implemented by Figure 2A 220-A and another layer is processed by video decoder 220-B) because the efficiency of the scheme is based on SVC and SHVC specific layer internal information. Thus, SVC and SHVC streams need to be decoded by a single decoder unit that has access to this internal information.
[0050] With newer multi-layer video coding schemes such as LCEVC, it becomes possible to have a multi-layer coding scheme in which different layers within the coding scheme are encoded according to different coding standards. For example, an LCEVC data stream may include a lower resolution H.264, H.265, or Versatile Video Coding (VVC) "base" layer stream and an LCEVC-specific "enhancement layer" stream, where the LCEVC-specific "enhancement layer" stream may in turn include different sub-layers. Thus, the "base" layer stream may take a variety of forms and may include pre-existing and / or independent streams, for example, a video distributor may provide an LCEVC-specific "enhancement layer" stream "on top" of an existing and / or independent "base" layer stream. In LCEVC, the sub-layers include encoded residual data for application to the decoded output of the "base" layer stream.
[0051] Figure 3 An example 300 is shown that provides efficient decoding of a multi-layer video stream, where different layers may be encoded according to different coding formats. Figure 3 The example 300 can be considered as Figure 2A An adaptation or extension of Example 200. Figure 2A The components of the functions described are Figure 3 Corresponding reference numerals are provided in FIG. Figure 3 A first example is shown, where different layers in a multi-layer stream received as separate elementary streams can be combined at the decoding device to form a joint elementary stream that can be passed to a multi-layer decoder. This is one possible solution. Fig.11Other alternative solutions are described. In these latter cases, the joint elementary stream may be generated at the encoder or transmitter of the transport stream and sent as a single elementary stream (eg, with a PID for the first layer).
[0052] Steering Figure 3 An example of this is shown in the figure as reference Figure 2A The stream receiver 210 receives the stream data 205. Figure 3 In Figure 2A An audio decoding pipeline is shown implemented, for example, via audio decoder 240 and audio mixer 250. The audio decoding pipeline may be used to decode and output one or more audio streams or tracks corresponding to a decoded video stream. Figure 3 Also shown is the Figure 2A The decoded video data is displayed at the display output synthesizer 230. Figure 3 In example 300, Figure 2A The arrangement of the video decoder 220 in is modified to provide efficient decoding of multi-layer video streams. In this context, some existing single-layer video decoders 220 that are not used by multi-layer video streams can be used as shown in reference Figure 2A This minimizes changes to existing media players. However, in order to provide decoding of multi-layer video streams, Figure 3 Example 300 shows a stream generator 310 , a first layer decoder 322 , a second layer decoder 324 , and a multi-layer controller 326 . Figure 3 The components shown in may form part of a video decoder, such as a client device implementing a media player.
[0053] The stream receiver 210 in example 300 performs the same operation as reference Figure 2A and Figure 2B In this example 300, the stream receiver 210 coordinates the reception of the first packet substream and the second packet substream corresponding to the first layer and the second layer of the multi-layer video coding. For example, the first packet substream and the second packet substream may be received as a transport stream (such as Figure 1 or Figure 5 The first packet substream and the second packet substream may be multiplexed within the stream data 205. Figure 5 shows a graph with a first PID ( Figure 5 The first packet sub-stream 104 is in the form of a PID stream having a PID "B" in . Figure 5 Also shown is a second PID ( Figure 5 The stream receiver 210 may receive the second packet substream 504 in the form of a PID stream having a PID "L" in the stream. Figure 5 The two sub-streams multiplexed in the transport stream 102 are shown. Figure 1 , in this example and if Figure 5 As shown, each packet in the first packet substream and the second packet substream includes a header 112 and a data payload 114. The first packet stream is identified via a first packet identifier (e.g., PID "B") indicated in the header, and the second packet stream is identified via a second packet identifier (e.g., PID "L") indicated in the header. In this example, the identifiers "B" and "L" may refer to the "base" layer and "LCEVC" layer of the multi-layer stream, respectively. Although examples are presented herein with reference to multiple PID streams, in alternative examples, the stream receiver 210 may be adapted to extract multiple encoded data streams according to other methods, including based on tags or identifiers in the elementary streams.
[0054] exist Figure 3 In an example of , the stream generator 310 receives a first packet substream and a second packet substream and is configured to generate a joint elementary packet stream from the substreams. In one case, the stream generator 310 may receive the first packet substream and the second packet substream as separate packet substreams. In this case, it may be desirable to view the multi-layer stream indicated by the highest layer stream (e.g., the "L" PID stream 504). In this case, the user may select a program associated with the "L" PID stream 504. This may be identified as an "enhanced" version of the "B" PID stream 104. The stream receiver 210 may first determine the content type of the selected "L" PID stream 504 and extract metadata therefrom. The extracted metadata may then include a reference to the PID of the "B" PID stream 104 (e.g., the header data for the "L" PID stream 504 may identify the base PID as "B"). The stream receiver 210 may then be configured to also determine the content type of the determined "B" PID stream 104 and extract metadata therefrom. In another case, the user may select a program associated with the "L" PID stream 504, and that PID and the output from the stream receiver 210 for that stream may be passed to the stream generator 310 along with the transport stream. The stream generator 310 may then be configured to determine the lower layer (e.g., base) PID stream associated with the higher layer (e.g., enhancement or LCEVC) stream, and demultiplex the two PID streams accordingly. In both cases, the stream generator 310 ends up with two packet substreams in the form of "B" and "L" PID streams.
[0055] In this example 300, the stream generator 310 takes two packet substreams and generates a joint elementary packet stream that includes a sequence of packets that include data for both the first layer and the second layer. The joint elementary packet stream may include Figure 5508 is shown, where a PES payload 512 for a first layer (e.g., a "B" base layer) is extracted together with a PES payload 514 for a second layer (e.g., an "L" enhancement layer). In this case, the PES payload 512 for the first layer and the PES payload 514 for the second layer are arranged as a payload 516 for the PES joint stream 508. The PES joint stream 508 also includes a PES header 518. Thus, the PES joint stream 508 appears as a normal PES, such as Figure 1 PES106 in. Figure 5 As shown, the combined elementary packet stream can be Figure 1 The NALU stream 108 in is similarly composed of a sequence of NAL units 520; however, in the present case, the NAL units 520 include a first sequence of NAL units 522 for a first layer and a second sequence of NAL units 524 for a second layer (eg, concatenated therewith).
[0056] Return to Figure 3 , the stream generator 310 provides the same joint elementary packet stream to both the first layer decoder 322 and the second layer decoder 324. The stream generator 310 outputs the joint elementary packet stream in a normal elementary packet stream format (e.g., as PES). Thus, the first layer decoder 322 and the second layer decoder 324 can be configured to receive the same Figure 2A The video decoder 220 in FIG. 1 receives an input in the same format as that in FIG. 2 , only in the present case the decoder receives additional data in the form of a joint stream.
[0057] The first layer decoder 322 is configured to receive the joint elementary packet stream generated by the stream generator and output a decoded representation of the data for the first layer within the joint elementary packet stream. The first layer decoder 322 may include a normal single layer decoder (e.g., an H.264 or H.265 decoder). In this case, the NAL units for the second layer in the joint elementary packet stream (such as Figure 5The NAL units 524 in the header may include a NAL unit type field in the header of each NAL unit. The NAL unit type field may be set to a reserved value (such as 28 to 30), where the first layer decoder 322 may be configured to ignore NAL units within the reserved range (e.g., 28 and above). For example, a decoding device compliant with the H.264 or H.265 standard, etc., is configured to ignore NAL units containing header data outside of a specified value range defined in these standards. In other cases, other NAL unit fields may be set to indicate that the NAL unit 524 is to be ignored when processing the first layer. With respect to LCEVC, further details may be found in Section 7.4.2.2 of the LCEVC standard (ISO / IEC FDIS23094-2:2021), which is incorporated herein by reference. Thus, the first layer decoder 322 uses the same Figure 1 The data including the first sequence of NAL units 522 is processed in a similar manner to the original sequence of NAL units 108 shown in FIG. Decoding the data for the first layer may include a decoded picture or frame for the first layer, such as an H.264, H.265, or VVC decoded frame.
[0058] The second layer decoder 324 also receives the joint elementary packet stream generated by the stream generator 310. The second layer decoder 324 decodes data for the second layer, for example, data contained in the NAL unit 524, to output a decoded version of the data for the second layer within the joint elementary packet stream. In the examples described herein, the second layer decoder 324 is different from the first layer decoder 322. For example, the first layer decoder 322 may be a legacy hardware and / or software video decoder that complies with a first video coding standard (e.g., H.264, H.265, VVC, etc.), and the second layer decoder 324 may be an enhanced hardware and / or software video decoder that complies with a second, different video coding standard (e.g., LCEVC). The second layer decoder 324 may decode a residual signal, while the first layer decoder 322 may decode a video (non-residual) signal. The residual signal may include multiple sub-layers representing different quality levels (e.g., different spatial resolutions).
[0059] In this example, the multi-layer controller 326 is communicatively coupled to the first layer decoder 322 and the second layer decoder 324. The multi-layer controller 326 is configured to combine the output of the first layer decoder 322 and the output of the second layer decoder 324 to provide a multi-layer reconstruction of the video signal. Although shown as separate components, in some implementations, the multi-layer controller 326 may form a part of the second layer decoder 324. For example, the enhancement decoder may include a second layer decoder in the form of a residual decoder and a controller that applies the decoded residual data to the decoded video frame from the first layer decoder 322. The multi-layer controller 326 may directly or indirectly receive the output of the first layer decoder 322 and the second layer decoder 324. In the latter case, the multi-layer controller 326 may have access to a shared memory, where the decoded output of one or more of the first layer decoder 322 and the second layer decoder 324 is available. The shared memory may include a frame buffer that accommodates the one or more frames when the one or more frames are decoded. In the case of LCEVC, the first layer decoder 322 may include a base decoder that outputs a lower quality picture or frame (e.g., a lower resolution frame), and the second layer decoder 324 may include an LCEVC decoder that receives, for example, residual data for a higher quality picture or frame at a higher resolution and decodes it. In this case, the multi-layer controller 326 may be configured to upsample the output of the first layer decoder 322 and apply one or more sub-layers of the residual data. In one configuration, the multi-layer controller 326 may apply a first sub-layer of the decoded residual data to the output of the first layer decoder 322, upsample the result, and then apply a second sub-layer of the decoded residual data before outputting a multi-layer reconstruction of a video signal at an upsampled resolution. In LCEVC, the sub-layers and upsampling operations may be flexibly configured, and therefore different multi-layer reconstruction configurations are possible. Typically, multiple sub-layers of the second layer may be decoded in parallel by a common (e.g., a single) second layer decoder.
[0060] exist Figure 3 In example 300, according to Figure 2A The first layer decoder 322 provides a decoded output to the display output compositor 230. The output may be at a lower quality level, for example, a video data frame at a reduced resolution or encoding quality (e.g., high quantization for a lower bit rate). However, according to Figure 2A The video is decoded as shown in FIG. 1 , and it is still viewable. The accompanying audio stream may also be decoded by the audio decoder 240 and output by the audio mixer 250, as shown in FIG. Figure 2A In Figure 3In the embodiment of the present invention, the multi-layer reconstruction output by the multi-layer controller 326 is also available to the display output compositor 230. The multi-layer reconstruction can be an improved or enhanced version of the output of the first layer decoder 322. For example, it can include a higher resolution version of the output or a version with finer details and fewer compression artifacts. Thus, the multi-layer reconstruction can be viewed on a display device as a sequence of decoded frames.
[0061] In some examples, the stream receiver 210 may include a demultiplexer to receive and demultiplex a multiplexed transport stream including a first packet substream and a second packet substream, such as the transport stream 102. The data payload of the first packet substream may form a first packetized elementary stream (i.e., a first PES), and the data payload of the second packet substream may form a second packetized elementary stream (i.e., a second PES), wherein a joint elementary packet stream is generated including a third packetized elementary stream (i.e., a joint PES), the third packetized elementary stream having a header including a presentation timestamp, and the data payload after the header including the data payload from the first packetized elementary stream and the second packetized elementary stream associated with the presentation timestamp. In some cases, the first packet substream and the second packet substream may be sent so that the data payload of the second packet substream arrives no later than the data payload of the corresponding portion of the first packet substream. This may assist in synchronizing the two PID streams.
[0062] Figure 4 Shows Figure 2C Another example 400 of how components of may be adapted to allow efficient decoding of multi-layer video streams. Figure 4 An elementary stream (ES) parser union 410 is shown supplying stream data to a first decoder 420 and a second decoder 430. According to the example 300, in Figure 4 In the example 400 of , a solution with a dual PID combiner is adopted. The ES parser combiner 410 is fed with multiple PES or PID streams and outputs a single PES whose payload includes the concatenated payloads of the multiple streams. This single PES is supplied to a downstream decoding chain of components including a first decoder 420 and a second decoder 430. Both decoders include a serially connected AU generator and renderer, similar to Figure 2C In this case, the ES parser union 410 replaces the separate ES parsers 222 of the respective video decoders. The first decoder 420 includes a first AU generator 424 and a first renderer 426. The second decoder 430 includes a second AU generator 434 and a second renderer 436. The first decoder 420 may be reused. Figure 2C The AU generator 224 and renderer 226 shown in FIG. 2 provide the AU generator 224 and renderer 226 shown in FIG. Figure 2CThe decoded video output (e.g., decoded video frame) is described. In one case, the first AU generator 424 receives a single PES with a joint payload and processes the payload data to generate an access unit from a sequence of NAL units. Since the first AU generator 424 can be configured based on a legacy or known video decoding standard, it can ignore the following when generating access units: Figure 5 A second sequence of NAL units 524 is shown. In short, the first AU generator 424 may behave in the same manner as if the payload originally came from a single PID stream.
[0063] exist Figure 4 , the second decoder 430 is then configured to decode the enhancement of the output of the first decoder 420. In this example, the second decoder 430 incorporates the features of the multi-layer controller 326 as part of the renderer 436. The second AU generator 434 is configured to generate an access unit including a second sequence of NAL units 524 for supply to the second renderer 436 for decoding. The second renderer 436 receives the output of the first decoder 420, decodes the enhancement carried within the access unit, and then combines the output and the enhancement to output an enhanced video for display. Therefore, the first renderer 426 can form a first layer renderer to render the output of the first layer decoder on a display device, and the second renderer 436 can form a multi-layer renderer to render a multi-layer reconstruction on a display device.
[0064] As mentioned above, Figure 5 The process of combining two PID streams or the first PES and the second PES is shown. Figure 1 A multiplexed transport stream 102 is shown. The transport stream 102 includes a plurality of different PID sub-streams, including a base "B" PID stream 104 and an enhancement or LCEVC "L" stream 504. Figure 5 The two PID streams 104, 504 are shown after demultiplexing. Figure 5 Also shown is the result of processing two PID streams to generate a single PES (i.e., a single packetized elementary stream). This may include combining the payloads of multiple TS packets with a common PTS value (in Figure 5 The PTS may also be used to synchronize two PID streams, for example, the PES headers shown for the "B" PES 106 and the "L" PES 506 both have corresponding PTS values. Figure 5The generated joint elementary packet stream (e.g., joint PES) is then shown with a PES header 518 from PID "B" and a payload 512 from PID "B", followed by a payload 514 of PID "L". This joint elementary packet stream 508 forms an access unit for two layers for multi-layer coding, with a first sequence of NAL units 522 containing first layer coded data and a second sequence of NAL units 524 containing second layer coded data.
[0065] Figure 6 A method 600 for decoding a multi-layer video stream according to an example is shown. The method 600 may be performed by Figure 3 or Figure 4 The system 300 or 400 of the present invention or is implemented on another decoding system. In this example, the multi-layer video stream encodes the video signal and includes at least a first layer and a second layer. The first layer can be a normal single-layer video stream, for example, an H.264, H.265 or VVC stream at a predefined resolution and / or bit rate. The second layer may include an enhancement layer encoded using a different coding method (such as layered residual coding). The second layer may include an LCEVC stream.
[0066] At block 602, method 600 includes receiving a first packet substream for a first layer. The first packet substream may be a PID stream or a PES, such as Figure 5 The first packet substream may be a PID stream 104 or a PES 106 in the first packet substream. If the first packet substream is a PID stream, each packet in the first packet substream includes a header and a data payload. The first packet substream is identified via a first packet identifier (e.g., PID) indicated in the header. The first packet substream may be received as part of a digital multimedia container (such as a transport stream).
[0067] At block 604, method 600 includes receiving a second packet substream for a second layer. The second packet substream may be a PID stream or a PES, such as Figure 5 504 or PES 506 in the header. If the second packet substream is a PID stream, each packet in the second packet substream includes a header and a data payload. The second packet substream is identified via a second packet identifier (e.g., PID) indicated in the header. The second packet substream can be received as part of a digital multimedia container (such as a transport stream). The digital multimedia container can be the same as or different from the digital multimedia container containing the first packet substream. In one case, both the first packet substream and the second packet substream are received as part of a transport stream. Figure 5 Blocks 602 and 604 may be performed by one or more of the stream receiver 210, stream generator 310, and ES parser union 410 of the previous examples.
[0068] At block 606, packets from the first packet substream and the second packet substream are joined to generate a joint elementary packet stream. The joint elementary packet stream includes a sequence of packets that include data for both the first layer and the second layer. These packets may be NAL units, such as Figure 5 NAL units 522 and 524 in. They can be encapsulated in a single PES, such as Figure 5 PES 508 in.
[0069] At block 608, the joint elementary packet stream is provided to a first layer decoder for decoding data for the first layer within the joint elementary packet stream. This may include decoding the data for the first layer using an H.264, H.265, or VVC decoder. The first layer decoder may ignore packets including data for the second layer. The first layer decoder may include Figure 3 The first layer decoder 322 or Figure 4 The first decoder 420.
[0070] At block 610, the joint elementary packet stream is also provided to a second layer decoder for decoding at least the data for the second layer within the joint elementary packet stream. The second layer decoder may include Figure 3 The second layer decoder 324 or Figure 4 The second decoder 430 of FIG. 608 and FIG. 610 may be performed in parallel. In a preferred example, the first layer decoder is different from the second layer decoder. For example, the second layer decoder may include an LCEVC decoder.
[0071] At block 612, method 600 includes combining the output of the first layer decoder and the output of the second layer decoder to provide a multi-layer reconstruction of the video signal. For example, this may be accomplished by Figure 3 Multilayer controller 326 or Figure 4 The multi-layer reconstruction may include an enhanced or augmented version of the reconstruction output by the first layer decoder at block 608. The method may be repeated frame by frame, where different color planes may be processed serially or in parallel per frame. The result may be an enhanced or augmented video rendered on a display for viewing.
[0072] In one case, the data payload of the first packet substream forms a first packetized elementary stream (i.e., the first PES), and the data payload of the second packet substream forms a second packetized elementary stream (i.e., the second PES). Therefore, the joint elementary packet stream includes a third packetized elementary stream (i.e., the third PES). The third packetized elementary stream has a header including a presentation timestamp (PTS), and the data payload after the header (i.e., the payload of the third PES) includes data payloads from the first packetized elementary stream and the second packetized elementary stream associated with the presentation timestamp. In this case, the presentation timestamp is used to synchronize data for a particular picture or frame. In one case, the first packet substream and the second packet substream can be sent so that the last packet in the second packet substream arrives no later than the last packet in the first packet substream. In this way, data for all layers of multi-layer coding for a given picture or frame is available at the decoder to be synchronized, decoded and combined as described.
[0073] The payload of the joint elementary packet stream may include a sequence of network abstraction layer (NAL) units for the first layer and a sequence of NAL units for the second layer. The first layer decoder may be configured to ignore the network abstraction layer units for the second layer based on a unit type data value within a header of the network abstraction layer units for the second layer.
[0074] The output of the first layer decoder can be rendered independently of the multi-layer reconstruction of the video signal. For example, the output of the first layer decoder and the multi-layer reconstruction of the video signal can be viewed. In some cases, the displayed video rendering can switch between the two outputs based on the availability of the second packet substream. The first layer can include a "base" video stream, and the second layer can include a corresponding "enhanced" video stream. The base video stream can have a first encoding parameter, and the enhanced video stream can have a second encoding parameter. In one case, the multi-layer reconstruction of the video signal includes a higher quality reproduction of the base video signal decoded from the base video stream. For example, the first packet substream can represent a video encoding at a first resolution, such as high-definition (HD) H.264 encoding, and the second packet substream can represent an enhanced encoding at a second higher resolution, such as ultra-high-definition (UHD) LCEVC encoding. In addition to different resolutions, the two packet substreams can also represent encodings at one or more of the following: different bit rates, different color depths, different quantization configurations, and different bit depths.
[0075] In one case, the method 600 further includes receiving a multiplexed transport stream including a first packet substream and a second packet substream, and demultiplexing the multiplexed transport stream to extract the first packet substream and the second packet substream. For example, this can be performed by one of the stream receiver 210, the stream generator 310, and the ES parser union 410 of the previous example.
[0076] In some examples, the data for the second layer includes frames of residual data combined with frames of a base video signal such as decoded from a base video stream. For example, the second layer may include a low complexity enhanced video coding (LCEVC) video stream. In some cases, the second layer may include a watermark stream, for example, a stream with data to be added to the original video stream to mark, identify or protect the original video stream visibly or invisibly. In this case, the second layer may include data combined with the original video stream, but wherein the data does not include residual data. The second layer may also include a metadata stream accompanying the original first layer video stream. For example, the second layer may include localized metadata associated with an object or person within the original first layer video stream, such as a unique identifier or hyperlink to a data source. There may also be multiple base layers and / or multiple enhancement layers as part of a multi-layer video stream, and these layers may be processed similarly to the single base example and the single enhancement example described herein.
[0077] In one case, the method 600 described herein can be applied as part of an adapted media player implementation. Frames 602 and 604 can be implemented by a PMT reader or parser that extracts an elementary stream (ES) from a digital multimedia container (such as a transport stream). For example, the transport stream can be detected and the PMT can be parsed to extract the mapping of a directory or program to a set of PIDs. Frames 602 and 604 can also involve a transport stream extractor that creates a PES reader for one or more of the identified PID streams. The transport stream extractor can perform functions similar to stream receiver 210 or stream generator 310. A PES reader can be generated based on stream type and stream information (e.g., PES information). The PID for each ES / PES can also be extracted and passed to the corresponding PES reader. In the case of enhanced decoding, if the corresponding basic PID is signaled, a PES reader for an elementary stream (e.g., with a basic PID) can be shared between a basic decoder and an enhanced decoder. When reading ES metadata, if a reference base PID is signaled, the ES for that base is sent to the "base" PES reader. In the current case, all PES readers can act as consumers with a combiner interface that can provide a joint elementary packet stream. Each PES reader may be unaware of the join and see a sequence of NAL units. The combiner interface can provide data to a specific data consumer, such as a reader (i.e., decoder) for a specific video format. The combiner interface can be provided by an ES parser that is fed by two inputs based on two PID streams, but based on a comparison of ES parsers such as Figure 2C 222 in has a single output. This can be achieved using an internal consumer with a combiner interface as an output. The combiner interface can then output to the PES reader consumer, just like in the single PID case.
[0078] In some cases, the NAL units of the second layer may be interleaved with the NAL units of the first layer. In this case, the second layer decoder or the second layer preprocessor may parse the sequence of NAL units to extract the NAL units of the second layer from the joint stream.
[0079] In some cases, the joint stream may only need data for the first layer before release. In this case, if the second layer data exists, it can be added, but if it does not exist (for example, all or part), it can be omitted and only the first layer is decoded and viewed. Therefore, the second layer decoder can skip enhancement and / or provide a pass-through of the first layer output. Therefore, the enhancement layer can be flexibly added to the base layer. The current example allows the easy transformation of the existing stream processing pipeline to manage multi-layer streams. For example, each video decoder expects a single PES stream as input, and therefore a common interface can be provided, regardless of whether a single layer stream or a multi-layer stream is being decoded. The advanced logic for the upper layer of the multi-layer stream can therefore be incorporated into the second layer decoder and / or multi-layer controller that can be easily added to the existing options for stream parsing and decoding.
[0080] In some cases, one or more of the example systems 300 and 400 or the method 600 may be implemented via instructions retrieved from a computer-readable medium and executed by a processor of a decoding system, such as a client device.
[0081] Some general information related to an example enhancement coding scheme will now be described. This information provides examples of specific multi-layer coding schemes.
[0082] It should be noted that examples are presented herein with reference to a signal being a sequence of samples (i.e., a two-dimensional image, a video frame, a video field, a sound frame, etc.). For simplicity, the non-limiting examples described herein generally refer to a signal displayed as a set of 2D planes (e.g., a 2D image in a suitable color space), such as, for example, a video signal. In a preferred case, the signal comprises a video signal. Reference Fig. 9 An example video signal is described in more detail.
[0083] The terms "picture", "frame" or "field" are used interchangeably with the term "image" to indicate temporal samples of a video signal: any concepts and methods described for a video signal consisting of frames (progressive video signals) can also be easily applied to a video signal consisting of fields (interlaced video signals), and vice versa. Although the examples described herein focus on images and video signals, a person skilled in the art can easily understand that the same concepts and methods are also applicable to any other type of multi-dimensional signal (e.g., audio signals, volumetric signals, stereoscopic video signals, 3DoF / 6DoF video signals, plenoptic signals, point clouds, etc.). Although image or video encoding examples are provided, the same methods can be applied to signals with dimensions less than two (e.g., audio or sensor streams) or greater than two (e.g., volumetric signals).
[0084] In the specification, the terms "image", "picture" or "plane" (meaning the broadest sense of "hyperplane", i.e., an array of elements with arbitrary dimensions and a given sampling grid) will generally be used to identify a digital reproduction of signal samples along a sequence of samples, wherein each plane has a given resolution for each of its dimensions (e.g., X and Y) and comprises a set of plane elements (or "elements" or "pixels" or display elements commonly referred to as "pixels" for two-dimensional images, display elements commonly referred to as "voxels" for volumetric images, etc.) characterized by one or more "values" or "settings" (e.g., by way of non-limiting example, a color setting in a suitable color space, a setting indicating a density level, a setting indicating a temperature level, a setting indicating an audio pitch, a setting indicating an amplitude, a setting indicating a depth, a setting indicating an alpha channel transparency level, etc.). Each plane element is identified by a set of suitable coordinates indicating the integer position of the element in the sampling grid of the image. Signal dimensions may comprise only spatial dimensions (e.g., in the case of images) or temporal dimensions (e.g., in the case of signals evolving over time, such as video signals). In one case, it can be seen that a frame of a video signal includes a two-dimensional array having three color component channels, or a three-dimensional array having two spatial dimensions (e.g., spatial dimensions indicating resolution - lengths equal to the respective width and height of the frame) and one color component size (e.g., length 3). In some cases, the processing described herein is performed separately for each plane of color component values that make up the frame. For example, planes representing pixel values for each of the Y, U, and V color components can be processed in parallel using the methods described herein.
[0085] Certain examples described herein use a scalability framework that uses base encoding and enhancement encoding. The video decoding system described herein operates on a received decoding of a base encoding (e.g., frame-by-frame or complete base encoding) and adds one or more of spatial, temporal or other quality enhancements via enhancement layers. The base encoding can be generated by a base layer, which can use a different decoding scheme than the enhancement layer, and in some cases can include a legacy or comparative (e.g., older) coding standard.
[0086] Figures 7 to 9 A spatially scalable coding scheme is shown that uses a downsampled source signal encoded with a base codec, adds a first correction or enhancement layer data to the decoded output of the base codec to generate a corrected picture, and then adds another correction or enhancement layer data to an upsampled version of the corrected picture. Thus, the spatially scalable coding scheme can generate an enhancement stream with two spatial resolutions (higher and lower) that can be combined with the base stream at the lower spatial resolution.
[0087] In a spatially scalable coding scheme, the method and apparatus may be based on an overall algorithm built from existing encoding and / or decoding algorithms (e.g., MPEG standards such as AVC / H.264, HEVC / H.265, and non-standard algorithms such as VP9, AV1, etc.) that are used as a baseline for enhancement layers. The enhancement layers operate according to different encoding and / or decoding algorithms. The idea behind the overall algorithm is to encode / decode video frames in a hierarchical manner, as opposed to using a block-based approach as done in the MPEG family of algorithms. Encoding frames in a hierarchical manner includes generating residuals for full frames, and then generating residuals for reduced or decimated frames, etc.
[0088] Figure 7 The system configuration of the example spatial scalable coding system 700 is shown. The coding process is split into two halves as shown by the dotted lines. Each half can be implemented separately. Below the dotted line is the base level, and above the dotted line is the enhancement level, which can be usefully implemented in software. The coding system 700 can include only the enhancement level process, or a combination of the base level process and the enhancement level process as needed. The topology of the coding system 700 at the general level is as follows. The coding system 700 includes an input I for receiving an input signal 701. The input I is connected to a downsampler 705D. The downsampler 705D outputs to the base encoder 720E at the base level of the coding system 700. The downsampler 705D also outputs to the residual generator 710-S. The encoded base stream is directly created by the base encoder 720E and can be quantized and entropy encoded as needed according to the base coding scheme. The encoded base stream can be the base layer as described above, for example, the lowest layer in the multi-layer decoding scheme.
[0089] Above the dotted line is a series of enhancement layer processes that generate an enhancement layer of a multi-layer decoding scheme. In this example, the enhancement layer includes two sub-layers. In another example, one or more sub-layers may be provided. Figure 7In order to generate a coded sublayer 1 enhancement stream, the coded base stream is decoded via a decoding operation applied at a base decoder 720D. In a preferred example, the base decoder 720D can be a decoding component that supplements the encoding component in the form of a base encoder 720E within the base codec. In other examples, the base decoding block 720D can instead be part of the enhancement layer. Via a residual generator 710-S, a difference between the decoded base stream output from the base decoder 720D and the downsampled input video is created (i.e., a subtraction operation 710-S is applied to the frames of the downsampled input video and the frames of the decoded base stream to generate a first set of residuals). Here, the residual represents the error or difference between the reference signal or frame and the desired signal or frame. The residuals used in the first enhancement layer can be regarded as correction signals because they are able to 'correct' future frames of the decoded base stream. This is useful because it can correct for quirks or other characteristics of the base codec. These characteristics include, among other things, the motion compensation algorithm applied by the base codec, the quantization and entropy coding applied by the base codec, and the block adjustment applied by the base codec.
[0090] exist Figure 7 In , the first set of residuals is transformed, quantized, and entropy coded to produce a coded enhancement layer sub-layer 1 stream. Figure 7 , a transform operation 710-1 is applied to a first set of residuals; a quantization operation 720-1 is applied to the transformed set of residuals to generate a set of quantized residuals; and an entropy coding operation 730-1 is applied to the quantized set of residuals to generate an encoded enhancement layer sublayer 1 stream (e.g., at a first enhancement level). However, it should be noted that in other examples, only the quantization step 720-1, or only the transform step 710-1 may be performed. Entropy coding may not be used, or may optionally be used in addition to one or both of the transform step 710-1 and the quantization step 720-1. The entropy coding operation may be any suitable type of entropy coding, such as a Huffmann coding operation or a run length encoding (RLE) operation, or a combination of both a Huffmann coding operation and an RLE operation (e.g., RLE followed by Huffmann or prefix coding).
[0091] To generate the coded enhancement layer sublayer 2 stream, another enhancement level information is created by generating and encoding another set of residuals via a residual generator 700-S. The other set of residuals is the difference between an upsampled version (via upsampler 705U) of the corrected version of the decoded base stream (reference signal or frame) and the input signal 701 (desired signal or frame).
[0092] To achieve this as will be done in a decoder (e.g. Figure 8700) reconstructs a corrected version of the decoded base stream generated at the decoder, undoing at least some of the sub-layer 1 encoding operations to simulate the process of the decoder, and taking into account at least some of the losses and quirks of the transform and quantization processes. To this end, the first set of residuals is processed by a decoding pipeline including an inverse quantization block 720-1i and an inverse transform block 710-1i. The quantized first set of residuals is inversely quantized at the inverse quantization block 720-1i in the encoding system 700 and inversely transformed at the inverse transform block 710-1i to regenerate a decoder-side version of the first set of residuals. The decoded base stream from the decoder 720D is then combined with the decoder-side version of the first set of residuals (i.e., the decoder-side version of the decoded base stream and the first set of residuals is summed 710-C). The summation 710-C generates a reconstruction of a downsampled version of the input video that is likely to be generated at the decoder - that is, the reconstructed base codec video). The reconstructed base codec video is then upsampled by an upsampler 705U. The processing in this example is typically performed frame by frame. Each color component of the frame may be processed in parallel or serially as shown.
[0093] The upsampled signal (i.e., the reference signal or frame) is then compared to the input signal 701 (i.e., the desired signal or frame) to create another set of residuals (i.e., the difference operation is applied by the residual generator 700-S to the upsampled recreated frame to generate another set of residuals). The other set of residuals is then processed through a coding pipeline that is a mirror image of the first set of residuals to become an encoded enhancement layer sublayer 2 stream (i.e., the coding operation is then applied to the other set of residuals to generate another encoded enhancement stream). In particular, the other set of residuals is transformed (i.e., the other set of residuals is subjected to a transform operation 710-0 to generate another transformed set of residuals). The transformed residuals are then quantized and entropy encoded in the manner described above for the first set of residuals (i.e., a quantization operation 720-0 is applied to the transformed set of residuals to generate another set of quantized residuals; and, an entropy coding operation 730-0 is applied to the quantized other set of residuals to generate an encoded enhancement layer sublayer 2 stream containing another enhancement level information). In some cases, the operation may be controlled, for example, so that only the quantization step 720-1, or only the transform and quantization steps may be performed. Optionally, entropy coding may be used as a supplement. Preferably, the entropy coding operation may be a Huffman coding operation or a run length encoding (RLE) operation or both (e.g., RLE followed by Huffman coding). The transform applied to blocks 710-1 and 710-0 may be a Hadamard transform applied to a 2×2 or 4×4 residual block.
[0094] Figure 7 The encoding operation in does not result in dependencies between local blocks of the input signal (e.g., in contrast to many known coding schemes that apply inter-frame or intra-frame prediction to macroblocks and thus introduce macroblock dependencies). Figure 7 The operations shown in can be performed in parallel on 4×4 or 2×2 blocks, which greatly improves the encoding efficiency on a multi-core central processing unit (CPU) or graphics processing unit (GPU).
[0095] like Figure 7 As shown, the output of the spatial scalable encoding process is one or more enhancement streams for the enhancement layer, which preferably includes a first enhancement level and another enhancement level. This can then be combined with the base stream at the base level (e.g., via multiplexing or other means), for example, into the transport stream 102 as described above or as multiple tracks within another digital container. The first enhancement level (sublayer 1) can be considered to implement the corrected video at the base level, that is, for example, correcting the encoder quirks. The second enhancement level (sublayer 2) can be considered to be another enhancement level that is used to convert the corrected video to the original input video or an approximation close to it. For example, the enhancement of the second level can add fine details lost during downsampling and / or help correct errors introduced by one or more of the transform operation 710-1 and the quantization operation 720-1.
[0096] Figure 8 A corresponding example decoding system 800 for an example spatially scalable coding scheme is shown. The enhancement layer processing shown above the dashed line may be implemented by a second layer decoder as described herein, for example, Figure 3 The second layer decoder 324 and the multi-layer controller 326 or Figure 4 The base layer processing shown below the dashed line may be implemented by a first layer decoder as described herein, for example, Figure 3 The first layer decoder 324 or Figure 4 The first decoder 420.
[0097] exist Figure 8 , the encoded base stream is decoded at base decoder 820 in order to produce a base reconstruction of the input signal 701. This base reconstruction may be used in practice to provide a viewable reproduction of the signal 701 at a lower quality level. However, the main purpose of this base reconstruction signal is to provide a basis for a higher quality reproduction of the input signal 701. To this end, the decoded base stream is provided for enhancement layer sub-layer 1 processing (i.e., sub-layer 1 decoding). Figure 8The sub-layer 1 processing in includes an entropy decoding process 830-1, an inverse quantization process 820-1, and an inverse transform process 810-1. Optionally, only one or more of these steps may be performed depending on the operations performed at the corresponding block 700-1 at the encoder. By performing these corresponding steps, a decoded enhancement layer sub-layer 1 stream including a first set of residuals is made available at the decoding system 800. The first set of residuals is combined with the decoded base stream from the base decoder 820 (i.e., a summation operation 810-C is performed on the frames of the decoded base stream and the frames of the decoded first set of residuals to generate a reconstruction of a downsampled version of the input video - i.e., the reconstructed base codec video). The frames of the reconstructed base codec video are then upsampled by an upsampler 805U.
[0098] In addition, and optionally in parallel, the encoded enhancement layer sublayer 2 stream is processed to produce another set of decoded residuals. Similar to the sublayer 1 processing, the enhancement layer sublayer 2 processing includes an entropy decoding process 830-0, an inverse quantization process 820-0, and an inverse transform process 810-0. Of course, these operations will correspond to those performed at block 700-0 in the encoding system 700, and one or more of these steps may be omitted as desired. Block 800-0 produces a decoded enhancement layer sublayer 2 stream including another set of residuals, and these are summed at operation 800-C with the output from the upsampler 805U to create an enhancement layer sublayer 2 reconstruction of the input signal 701, which can be provided as an output of the decoding system 800. Thus, as Figure 7 and Figure 8 As shown, the output of the decoding process may include up to three outputs: a base reconstruction, a corrected lower resolution signal, and a reconstruction of the original signal at a higher resolution for a multi-layer coding scheme.
[0099] refer to Figure 3 300, the residual "decoding" process 710, 720, 730 may be performed by the second layer decoder 324, where the two sub-layers are provided as part of the enhancement layer NAL unit.
[0100] Typically, the examples described herein operate within an encoding and decoding pipeline that includes at least a transform operation. The transform operation may include a DCT or a variant of the DCT, a fast Fourier transform (FFT), or in a preferred example, a Hadamard transform as implemented by LCEVC. The transform operation may be applied block by block. For example, the input signal may be divided into a plurality of different continuous signal portions or blocks, and the transform operation may include a matrix multiplication (i.e., a linear transform) applied to the data from each of these blocks (e.g., as shown in a 1-dimensional vector). In this specification and in the art, it may be said that the transform operation produces a set of values of a predefined number of data elements, which, for example, represent positions in a transformed composite vector. These data elements are referred to as transform coefficients (or sometimes simply referred to as "coefficients").
[0101] As described herein, in the case where the enhancement data includes residual data, the reconstructed set of coefficient bits may include transformed residual data, and the decoding method may further include directing the combination of the residual data obtained from further decoding the reconstructed set of coefficient bits with a reconstruction of the input signal generated by a representation of the input signal at a lower quality level to generate a reconstruction of the input signal at a first quality level. The representation of the input signal at the lower quality level may be a decoded base signal, and the decoded base signal may be optionally upgraded before being combined with the residual data obtained from further decoding the reconstructed set of coefficient bits, the residual data being at a first quality level (e.g., a first resolution). Decoding may further include receiving and decoding the residual data associated with the second sublayer, for example, obtaining the output of the inverse transform and inverse quantization component and combining it with data derived from the aforementioned reconstruction of the input signal at the first quality level. The data may include data derived from an upgraded version of the reconstruction of the input signal at the first quality level, i.e., upgraded to the second quality level.
[0102] Further details and examples of the two sub-layer enhancement encoding and decoding systems can be obtained from the published LCEVC documents. Although the examples have been described with reference to a layer-based layered decoding scheme in the form of LCEVC, the methods described herein may also be applied to other layer-based layered decoding schemes, such as VC-6 as described in PCT / GB2018 / 053552: SMPTE VC-6ST-2117 and / or associated published standard documents, both of which are incorporated herein by reference.
[0103] Fig. 9 An example 900 is shown of how a video signal can be decomposed into different components and then encoded. Fig. 9In the example of , a video signal 902 is encoded. The video signal 902 includes a plurality of frames or pictures 904, for example, where the plurality of frames represent an action over time. In this example, each frame 904 is composed of three color components. The color components can be in any known color space. Fig. 9 In the embodiment of the present invention, the three color components 906 are Y (luminance), U (first chrominance opponent color) and V (second chrominance opponent color). Each color component can be considered as a plane of values 908. The plane 908 can be decomposed into a set of n×n signal data blocks 910. For example, in LCEVC, n can be 2 or 4; in other video decoding technologies, n can be 8 to 32.
[0104] In LCEVC and certain other decoding techniques, the video signal fed into the base layer is a degraded version of the input video signal (e.g., 701). In this case, the signals fed to the two sublayers of the enhancement layer include a residual signal, which includes residual data. The residual data plane can also be organized into groups of n×n signal data blocks 910. Residual data can be generated by comparing data derived from an encoded input signal (e.g., video signal 701) and data derived from a reconstruction of the input signal, the reconstruction of the input signal being generated by a representation of the input signal at a lower quality level. The comparison can include subtracting the reconstruction from the downsampled version. The comparison can be performed on a frame-by-frame (and / or block-by-block) basis. The comparison can be made at a first quality level; if the base quality level is lower than the first quality level, the reconstruction from the base quality level can be upgraded before the comparison. In a similar manner, the input signal of the second sublayer, such as the input to the second sublayer transform and quantization component, may include residual data generated by a comparison of the input video signal 701 at the second quality level (which may include a full quality original version of the video signal) and a reconstruction of the video signal at the second quality level. As previously described, the comparison may be performed on a frame-by-frame (and / or block-by-block) basis and may include subtraction. The reconstruction of the video signal may include a reconstruction generated by a decoded version of the decoded coded base bit stream and the first sublayer residual data stream. The reconstruction may be generated at a first quality level and may be upsampled to a second quality level.
[0105] Thus, the data plane 908 for the first sublayer may include residual data arranged in n×n signal blocks 910 . Fig. 9 Such a 2×2 signal block is shown in more detail in FIG. 9 (for ease of explanation, n is chosen to be 2), where for the color plane, the block may have a value 912 with a set bit length (e.g., 8 or 16 bits). Each n×n signal block may be represented as a length of n 2The flattened vector 914 represents the signal data block. To perform the transform operation, the flattened vector 914 can be multiplied by the transform matrix 916 (i.e., the dot product is taken). This then generates a length n 2 Another vector 918 representing different transform coefficients for a given signal block 910. Fig. 9 An example similar to LCEVC is shown, where the transform matrix 916 is a Hadamard matrix of size 4×4, resulting in a transform coefficient vector 918 having four elements, each with its own value. These elements are sometimes referred to by the letters A, H, V, and D, because they can represent the average, horizontal difference, vertical difference, and diagonal difference. This transform operation may also be referred to as directional decomposition. When n=4, the transform operation may use a 16×16 matrix and is referred to as directional decomposition squared.
[0106] like Fig. 9 As shown, a set of values for each data element in the entire group of signal blocks 910 of the plane 908 can itself be represented as a plane or surface 920 of coefficient values. For example, the values of the "H" data elements of the signal block group can be combined into a single plane, where the original plane 908 is then represented as four separate coefficient planes 922. For example, the coefficient plane 922 shown contains all "H" values. These values are stored in a predefined bit length (e.g., bit length B), which can be 8, 16, 32 or 64 depending on the bit depth. A 16-bit example is given below, but this is not limiting. Thus, the coefficient plane 922 can be represented as a sequence of 16-bit or 2-byte values 924 (e.g., in memory), which represents the value of one data element from the transformed coefficient. These values can be referred to as coefficient bits. These coefficient bits can be quantized and then entropy encoded as discussed to then generate the encoded enhancement layer data (e.g., "L" PID stream 504) as described above.
[0107] In some cases, for LCEVC video streams, for example, as referenced above Figures 6 to 9 As described, the PTS may be present in the PES packet header. In one case, the PTS timestamp may refer only to one LCEVC access unit starting in the PES packet. In some cases, the decoding timestamp (DTS) may not be present in the PES packet header because LCEVC decoding is performed in presentation order.
[0108] A set of additional examples will now be described. These examples operate within a similar context to the examples set forth above, but differ in certain respects. These examples may or may not be similar to those described in reference to Figures 7 to 9 Use with multi-layer schemes such as those described above.
[0109] In one example, a method for processing a multi-layer video stream is provided. The multi-layer video stream encodes a video signal and includes at least a first layer and a second layer. In this example, the method includes: receiving a first packet substream for the first layer; receiving a second packet substream for the second layer; and, combining packets from the first packet substream and the second packet substream to generate a joint basic packet stream, the joint basic packet stream including a sequence of packets, the sequence of packets including data for both the first layer and the second layer. In this case, each packet in the first packet substream may include a header and a data payload, wherein the data payload includes encoded data for the first layer, and each packet in the second packet substream may include a header and a data payload, the data payload including encoded data for the second layer. For example, the method may include a method similar to the method performed by the stream generator 310. However, the method may be performed at the encoder so that the joint basic stream is sent to the decoder as a single PID stream, for example, as part of a transport stream.
[0110] A variation of the above example is Fig.10 and Fig.11 In Fig.10 In , a transport stream 1002 is generated that includes at least the following two elementary streams: a first elementary stream has a "C" PID, and a second elementary stream has a "D" PID. The transport stream 1002 may be generated at an encoder for supply to one or more client devices. At the client device, packets of the transport stream 1002 with a common (i.e., equal or shared) PID may be used to reconstruct a particular elementary stream. In Fig.10 1004. As in the previous example, the payload data from the packets in the PID stream 1004 can be combined to form a packetized elementary stream 1008. For example, the packetized elementary stream 1008 can have packets with a header 1018 and a payload 1016 corresponding to a frame (or plane) of a video stream. In this case, the payload 1016 includes a NALU stream 1020, which includes NAL units for a first layer 1022 and NAL units for at least a second layer 1024. Thus, in this case, a stream similar to Figure 5520 in the NALU or data stream 1020. However, in this example, the NALU or data stream 1020 is derived from a single PID stream (or at least a smaller number of PID streams than the number of layers in the multi-layer scheme). For example, a component similar to the stream generator 310 can receive encoded data for multiple layers, for example from different encoders or encoding systems, and generate the elementary stream 1008 by interleaving the encoded data across multiple frames. The PID stream 1004 can then be generated by packetizing the elementary stream 1008 and multiplexing the resulting packets in the transport stream 1002. In this case, the decoder can be based on Figure 3 , but omitting the stream generator 310, passing the frame-interleaved multi-layer NALU stream from the stream receiver 210 to the first layer decoder 322 and the second layer decoder 324, or omitting Figure 4 ES parser union 410 in. In this solution, different bit streams for multiple layers of the multi-layer decoding scheme are interleaved and used to provide data streams to different decoders. The legacy decoder (such as the first layer decoder described above) is therefore able to decode the first layer of encoded data in the interleaved or combined stream and generate a first layer of video output; the enhanced decoder (such as the second layer decoder described above) can also use data from the same interleaved or combined stream to generate a second layer of video output. The outputs for the two layers can then be combined to provide a multi-layer or enhanced output video.
[0111] Fig.11 A method 1100 that can be performed at an encoding device is shown. At box 1102, a first layer packet stream is received. At box 1104, a second layer packet stream is received. For example, these can be output by a corresponding first layer encoder and a second layer encoder. The packet can be a NAL unit, a PES packet, and / or a PID stream packet. At box 1106, the packets are combined to generate a joint packet stream. The joint packet stream can be a joint NALU or an elementary stream. At box 1108, the joint packet stream is packaged as if it were a first layer data stream. This can include setting metadata for the joint packet stream to indicate that it has a first layer format. Additional metadata may or may not be set to indicate that the joint packet stream is part of a multi-layer encoding (depending on the implementation). In one case, a first layer decoder receiving the joint packet stream ignores NAL units for any layer that is not the first layer based on a differential value in the NAL header (e.g., a NAL unit for the first layer can indicate a first NALU type, and a NAL unit for other layers can indicate a reserved, unused, or other NALU type).
[0112] Although examples are presented herein in the form of transmitted streams, static media files are also based on the same framework, and thus the methods described herein can also be applied to media "containers", such as those that package encoded media content.
[0113] In some variations of the above examples (e.g., see Fig.10 and Fig.11 Those described), the joint elementary packet stream can be parsed by a first layer decoder to reconstruct data for the first layer, and can be parsed by a second layer decoder to reconstruct data for the second layer, wherein the outputs of the first layer decoder and the second layer decoder can be combined to reconstruct the video output from the multi-layer video stream.
[0114] In some variants, a single packet identifier is assigned to the joint elementary packet stream. Fig.10 The "C" in . This allows the joint elementary packet stream to be sent and / or broadcast as if it were a first layer coded stream. The joint elementary packet stream can then be decoded and rendered by the first layer decoder according to the normal legacy first layer stream. However, the second layer (and above) decoders can extract the additional second layer (and above) coded data from the same stream, for example, they can be configured to check the first layer stream for this data and / or this can be indicated in the metadata for the first layer stream being ignored by the first layer decoder.
[0115] In some variations, the method set forth above may include sending the joint elementary packet stream to one or more video decoders as part of a packetized transport stream, data for the joint elementary packet stream being indicated by a single packet identifier in a packet header of the packetized transport stream. Fig.10 In this case, a single packet identifier may be associated with the first packet substream for the first layer in the metadata for the joint elementary packet stream.
[0116] In certain aspects, a method of processing a multi-layer video stream is provided. Again, the multi-layer video stream encodes a video signal and includes at least a first layer and a second layer. In this aspect, the method includes receiving encoded data for the first layer; receiving encoded data for the second layer; and combining the encoded data for the first layer and the encoded data for the second layer as a single elementary packet stream with a single packet identifier that is associated with the first layer within metadata for the single elementary packet stream. The method can be performed at an encoder (e.g., according to the recent example) or at a decoder (e.g., according to Figures 3 to 6 ) is performed.
[0117] In this aspect, the method may further include, for example, at a service provider server, sending the single elementary packet stream as part of the transport stream to one or more decoding devices. This may occur if the joint stream or the single stream is generated at the encoder.
[0118] In this regard, the coded data for the first layer and the coded data for the second layer may be interleaved. For example, a PES payload for a frame or plane of a video signal may result in data being grouped in a BLBLBLBLBLB ... format, where B indicates data (e.g., NAL units) for the first layer or base layer, and L indicates data (e.g., NAL units) for the second layer or LCEVC / enhancement layer. Interleaving may enable simple synchronization of different layers and provide robustness to reduce streaming delays.
[0119] In the above case, a method for decoding a transport stream as generated by an encoder that generates a single encoded data stream may include: extracting an elementary packet stream from the transport stream based on a single packet identifier; transmitting data from the elementary packet stream to a first layer decoder based on a mapping between the single packet identifier and the first layer; transmitting data from the elementary packet stream to a second layer decoder to determine whether the elementary packet stream includes encoded data for the second layer; and, in response to a determination that the elementary packet stream includes encoded data for the second layer, combining the outputs of the first layer decoder and the second layer decoder to provide a multi-layer reconstruction of the video signal. For example, metadata for a program represented by the elementary packet stream, such as one or more of PSI, PAT, PMT, and descriptor data, may indicate that the elementary packet stream also includes data for one or more additional layers and / or the second layer decoder may check data packets for the elementary packet stream to determine whether they contain metadata (such as header data) indicating the presence of encoded data for other layers. In some cases, the first layer decoder may be a legacy hardware and / or software decoder that cannot be updated with new functionality, wherein other layer decoders (such as the second layer decoder) may be able to be updated with new functionality and therefore may include additional logic for parsing the combined data stream. In one case, the second layer decoder may pass at least a portion of the data from all compatible first layer data streams, and may be activated only when second layer data is detected within those streams.
[0120] In one variation, the second layer decoder is configured to examine header data from one or more network abstraction layer units derived from the elementary packet stream to determine whether the elementary packet stream includes coded data for the second layer. In this case, the first layer decoder may be configured to ignore the NAL unit containing coded data for the second layer based on a value within the header of the NAL unit.
[0121] Fig.12Another example method 1200 for processing a multi-layer stream is shown. The multi-layer video stream may encode a video signal and include at least a first layer and a second layer. The method 1200 includes a first block 1202 for receiving a first encoded data stream for a first layer of the multi-layer video stream. The first encoded data stream may be an encoded data stream read from a media file or received as part of a multiplexed stream. In one case, it may be received as a PID stream 104, such as Figure 5 In another case, it may be part of a combined stream, such as Fig.10 1008. The receiving may be performed over a network and / or from a local memory.
[0122] At block 1204, a descriptor field of the first coded data stream is parsed to extract an identifier for the first coded data stream. The descriptor field may be a descriptor field as defined as part of metadata (such as PSI data) for the first coded data stream. The descriptor field may be set to a value ignored by the first layer decoder, such as a reserved value for one or more first layer coding standards.
[0123] At block 1206, a second coded data stream for a second layer of the multi-layer video stream is received. The second coded data stream may be received in a similar manner as the first layer of the multi-layer video stream, for example, as Figure 5 PID stream 504 in or such as Fig.10 At block 1208, a descriptor field of the second coded data stream is parsed to determine if an identifier for the first coded data stream exists. The descriptor field may be the same descriptor field used to carry the identifier extracted at block 1204, or may include additional descriptor fields added for the second coded data stream. In one case, the same descriptor field for all relevant layers in a multi-layer coding scheme is set to carry a value used by the lowest layer in the coding scheme (e.g., a base identifier or tag).
[0124] At block 1210, conditional logic is applied based on the parsing performed at blocks 1204 and 1208. For example, in response to the presence of an identifier for the first coded data stream as determined in block 1208, the first coded data stream is paired with the second coded data stream as set forth in block 1212, and decoding of the multi-layer video stream is directed based on the paired data. For example, the decoding may include Figure 81204, wherein the paired data streams are supplied as base layer data and enhancement layer data, respectively. If no identifier is detected in the second coded data stream and / or the identifier exists but does not match the specific first layer identifier extracted at block 1204, the first coded data can be encoded by the first layer decoding and rendered according to normal single layer decoding.
[0125] Therefore, the method of Figure 1200 is an example of a general method of using a tag in the descriptor of an enhancement layer to reference an associated base layer. Therefore, the base layer and the enhancement layer (and one or more of the two may be present) can be grouped based on the data carried in the stream metadata. In this way, the first layer data stream can be sent and decoded in a normal manner for the first layer data stream, and the second layer data stream (e.g., for enhancement) can be flexibly bound to the first layer data stream so that the decoded second layer data can be combined with the decoded first layer data. In these cases, the enhancement layer can reference the base layer via a tag such as carried in the descriptor field, but the base layer may not reference the enhancement layer. This allows enhancements to be flexibly added to different pre-existing or legacy base streams. In some cases, tags can be provided in two data streams, but the two data streams can be sent as independent PID streams.
[0126] In some variants, such as the variant of Figure 1200, the first coded data stream and the second coded data stream are part of a joint elementary stream with a single packet identifier. In this case, the joint elementary stream is identified as an elementary stream according to the format of the first layer, so that the joint elementary stream can be parsed by the first layer decoder. In other variants, the first coded data stream and the second coded data stream are separate elementary streams with different packet identifiers.
[0127] In some cases, the first coded data stream for the first layer (eg, as read from a file as a media track or received as a PID stream) appears as a first layer coded bitstream. However, it actually carries interleaved coded data for multiple layers.
[0128] In certain aspects, a method for decoding a multi-layer video stream includes accessing a media track of a data file structure, the media track being identified by an identifier, the media track carrying a multi-layer video stream, the multi-layer video encoding a video signal and including data representing a first layer and data representing a second layer; parsing the identifier to guide decoding of the data representing the first layer using a first layer decoder, wherein the identifier is defined according to an encoding format of the first layer, wherein the data within the media track is accessed by the first layer decoder; and parsing the identifier to guide decoding of the data representing the second layer using a second layer decoder, wherein the outputs of the first layer decoder and the second layer decoder can be combined to reconstruct the output for the multi-layer video stream. In this case, as an adaptation to the method of Figure 1200 as applied to a media container, instead of storing multiple layers of the multi-layer encoding as separate tracks with a media file (e.g., similar to different PID streams within a broadcast), the multiple layers are stored as one track, which is similar to the lowest layer or legacy layer used for encoding. This then enables the data to be read by the lowest or legacy decoder, wherein data for other layers is ignored or skipped based on identifying metadata (such as NALU header data). With respect to the previous example, a higher layer decoder (eg, a second layer decoder) may be configured to monitor the activity of a first layer decoder and decode data from the same track to enhance the first layer.
[0129] In some examples as described herein, different video encoders are used to generate the encoded data for the first layer and the encoded data for the second layer. For example, a selectable base codec generates the first layer and an LCEVC codec generates the additional layers.
[0130] In some examples described herein, the first layer of video streams can form the base layer for multiple enhancement streams (e.g., multiple second layer streams). In this case, each enhancement stream can have different functions and / or can carry differentiated content. For example, the enhancement stream can provide and / or can include different content to be covered on the base stream content with different quality levels (such as different bit rates, color depths and / or resolutions). For example, the enhancement stream can provide different texts for different languages, or provide different advertising content for different users or regions. In one case, each LCEVC stream can encode different logo content for visible surfaces (such as sports billboards or billboards in the video) in the base video stream. This method and the use of more generally descriptors can be applied to both video streams through the network and file-based content (e.g., as streams recorded in files as bit sequences).
[0131] In the case where there is one first layer or base stream and multiple second layers or enhancement streams, a descriptor may be provided that has a looping function and defines multiple additional streams associated with the first layer or base stream. This descriptor with the looping function may be provided as part of the first layer or base stream, thereby allowing any decoder receiving the first layer or base stream to have access to a set of identifiers for the available second layers or enhancement streams. Each second layer or enhancement stream may have a descriptor with an identifier for that stream. Thus, a decoder may pair a base stream with multiple different enhancement streams, but a legacy base decoder may ignore the additional descriptors accompanying the base stream and simply decode the base stream according to the single layer case.
[0132] In the case of LCEVC, LCEVC video extension descriptors can be defined. Each LCEVC video stream (e.g., "second layer" stream as discussed in the examples of this article) can have an LCEVC video descriptor present in the descriptor loop of the program map section for the LCEVC video stream. The base video stream (e.g., the first layer stream as discussed in the examples of this article) can constitute a base video stream for more than one LCEVC video stream. The base video stream can also include an LCEVC video extension descriptor. As described above, the base video stream can include multiple LCEVC video extension descriptors (in the so-called descriptor "loop"), wherein each LCEVC video extension descriptor includes an identifier that identifies the association with different LCEVC encoded video streams. The LCEVC video extension descriptor can be identified using an extension descriptor tag, for example, a tag with a value of "0x17" previously defined as "reserved". Thus, the legacy base decoder can simply ignore the LCEVC video extension descriptor because the value is considered not to be used in their configuration.
[0133] Each LCEVC video extension descriptor may have a form similar to that set forth in the following table:
[0134]
[0135] In this case, the lcevc_stream_tag field is an 8-bit field that specifies the identifier of the association between the basic encoded video stream and the enhanced encoded video stream. In other alternative examples, lcevc_stream_tag can be replaced with the PID of the elementary stream to form a connection between the streams. Other fields can then provide additional optional information about the LCEVC video stream, such as the attributes of the LCEVC video stream.
[0136] A LCEVC registration descriptor may also be provided that defines a set of available lcevc_stream_tags that may be used. For example, the LCEVC registration descriptor may have the following form:
[0137]
[0138] This descriptor can be used for a base stream, where num_lcevc_stream_tags defines the number of associated enhancement streams, and a built-in loop repeats lcevc_stream_tags for each enhancement stream in turn. As before, the lcevc_stream_tag tag value allows a video elementary stream to be indicated as the basis for an LCEVC video stream that carries the same tag value in its LCEVC video descriptor. In the case of a single base stream and enhancement streams, the base descriptor may contain only one lcevc_stream_tag.
[0139] In some cases, one or more of the example systems 300 and 400, the method 600, the method 1100, the method 1200, or any other example described herein can be implemented via instructions retrieved from a computer-readable medium. These instructions can be executed by a processor of a decoding system (such as a client device). In one case, the examples related to method 1100 can be implemented by instructions retrieved from a computer-readable medium and executed by a processor of an encoding system (such as an encoding server).
[0140] The techniques described herein may be implemented in software or hardware, or may be implemented using a combination of software and hardware. They may include configuring devices to implement and / or support any or all of the techniques described herein. The above examples are to be understood as illustrative. Further examples are contemplated. It should be understood that any feature described with respect to any one example may be used alone, or in combination with other features described, and may also be used in combination with one or more features of any other example, or in combination with any combination of any other example. In addition, equivalents and modifications not described above may also be used without departing from the scope of the invention as defined in the appended claims.
Claims
1. A method for decoding a multi-layer video stream, wherein the multi-layer video stream encodes a video signal and comprises at least a first layer and a second layer, the method comprising: receiving a first packet substream for the first layer, each packet in the first packet substream comprising a header and a data payload, the first packet substream being identified via a first packet identifier indicated in the header and being received as part of a transport stream; receiving a second packet substream for the second layer, each packet in the second packet substream comprising a header and a data payload, the second packet substream being identified via a second packet identifier indicated in the header and received as part of the transport stream; combining packets from the first packet substream and the second packet substream to generate a combined elementary packet stream, the combined elementary packet stream comprising a sequence of packets including data for both the first layer and the second layer; providing the joint elementary packet stream to a first layer decoder for decoding the data for the first layer within the joint elementary packet stream; providing the joint elementary packet stream to a second layer decoder for decoding at least the data for the second layer within the joint elementary packet stream, wherein the first layer decoder is different from the second layer decoder; as well as An output of the first layer decoder and an output of the second layer decoder are combined to provide a multi-layer reconstruction of the video signal.
2. The method of claim 1 , wherein the data payload of the first packet substream forms a first packetized elementary stream and the data payload of the second packet substream forms a second packetized elementary stream, wherein the joint elementary packet stream comprises a third packetized elementary stream having a header comprising a presentation timestamp, the data payload following the header comprising data payloads from the first packetized elementary stream and the second packetized elementary stream associated with the presentation timestamp.
3. The method of claim 1 or claim 2, wherein the first packet substream and the second packet substream are sent such that the data payload of the second packet substream arrives no later than the data payload of the corresponding portion of the first packet substream.
4. The method according to any of the preceding claims, wherein the payload of the joint elementary packet stream comprises a sequence of network abstraction layer units for the first layer and a sequence of network abstraction layer units for the second layer.
5. The method of claim 4, wherein the first layer decoder is configured to ignore the network abstraction layer unit for the second layer based on a unit type data value within a header of the network abstraction layer unit for the second layer.
6. A method according to any one of the preceding claims, wherein the output of the first layer decoder is renderable independently of the multi-layer reconstruction of the video signal.
7. A method according to any one of the preceding claims, comprising: receiving the transport stream comprising the first packet substream and the second packet substream; as well as The transport stream is demultiplexed to extract the first packet substream and the second packet substream.
8. A method according to any of the preceding claims, wherein the first layer comprises a base video stream and the second layer comprises an enhancement video stream, and wherein the multi-layer reconstruction of the video signal comprises a higher quality rendition of a base video signal decoded from the base video stream.
9. The method of claim 8, wherein the data for the second layer comprises frames of residual data combined with frames of the base video signal as decoded from the base video stream.
10. The method of claim 9, wherein the second layer comprises a Low Complexity Enhancement Video Coding (LCEVC) video stream.
11. A method according to any preceding claim, wherein the second layer comprises a watermark stream.
12. A video decoder, comprising: a stream receiver to coordinate reception of a first packet substream and a second packet substream corresponding to a first layer and a second layer of a multi-layer video coding of a video signal, each packet of the first packet substream and the second packet substream comprising a header and a data payload, the first packet stream being identified via a first packet identifier indicated in the header and the second packet stream being identified via a second packet identifier indicated in the header, the first packet substream and the second packet substream being received as part of a transport stream; a stream generator to receive the first packet substream and the second packet substream and generate a joint elementary packet stream, the joint elementary packet stream comprising a sequence of packets, the sequence of packets comprising data for both the first layer and the second layer; a first layer decoder for receiving the joint elementary packet stream generated by the stream generator and outputting a decoded copy of the data for the first layer within the joint elementary packet stream; a second layer decoder configured to receive the joint elementary packet stream generated by the stream generator and output a decoded version of the data for the second layer within the joint elementary packet stream, the second layer decoder being different from the first layer decoder; as well as A multi-layer controller is configured to combine the output of the first layer decoder and the output of the second layer decoder to provide a multi-layer reconstruction of the video signal.
13. The video decoder of claim 12, wherein the stream receiver comprises a demultiplexer configured to receive and demultiplex the transport stream comprising the first packet substream and the second packet substream.
14. A video decoder according to claim 12 or claim 13, wherein the data payload of the first packet substream forms a first packetized elementary stream and the data payload of the second packet substream forms a second packetized elementary stream, wherein the joint elementary packet stream comprises a third packetized elementary stream, the third packetized elementary stream carries a header including a presentation timestamp, the data payload after the header comprises data payloads from the first packetized elementary stream and the second packetized elementary stream associated with the presentation timestamp, wherein the first packet substream and the second packetized substream are sent so that the data payload of the second packet substream arrives no later than the data payload of the corresponding part of the first packet substream.
15. A video decoder according to any one of claims 12 to 14, wherein the payload of the joint elementary packet stream comprises a sequence of network abstraction layer units for the first layer and a sequence of network abstraction layer units for the second layer, and wherein the first layer decoder is configured to ignore the network abstraction layer units for the second layer based on a unit type data value within a header of the network abstraction layer units for the second layer.
16. The video decoder according to any one of claims 12 to 15, comprising: A first layer renderer, configured to render the output of the first layer decoder on a display device; as well as A multi-layer renderer is used to render the multi-layer reconstruction on the display device.
17. A video decoder according to any one of claims 12 to 16, wherein the first layer comprises a base video stream and the second layer comprises an enhancement video stream, and wherein the multi-layer reconstruction of the video signal comprises a higher quality reproduction of a base video signal decoded from the base video stream.
18. The video decoder of claim 17, wherein the data for the second layer comprises frames of residual data combined with frames of the base video signal as decoded from the base video stream.
19. The video decoder of claim 18, wherein the second layer comprises a Low Complexity Enhancement Video Coding (LCEVC) video stream.
20. A computer readable medium comprising instructions which, when executed, cause a processor to perform the method according to any one of claims 1 to 11.
21. A method for processing a multi-layer video stream, wherein the multi-layer video stream encodes a video signal and comprises at least a first layer and a second layer, the method comprising: receiving a first substream of packets for the first layer, each packet in the first substream of packets comprising a header and a data payload, the data payload comprising encoded data for the first layer; receiving a second substream of packets for the second layer, each packet in the second substream of packets comprising a header and a data payload, the data payload comprising encoded data for the second layer; as well as Packets from the first packet substream and the second packet substream are joined to generate a joint elementary packet stream comprising a sequence of packets including data for both the first layer and the second layer.
22. The method of claim 21, wherein the joint elementary packet stream is capable of being parsed by a first layer decoder to reconstruct data for the first layer, and is capable of being parsed by a second layer decoder to reconstruct data for the second layer, wherein the outputs of the first layer decoder and the second layer decoder are capable of being combined to reconstruct a video output from the multi-layer video stream.
23. A method according to claim 21 or claim 22, comprising: A single packet identifier is assigned to the joint elementary packet stream.
24. The method according to claim 23, comprising: The joint elementary packet stream is sent to one or more video decoders as part of a packetized transport stream, data for the joint elementary packet stream being indicated by the single packet identifier in a packet header of the packetized transport stream.
25. A method according to claim 23 or claim 24, wherein the single packet identifier is associated with the first packet substream for the first layer in metadata for the joint elementary packet stream.
26. A method for processing a multi-layer video stream, wherein the multi-layer video stream encodes a video signal and comprises at least a first layer and a second layer, the method comprising: receiving encoded data for the first layer; receiving encoded data for the second layer; as well as The encoded data for the first layer and the encoded data for the second layer are combined as a single elementary packet stream with a single packet identifier associated with the first layer within metadata for the single elementary packet stream.
27. The method according to claim 26, comprising: The single elementary packet stream is sent as part of a transport stream to one or more decoding devices.
28. The method of any one of claims 21 to 27, wherein the encoded data for the first layer and the encoded data for the second layer are interleaved.
29. A method of decoding a transport stream as generated by the method according to any one of claims 24, 25 or 27, comprising: extracting the elementary packet stream from the transport stream based on the single packet identifier; transmitting data from the elementary packet stream to a first layer decoder based on a mapping between the single packet identifier and the first layer; transmitting data from the elementary packet stream to a second layer decoder to determine whether the elementary packet stream includes coded data for the second layer; as well as Responsive to a determination that the elementary packet stream includes encoded data for the second layer, outputs of the first layer decoder and the second layer decoder are combined to provide a multi-layer reconstruction of the video signal.
30. The method of claim 29, wherein the second layer decoder is configured to examine header data from one or more network abstraction layer units derived from the elementary packet stream to determine whether the elementary packet stream includes encoded data for the second layer.
31. The method of claim 29 or claim 30, wherein the first layer decoder is configured to ignore network abstraction layer units containing encoded data for the second layer based on a value within the header of the network abstraction layer unit.
32. A method for processing a multi-layer video stream, wherein the multi-layer video stream encodes a video signal and comprises at least a first layer and a second layer, the method comprising: receiving a first encoded data stream for the first layer of the multi-layer video stream; parsing a descriptor field of the first coded data stream to extract an identifier for the first coded data stream; receiving a second encoded data stream for the second layer of the multi-layer video stream; parsing a descriptor field of the second coded data stream to determine whether the identifier for the first coded data stream is present; as well as The first encoded data stream and the second encoded data stream are paired responsive to the presence of the identifier for the first encoded data stream, and decoding of the multi-layer video stream is directed based on the paired data.
33. The method of claim 32, wherein the first coded data stream and the second coded data stream are part of a joint elementary stream with a single packet identifier.
34. The method of claim 33, wherein the joint elementary stream is identified as an elementary stream according to a format of the first layer, so that the joint elementary stream can be parsed by a first layer decoder.
35. The method of claim 32, wherein the first encoded data stream and the second encoded data stream are separate elementary streams with different packet identifiers.
36. The method of any one of claims 32 to 35, wherein the first coded data stream has a plurality of descriptor fields, each descriptor field referring to a different coded data stream for the second layer.
37. The method of any one of claims 32 to 36, wherein the identifier for the first encoded data stream comprises a second layer stream tag identifying an association between the first encoded data stream and the second encoded data stream.
38. A method for decoding a multi-layer video stream, comprising: accessing a media track of a data file structure, the media track being identified by an identifier, the media track carrying the multi-layer video stream, the multi-layer video stream encoding a video signal and comprising data representing a first layer and data representing a second layer; parsing the identifier to direct decoding of the data representing the first layer using a first layer decoder, wherein the identifier is defined according to an encoding format of the first layer, wherein the data within the media track is accessed by the first layer decoder; as well as parsing the identifier to direct decoding of the data representing the second layer using a second layer decoder, The outputs of the first layer decoder and the second layer decoder can be combined to reconstruct the output for the multi-layer video stream.
39. The method according to any one of claims 1 to 11 or 21 to 38, wherein the encoded data for the first layer and the encoded data for the second layer are generated using different video encoders.
40. A computer readable medium comprising instructions which, when executed, cause a processor to perform the method of any one of claims 21 to 39.
