Synchronized frame decoding in multi-layer video stream
By decoding the first and second layers respectively in the multi-layer video stream and pairing with time stamps and timing metadata, the problem of poor adaptability of multi-layer video stream decoding in the prior art is solved, and the flexibility of video signals is realized Reconstruction and efficiency improvement.
Patent Information
- Application Number
- CN202280091992.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2021-12-20
- Filing Date
- 2022-12-20
- Publication Date
- 2024-11-29
AI Technical Summary
The existing multi-layer video encoding scheme has poor adaptability during the decoding process, making it difficult to effectively process multi-layer encoded streams, resulting in the advantages of multi-layer encoding not being fully realized, and many developers have turned to the single-layer H.264 video stream. .
By using different decoding methods on the first and second layers of the multi-layer video stream, using timestamps and timing metadata for pairing and combining, the video signal reconstruction is realized, allowing different decoding methods to process data separately from each layer. And combine.
It realizes flexible decoding of multi-layer video streams, supports video encoding methods at different levels, enhances the adaptability and efficiency of video decoding, and overcomes the problem of poor adaptability of multi-layer encoding schemes in the prior art.
Smart Images

Figure CN119054285A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to decoding of multi-layer video streams. In particular, the present invention relates to pairing or synchronizing decoded data across multiple layers of a multi-layer video stream. Background Art
[0002] Multi-layer video coding schemes have been around for many years, but have encountered problems in widespread adoption. Much of the video content on the Internet is still encoded using H.264 (also known as MPEG-4 Part 10, Advanced Video Coding - MPEG-4 AVC), which is used for 80%-90% of online video content. This content is usually provided to the decoding device as a single video stream with a one-to-one relationship with an available hardware and / or software video decoder, such as a single stream that is received, parsed and decoded by a single video decoder to output a reconstructed video signal. Therefore, many video decoder implementations have been developed based on this framework. In order to support different encodings, the decoder is generally configured with a simple conversion mechanism driven by metadata that identifies the stream format.
[0003] Existing multi-layer coding schemes include the Scalable Video Coding (SVC) extension to H.264, the scalable extension to H.265 (MPEG-H Part 2 High Efficiency Video Coding - SHVC), and newer standards such as MPEG-5 Part 2 Low Complexity Enhanced Video Coding (LCEVC). While H.265 is an evolution of the coding framework used by H.264, LCEVC takes a different approach to scalable video. SVC and SHVC operate by creating different coding layers and feeding each of these coding layers at different spatial resolutions. Each layer encodes the input according to a normal AVC or HEVC encoder, where it is possible to utilize information generated by lower coding layers. On the other hand, LCEVC generates one or more enhanced residual layers compared to the base coding, where the base coding can have a lower spatial resolution.
[0004] One reason for the slow implementation of multi-layer coding schemes is that it is difficult to adapt existing and new decoders to handle multi-layer coded streams. As discussed above, whether implemented in hardware or software or a combination of the two, a video stream is typically a single stream of data with a one-to-one pairing with a suitable decoder. Therefore, client devices and media players (including Internet browsers) are constructed as streams that receive data, determine what video encoding the stream uses, and then pass the stream to a suitable video decoder. Within this framework, multi-layer schemes such as SVC and SHVC are typically encapsulated as larger single video streams containing multiple layers, wherein these streams can be detected with "SVC" or "SHVC", and multiple layers are extracted from a single stream and passed to an SVC or SHVC decoder for reconstruction. However, this approach typically weakens some of the benefits of multi-layer coding. Therefore, many developers and engineers conclude that multi-layer coding schemes are too cumbersome and return to the multicast of a single H.264 video stream.
[0005]
[0006] It would therefore be desirable to have an improved method and system for decoding multi-layer video data that overcomes some of the disadvantages discussed above and allows for more advantages of multi-layer encoding schemes to be realized.
[0006] An overview of the SVC extension is provided in the paper “The Scalable Video Coding Extension of the H.264 / AVC Standard” by Heiko Schwarz and Mathias Wien, IEEE Signal Processing Magazine, March 2008, p. 135.
[0007] An overview of SHVC extensions is provided in the paper “Overview of SHVC: Scalable Extensions of the High Efficiency Video Coding Standard,” by Jill Boyce, Yan Ye, Jianle Chen, and Adarsh K. Ramasubramonian, published in the January 2016 issue of IEEE Transactions on Circuits and Systems for Video Technology, Vol. 26, No. 1.
[0008] The decoding technology of LCEVC is described in the draft text of ISO / IEC FDIS23094-2 and the final approved text published at the 129th MPEG meeting held in Brussels in January 2020 and WO 2020 / 188273 A1. Figure 29B of WO2020 / 188273 A1 describes a hypothetical reference decoder in which a demultiplexer provides a base bitstream to a base decoder and an enhanced bitstream to an enhanced decoder.
[0009] US2010 / 0272190 A1 describes a scalable transmission / reception device and method for improving the availability of broadcast services, which can allow the receiver to select the optimal video according to the attenuation degree of the broadcast signal by scalably encoding the video data and sending it through different transmission schemes for each layer. US2010 / 0272190 A1 uses the H.264 scalable video encoder (i.e., using SVC) to encode HD and SD video streams, and uses different packet streams to generate different layers of SVC encoding. At the decoding device, the DVB-S2 receiver / demodulator receives / demodulates the satellite broadcast signal from the transmitting satellite and recovers the first layer packet stream and the second layer packet stream. At the decoding device, the scalable combiner combines the recovered first layer substream group and the second layer packet stream in the input order to generate a single transport stream. The subsequent demultiplexer demultiplexes and assembles the combined transport stream, and splits it into the first layer video stream and the second layer video stream, and then passes it to the H.264 scalable video decoder for decoding and generating the reconstruction of the original HD video stream.
[0010] WO 2017 / 141038 A1 describes a physical adapter configured to receive a data stream comprising data usable to derive a reproduction of a signal at a first quality level and reconstruction data produced by processing the reproduction of the signal at a second, higher quality level and indicating how to use the reproduction at the first quality level to reconstruct the reproduction at the second quality level. WO 2017 / 141038 A1 describes how presentation timestamps (PTS) can be used to synchronize different elementary streams, a first elementary stream having a first packet identifier (PID) and a second elementary stream having a second packet identifier (PID).
[0011] All of the above publications set forth above are incorporated herein by reference. Summary of the invention
[0012] Various aspects of the invention are set out in the attached independent claims. Variations to these aspects are set out in the attached dependent independent claims.
[0013] According to a first example, a method for decoding a multi-layer video stream can be provided, the multi-layer video stream encodes a video signal and includes at least a first layer and a second layer, the first layer is decoded using a first decoding method, the second layer is decoded using a second decoding method, the first decoding method uses data that the second decoding method cannot access, the method comprising: obtaining second layer decoded data for the multi-layer video stream using the second decoding method, the second layer decoded data is associated with at least one frame of the video signal, the second layer decoded data is indexed using a timestamp derived from the multi-layer video stream; receiving a callback indicating the availability of the first layer decoded data for a frame of the first layer of the multi-layer video stream from the first decoding method; obtaining timing metadata for the first layer decoded data, the timing metadata being associated with rendering of the frame for the first layer; comparing the timing metadata with one or more timestamps for the second layer decoded data to pair the first layer decoded data for the frame with the second layer decoded data for the frame; and combining the first layer decoded data for the frame with the paired second layer decoded data to output a reconstruction of the frame of the video signal.
[0014] According to a second example, a system for decoding a multi-layer video stream can be provided, the multi-layer video stream encoding a video signal and including at least a first layer and a second layer, the system including: a second layer decoder for decoding the second layer of the multi-layer video stream; a memory for storing the output of the second layer decoder, the output being indexed using a timestamp derived from the multi-layer video stream; and a decoding controller communicatively connected to the first layer decoder, the first layer decoder using data that the second layer decoder cannot access, the decoding controller being configured to: receive a callback indicating the availability of first layer decoded data from the first layer decoder for a frame of the first layer; obtain timing metadata for the first layer decoded data, the timing metadata being associated with rendering of the frame for the first layer; compare the timing metadata with one or more timestamps for the output of the second layer decoder to pair the first layer decoded data for the frame with the second layer decoded data for the frame; and combine the first layer decoded data for the frame and the paired second layer decoded data to output a multi-layer reconstruction of the frame of the video signal. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 is a schematic diagram showing how encoded video data may be transmitted within various data streams.
[0016] Figure 2A and Figure 2B is a schematic diagram illustrating an example system for receiving and decoding multiple streams of encoded data.
[0017] Figure 3 is a schematic diagram illustrating an example browser implementation.
[0018] Figure 4 is a flow chart illustrating an example method of decoding a multi-layer video stream.
[0019] Figure 5 and Figure 6 are schematic diagrams illustrating example multi-layer encoder and decoder configurations, respectively.
[0020] Figure 7 is a schematic diagram illustrating certain data processing operations performed by an example multi-layer encoder. DETAILED DESCRIPTION
[0021] Certain examples described herein allow decoding devices to be easily adapted to handle multi-layer video coding schemes. Certain examples are described with reference to LCEVC multi-layer video streams, but the general concept can be applied to other multi-layer video schemes including SVC and SHVC, as well as multi-layer watermarking and content delivery schemes. Certain examples described herein are particularly useful in the case where the first layer of a multi-layer video stream is decoded by a first layer decoder that implements a first decoding method, and the second layer of the multi-layer video stream is decoded by a second layer decoder that implements a second decoding method. In order to allow flexibility in multi-layer configurations, the first layer can be encoded using various video coding methods (such as H.264 and H.265 as described above) and new and / or unimplemented video coding methods (such as universal video coding (VVC or H.266)). Therefore, the first layer decoder can vary for different encoded video streams. In the examples described herein, support is provided for a fully encapsulated first layer decoder that provides restricted access to internal data. For example, a first layer decoder may include a hardware component such as a secure hardware decoder chipset, in which other processes within a client device that performs decoding cannot access the data provided in the packet for the first layer. Instead, these processes can only access the decoded output of the first layer decoder. Thus, the examples described herein allow data for the second layer of a multi-layer video stream to be decoded separately using different decoding methods, but then combined with the appropriate output of the first layer decoder, for example to provide an enhanced video output.
[0022] Specifically, examples described herein enable the output of a second layer decoder to be combined with the output of a first layer decoder, where the first layer decoder uses data that the second layer decoder cannot access. For example, a presentation timestamp (PTS) provided in a packet of a first layer stream received and decoded by a first layer decoder may be inaccessible to a second layer decoder that receives the second layer stream for decoding. Therefore, the second layer decoder may not be able to use the PTS to synchronize its decoded output with the output of the first layer decoder.
[0023] In certain examples described herein, a callback indicating the availability of first layer decoded data from a first layer decoder for a frame of the first layer is used to obtain timing metadata for the first layer decoded data. For example, this may include a display or rendering time for the first layer decoded data. The second layer decoder then uses a timestamp (such as a PTS) that it can access via a second layer stream to find a match within the first layer decoded data. This can be performed by comparing the timing metadata to the timestamp. The matched first layer decoded data can then be combined with the second layer decoded data to output a reconstruction of a frame of a video signal. For example, a decoded frame of video data from a first layer decoder can be combined with one or more decoded frames from a second layer decoder. This example can be beneficial in the case where the second layer includes residual data, watermark data, and / or locally embedded metadata.
[0024] In some variations, the second layer decoded data may have two or more sub-layers at two or more resolutions (e.g., spatial resolutions). In these cases, the matched first layer decoded data may be upsampled to a higher resolution with or without correction at the decoded resolution to provide enhancement via the second layer. In some implementations, the first layer decoded data may also be generally available for output processing and combined reconstruction, providing different options for viewing.
[0025] In some examples described herein, different layers of multi-layer video coding may be sent as separate packets multiplexed within a transport stream. This allows different layers to be efficiently provided separately and, for enhancement layers, to be easily added to a pre-existing or pre-configured base layer. At a decoding device, different packet substreams may be received and parsed, for example, based on a packet identifier (PID) within a packet header.
[0026] In the following description, reference is made to Figure 1 A first example of an encoded video stream is described. Figure 2A , Figure 2B , Figure 3 and Figure 4 Example systems and methods for decoding multi-layer video streams are described. Figures 5 to 7 An example of a specific multi-layer encoding scheme is described.
[0027] Figure 1An example 100 of a transport stream (TS) 102 that can be used to send encoded video data to one or more decoding devices is shown. The transport stream 102 includes a sequence of TS packets 110 of fixed length 188 bytes. Each TS packet 110 has a header 112, which can have a variable length, and a payload 114. The header 112 contains one or more data fields. One of these data fields provides a packet identifier (PID) 116. The PID is used to distinguish different substreams within the transport stream. The PID can be a fixed length of multiple bits (e.g., 13 bits) storing a numeric or alphanumeric identifier (usually expressed as a hexadecimal value). For example, the PID 116 can be used to identify different video streams that are multiplexed together into a single stream that forms the transport stream 102. Example transport stream specifications are set forth in MPEG-2 Part 1 and are defined as part of ISO / IEC Standard 13818-1 or ITU-T Rec. H.222.0.
[0028] Figure 1 Also shown is one of the so-called PID streams 104 that can be extracted from the transport stream 102. The PID stream 104 includes a stream of consecutive packets 110 having a common (i.e., shared) PID value. The PID stream 104 can be created by demultiplexing the transport stream 102 based on the PID value. Thus, the PID stream 104 represents a substream of the transport stream 102.
[0029] In some cases, there may be special PID values reserved for index tables. In one case, a PID value may be reserved for a program association table (PAT) that contains a directory listing for a collection of program map tables (PMTs), which include mappings between one or more PID values and specific "programs." Originally, a "program" was associated with a specific broadcast program, but in the case of Internet streaming, the term is used more broadly to relate to the content of a specific video stream.
[0030] Figure 1 Also shown is a packetized elementary stream (PES) 106 constructed based on the payload data of multiple TS packets 110. The PES includes data from the payload of the PID stream, which carries the media sample data. The media sample data may include video data as well as other modalities such as audio data, subtitle data, or volume data. Figure 1In , PES is generated by combining the payloads 114 of multiple media TS packets 110 associated with a common (i.e., shared) PID value. PES includes a packet stream, where each PES packet consists of a header (i.e., PES header) 122 and a payload 124, which carries the combined data. The start of a new PES packet is indicated by a one-bit field from the TS header 112, called the payload unit start indicator (PUSI) 118. When the PUSI is set, the first byte of the TS packet payload 114 indicates the location where a new PES payload unit starts. This allows a decoding device that begins receiving data midway through a transmission to determine when to start extracting data. The PES header 122 contains a presentation timestamp (PTS) 128. This indicates the presentation time of the corresponding media segment encapsulated in the payload 124.
[0031] Figure 1 Finally, the content of a PES payload 124 for a video stream is shown. In this case, the PES payload 124 comprises a sequence 108 (i.e., a NALU stream) of NAL units 130. These may form part of an access unit of a video stream, i.e., a collection of NAL units that are associated with a specific output time, are consecutive in decoding order, and contain coded pictures or frames. Figure 1 A NALU stream 108 is shown which may be provided to a suitable video decoder for decoding.
[0032] Figure 1 The example of shows how different layers of a multi-layer video stream (eg, an encoded video stream) can be communicated to a decoding device. For example, the transport stream 102 can be sent via one or more communication channels, including over-the-air transmission and network communication. Figure 1 In an example of , different layers of a multi-layer video stream may be delivered as different PID substreams, e.g., packets with a PID of "B" may represent encoded data for the first layer ("base") stream, while packets with a PID of "L" may represent encoded data for the second layer ("LCEVC" or enhancement) stream. The different substreams of each layer may be demultiplexed and provided to a video decoder as PES 106. Although Figure 1 An example transport stream is shown, but other digital media containers (such as "tracks" on a computer-readable medium such as a disk or solid-state memory) may also be used to provide the encoded layer data. In some cases, the client device file system may be used to read the encoded data for different layers (e.g., as different tracks from the provided media).
[0033] Figure 2AAn example system 200 for decoding a multi-layer video stream is shown. The system 200 can be implemented as part of a client computing device such as a smart phone, laptop, smart TV, or other media receiver and / or player. The multi-layer video stream encodes a video signal and includes at least a first layer and a second layer. The first layer may include encoding according to a first encoding method, scheme, or standard (such as one of H.264, H.265, or H.266, etc.). The first layer may be referred to as a "base" layer. The first layer may be encoded according to a first quality level (such as a first spatial resolution, a first quantization level, a first specified or desired bit rate, and / or a first temporal resolution). The first layer may be a complete video encoding, i.e., encoded data that can be received, decoded, and rendered regardless of whether there are additional layers. The second layer may include encoding according to a second encoding method, scheme, or standard. For example, the second layer may include an "enhancement" layer for enhancing or otherwise strengthening the "base" layer. The second layer may be encoded using an enhanced encoding method such as LCEVC. The second layer may include encoding of a residual data stream, such as for combination with the first layer. The second layer may be encoded according to a second quality level, such as a second spatial resolution, a second quantization level, a second specified or desired bit rate, and / or a second temporal resolution. The second quality level may be higher than the first quality level, for example, to provide enhancement. Each layer may be received as a series of NAL units, such as Figure 1 108 in which the NAL unit includes coded data for a layer for a particular frame. Each layer may include a sequence of coded data for consecutive frames. In some cases, the sequence of coded data may be related to a group of pictures.
[0034] exist Figure 2A In , system 200 receives a first layer stream 202 and a second layer stream 204. The term "stream" is used herein to refer to a continuous portion of data being received or accessed. The first layer stream 202 and the second layer stream 204 may be generated as a result of a demultiplexing operation, such as an operation performed on transport stream 102 to extract one of PID stream 104 or PES 106, or may include data read from one or more files. Figure 2AIn the system 200, the first layer stream 202 is provided to the first layer video decoder 212, and the second layer stream 204 is provided to the second layer decoder 214. The first layer video decoder 212 is configured to decode the first layer of the multi-layer video stream, and the second layer video decoder 214 is configured to decode the second layer of the multi-layer video stream. The first layer video decoder 212 may be implemented using a video codec. The video codec may be hardware and / or software based. In some cases, the first layer video decoder 212 may use hardware acceleration, where one or more actions performed as part of the decoding are implemented using a specially configured hardware device (such as a specific video decoding chipset). In some cases, the first layer video decoder 212 may use one or more operating system services (e.g., as an operating system (such as or The second layer video decoder 214 may be an LCEVC decoder, for example, a software decoder implemented according to the LCEVC standard.
[0035] exist Figure 2A In the embodiment of the present invention, the second layer video decoder 214 is communicatively coupled to the memory 216. The memory 216 is configured to store the output of the second layer decoder 214. The memory 216 may include a dedicated hardware buffer and / or a portion of a shared system memory reserved for video decoding. The second layer video decoder 214 is configured to decode frames 224 of data for the second layer and store the frames in the memory 216. If the second layer video decoder 214 is configured to receive and decode one or more sub-layers of residual data (as in LCEVC), the frames 224 of data may include one or more frames at one or more corresponding resolutions.
[0036] Figure 2AA decoding controller 230 is also shown. The decoding controller 230 may form part of a decoder integration layer that is configured to control decoding of multiple layers of video, for example, in association with the first layer video decoder 212 and the second layer video decoder 214. In one case, the first layer video decoder 212 may be implemented as an independent video decoder (e.g., an independent codec) for the first layer (e.g., capable of decoding the first layer data in the absence of the second layer data), and the decoding controller 230 and the second layer video decoder 214 may be implemented as part of a multi-layer decoder configured to enhance the first layer with one or more additional layers of video encoding. The decoding controller 230 may be implemented in software (e.g., executed by a processor of a client device) and / or using dedicated hardware. In one case, the decoding controller 230 and the second layer video decoder 214 may be implemented as software (including firmware) enhancements to existing or legacy video decoding systems (including those with hardware acceleration for the first layer). An example decoding system that may be suitable for providing the functionality of the second layer video decoder 214 and the decoding controller 230 is described in PCT / GB2021 / 051940 (which is incorporated herein by reference).
[0037] exist Figure 2A , the decoding controller 230 is communicatively coupled to the first layer decoder 212. The decoding controller 230 is configured to receive a callback 222 indicating the availability of the first layer decoded data for the first layer frame from the first layer decoder 212. For example, the decoding controller 230 may request a callback from the first layer decoder 212 whenever the decoded data for the first layer is ready for rendering. The term "callback" is used herein to refer to a communication or signal sent between hardware and / or software components to indicate that an event has occurred. In hardware, a callback may include a physical signal and / or a change in a register value (e.g., representing a binary flag) sent over a communication bus or channel. In software, a callback may include an asynchronous function return and / or a change in a monitored value in a memory. The callback 222 may include a callback for rendering the output of the first layer decoder 212 in the case of a comparison. For example, the callback 222 may indicate that a frame of a video encoded using the first layer has been decoded from the first layer stream 202 and is ready to be displayed as part of the rendered video. In some cases, callback 222 may include data for a decoded frame (eg, frame layer 1 - FL1); in other cases, callback 222 may include a reference indicating a location (such as a memory address) where decode controller 230 may access the decoded data.
[0038] Upon receiving the callback 222, the decoding controller 230 is configured to further obtain timing metadata 232 for the first layer of decoded data (e.g., a decoded first layer frame). The timing metadata 232 is associated with rendering of the frame for the first layer. The timing metadata 232 can be generated by the first layer decoder 212. For example, the timing metadata 232 can be generated to help downstream processes render or otherwise display the first layer of decoded data. In one case, the timing metadata 232 can include one of a media time derived from the first layer of decoded data for a frame or a current playback time derived from the first layer of decoded data for a frame. The timing metadata 232 can be provided as part of the callback 222 and / or can be accessed using the first layer of decoded data (e.g., from a memory address associated with a memory address of the first layer of decoded data).
[0039] After receiving callback 222 and having obtained timing metadata 232, decoding controller 230 is configured to compare the timing metadata with one or more timestamps for the output of the second layer decoder to pair the first layer decoded data for the frame with the second layer decoded data for the frame. Figure 2A In an example of , the decoding controller 230 performs a query 236 on the second layer decoded data 224 stored in the memory 216. Specifically, the decoding controller 230 looks for a match between a timestamp stored with each portion of the second layer decoded data (such as each decoded frame of the second layer data) and the timing metadata 232 (e.g., media or playback time) associated with the ready form of the first layer decoder data. For example, the decoding controller 230 can be configured to search for second layer decoded data having a timestamp that falls within a range defined with reference to the time indicated by the timing metadata. The range can be set based on a configurable drift offset. In this case, the decoding controller 230 can search for data in the memory where the timestamp of the data plus a small drift deviation equals the time indicated in the timing metadata. The configurable drift offset can be a small number of milliseconds (e.g., 10ms) and can be set based on the frame rate of the rendering (e.g., can be reduced for higher frame rates).
[0040] exist Figure 2A In the example, the decoding controller 230 retrieves the second layer decoded data 234 based on the comparison and combines it with the first layer decoded data 222 to output a reconstruction 238 of a frame of the video signal. The reconstruction 238 may be an enhanced frame, whereby the second layer decoded data 234 in the form of residual data of the frame is combined with the decoded frame for the first layer. The decoding controller 230 may apply one or more enhancement sub-layers based on the second layer decoded data 234, for example, the second layer decoded data 234 may include two residual data sub-layers, as described later in Figures 5 to 7 As described. In some configurations, if no second layer decoded data 234 is located, for example because there has been a problem with the reception of data for the second layer stream or network congestion, the decode controller 230 can simply output the first layer decoded data 222 (e.g., without enhancement), such as acting in a pass-through mode. The output of the decode controller 230 can be rendered on a display that forms part of the client device implementation system 200 or is communicatively coupled to the client device implementation system. In other cases, the output of the decode controller 230 can be made available to other processes, such as in the memory 216 or another buffer.
[0041] Figure 2B An alternative configuration 240 of the system 200 is shown. Figure 2A In the configuration of , each of the first layer video decoder 212, the second layer video decoder 214, and the decoding controller 230 accesses data stored in the memory 216. In this case, the decoding controller 230 is configured to receive callbacks from both the first layer video decoder 212 and the second layer video decoder 214. The first layer video decoder 212 sends a first callback 232 to the decoding controller 230 to indicate that a new frame of the first layer data 242 is ready. The second layer video decoder 214 sends a callback 252 to the decoding controller 230 to indicate that a new frame of the second layer data 224 is ready. Both the first layer video decoder 212 and the second layer video decoder 214 can be configured to buffer decoded frames in the memory 216. Although the memory 216 is Figure 2B It is shown as a shared memory in , but in other implementations it may include separate memories accessible by appropriate components (e.g., dedicated hardware or software frame buffers for each decoder).
[0042] for Figure 2A , the frame of the first layer data 242 uses the timing metadata value ( Figure 2B It is shown as the media playback time t MPB ) is used for indexing, and the frames of the second layer data 224 are indexed using the timestamps (e.g., PTS values) from the second layer stream 204. Figure 2B 2, decode controller 230 is configured to coordinate the receipt of callback 232 and callback 252 and to compare the timing metadata and timestamp values to locate corresponding frames of first layer data 242 and second layer data 224. In some cases, decode controller 230 may act conditionally upon receipt of both callback 232 and callback 252. After the comparison is made, the corresponding frames are then combined, for example, by decode controller 230 or via an algorithm executed in memory 216 to output a multi-layer reconstruction 238. Figure 2BIn the multi-layer reconstruction 238, the multi-layer reconstruction 238 can then be used for rendering, for example, it can be copied to a dedicated frame buffer of a display device. As described above, the frames of the first layer data 242 and the second layer data 224 can be found by finding a timestamp value equal to the timestamp value plus a configurable drift offset (e.g., t MPB =PTS+Offset) timing metadata value for matching.
[0043] Figure 3 is a schematic diagram of an example system 300 for decoding a coded multi-layer video stream, such as a coded multi-layer video stream encoded using LCEVC. The system 300 may be implemented within a browser Figure 2A and Figure 2B The browser may be any browser capable of accessing information on the World Wide Web, examples of which include but are not limited to Google Microsoft and The browser may be implemented in a client device. Example client devices include, but are not limited to, mobile devices, computing devices, tablet devices, smart TVs, etc. The client device includes an operating system (OS), and the OS includes a browser.
[0044] One function of a browser is to transform a document written in a markup scripting language (sometimes referred to as a markup language) into a visual representation of a web page. The markup scripting language is used to control the display of data in a rendered web page. The markup language may include a markup video element, which, when processed by a browser, becomes a video display area. For example, a user of a browser may navigate to a web page containing an embedded video. When a browser renders a web page, it receives data corresponding to the video. For example, a browser may include resources necessary for decoding and playback of a video so that the video is displayed to the user in a video display area by rendering the browser on a display of a client device. Examples of markup scripting languages include any version of Hypertext Markup Language (HTML), such as HTML5, and Extensible Hypertext Markup Language (XHTML).
[0045] For example, the tag video element indicates attributes associated with the display of the video in the web page, such as the size of the video within the web page and whether the video will automatically play when the web page is loaded. For example, the tag video element also includes an indication of the video encoding format used to encode the video. This indicates to the browser which decoder to use to decode the encoded video. Then, as discussed further below, the browser can perform a call to at least one of the decoding functions within the browser native resources (which can be considered to be browser native browser native resources) or the decoding functions implemented in the OS.
[0046] Figure 3The system 300 includes a source buffer 302 for receiving a coded multi-layer video stream. In this example, the source buffer 302 may receive Figure 2A and Figure 2B In the embodiment of the present invention, the first layer stream 202 and the second layer stream 204 are encoded, and are the sources of the first layer stream and the second layer stream. In other examples, each stream may have a separate corresponding source buffer, rather than a joint buffer for two streams. The source buffer is a section of a memory, such as a browser that can access it. For example, the source buffer may be a media source extension (MSE) application programming interface (API) source buffer. The encoded multi-layer video stream in this example includes an encoded base stream and an encoded enhancement stream. The encoded base stream includes video content encoded by any basic encoder (also referred to as a compressor), such as Advanced Video Coding (AVC), High Efficiency Video Coding (HEVC), VP9, MPEG-5 Basic Video Coding (EVC) or Open Media Alliance Video (AOMedia Video 1, AV1) encoder.
[0047] The system 300 further includes an HTML media element 304 that implements a first layer (eg, base stream) video decoder, also referred to as a base stream decompressor. The HTML media element 304 may instruct or otherwise implement the Figure 2A and Figure 2B The first layer video decoder 212 described. In this example, the encoded base stream is extracted from the source buffer 302 and decoded using the HTML media element 304. The HTML media element 304 is a tagged video element that provides an interface between the markup language and the decoding resource. For example, HTML5 includes a tagged video element that can be used to embed video content in a web page. Another example is a JavaScript library that builds a set of custom controls on top of the HTML5 video element, which can be regarded as playing the role of a JavaScript player. It should be understood that tagged video elements such as HTML5 video elements can be modified by adding additional resources such as a multi-layer video stream decoding library, a WebAssembly decoding library and / or a web worker function that can be accessed by the tagged video element.
[0048] In the case of LCEVC, the enhancement stream may be carried in a set of supplemental enhancement information (SEI) messages accompanying and associated with the base stream or in a separate network abstraction layer (NAL) unit stream, such as Figure 1, carried in the PID stream. If the SEI message or NAL unit stream contains information that they cannot interpret (such as header information indicating a specific packet type), the base stream decoder can be configured to ignore the SEI message or NAL unit stream. Therefore, in this case, the HTML media element 304 can retrieve data related to the base stream from the source buffer 302 in a default manner, where both the enhanced and non-enhanced base streams are processed in a common manner. In this case, the HTML media element 304 can ignore the SEI message or NAL unit carrying the enhanced stream residing in the source buffer 302.
[0049] exist Figure 3 In some implementations of the example system 300, the tagged video element includes an indication of a video encoding format associated with the encoded elementary stream. When processed by a browser, the tagged video element locates the appropriate elementary stream decoder associated with the video encoding format and decodes the encoded elementary stream. The HTML media element 304 can be implemented within a browser, using functions of the OS of the client device including the browser, or utilizing resources of both the browser and the OS. For example, the OS can utilize hardware acceleration to decode the encoded elementary stream, which can reduce power consumption and the number of calculations performed by the CPU compared to decoding using software alone.
[0050] The decoded elementary stream is rendered in the first marked video display area 306. For example, the first marked video display area 306 corresponds to the area in the web page where the video is expected to be displayed. The first marked video display area 306 may include a <video> tag display area. Rendering the decoded elementary stream allows access to the elementary stream video data (e.g., decoded frames of the underlying encoded video). By rendering the elementary stream video data, the elementary stream video data can be accessed by other decoding processes within the browser, for example, even if the decoding of the elementary stream is performed by an inaccessible or protected method. In the present case, when a frame of the first layer of decoded data is ready, for example, as Figure 2A and Figure 2B As shown in 232, the HTML media element 304 can provide an option to register a callback.
[0051] The rendered decoded base stream is then combined with the decoded enhancement stream to generate a reconstructed video stream. In some cases, since the rendered base stream does not contain enhancement data from the enhancement stream at this time, the marked video display area is hidden. This ensures that the rendered video content corresponding to the base stream is not displayed in the web page and is therefore not visible to viewers of the web page. However, in some cases, there may be a user option to view the content. Rendering the decoded base stream ensures that the system 300 can still decode and render video streams that are not encoded using a multi-layer video coding format. For example, if this is the case, the marked video display area can be set to visible and the decoded base stream can be displayed based on a comparative non-enhanced video rendering. For example, if a web page contains a single-layer video stream that lacks an enhancement stream, then Figure 3 The system 300 may be used to display a decoded single-layer video stream, for example, by unhiding the first marked video display area 306 .
[0052] The system 300 further includes an enhanced stream decoder 308. The enhanced stream decoder 308 may implement reference Figure 2A and Figure 2B The functions of one or more of the second layer video decoder 214 and the decoding controller 230 described above. In the current case, the source buffer 302, which may include an MSE component, issues a callback to indicate that the data is ready for decoding. The callback may be received by the enhancement stream decoder 308 (for example, the enhancement stream decoder 308 may register the callback as part of the initial configuration). Upon receiving the callback, the encoded enhancement stream is extracted from the source buffer 302 and decoded by the enhancement stream decoder 308. For example, the enhancement stream decoder 308 may retrieve the encoded enhancement stream from the data stored in the source buffer 302 for a portion of the SEI message set or PID stream. In some cases, the enhancement stream decoder 308 may obtain the first layer stream and the second layer stream (i.e., the base stream and the enhancement stream), demultiplex the two streams, and then discard the first layer stream (for example, because this is obtained and decoded through the HTML media element 304). At this stage, the timestamp may be obtained by tracing the combined stream from the source buffer 302. For example, the PTS timestamp may be obtained from the base stream or from the source buffer callback.
[0053] exist Figure 3In the example of, the enhanced stream decoder 308 also obtains the decoded base stream from the first marked video display area 306, and combines the decoded base stream with the decoded enhanced stream to generate a reconstructed video stream. The reconstructed video stream can then be rendered in the second marked video display area 310 within the web page visible to the web page viewer. The second marked video display area 310 may include a <canvas> tag display area. In this example, when the frame of the base (i.e., first layer) stream is ready, the enhanced stream decoder 308 receives a callback from the first marked video display area 306. This may include a request video frame (requestVideoFrame) callback. When the enhanced stream decoder 308 obtains the callback from the first marked video display area 306, it obtains the timing metadata for the base frame as discussed above. For example, the timing metadata may include a "media time" variable or the current playback time provided with the frame rendered in the first marked video display area 306. As described above, the enhanced stream decoder 308 then compares the timed metadata with the originally traced timestamp to pair the first layer decoded data for the frame with the second layer decoded data for the frame. The combined paired data is then rendered as an enhanced frame in the second marked video display area 310.
[0054] The enhanced stream decoder 308 may be a multi-layer video stream decoder plug-in (DPI), such as an LCEVC decoder plug-in, which is configured to decode an LCEVC encoded video stream. The enhanced stream decoder 308 may provide the decoding capabilities of the second layer decoder 214 and the control capabilities of the decoding controller 230, as described in reference to FIG. Figure 2A and Figure 2B The enhanced stream decoder 108 may include a single component or two separate components having the same functional effect depending on the implementation. One or more components of the system 300 may be implemented in a browser. In one example, a browser including an enhanced stream decoder 308 is provided.
[0055] Figure 3A is a schematic diagram of an alternative example system 300 for decoding a coded multi-layer video stream, such as a coded multi-layer video stream encoded using LCEVC. Figure 3 Components and Features The common components and features function in the same manner as described above and will not be described in detail again.
[0056] As mentioned earlier, Figure 3A The system 300 includes a source buffer 302 to receive a coded multi-layer video stream, such as Figure 2A and Figure 2BThe first layer stream 202 and the second layer stream 204 in the source buffer 302. The encoded elementary stream is extracted from the source buffer 302 and decoded using the HTML media element 304. The decoded elementary stream is rendered in the first markup video display area 306.
[0057] The source buffer 302 issues a callback to indicate that the data is ready for decoding, and the callback is received by the enhancement stream decoder 308. Upon receiving the callback, the encoded enhancement stream is extracted from the source buffer 302 and decoded by the enhancement stream decoder 308. The timestamp, PTS timestamp, can be obtained from the base stream or from the source buffer callback.
[0058] The enhancement stream decoder 308 also obtains the decoded base stream from the first mark video display area 306 and combines the decoded base stream with the decoded enhancement stream to generate a reconstructed video stream. The reconstructed video stream is then rendered in the second mark video display area 310.
[0059] As discussed previously, when LCEVC is used, the enhancement stream extracted from the source buffer 302 is carried within a set of SEI messages accompanying and associated with the base stream or within a separate NAL unit stream.
[0060] If the SEI message or NAL unit stream contains information that the base stream decoder cannot interpret, the base stream decoder is usually configured to ignore these SEI messages or NAL unit streams. This means that the HTML media element 304 retrieves data related to the base stream from the source buffer 302 in a default manner, where both enhanced and non-enhanced base streams are processed in a common manner. The HTML media element 304 ignores SEI messages or NAL units carrying the enhancement stream residing in the source buffer 302.
[0061] As in Figure 3A As can be seen in FIG. 3 , the enhancement stream extracted from the source buffer 302 and carrying the SEI message set is first passed to the integration layer 312 before being passed to the enhancement stream decoder 308 for decoding.
[0062] Specifically, the integration layer 312 extracts LCEVC data from the enhancement stream based on the type of source buffer. The extracted data can be a NAL unit with LCEVC data and a real-time transport protocol (RTP) timestamp, or data attached to the source buffer from a media source extension. The LCEVC data is indexed using the PTS and stored for use when matching and combining the corresponding base stream with the LCEVC data using the PTS. As previously described, this is done by comparing the timing metadata with the PTS, and the matching LCEVC data can then be combined with the base stream to generate a reconstructed video stream. Figure 3AAs shown in , the relevant LCEVC data is passed to the enhancement stream decoder 308 together with the deviation generated based on the playback environment.
[0063] The offset is calculated by the offset calculation block 314 and can be done using two methods. The first method involves calculating the offset during source buffer appending and the second method involves calculating the offset during enhancement stream decoding.
[0064] Using the first method, there are instances where data with the same timestamp is appended to the source buffer multiple times during source buffer appending. In this case, the frame time is calculated based on the FPS of the enhancement stream, and then the frame time is added to the provided timestamp by the number of times the appending is repeated. For example, if the data is appended three times, the frame time is added to the provided timestamp three times. This is then used to store the LCEVC data used by the integration layer.
[0065] Using the second method, during enhancement layer decoding, the offset is calculated based on one or more of the operating system, browser, player, and container format. The offset is then added to the timestamp provided by the video stream to extract the relevant LCEVC data from the stored LCEVC data.
[0066] Figure 4 An example method 400 for decoding a multi-layer video stream is shown. The method 400 may utilize Figure 2A , Figure 2B and Figure 3 The system 200, system 240, system 300 of the embodiment of the present invention can be executed by one of the systems 200, system 240, and system 300, or can be implemented as a set of instructions executed by a processor, for example. As in the above example, the multi-layer video stream encodes a video signal and includes at least a first layer and a second layer. The first layer is decoded using a first decoding method, and the second layer is decoded using a second decoding method. The first decoding method uses data that the second decoding method cannot access. For example, the PTS for the decoded frame in the first layer of the multi-layer video stream may be available to the first decoding method, but not to the second decoding method, for example, due to the use of secure or restricted internal data.
[0067] At block 402, the method includes obtaining second layer decoded data for a multi-layer video stream using a second decoding method. The second layer decoded data is associated with at least one frame of the video signal. For example, the second layer decoded data may include residual or other enhancement data (such as watermark data) for a frame of the video. The second layer decoded data is indexed using a timestamp derived from the multi-layer video stream. For example, the second decoding method (e.g., as obtained by a second layer video decoder (such as Figure 2A and Figure 2B 214) in) can be implemented for the second layer stream from the input such as Figure 2A and Figure 2B The second layer flow 204 or Figure 1 The second layer decoded data may be obtained directly, such as received from a second layer video decoder, or indirectly, such as accessed from a shared memory. In one case, the second decoding method generates a callback to indicate that the second layer decoded data for the consecutive frame is ready.
[0068] At block 404, method 400 includes receiving a callback indicating the availability of first layer decoded data for a frame of a first layer of a multi-layer video stream from a first decoding method. The callback may contain the first layer decoded data or may indicate that it is available from accessible memory. The callback may be received from a video tag, such as a request for a video frame.
[0069] At box 406, method 400 includes obtaining timing metadata for the first layer of decoded data. The timing metadata is associated with the rendering of the frames for the first layer. For example, the timing metadata may include a media time derived from the first layer of decoded data for the frame or a current playback time derived from the first layer of decoded data for the frame. If the first decoding method decodes multiple frames in series and / or parallel (e.g., when they are received in a stream), each decoded frame may have an associated time within the timing metadata. It should be noted that the PTS timestamp for the first layer of decoded data is not output by the first decoding method at this time, for example, because it is only used internally within the security method. Therefore, in order to match the decoded frames of the first layer with the decoded data of the second layer, additional adaptation is required.
[0070] At block 408, method 400 includes comparing the timing metadata to one or more timestamps for the second layer decoded data to pair the first layer decoded data for the frame with the second layer decoded data for the frame. This may include determining whether a time value indicated in one of the timing metadata and the timestamp falls within a defined range of the other of the timing metadata and the timestamp. For example, the second layer decoded data may include buffered second layer decoded data for a series of frames, each frame having a time value from such as Figure 1 The corresponding time value derived from the PTS of the PES 106 shown, and the first layer of decoded data may include buffered first layer of decoded data for a series of frames, each frame having a corresponding time value derived from timing metadata such as presentation time. In this case, block 408 includes fuzzy matching of the time values to match the data for the corresponding frames.
[0071] At block 410, method 400 includes combining the first layer decoded data and the paired second layer decoded data for the frame to output a reconstruction of the frame of the video signal. For example, this may include adding residual data from the second layer decoded data to the first layer decoded data for the frame. The residual data may be added at one or more spatial resolutions, for example, a combination of the residual data and the first layer frame at a first resolution (e.g., standard definition - SD or high definition - HD) may be upsampled and then combined with other residual data at a second resolution (e.g., HD or ultra high definition - UHD). Reference is made below to Figures 5 to 7 Describes further details of reconstruction using residual data.
[0072] Preferably, the method is repeated for each frame of the video signal, for example at 30 or 60 frames per second (fps), to allow the video signal to be viewed as a video. In some cases, comparing the timing metadata to one or more timestamps for the second layer decoded data further includes searching for the second layer decoded data having a timestamp that falls within a range defined with reference to the time indicated by the timing metadata. In this case, the range can be set based on the frame rate, for example to a temporal resolution that is less than the frame rate. The range can be set based on a configurable drift offset, where the drift offset can be configured based on the frame rate and / or a matching tolerance. In some cases, additional available data can be used to assist in matching, such as resource load, other time values output by the first decoding method, data received from an upstream demultiplexer, etc.
[0073] In some cases, the second layer decoded data is stored in segments with a defined time length, where each segment has multiple data groups with a start time and an end time, each data group representing data for a frame. These data groups can then be matched with the first layer decoded frames based on the timing metadata.
[0074] In preferred examples, the first layer comprises a base video stream, and the second layer comprises an enhancement video stream, wherein the second layer "enhances" the first layer, e.g., increases resolution, provides sharper details, removes compression artifacts, etc. In these examples, the second layer decoded data may include frames of residual data combined with frames derived from the base video stream. For example, the second layer may comprise a Low Complexity Enhancement Video Coding (LCEVC) video stream. In other cases the second layer comprises a watermark stream, wherein watermark information is added to the first layer. The watermark information may be visible or invisible, e.g., incorporated as invisible metadata in the latter case.
[0075] In some cases, the callback at block 404 is received from a first decoding method, for example to indicate that a first layer frame is ready for rendering. Block 402 may also be performed in response to a callback from a second decoding method, for example to indicate that a surface or frame of second layer data is rendered for composition.
[0076] In some cases, the first decoding method may be implemented by an operating system service. In this case, internal data used by the operating system service may be inaccessible to other processes outside the operating system service, for example, the service may operate within a protected kernel memory space that is inaccessible to application-level processes. The operating system service may use hardware acceleration to decode the first layer, for example, using a driver and associated data that is accessible only to the operating system. The second decoding method may not form part of the operating system service, but may represent an application-level process for improving default or traditional decoding operations. Therefore, the second decoding method may allow the client device to be easily enhanced or upgraded using a fixed operating system or firmware.
[0077] In some cases, the first decoding method may include rendering the reconstruction of the frames of the video signal within a browser element. For example, the methods and systems described herein may be rendered within a hypertext markup language (HTML) document to render the multi-layer stream, such as rendered within an Internet browser. The HTML document may include an HTML5 document.
[0078] Although examples are described herein with reference to two layers, these methods can be applied to more than two layers. For example, there can be a base layer and multiple higher enhancement layers. As described herein, each enhancement layer can be matched with a base layer.
[0079] In some examples described herein, a first layer stream and a second layer stream may be synchronized, e.g., the last packet of a picture in the second layer stream may be specified to arrive no later than the last packet of the corresponding picture in the first layer stream (e.g., when both pictures are associated with the same underlying frame). In this case, the presentation timestamp (PTS) for the second layer may be taken from either the first layer or the second layer.
[0080] In some cases, a second layer decoder or second layer decoding method may receive a stream having data for two layers. In this case, the second layer decoder or decoding method may discard the first layer data and decode the second layer data. However, the second layer decoder or decoding method may retain the PTS value from the data for the first layer.
[0081] In some cases, a callback from a first layer decoder or first layer decoding method may indicate that a first layer frame is ready. The callback may provide frame metadata containing the media time for that (first layer) frame. This media time may be used to perform the matching described above. If the media time is not available, the current playback time (e.g., from a seek bar if rendering in a browser or media player) may be used as a fallback or replacement for the match.
[0082] In some cases, one or more of the example systems 200, 240, and 300, or method 400 may be implemented via instructions retrieved from a computer-readable medium and executed by a processor of a decoding system, such as a client device.
[0083] Certain examples described herein allow the upper layers of a multi-layer coding scheme to be efficiently decoded while maintaining the timing constraints imposed by the lower layers of the multi-layer coding scheme. Matching can be flexible so that when matching is not possible or when second layer decoded data is not available, the first layer frames are simply rendered without enhancement. If this only occurs intermittently, it is usually not noticeable.
[0084] Some general information related to an example enhancement coding scheme will now be described. This information provides an example of a specific multi-layer coding scheme.
[0085] It should be noted that examples are presented herein with reference to signals that are sequences of samples (i.e., two-dimensional images, video frames, video fields, sound frames, etc.). For simplicity, the non-limiting examples shown herein generally refer to signals that are displayed as a set 2D plane (e.g., a 2D image in a suitable color space), such as, for example, a video signal. In a preferred case, the signal comprises a video signal. Reference Figure 6 An example video signal is described in more detail. Although the examples are specifically described for video, the methods may also be used for other media forms, such as audio or subtitle data.
[0086] The terms "picture", "frame" or "field" are used interchangeably with the term "image" to indicate samples of a video signal in time: any concepts and methods shown for a video signal consisting of frames (progressive video signals) can also be easily applied to video signals consisting of fields (interlaced video signals), and vice versa. Although the examples shown herein focus on images and video signals, it can be easily understood by those skilled in the art that the same concepts and methods are also applicable to any other type of multi-dimensional signal (e.g., audio signals, volumetric signals, stereoscopic video signals, 3DoF / 6DoF video signals, plenoptic signals, point clouds, etc.). Although image or video encoding examples are provided, the same methods can be applied to signals with dimensions less than two (e.g., audio or sensor streams) or greater than two (e.g., volumetric signals).
[0087] In the specification, the terms "image", "picture" or "plane" (meaning the broadest sense of "hyperplane", i.e., an array of elements with arbitrary dimensions and a given sampling grid) will generally be used to identify a digital reproduction of signal samples along a sequence of samples, wherein each plane has a given resolution for each of its dimensions (e.g., X and Y) and comprises a set of plane elements (or "elements" or "pixels" or display elements commonly referred to as "pixels" for two-dimensional images, display elements commonly referred to as "voxels" for volumetric images, etc.) characterized by one or more "values" or "settings" (e.g., by way of non-limiting example, a color setting in a suitable color space, a setting indicating a density level, a setting indicating a temperature level, a setting indicating an audio pitch, a setting indicating an amplitude, a setting indicating a depth, a setting indicating an alpha channel transparency level, etc.). Each plane element is identified by a set of suitable coordinates indicating the integer position of the element in the sampling grid of the image. Signal dimensions may comprise only spatial dimensions (e.g., in the case of images) or temporal dimensions (e.g., in the case of signals evolving over time, such as video signals). In one case, it can be seen that a frame of a video signal includes a two-dimensional array having three color component channels, or a three-dimensional array having two spatial dimensions (e.g., spatial dimensions indicating resolution - lengths equal to the respective width and height of the frame) and one color component size (e.g., length 3). In some cases, the processing described herein is performed separately for each plane of color component values that make up the frame. For example, planes representing pixel values for each of the Y, U, and V color components can be processed in parallel using the methods described herein.
[0088] Certain examples described herein use a scalability framework using base coding and enhancement coding. The video coding system described herein operates on a decoded received base coding (e.g., frame-by-frame or complete base coding) and adds one or more of spatial, temporal or other quality enhancements via enhancement layers. The base coding may be generated by a base layer, which may use a different coding scheme than the enhancement layer, and in some cases may include a traditional or contrasting (e.g., older) coding standard.
[0089] Figures 5 to 7 A spatially scalable coding scheme is shown that uses a downsampled source signal encoded with a base codec, adds a first level of correction or enhancement data to the decoded output of the base codec to generate a corrected picture, and then adds another level of correction or enhancement data to an upsampled version of the corrected picture. Thus, the spatially scalable coding scheme can generate an enhancement stream with two spatial resolutions (higher and lower), which can be combined with a base stream at a lower spatial resolution.
[0090] In a spatially scalable coding scheme, the method and apparatus may be based on an overall algorithm built on top of existing encoding and / or decoding algorithms (e.g., MPEG standards such as AVC / H.264, HEVC / H.265, and non-standard algorithms such as VP9, AV1, etc.) used as a baseline for enhancement layers. The enhancement layers operate according to different encoding and / or decoding algorithms. The idea behind the overall algorithm is to encode / decode video frames in a hierarchical manner, as opposed to using a block-based approach as done in the MPEG family of algorithms. Encoding frames in a hierarchical manner includes generating residuals for full frames, and then generating residuals for reduced or decimated frames, etc.
[0091] Figure 5 The system configuration for an example spatial scalable coding system 500 is shown. The coding process is split into two halves as shown by the dotted lines. Each half can be implemented separately. Below the dotted line is the base level, and above the dotted line is the enhancement level, which can be usefully implemented in software. The coding system 500 can include only the enhancement level process, or include a combination of the base level process and the enhancement level process as required. The topology of the coding system 500 at a general level is as follows. The coding system 500 includes an input I for receiving an input signal 501. The input I is connected to a downsampler 505D. The downsampler 505D outputs to the base encoder 520E of the base level of the coding system 500. The downsampler 505D also outputs to the residual generator 510-S. The encoded base stream is directly created by the base encoder 520E, and can be quantized and entropy encoded when necessary according to the base coding scheme. The encoded base stream can be the base layer described above, such as the lowest layer in the multi-layer coding scheme.
[0092] Above the dotted line is a series of enhancement layer progressions used to generate an enhancement layer for a multi-layer coding scheme. In this example, the enhancement layer includes two sub-layers. In other examples, one or more sub-layers may be provided. Figure 5 In order to generate a coded sublayer 1 enhancement stream, the coded base stream is decoded via a decoding operation applied at a base decoder 520D. In a preferred example, the base decoder 520D may be a decoding component that supplements the encoding component in the form of a base encoder 520E within the base codec. In other examples, the base decoding block 520D may alternatively be part of the enhancement layer. Via a residual generator 510-S, a difference is created between the decoded base stream output from the base decoder 520D and the downsampled input video (i.e., a subtraction operation 510-S is applied to the frames of the downsampled input video and the frames of the decoded base stream to generate a first residual set). Here, the residual represents the error or difference between a reference signal or frame and a desired signal or frame. The residuals used in the first enhancement layer can be regarded as correction signals because they are able to 'correct' future frames of the decoded base stream. This is useful because this can correct for quirks or other characteristics of the base codec. These characteristics include, among other things, the motion compensation algorithm applied by the base codec, the quantization and entropy coding applied by the base codec, and the block alignment applied by the base codec.
[0093] exist Figure 5 In , the first set of residuals is transformed, quantized, and entropy encoded to produce an encoded enhancement layer, a sub-layer 1 stream. Figure 5 , a transform operation 510-1 is applied to a first set of residuals; a quantization operation 520-1 is applied to the transformed set of residuals to generate a set of quantized residuals; and an entropy coding operation 530-1 is applied to the quantized set of residuals to generate an encoded enhancement layer - sub-layer 1 stream (e.g., at a first enhancement level). However, it should be noted that in other examples, only the quantization step 520-1 may be performed, or only the transform step 510-1 may be performed. Entropy coding may not be used, or may optionally be used in addition to one or both of the transform step 510-1 and the quantization step 520-1. The entropy coding operation may be any suitable type of entropy coding, such as a Huffmann coding operation or a run length encoding (RLE) operation, or a combination of both a Huffmann coding operation and an RLE operation (e.g., RLE followed by Huffmann or prefix coding).
[0094] To generate the coded enhancement layer - sub-layer 2 stream, another enhancement level information is created by generating and encoding another set of residuals via the residual generator 500-S. The other set of residuals is the difference between the upsampled version (via upsampler 505U) of the corrected version of the decoded base stream (reference signal or frame) and the input signal 501 (desired signal or frame).
[0095] In order to achieve reconstruction of a corrected version of the decoded elementary stream to be generated at the decoder (e.g. Figure 6 ), at least some of the sub-layer 1 coding operations are reversed to mimic the process of the decoder, and to account for at least some of the losses and quirks of the transform and quantization processes. To this end, the first residual set is processed by a decoding pipeline including an inverse quantization block 520-1i and an inverse transform block 510-1i. The quantized first residual set is inverse quantized at the inverse quantization block 520-1i and inverse transformed at the inverse transform block 510-1i in the encoding system 500 to regenerate a decoder-side version of the first residual set. The decoded base stream from the decoder 520D is then combined with the decoder-side version of the first residual set (i.e., a summation operation 510-C is performed on the decoded base stream and the decoder-side version of the first residual set). The summation operation 510-C generates a reconstruction of a downsampled version of the input video as would likely be generated at the decoder - i.e., a reconstructed base codec video). The reconstructed base codec video is then upsampled by an upsampler 505U. The processing in this example is typically performed frame by frame. Each color component of the frame may be processed in parallel or serially as shown.
[0096] The upsampled signal (i.e., reference signal or frame) is then compared to the input signal 501 (i.e., desired signal or frame) to create another set of residuals (i.e., the difference operation is applied by the residual generator 500-S to the upsampled recreated frame to generate another set of residuals). The other set of residuals is then processed by mirroring the encoding pipeline for the first set of residuals to become an encoded enhancement layer - sub-layer 2 stream (i.e., the encoding operation is then applied to the other set of residuals to generate another encoded enhancement stream). Specifically, the other set of residuals is transformed (i.e., the transform operation 510-0 is performed on the other set of residuals to generate another transformed set of residuals). The transformed residual is then quantized and entropy encoded in the manner described above for the first set of residuals (i.e., the quantization operation 520-0 is applied to the transformed set of residuals to generate another quantized set of residuals; and the entropy encoding operation 530-0 is applied to the quantized other set of residuals to generate an encoded enhancement layer - sub-layer 2 stream containing another enhancement level information). In some cases, the operation may be controlled, for example so that only the quantization step 520-1 may be performed, or only the transform and quantization steps may be performed. Entropy coding may optionally be used as a supplement. Preferably, the entropy coding operation may be a Huffman coding operation or a run length encoding (RLE) operation or both (e.g., RLE followed by Huffman coding). Preferably, the entropy coding operation may be a Huffman coding operation or a run length encoding (RLE) operation, or both (e.g., RLE followed by Huffman coding). The transform applied at both blocks 510-1 and 510-0 may be a Hadamard transformation applied to a 2×2 or 4×4 residual block.
[0097] Figure 5 The encoding operation in does not result in dependencies between local blocks of the input signal (e.g., in contrast to many known encoding schemes that apply inter-frame or intra-frame prediction to macroblocks and thus introduce macroblock dependencies). Figure 5 The operations shown in can be performed in parallel on 4×4 or 2×2 blocks, which greatly improves the encoding efficiency on a multi-core central processing unit (CPU) or graphics processing unit (GPU).
[0098] like Figure 5As shown, the output of the spatial scalable encoding process is one or more enhancement streams for the enhancement layer, which preferably includes a first enhancement level and another enhancement level. This can then be combined with the base stream at the base level (e.g., via multiplexing or other means), for example, into the transport stream 102 described above or as multiple tracks within another digital container. The first enhancement level (sublayer 1) can be viewed as implementing the corrected video at the base level, that is, for example, correcting the encoder quirks. The second enhancement level (sublayer 2) can be viewed as another enhancement level that is used to convert the corrected video to the original input video or an approximation thereof. For example, the second enhancement level can add fine details lost during downsampling and / or help correct errors introduced by one or more of the transform operation 510-1 and the quantization operation 520-1.
[0099] Figure 6 A corresponding example decoding system 600 for an example spatially scalable coding scheme is shown. The enhancement layer processing shown above the dashed line can be implemented by the second layer decoder described herein, for example, Figure 2A or Figure 2B The second layer decoder 214 and the decoding controller 230, or Figure 3 The base layer processing shown below the dashed line can be performed by the first layer decoder described herein (e.g. Figure 2A and Figure 2B 1st layer decoder 212) or via Figure 3 The HTML media element 304 is implemented.
[0100] exist Figure 6 , the coded elementary stream is decoded at the elementary decoder 620 in order to produce a basic reconstruction of the input signal 501. This basic reconstruction can be used in practice to provide a visual reproduction of the signal 501 at a lower quality level. However, the main purpose of the basic reconstruction signal is to provide a basis for a higher quality reproduction of the input signal 501. To this end, the decoded elementary stream is provided for enhancement layer - sub-layer 1 processing (i.e., sub-layer 1 decoding). Figure 6The sub-layer 1 processing in includes an entropy decoding process 630-1, an inverse quantization process 620-1, and an inverse transform process 610-1. Optionally, only one or more of these steps may be performed depending on the operation performed at the corresponding box 500-1 at the encoder. By performing these corresponding steps, a decoded enhancement layer-sub-layer 1 stream including a first residual set is made available at the decoding system 600. The first residual set is combined with the decoded base stream from the base decoder 620 (i.e., a summation operation 610-C is performed on the frames of the decoded base stream and the frames of the decoded first residual set to generate a reconstruction of a downsampled version of the input video-i.e., the reconstructed base codec video). The frames of the reconstructed base codec video are then upsampled by the upsampler 605U.
[0101] In addition, and optionally in parallel, the encoded enhancement layer - sublayer 2 stream is processed to produce another set of decoded residuals. Similar to the sublayer 1 processing, the enhancement layer - sublayer 2 processing includes an entropy decoding process 630-0, an inverse quantization process 620-0 and an inverse transform process 610-0. Of course, these operations will correspond to the operations performed at box 500-0 in the encoding system 500, and one or more of these steps may be omitted if necessary. Box 600-0 produces a decoded enhancement layer - sublayer 2 stream including another set of residuals, and these are summed with the output from the upsampler 605U at operation 600-C to create an enhancement layer - sublayer 2 reconstruction of the input signal 501, which can be provided as an output of the decoding system 600. Therefore, as Figure 5 and Figure 6 As shown in , the output of the decoding process may include up to three outputs: a base reconstruction, a corrected lower resolution signal, and a higher resolution reconstruction of the original signal for a multi-layer coding scheme.
[0102] refer to Figure 2A and Figure 2B 200 and 240, the residual "decoding" process 610, process 620, process 630 may be performed by the second layer decoder 214, where the two sub-layers are provided as part of the enhancement layer NAL unit. Figure 3 In example 300 , “decoding” processes 610 , 620 , 630 may be performed by enhancement stream decoder 308 , where base layer and enhancement layer data are retrieved from source buffer 302 .
[0103] Typically, the examples described herein operate within a coding and decoding pipeline that includes at least a transform operation. The transform operation may include a DCT or a variant of the DCT, a fast Fourier transform (FFT), or in a preferred embodiment, a Hadamard transform such as implemented by LCEVC. The transform operation may be applied block by block. For example, the input signal may be divided into a plurality of different continuous signal portions or blocks, and the transform operation may include a matrix multiplication (i.e., a linear transform) applied to the data from each of these blocks (e.g., as shown in a 1-dimensional vector). In this specification and in the art, it can be said that the transform operation produces a set of values of a predefined number of data elements, which, for example, represent positions in a transformed composite vector. These data elements are referred to as transform coefficients (or sometimes simply referred to as "coefficients").
[0104] As described herein, in the case where the enhancement data includes residual data, the reconstructed set of coefficient bits may include transformed residual data, and the decoding method may further include instructions to combine the residual data obtained from further decoding of the reconstructed set of coefficient bits with a reconstruction of the input signal generated from a representation of the input signal at a lower quality level to generate a reconstruction of the input signal at a first quality level. The representation of the input signal at the lower quality level may be a decoded base signal, and the decoded base signal may optionally be upgraded before being combined with the residual data obtained from further decoding of the reconstructed set of coefficient bits, the residual data being at a first quality level (e.g., a first resolution). Decoding may further include receiving and decoding residual data associated with a second sublayer, such as obtaining an output of an inverse transformed and inverse quantized component, and combining it with data derived from the aforementioned reconstruction of the input signal at the first quality level. The data may include data derived from an upgraded version of the reconstruction of the input signal at the first quality level, i.e., upgraded to a second quality level.
[0105] Further details and examples of the two sub-layer enhancement encoding and decoding systems can be obtained from the published LCEVC documents. Although the examples have been described with reference to a layer-based hierarchical coding scheme in the form of LCEVC, the methods described herein can also be applied to other layer-based hierarchical coding schemes, such as VC-6: SMPTE VC-6ST-2117, as described in PCT / GB2018 / 053552 and / or related published standard documents, both of which are incorporated herein by reference.
[0106] Figure 7 An example 700 is shown of how a video signal can be decomposed into different components and then encoded. Figure 7In the example of , a video signal 702 is encoded. The video signal 702 includes a plurality of frames or pictures 704, for example, where the plurality of frames represent an action over time. In this example, each frame 704 is composed of three color components. The color components can be in any known color space. Figure 7 In the embodiment of the present invention, the three color components 706 are Y (luminance), U (first chrominance opposite color) and V (second chrominance opposite color). Each color component can be considered as a plane 708 of values. The plane 708 can be decomposed into a set of n times n signal data blocks 710. For example, in LCEVC, n can be 2 or 4; in other video coding techniques, n can be 8 to 32.
[0107] In LCEVC and certain other coding techniques, the video signal fed to the base layer is a scaled-down version of the input video signal, such as 501. In this case, the signals fed to the two sublayers of the enhancement layer include residual signals containing residual data. The planes of residual data may also be organized into n by n signal data blocks 710. The residual data may be generated by comparing data derived from the encoded input signal (e.g., video signal 501) with data derived from a reconstruction of the input signal, the reconstruction of the input signal being generated from a representation of the input signal at a lower quality level. The comparison may include subtracting the reconstruction from the downsampled version. The comparison may be performed on a frame-by-frame (and / or block-by-block) basis. The comparison may be performed at a first quality level; if the base quality level is lower than the first quality level, the reconstruction from the base quality level may be upgraded before the comparison. In a similar manner, the input signal to the second sublayer (e.g., the input for the second sublayer transform and quantized component) may include residual data generated by comparing the input video signal 501 at the second quality level (which may include a full quality original version of the video signal) with a reconstruction of the video signal at the second quality level. As previously described, the comparison may be performed on a frame-by-frame (and / or block-by-block) basis and may include subtraction. The reconstruction of the video signal may include a reconstruction generated from a decoded version of the decoded coded base bit stream and the first sublayer residual data stream. The reconstruction may be generated at the first quality level and may be upsampled to the second quality level.
[0108] Thus, the data plane 708 for the first sublayer may include residual data arranged in n by n signal blocks 710. Figure 7 , where for the color planes, the block may have a value 712 with a set bit length (e.g., 8 or 16 bits). Each n by n signal block may be represented as n, indicating that the length of the signal data block is n. 2The flattened vector 714 is a vector of length n. To perform a transform operation, the flattened vector 714 may be multiplied by a transform matrix 716 (i.e., a dot product is taken). This then generates a matrix of length n representing different transform coefficients for a given signal block 710. 2 Another vector 718 of . Figure 7 An example similar to LCEVC is shown, where the transform matrix 716 is a Hadamard matrix of size 4 by 4, resulting in a transform coefficient vector 718 having four elements with corresponding values. These elements are sometimes referred to by the letters A, H, V, and D, because they can represent the average, horizontal difference, vertical difference, and diagonal difference. This transform operation may also be referred to as directional decomposition. When n=4, the transform operation may use a 16 by 16 matrix, referred to as directional decomposition squared.
[0109] like Figure 7 As shown in , the set of values for each data element on the complete set of signal blocks 710 for the plane 708 can itself be represented as a plane or surface of coefficient values 720. For example, the values of the "H" data elements of the signal block set can be combined into a single plane, where the original plane 708 is then represented as four separate coefficient planes 722. For example, the coefficient plane 722 shown contains all the "H" values. These values are stored in a predefined bit length (e.g., bit length B), which can be 8, 16, 32 or 64 depending on the bit depth. A 16-bit example is given below, but this is not limiting. Therefore, the coefficient plane 722 can be represented as a sequence of 16-bit or 2-byte values 724 representing the value of one data element from the transformed coefficient (e.g., in a memory). These values can be referred to as coefficient bits. These coefficient bits can be quantized and then entropy encoded as discussed to subsequently generate the coded enhancement or second layer data described above.
[0110] The techniques described herein may be implemented in software or hardware, or using a combination thereof. The above examples are to be understood as illustrative. Further examples are contemplated. It should be understood that any feature described with respect to any one example may be used alone or in combination with other features described, and may also be used in combination with one or more features of any other example, or in combination with any combination of any other example. In addition, equivalents and modifications not described above may also be used without departing from the scope of the invention as defined in the attached patent application.
Claims
1. A method for decoding a multi-layer video stream, the multi-layer video stream encoding a video signal and comprising at least a first layer and a second layer, the first layer being decoded using a first decoding method and the second layer being decoded using a second decoding method, the first decoding method using data that the second decoding method cannot access, the method comprising: obtaining, using the second decoding method, second layer decoded data for the multi-layer video stream, the second layer decoded data being associated with at least one frame of the video signal, the second layer decoded data being indexed using a timestamp derived from the multi-layer video stream; receiving a callback indicating availability of first layer decoded data for a frame of the first layer of the multi-layer video stream from the first decoding method; obtaining, for the first layer of decoded data, timing metadata associated with rendering of the frame for the first layer; comparing the timing metadata to one or more timestamps for the second layer of decoded data to pair the first layer of decoded data for the frame with the second layer of decoded data for the frame; as well as combining the first layer of decoded data for the frame and the paired second layer of decoded data for the frame to output a multi-layer reconstruction of the frame of the video signal; and The method is implemented in the browser. The timestamp is obtained from data of the multi-layer video stream retrieved from a source buffer prior to decoding, The first decoding method includes rendering the first layer of decoded data in a first marked video display area, The callback indicating the availability of first layer decoded data is received from the first marked video display area, and The second decoding method includes rendering the reconstruction of the frame of the video signal in a second marked video display area. The method of claim 1 , wherein the method is repeated for each frame of the video signal.
3. A method according to any preceding claim, wherein the timestamp comprises a presentation timestamp.
4. A method according to any preceding claim, wherein the timing metadata comprises one of: a media time derived from the first layer decoded data for the frame; or A current playback time derived from the first layer of decoded data for the frame.
5. The method of any preceding claim, wherein comparing the timing metadata to one or more timestamps for the second layer of decoded data further comprises: A search is performed for second layer decoded data having a timestamp falling within a range defined with reference to a time indicated by the timing metadata. The method of claim 5 , wherein the range is set based on a configurable drift offset. The method of claim 6 , wherein the configurable drift offset is based on a frame rate of the rendering.
8. The method according to claim 6 or claim 7, comprising: A search is performed for second layer decoded data, wherein the timestamp of the second layer decoded data plus the configurable drift offset equals the time indicated within the timing metadata.
9. A method according to any preceding claim, wherein the second layer decoded data is stored in segments having a defined time length, wherein each segment has a plurality of data groups having a start time and an end time, each data group representing data for a frame.
10. A method according to any preceding claim, wherein: The first layer includes a base video stream, and The second layer includes an enhancement video stream.
11. The method of claim 10, wherein the second layer of decoded data comprises frames of residual data combined with frames derived from the base video stream.
12. The method of claim 11, wherein the second layer comprises a Low Complexity Enhancement Video Coding (LCEVC) video stream.
13. The method according to claim 12, wherein before obtaining the second layer decoded data, the method comprises: Passing the LCEVC video stream to the integration layer; extracting LCEVC data from the LCEVC video stream, wherein the LCEVC data includes a timestamp from the LCEVC video stream; LCEVC data is indexed using the timestamp from the LCEVC video stream.
14. A method according to any preceding claim, wherein the second layer comprises a watermark stream.
15. A method according to any preceding claim, wherein the first decoding method is implemented by an operating system service, internal data used by the operating system service being inaccessible to processes other than the operating system service.
16. A method according to any preceding claim, wherein the timed metadata is generated by a first layer decoder.
17. A method according to any preceding claim, wherein the timing metadata is provided as part of the callback.
18. A method according to any preceding claim, wherein the second layer decoded data is in the form of residual data for the frame and the reconstruction is an enhanced frame of the video signal.
19. A method according to any preceding claim, wherein the comparison comprises querying the second layer decoded data to find a match between a timestamp stored with each frame of the second layer decoded data and the timing metadata associated with the first layer decoder data.
20. A method as claimed in any preceding claim, wherein the frames of first layer decoded data are indexed using the timing metadata and the frames of second layer decoded data are indexed using the timestamps from a second layer stream.
21. A system for decoding a multi-layer video stream, the multi-layer video stream encoding a video signal and comprising at least a first layer and a second layer, the system comprising: a second layer decoder for decoding the second layer of the multi-layer video stream; a memory for storing an output of the second layer decoder, the output being indexed using a timestamp derived from the multi-layer video stream; as well as A decoding controller communicatively coupled to a first layer decoder, the first layer decoder using data that is not accessible to the second layer decoder, the decoding controller being configured to: receiving a callback indicating availability of first layer decoded data for a frame of the first layer from the first layer decoder; obtaining, for the first layer of decoded data, timed metadata associated with rendering of the frame for the first layer; comparing the timed metadata to one or more timestamps for the output of the second layer decoder to pair the first layer decoded data for the frame with second layer decoded data for the frame; as well as combining the first layer of decoded data for the frame with paired second layer of decoded data to output a multi-layer reconstruction of the frame of the video signal; and in: The first layer decoder outputs the first layer decoded data to a first markup video display area of a browser; and The second layer decoder receives the callback from the first tagged video display area of the browser and outputs the reconstruction of the frame of the video signal in a second tagged video display area of the browser.
22. The system of claim 21, wherein the timestamp comprises a presentation timestamp, and the timed metadata comprises one of: a media time derived from the first layer decoded data for the frame; or A current playback time derived from the first layer of decoded data for the frame.
23. A system according to claim 21 or claim 22, wherein the decoding controller is configured to search for second layer decoded data having a timestamp falling within a range defined with reference to a time indicated by the timing metadata.
24. The system of any one of claims 21 to 23, wherein the first layer decoder is a base decoder and the second layer decoder is an enhancement decoder, wherein the second layer decoded data comprises one or more of: residual data for a frame of reconstructed video; and Watermark data for frames of the reconstructed video.
25. The system of any one of claims 21 to 24, wherein the second layer decoder is a Low Complexity Enhancement Video Coding (LCEVC) decoder.
26. The system of claim 25, further comprising an integration layer communicatively coupled to the LCEVC decoder.
27. A computer readable medium comprising instructions which, when executed, cause a processor to perform the method according to any one of claims 1 to 20.
