Rendering of layered video signals

Abstract

A method of rendering a representation of a reference video signal, the method comprising: obtaining a first rendered video signal representing the reference video signal at a first level of quality; and superimposing one or more enhancement overlay streams on the first rendered video signal to generate a second rendered video signal representing the reference video signal at a second level of quality, the second level of quality being a higher level of quality than the first level of quality.

Description

Technical Field

[0001] The present invention relates to a method for rendering a representation of a reference video signal, a system for the method, and a computer-readable medium. Background Technology

[0002] Encoding and decoding video content are considerations in many known systems. Video content can be encoded for transmission, for example, over data communication networks. When decoding this video content, it may be necessary to increase the quality level of the video and / or recover as much information as possible contained in the original video. Many video decoding formats and their associated codecs have been developed in an attempt to achieve these desired characteristics, but often require significant software updates and / or hardware upgrades at the operating system level. Furthermore, increasing the quality of decoded video content typically requires increasing the complexity of the encoding and decoding procedures, which may increase power consumption and latency in delivering the video content.

[0003] One approach to improving the quality of decoded video content is to employ a multi-layer video decoding scheme. Although these technologies have existed for many years, they have encountered problems in widespread adoption. The majority of video content on the Internet is still encoded using H.264 (also known as MPEG-4 Part 10 Advanced Video Decoding – MPEG-4 AVC), a format used for 80% to 90% of online video content. This content is typically supplied to a decoding device as a single video stream, with a one-to-one relationship between the stream and an available hardware and / or software video decoder. For example, a single stream is received, parsed, and decoded by a single video decoder to output a reconstructed video signal. Therefore, many video decoder implementations are developed based on this framework. To support different encodings, decoders are typically configured with a simple switching mechanism driven by metadata identifying the stream format.

[0004] Existing multi-layer decoding schemes include Scalable Video Coding (SVC) extensions to H.264, Scalable Extensions to H.265 (MPEG-H Part 2 High-Efficiency Video Coding - SHVC), and newer standards such as MPEG-5 Part 2 Low Complexity Enhancement Video Coding (LCEVC). While H.265 is an evolution of the decoding framework used in H.264, LCEVC takes a different approach to scalable video. SVC and SHVC operate by creating different coding layers and feeding each of these layers with a different spatial resolution. Each layer encodes the input according to a normal AVC or HEVC encoder, potentially utilizing information generated by lower coding layers. On the other hand, LCEVC generates one or more layers of enhancement residuals compared to the base coding, where the base coding can have a lower spatial resolution.

[0005] One reason for the slow adoption of multi-layer decoding schemes is the difficulty in adapting existing and new decoders to handle multi-layer encoded streams. As discussed above, a video stream is typically a single data stream paired one-to-one with a suitable decoder, whether that decoder is implemented in hardware, software, or a combination of both. Client devices and media players (including internet browsers) are therefore built to receive the data stream, determine what video encoding the stream uses, and then pass the stream to the appropriate video decoder. Within this framework, multi-layer schemes (such as SVC and SHVC) are typically packaged into a larger single video stream containing multiple layers, which can be detected as "SVC" or "SHVC," and multiple layers are extracted from the single stream and passed to an SVC or SHVC decoder for reconstruction. However, this approach fails to provide some of the benefits of multi-layer encoding. Therefore, many developers and engineers have concluded that multi-layer decoding schemes are too cumbersome and instead revert to multicasting a single H.264 video stream.

[0006] It may also be necessary to embed the video content within a webpage for playback by end users of Global Times. To display the video content within a webpage, a media element can be included in a Hypertext Markup Language (HTML) document. This media element embeds a media player into the webpage and allows the video content to be played. For example, the latest version of HTML (i.e., HTML5) includes a video element for embedding video content. However, browsers may not be able to render video content in a specific video decoding format. Specifically, browsers may not be able to render video content encoded using a multi-layered decoding scheme.

[0007] Further problems arise when using existing multi-layer decoding schemes to encode protected content.

[0008] Generally, there are two common methods for protecting video delivery: Conditional Access (CA) and Digital Rights Management (DRM). Conditional Access is used in the more conventional broadcast world and has a physical authentication system (usually a smart card). For online content distribution, operators typically use Digital Rights Management (DRM). The purpose of both methods is to prevent consumers from unauthorized access to content, as this would allow them to freely distribute the content to others.

[0009] In terms of protection, three aspects generally need to be considered: the compressed video stream should be encrypted; then the output of the decrypted video stream to the display should go through a protected channel, such as using High Bandwidth Digital Content Protection (HDCP); and software should be unable to capture content from the decoded and decrypted video. The latter can be achieved through a secure platform that prevents the execution of unauthorized software or by utilizing a secure memory controller that prevents general access to secure memory.

[0010] Therefore, implementing multi-level decoding schemes (such as LCEVC) for protected content must address the security protections used to prevent unauthorized content capture.

[0011] In standard methods for encoding protected content using LCEVC, only the base layer is security vulnerable because the residual data in the enhancement layers is sparse and not a concern without decoding the base video. However, the encryption of the base layer and the security protections applied to the display of the rendered base video make it difficult to apply enhancements to the base video. For example, standard methods for implementing LCEVC involve combining frames of the base video with frames of one or more enhancement layers, which is impossible when security protections do not allow access to individual frames of the base video.

[0012] Therefore, there is a need for an improved method and system for decoding multi-layer video data that overcomes some of the drawbacks discussed above and allows for more advantages of implementing multi-layer coding schemes. Summary of the Invention

[0013] According to a first aspect of the present invention, a method for rendering a representation of a reference video signal is provided, the method comprising: obtaining a first rendered video signal representing the reference video signal at a first quality level; and superimposing one or more enhancement overlay streams onto the first rendered video signal to generate a second rendered video signal representing the reference video signal at a second quality level, the second quality level being a higher quality level than the first quality level.

[0014] When applying enhancement to the first rendered video signal by overlaying one or more enhancement overlay streams, the present invention eliminates the need to access data within the first rendered video signal itself. Compared to conventional layered coding schemes where access to the base layer is required to render the enhanced video signal, in the present invention, the first rendered video signal and the one or more enhancement overlay streams can be processed entirely independently. This means that the present invention finds particular advantage in rendering representations of reference video signals protected by security measures such as Conditional Access (CA) or Digital Rights Management (DRM).

[0015] The present invention is particularly useful when rendering a representation of a reference video signal encoded according to LCEVC technology, in which case the first rendered video signal corresponds to the rendered base video signal. In conventional methods of decoding video encoded according to LCEVC technology, the enhanced video signal is generated frame-by-frame by combining each frame of the base video signal with corresponding frames of one or more enhanced streams. However, since this requires access to frames of the base video signal, this method may be impossible if the base video signal is securely protected. Conversely, the present invention allows the enhancement to be applied without accessing the base video signal.

[0016] The step of obtaining the first rendered video signal preferably includes: obtaining a first decoded video stream; and rendering the first rendered video signal in a first marked video display area based on the first decoded video stream.

[0017] More preferably, obtaining the first decoded video stream includes: receiving a first encoded video stream; and, during a first decoding process, decoding the first encoded video stream to obtain the first decoded video stream. The first decoding process can be performed according to a standard encoding scheme (such as H.264). However, security protections associated with protected content may limit the decoding process that can be used to obtain the first rendered video signal.

[0018] Advantageously, this method is used to render the first encoded video signal in the form of an HTMLVideoElement. Thus, the first marked video display area can be an HTML element. <video>element.

[0019] The step of overlaying the one or more enhanced overlay streams preferably includes: obtaining one or more decoded enhanced streams; and rendering the one or more enhanced overlay streams in a second marked video display area based on the one or more decoded enhanced streams, the second marked video display area coinciding with the first marked video display area, that is, the second marked video display area is overlaid on the first marked video display area, and the first and second marked video display areas have the same size. As will be understood, the first rendered video signal fills the first marked video display area and the one or more enhanced overlay streams fill the second marked video display area, such that the one or more enhanced overlay streams are aligned with the first rendered video signal.

[0020] More preferably, obtaining one or more decoded enhanced streams includes: receiving one or more encoded enhanced streams; and, in the second decoding process: decoding the one or more encoded enhanced streams to obtain one or more decoded enhanced streams.

[0021] One or more encoded augmented streams may be received as part of the same data stream as the first encoded video stream, or they may be received in one or more separate data streams.

[0022] In most embodiments, the first and second decoding processes are performed according to different video decoding schemes. Furthermore, the first and second decoding processes are preferably performed by different decoding elements. For example, if the method is implemented within a browser, the first and second decoding processes can be performed by separate elements implemented by the browser. In one such implementation, the first decoding process can be performed by an HTML media element, while the second decoding process can be performed by an enhanced streaming decoder.

[0023] The second marked video display area can be HTML. <canvas>element. When the first tag video display area is HTML <video>This is the preferred option when considering elements.

[0024] As described above, the present invention is particularly useful when used to restrict access to data in a first rendered video signal for security protection purposes that prevent unauthorized capture of content. Thus, the first rendered video signal can include data that is inaccessible to a second decoding method. Nevertheless, the second decoding method can be based on other data obtained from the first rendered video signal, such as metadata including the playback time of the rendered video signal.

[0025] One or more encoded augmented streams may include one or more sets of encoded residuals, and decoding one or more encoded augmented streams may include decoding one or more sets of encoded residuals to obtain one or more sets of decoded residuals. The one or more sets of decoded residuals can then be used to render one or more augmented overlay streams.

[0026] Specifically, one or more encoded enhancement streams may include at least a second set of encoded residuals in a second enhancement sublayer of a hierarchical coding scheme. As described above, the present invention can be used to render video encoded using LCEVC technology. As will be understood, the LCEVC decoding scheme is based on representing a reference video signal using a base stream and first and second enhancement streams. However, the representation of the reference video signal can be rendered based solely on the base stream and the second enhancement stream. The present invention is compatible with this method.

[0027] In a preferred embodiment, at least one set of decoded residuals in one or more sets of decoded residuals includes positive residuals and negative residuals, and decoding one or more encoded enhanced streams includes: obtaining one or more decoded positive residual streams from positive residuals; and obtaining one or more decoded negative residual streams from negative residuals.

[0028] In a more preferred embodiment, rendering one or more enhanced overlay streams includes: rendering one or more positive residual overlay streams in a second marked video display area based on a decoded positive residual stream; and rendering one or more negative residual overlay streams in the second marked video display area based on a decoded negative residual stream.

[0029] For example, a decoded residual can represent a modification to the luminance component of a video signal, where a positive residual indicates that the luminance of a pixel should be increased, and a negative residual indicates that the luminance of a pixel should be decreased. To most effectively implement these positive and negative residuals in one or more enhancement overlay streams, the residuals are preferably split into separate decoded positive and negative residual streams, which can then be used individually to render the positive and negative residual overlay streams in a second marked video display area.

[0030] While in some embodiments there may be only one positive residual superposition stream and one negative residual superposition stream, any number of positive and negative residual superposition streams can exist. For example, there can be two pairs of positive and negative residual superposition streams, each pair corresponding to an enhancement sublayer of a hierarchical coding scheme (such as LCEVC). There can also be more or fewer positive residual superposition streams than negative residual superposition streams.

[0031] In some embodiments, at least one of one or more positive residual overlay streams can encode modifications to the luminance component of the rendered base video, the at least one positive residual overlay stream comprising an array of values ​​between 0 and 1, each of which corresponds to a pixel of the second rendered video signal, and rendering the at least one positive residual overlay stream in the second marked video display area comprises, for each pixel of the second rendered video signal, rendering a white pixel having an opacity equal to the corresponding value in the array of values.

[0032] In a further embodiment, at least one of one or more negative residual overlay streams can encode modifications to the luminance component of the rendered base video, the at least one negative residual overlay stream comprising an array of values ​​between 0 and 1, each of which corresponds to a pixel of the second rendered video signal, and rendering the at least one negative residual overlay stream in the second marked video display area comprises, for each pixel of the second rendered video signal: averaging the corresponding value from the value array with the luminance value of the underlying pixel in the first rendered video signal, and rendering the pixel having the average luminance value.

[0033] Advantageously, overlaying one or more enhanced overlay streams may include synchronizing one or more enhanced overlay streams with the first rendered video signal based on the playback time of the first rendered video signal. Preferably, each of the one or more enhanced overlay streams includes a series of frames indexed using a corresponding series of timestamps; and, for each of the one or more enhanced overlay streams, synchronizing the enhanced overlay stream with the first rendered video signal includes: comparing the current playback time of the first rendered video signal with the timestamps indexed to the frames of the enhanced overlay stream; identifying, based on the comparison, a frame of the enhanced overlay stream corresponding to the current playback time; and overlaying the frame of the enhanced overlay stream corresponding to the current playback time onto the first rendered video signal. The timestamps may be obtained from the first rendered video signal. In embodiments where enhanced overlay streams are derived from one or more encoded enhanced streams, timestamps may be derived from one or more encoded enhanced streams in addition to obtaining timestamps from the first rendered video signal, or as an alternative to obtaining timestamps from the first rendered video signal.

[0034] Comparing the current playback time of the first rendered video signal with the timestamps of the indexed frames of the enhanced overlay stream can include searching for frames in the enhanced overlay stream with timestamps falling within a range defined by the reference current playback time. This range can be set based on a configurable drift offset, which in turn can be based on the frame rate of the first rendered video signal. In this case, searching for frames in the enhanced overlay stream whose timestamps fall within the range defined by the reference current playback time can include searching for frames in the enhanced overlay stream where the frame's timestamp plus the configurable drift offset equals the current playback time.

[0035] The second rendered video signal represents the reference video signal at a second quality level, which is a higher quality level than the first quality level. For example, the second rendered video signal may have a higher resolution compared to the first rendered video signal, and / or the second rendered video signal may have a higher dynamic range compared to the first rendered video signal. The second rendered video signal may be, for example, HDR video, while the first rendered video signal may be SDR video.

[0036] One or more of the enhancement overlay streams can encode modifications to the luminance and / or chrominance components of the rendered base video.

[0037] In some embodiments, the method can be advantageously implemented within a browser.

[0038] According to a second aspect of the invention, a system for rendering a representation of a reference video signal can be provided, the system comprising: a video renderer configured to obtain a first rendered video signal representing the reference video signal at a first quality level; and an enhancement decoder configured to superimpose one or more enhancement overlay streams onto the first rendered video signal to generate a second rendered video signal representing the reference video signal at a second quality level, the second quality level being a higher quality level than the first quality level.

[0039] The system provided herein can be configured to implement the method according to the first aspect of the invention.

[0040] According to a third aspect of the invention, a computer-readable medium may be provided, the computer-readable medium comprising instructions that, when executed, cause a processor to perform a method according to a first aspect of the invention. Attached Figure Description

[0041] The invention will now be described with reference to the accompanying drawings, in which: Figure 1 This is a diagram illustrating how encoded video data is transmitted across various data streams; Figure 2A and Figure 2B This is a schematic diagram illustrating a conventional system for receiving and decoding multiple encoded data streams; Figure 3A and Figure 3B This is a schematic diagram illustrating an exemplary browser implementation of a system for receiving and decoding multiple encoded data streams; Figure 4A and Figure 4B This is a schematic diagram illustrating an exemplary browser implementation of a system for receiving and decoding multiple encoded data streams according to an embodiment of the present invention; Figure 5 This is a flowchart illustrating an exemplary method for decoding multi-layer video streams according to an embodiment of the present invention; and Figure 6 This is a schematic diagram illustrating an exemplary multilayer encoder configuration. Detailed Implementation

[0042] Some of the examples described herein allow decoding devices to be easily adapted to handle multi-layer video decoding schemes. While some examples are described with reference to LCEVC multi-layer video streams, the general concepts can be applied to other multi-layer video schemes, including SVC and SHVC, as well as multi-layer watermarking and content delivery schemes. Some of the examples described herein are particularly useful when the first layer of a multi-layer video stream is decoded by a first-layer decoder implementing a first decoding method, and the second layer of the multi-layer video stream is decoded by a second-layer decoder implementing a second decoding method. To allow for flexibility in multi-layer configuration, the first layer can be encoded using various video decoding methods (such as H.264 and H.265 as described above) as well as new and / or unimplemented video decoding methods (e.g., generic video decoding (VVC or H.266)). Therefore, the first-layer decoder can vary for different encoded video streams.

[0043] In the example described herein, support is provided for a first-layer decoder that fully encapsulates access to internal data. For example, the first-layer decoder may include hardware components such as a secure hardware decoder chipset, where other processes within the client device performing the decoding cannot access the data provided in the packets used for the first layer. Furthermore, the protection applied to the data in the first layer can be applied to the decoding output of the first-layer decoder, for example, by preventing access to individual frames of the rendered video. Thus, the example described herein allows the data of the second layer of a multi-layer video stream to be decoded independently using different decoding methods, but subsequently combined with the appropriate output of the first-layer decoder, for example, to provide an enhanced video output.

[0044] Specifically, the examples described herein enable the output of the second-layer decoder to be combined with the output of the first-layer decoder, where the first-layer decoder uses data inaccessible to the second-layer decoder. For example, while conventional methods of combining the output of the second-layer decoder with the output of the first-layer decoder may involve combining decoded frames of video data from the first-layer decoder with one or more decoded frames from the second-layer decoder, the second-layer decoder may not have access to the data of individual frames of the video data from the first-layer decoder. This disclosure provides techniques for addressing these limitations. This example may be advantageous when the second layer includes residual data, watermarked data, and / or localized embedded metadata.

[0045] In some variations, the second-layer decoded data may have two or more sub-layers, each with two or more resolutions (e.g., spatial resolution). In these cases, the matched first-layer decoded data, regardless of whether it is corrected at the decoding resolution, can be upsampled to a higher resolution to provide enhancement via the second layer. In some implementations, the first-layer decoded data may also be used in the output process and in combined reconstructions to provide different viewing options. For example, video may be rendered by the first-layer decoder and presented without enhancement.

[0046] In some of the examples described herein, different layers of multi-layer video decoding can be transmitted as separate packets multiplexed within the transport stream. This allows different layers to be efficiently and individually supplied, and allows enhancement layers to be easily added to a pre-existing or pre-configured base layer. At the decoding device, different packet substreams can be received and parsed, for example, based on the packet identifier (PID) in the packet header.

[0047] In the following description, refer to Figure 1 A first example describing an encoded video stream. Then refer to... Figure 2A , Figure 2B , Figure 3A and Figure 3B Examples of common systems and methods for decoding multi-layer video streams are described. Then refer to Figure 4 and... Figure 5 Examples of systems and methods for decoding multi-layer video streams according to embodiments of the present invention are described below. Reference is then made to... Figure 6 Examples describing a specific multi-layer decoding scheme.

[0048] Figure 1 Example 100 of transport stream (TS) 102 is shown, which can be used to send encoded video data to one or more decoding devices. Transport stream 102 includes a sequence of fixed-length 188-byte TS packets 110. Each TS packet 110 has a header 112, which may have a variable length, and a payload 114. The header 112 includes one or more data fields. One of these data fields provides a packet identifier (PID) 116. The PID is used to distinguish different substreams within the transport stream. The PID can be a fixed-length (e.g., 13 bits) number of bits storing a numeric or alphanumeric identifier (typically represented as a hexadecimal value). For example, PID 116 can be used to identify different video streams that are multiplexed together to form a single stream of transport stream 102. The example transport stream specification is set forth in MPEG-2 Part 1 and is defined as part of ISO / IEC Standard 13818-1 or ITU-T Rec.H.222.0.

[0049] Figure 1 One of the so-called PID streams 104 that can be extracted from transport stream 102 is also shown. PID stream 104 comprises streams of consecutive groups 110 having a common (i.e., shared) PID value. PID stream 104 can be created by decompiling transport stream 102 based on the PID value. Therefore, PID stream 104 represents a sub-stream of transport stream 102.

[0050] In some cases, there may be special PID values ​​reserved for index tables. In one instance, a PID value can be reserved for a Program Association Table (PAT), which contains a directory list of program mapping tables (PMTs) that include mappings between one or more PID values ​​and specific "programs." Originally, "program" referred to something associated with a specific broadcast program, but with the advent of internet streaming, the term is widely used to refer to content associated with a specific video stream.

[0051] Figure 1 Also shown is a packetized basic stream (PES) 106 constructed based on payload data from multiple TS packets 110. The PES includes payload data from a PID stream carrying media sample data. The media sample data may include video data as well as other modalities such as audio data, caption data, or volumetric data. Figure 1 In this process, a PES is generated by combining the payloads 114 of multiple media TS packets 110 associated with a common (i.e., shared) PID value. The PES comprises a packet stream, where each PES packet consists of a header (i.e., the PES header) 122 and a payload 124 carrying the combined data. The start of a new PES packet is indicated by a one-bit field from the TS header 112, referred to as the Payload Unit Start Indicator (PUSI) 118. When the PUSI is set, the first byte of the TS packet payload 114 indicates the position where a new PES payload unit begins. This allows a decoding device that begins receiving data midway through transmission to determine when to begin extracting data. The PES header 122 contains a Presentation Timestamp (PTS) 128. This indicates the presentation time for the corresponding media segment encapsulated within the payload 124.

[0052] Figure 1 Finally, the contents of the PES payload 124 for the video stream are shown. In this case, the PES payload 124 includes a sequence 108 of NAL units 130 (i.e., a NALU stream). These can form part of the access units for the video stream, i.e., a set of NAL units that are associated with a specific output time, are sequential in decoding order, and contain decoded pictures or frames. Figure 1 A NALU stream 108 is shown that can be provided to a suitable video decoder for decoding.

[0053] Figure 1 The example illustrates how different layers of a multi-layer video stream (e.g., an encoded video stream) can be transmitted to a decoding device. For example, transport stream 102 can be transmitted via one or more communication channels, including wireless transmission and network communication. Figure 1 In the example, different layers of a multi-layer video stream can be delivered as different PID sub-streams. For instance, a group with a PID of "B" can represent the encoded data of the first layer ("base") stream, while a group with a PID of "L" can represent the encoded data of the second layer ("LCEVC" or enhanced) stream. The different sub-streams of each layer can be demultiplexed and provided as PES 106 to the video decoder. Although Figure 1 An exemplary transport stream is shown, but other digital media containers (such as "tracks" on computer-readable media (such as optical discs or solid-state storage) can also be used to provide encoded layer data. In some cases, encoded data at different layers (e.g., as different tracks from the provided medium) can be read using the client device's file system.

[0054] Before describing examples of restricting access to data in the first-layer stream, it will be enlightening to first describe the common techniques used in situations with fewer restrictions.

[0055] Figure 2A An example of a conventional system 200 for decoding multi-layer video streams is shown. According to this example, access to data in the first-layer stream is unrestricted, and the first-layer decoded data can be enhanced by pairing first-layer decoded data of a specific frame with second-layer decoded data of said frame.

[0056] System 200 is typically implemented as part of a client computing device, such as a smartphone, laptop computer, smart TV, or other media receiver and / or player. The multi-layer video stream encodes the video signal and includes at least a first layer and a second layer. The first layer may include encoding according to a first decoding method, scheme, or standard, such as one of H.264, H.265, or H.266. The first layer may be referred to as the "base" layer. The first layer may be encoded according to a first quality level, such as a first spatial resolution, a first quantization level, a first specified or desired bit rate, and / or a first temporal resolution. The first layer may be complete video encoding, i.e., the encoded data can be received, decoded, and rendered regardless of the presence of other layers. The second layer may include encoding according to a second encoding method, scheme, or standard. For example, the second layer may include an "enhancement" layer for enhancing or otherwise extending the "base" layer. The second layer may be encoded using an enhancement encoding method such as LCEVC. The second layer may include encoding of residual data streams, for example, for combination with the first layer. The second layer can be encoded according to a second quality level (such as a second spatial resolution, a second quantization level, a second specified or desired bit rate, and / or a second temporal resolution). The second quality level can be higher than the first quality level, for example, to provide enhancement. Each layer can be a series of NAL units (such as...) Figure 1 (108) is received, wherein the NAL unit includes encoded data for the layer for a specific frame. Each layer may include a sequence of encoded data for consecutive frames. In some cases, the sequence of encoded data may be associated with a group of images.

[0057] exist Figure 2A In this system, system 200 receives a first layer stream 202 and a second layer stream 204. The term "stream" is used herein to refer to a continuous portion of data that is received or accessed. The first layer stream 202 and the second layer stream 204 may be generated by demultiplexing operations, such as operations performed on transport stream 102 to extract one of the PID stream 104 or PES 106, or may include data read from one or more files.

[0058] exist Figure 2A In system 200, a first-layer stream 202 is provided to a first-layer video decoder 212 and a second-layer stream 204 is provided to a second-layer decoder 214. The first-layer video decoder 212 is configured to decode the first layer of a multi-layer video stream, and the second-layer video decoder 214 is configured to decode the second layer of the multi-layer video stream. The first-layer video decoder 212 can be implemented using a video codec. The video codec can be hardware-based and / or software-based. In some cases, the first-layer video decoder 212 can use hardware acceleration, where one or more actions performed as part of decoding are implemented using specially configured hardware devices, such as a specific video decoding chipset. In some cases, the first-layer video decoder 212 can be implemented using one or more operating system services, such as features provided as part of an operating system like iOS®, Windows®, or Linux®. The second-layer video decoder 214 can be an LCEVC decoder, such as a software decoder implemented according to the LCEVC standard.

[0059] exist Figure 2A In this configuration, the second-layer video decoder 214 is communicatively coupled to memory 216. Memory 216 is configured to store the output of the second-layer decoder 214. Memory 216 may include a dedicated hardware buffer and / or a portion of shared system memory reserved for video decoding. The second-layer video decoder 214 is configured to decode second-layer data frames 224 and store them in memory 216. If the second-layer video decoder 214 is configured to receive and decode one or more sublayers of residual data (as in LCEVC), the data frames 224 may include one or more frames at one or more corresponding resolutions.

[0060] Figure 2A A decoding controller 230 is also shown. The decoding controller 230 may form part of a decoder integration layer configured to control the decoding of multiple layers of video, for example, in association with a first-layer video decoder 212 and a second-layer video decoder 214. In one embodiment, the first-layer video decoder 212 may be implemented as a standalone video decoder (e.g., a standalone codec) for the first layer (e.g., capable of decoding first-layer data without second-layer data), and the decoding controller 230 and the second-layer video decoder 214 may be implemented as part of a multi-layer decoder configured to enhance the first layer using one or more additional video coding layers. The decoding controller 230 may be implemented in software (e.g., executed by a processor of a client device) and / or using dedicated hardware. In one embodiment, the decoding controller 230 and the second-layer video decoder 214 may be implemented as software (including firmware) enhancements to existing or conventional video decoding systems, including systems with hardware acceleration for the first layer. An exemplary decoding system that can be adapted to provide the functionality of a second-layer video decoder 214 and a decoding controller 230 is described in PCT / GB2021 / 051940, which is incorporated herein by reference.

[0061] exist Figure 2A In this configuration, decoding controller 230 is communicatively coupled to first-layer decoder 212. Decoding controller 230 is configured to receive callback 222 from first-layer decoder 212 indicating the availability of first-layer decoded data for frames of the first layer. For example, decoding controller 230 may request a callback from first-layer decoder 212 whenever first-layer decoded data is ready for rendering. The term "callback" is used herein to refer to communication or signals sent between hardware and / or software components to indicate that an event has occurred. In hardware, a callback may include physical signals and / or changes in register values ​​(e.g., representing binary flags) sent via a communication bus or channel. In software, a callback may include asynchronous function returns and / or changes in monitored values ​​in memory. Callback 222 may include a callback used for rendering the output of first-layer decoder 212 in comparative cases. For example, callback 222 may indicate that a video frame encoded using the first layer has been decoded from first-layer stream 202 and is ready to be displayed as part of a rendered video. In some cases, callback 222 may include data for the decoded frame (e.g., frame layer 1-FL1); in other cases, callback 222 may include a reference indicating where the decoding controller 230 can access the decoded data, such as a memory address.

[0062] Upon receiving callback 222, decoding controller 230 is configured to further obtain timing metadata 232 for the first-layer decoded data (e.g., the decoded first-layer frame). Timing metadata 232 is associated with the rendering of the first-layer frame. Timing metadata 232 may be generated by the first-layer decoder 212. For example, timing metadata 232 may be generated to assist downstream processes in rendering or otherwise displaying the first-layer decoded data. In one case, timing metadata 232 may include either the media time of a frame derived from the first-layer decoded data or the current playback time of a frame derived from the first-layer decoded data. Timing metadata 232 may be provided as part of callback 222 and / or may be accessed along with the first-layer decoded data, for example, from a memory address associated with the memory address of the first-layer decoded data.

[0063] Upon receiving callback 222 and having acquired timing metadata 232, decoding controller 230 is configured to compare the timing metadata with one or more timestamps of the second-layer decoder output to pair the first-layer decoded data of the frame with the second-layer decoded data of the frame. This is possible in conventional techniques because the data in the first-layer stream 202 is accessible.

[0064] exist Figure 2A In the example, decoding controller 230 queries 236 the second-layer decoded data 224 stored in memory 216. Specifically, decoding controller 230 searches for a match between the timestamp stored with each part of the second-layer decoded data (such as each decoded frame of the second-layer data) and the timing metadata 232 (e.g., media or playback time) associated with a ready frame of the first-layer decoded data. For example, decoding controller 230 can be configured to search for second-layer decoded data with timestamps falling within a range defined by a reference time indicated by the timing metadata. This range can be set based on a configurable drift offset. In this case, decoding controller 230 can search for data within memory where the timestamp of the data plus a small drift offset equals the time indicated within the timing metadata. The configurable drift offset can be a small number of milliseconds (e.g., 10 ms) and can be set based on the rendered frame rate (e.g., it can be reduced for higher frame rates).

[0065] exist Figure 2A In this process, the decoding controller 230 retrieves second-layer decoded data 234 based on comparison and combines it with first-layer decoded data 222 to output a reconstructed frame 238 of the video signal. The reconstructed frame 238 may be an enhanced frame, whereby the second-layer decoded data 234 in the form of residual data of the frame is combined with the first-layer decoded frame. The decoding controller 230 may apply one or more enhancement sub-layers based on the second-layer decoded data 234; for example, the second-layer decoded data 234 may include two sub-layers of residual data, as referred to later. Figure 6 As described. In some configurations, if the second-layer decoded data 234 is not located, for example due to problems with receiving the second-layer stream or network congestion, the decoding controller 230 may simply output the first-layer decoded data 222 (e.g., without enhancement), for example, operating in pass-through mode. The output of the decoding controller 230 may be rendered as part of a client device forming the implementation of system 200 or communicatively coupled to a display of said client device. In other cases, the output of the decoding controller 230 may be made available to other processes, for example, in memory 216 or another buffer.

[0066] Figure 2B Another configuration 240 of the conventional system 200 is shown. Figure 2A In the system configuration, each of the first-layer video decoder 212, the second-layer video decoder 214, and the decoding controller 230 accesses data stored in memory 216. Figure 2A The system configuration is the same, access to data in the first layer stream is unrestricted, and the first layer decoded data can be enhanced by pairing the first layer decoded data of a specific frame with the second layer decoded data of the frame.

[0067] In this configuration, the decoding controller 230 is configured to receive callbacks from both the first-layer video decoder 212 and the second-layer video decoder 214. The first-layer video decoder 212 sends a first callback 232 to the decoding controller 230 to indicate that a new frame of the first-layer data 242 is ready. The second-layer video decoder 214 sends a callback 252 to the decoding controller 230 to indicate that a new frame of the second-layer data 224 is ready. Both the first-layer video decoder 212 and the second-layer video decoder 214 can be configured to buffer the decoded frames in memory 216. Although memory 216... Figure 2B It is shown as shared memory, but it can include separate memory that can be accessed by appropriate components (e.g., hardware or software frame buffers dedicated to each decoder).

[0068] and Figure 2A Similarly, the frames of the first-layer data 242 are indexed using timed metadata values, where the timed metadata values ​​are... Figure 2B The time shown in the middle is the media playback time t. MPB And the frames of the second-layer data 224 are indexed using timestamps (e.g., PTS values) from the second-layer stream 204. Figure 2B In this process, the decoding controller 230 is configured to coordinate the reception of callbacks 232 and 252 and compare timing metadata and timestamp values ​​to locate corresponding frames of the first-layer data 242 and the second-layer data 224. In some cases, the decoding controller 230 may conditionally act upon receiving both callbacks 232 and 252. After the comparison, the corresponding frames are then combined, for example by the decoding controller 230 or via arithmetic performed through memory 216, to output the multi-layer reconstruction 238. Figure 2B In this process, multi-layer reconstruction 238 can then be used for rendering, for example, it can be copied to a dedicated frame buffer of the display device. As described above, frames of the first layer data 242 and the second layer data 224 can be obtained by looking up a timing metadata value (e.g., t) equal to a timestamp value plus a configurable drift offset. MPB = PTS + offset) to match.

[0069] Figure 3A This is a schematic diagram of another conventional system 300 used for decoding encoded multi-layer video streams, such as encoded multi-layer video streams using LCEVC. System 300 can be implemented within a browser. Figure 2A and Figure 2B The browser can be one of systems 200 and 240. It can be any browser capable of accessing information on the Global Network, examples of which include (but are not limited to) Google Chrome®, Microsoft Edge®, Safari®, Firefox®, and Opera®. The browser can be implemented on a client device. Examples of client devices include (but are not limited to) mobile devices, computing devices, tablet devices, smart TVs, etc. Client devices typically include an operating system (OS), and the OS includes a browser.

[0070] One function of a browser is to transform documents written in markup scripting languages ​​(sometimes called markup languages) into a visual representation of web pages. Markup scripting languages ​​are used to control the display of data in a rendered web page. A markup language can contain markup video elements, which then become the video display area when processed by the browser. For example, a browser user can navigate to a web page containing embedded video. When the browser renders the web page, it receives data corresponding to the video. The browser may contain the resources necessary to decode and play the video so that it can be displayed to the user, for example, within the video display area rendered by the browser on the client device's monitor. Examples of markup scripting languages ​​include any version of Hypertext Markup Language (HTML), such as HTML5 and Extensible Hypertext Markup Language (XHTML).

[0071] For example, the video tag indicates properties associated with the display of a video on a webpage, such as the size of the video within the webpage and whether the video will play automatically after the webpage loads. The video tag may also include an indication of the video decoding format used to encode the video. This instructs the browser which decoders to use to decode the encoded video. The browser may then make a call to at least one of the decoding functions within its own resources (which can be considered browser-native resources, on the browser's own machine) or to decoding functions implemented in the OS, as discussed further below.

[0072] Figure 3A System 300 includes a source buffer 302 to receive encoded multi-layer video streams. In the example described, source buffer 302 can receive... Figure 2A and Figure 2B The first layer stream 202 and the second layer stream 204 are the source of each other. In other examples, each stream may have a separate corresponding source buffer instead of a joint buffer for both streams. A source buffer is a segment of memory, for example, that is accessible by a browser. A source buffer may be, for example, a Media Source Extension (MSE) Application Programming Interface (API) SourceBuffer. In this example, the encoded multi-layer video stream includes an encoded base stream and an encoded enhancement stream. The encoded base stream includes video content encoded by any base encoder (also known as a compressor), such as Advanced Video Decoder (AVC), High Efficiency Video Decoder (HEVC), VP9, ​​MPEG-5 Basic Video Decoder (EVC), or AOMedia Video 1 (AV1) encoder.

[0073] System 300 also includes an HTML media element 304 that implements a first-layer (e.g., a basic stream) video decoder, also known as a basic stream decompressor. The HTML media element 304 may indicate or otherwise implement the first-layer video decoder 212, as referenced in [reference]. Figure 2A and Figure 2B As described. In the example, the encoded underlying stream is extracted from source buffer 302 and decoded using an HTML media element 304. The HTML media element 304 is a markup video element that provides an interface between the markup language and the decoding resources. For example, HTML5 includes markup video elements that can be used to embed video content in web pages. Another example is a JavaScript library that builds a set of custom controls on top of the HTML5 video element, which can be thought of as acting as a JavaScript player. It should be understood that markup video elements, such as the HTML5 video element, can be modified by adding additional resources that can be accessed by the markup video element, such as multi-layer video streaming decoding libraries, WebAssembly decoding libraries, and / or web worker functionality.

[0074] When using LCEVC, the enhancement stream can be carried within a set of Supplemental Enhancement Information (SEI) messages that accompany and are associated with the base stream, or within a separate Network Abstraction Layer (NAL) unit stream, for example, as Figure 1 As shown, this is carried in the PID stream. The basic stream decoders can be configured to ignore SEI messages or NAL unit streams if they contain information they cannot interpret, such as header information indicating a specific packet type. Therefore, in this case, the HTML media element 304 can retrieve data related to the basic stream from the source buffer 302 in a default manner, where both enhanced and non-enhanced basic streams are processed in a common way. In this case, the HTML media element 304 can ignore SEI messages or NAL units carrying the enhanced stream residing in the source buffer 302.

[0075] exist Figure 3A In some implementations of the exemplary system 300, the marked video element contains an indication of the video decoding format associated with the encoded underlying stream. When processed by a browser, the marked video element locates the appropriate underlying stream decoder associated with the video decoding format and decodes the encoded underlying stream. The HTML media element 304 can be implemented within a browser, utilizing the functionality of the OS of a client device including the browser, or leveraging the resources of both the browser and the OS. For example, the OS can utilize hardware acceleration to decode the encoded underlying stream, which reduces power consumption and the number of computations performed by the CPU compared to software decoding alone.

[0076] The decoded underlying stream is rendered in the first marked video display area 306. For example, the first marked video display area 306 corresponds to the area on a webpage where the video is expected to be displayed. The first marked video display area 306 may include markers. <video>Elements, such as HTML <video>The `<base>` element. In the conventional method, rendering the decoded base stream allows access to the base stream video data, such as decoded frames of the base encoded video. Even if the base stream is decoded using inaccessible or protected methods, rendering the base stream video data in the conventional method makes the base stream video data accessible to other decoding processes within the browser. For example, the HTML media element 304 can provide an option to register a callback when frames of the first layer of decoded data are ready, such as... Figure 2A and Figure 2B As indicated by 232 in [reference needed]. However, as will be discussed further, the method relies on access to the underlying stream video data that becomes available through the rendering of the decoded underlying stream. Specifically, the enhancement of the underlying stream video data is based on access to individual frames of the underlying stream video.

[0077] The rendered, decoded base stream is then combined with the decoded enhanced stream to generate a reconstructed video stream. In some cases, because the rendered base stream does not include enhancement data from the enhanced stream at this time, the first marked video display area is hidden. This ensures that the rendered video content corresponding to the base stream is not displayed on the webpage and is therefore invisible to the viewer of the webpage. However, in some cases, there may be a user option to view this content. Rendering the decoded base stream ensures that the system 300 can still decode and render video streams that are not encoded using a multi-layer video decoding format. For example, if this is the case, the first marked video display area can be set to be visible, and the decoded base stream can be displayed according to the contrastive non-enhanced video rendering. For example, if the webpage contains a single-layer video stream lacking the enhanced stream, then... Figure 3A The system 300 can be used, for example, to display a decoded single-layer video stream by unhiding the first marker video display area 306.

[0078] System 300 further includes an enhanced stream decoder 308. The enhanced stream decoder 308 can implement reference... Figure 2A and Figure 2B The functionality of one or more of the second-layer video decoder 214 and decoding controller 230 is described. In the present case, the source buffer 302, which may include an MSE component, may issue a callback to indicate that data is ready for decoding. This callback may be received by the enhancement stream decoder 308 (e.g., the enhancement stream decoder 308 may register the callback as part of its initial configuration). When the callback is received, the encoded enhancement stream is retrieved from the source buffer 302 and decoded by the enhancement stream decoder 308. For example, the enhancement stream decoder 308 may retrieve the encoded enhancement stream from a portion of data in a set of SEI messages or PID streams stored in the source buffer 302. In some cases, the enhancement stream decoder 308 may obtain the first-layer stream and the second-layer stream (i.e., the base stream and the enhancement stream), demultiplex these two streams, and then discard the first-layer stream (e.g., because it is obtained and decoded via an HTML media element 304). At this stage, a timestamp can be obtained by retrieving the combined stream from the source buffer 302. For example, the PTS timestamp may be obtained from the base stream or from a callback in the source buffer.

[0079] exist Figure 3A In the example, the enhanced stream decoder 308 also obtains the decoded base stream from the first marked video display area 306 and combines the decoded base stream with the decoded enhanced stream to generate a reconstructed video stream. The reconstructed video stream can then be rendered in a second marked video display area 310 visible to the viewer of the webpage. The second marked video display area 310 may include markers. <canvas>Elements, such as HTML <canvas>Elements. In this example, when frames of the base (i.e., first layer) stream are ready, the enhanced stream decoder 308 receives a callback from the first marked video display area 306. This can include... requestVideoFrame Callback. When the enhanced streaming decoder 308 receives a callback from the first marked video display area 306, it obtains the timing metadata of the base frame as described above. For example, the timing metadata may include a "media time" variable or the current playback time provided along with the frame rendered in the first marked video display area 306. As described above, the enhanced streaming decoder 308 then compares the timing metadata with the originally acquired timestamp to pair the first layer decoded data of the frame with the second layer decoded data of the frame. The combined paired data is then rendered in the second marked video display area 310 as the enhanced frame.

[0080] The enhanced stream decoder 308 can be a multi-layer video stream decoder plugin (DPI), such as an LCEVC decoder plugin, configured to decode LCEVC-encoded video streams. The enhanced stream decoder 308 can provide the decoding capabilities of the second-layer decoder 214 and the control capabilities of the decoding controller 230, as shown in the reference... Figure 2A and Figure 2B As described. Depending on the implementation, the enhanced stream decoder 308 may include a single component or two separate components with the same functional effect. One or more components of system 300 may be implemented in a browser. In one instance, a browser including the enhanced stream decoder 308 is provided.

[0081] Figure 3B This is a schematic diagram of another conventional system 300 used for decoding encoded multi-layer video streams, such as encoded multi-layer video streams encoded using LCEVC. Figure 3A The common components and features function in the same way as previously described, and therefore will not be described in detail again.

[0082] As mentioned above, Figure 3B The system 300 includes a source buffer 302 for receiving encoded multi-layer video streams, such as... Figure 2A and Figure 2B The first layer stream 202 and the second layer stream 204 are used. The encoded base stream is extracted from the source buffer 302 and decoded using the HTML media element 304. The decoded base stream is rendered in the first marked video display area 306.

[0083] The source buffer 302 issues a callback to indicate that data is ready for decoding, and the callback is received by the enhanced stream decoder 308. Upon receiving the callback, the encoded enhanced stream is extracted from the source buffer 302 and decoded by the enhanced stream decoder 308. The timestamp (e.g., a PTS timestamp) is obtained from the underlying stream or from the source buffer callback.

[0084] The enhanced stream decoder 308 also obtains the decoded base stream from the first marked video display area 306 and combines the decoded base stream with the decoded enhanced stream to generate a reconstructed video stream. The reconstructed video stream is then rendered in the second marked video display area 310.

[0085] As discussed above, when using LCEVC, the enhanced stream extracted from source buffer 302 is carried within a set of SEI messages that accompany and are associated with the base stream, or within a separate NAL unit stream.

[0086] The underlying stream decoder is typically configured to ignore SEI messages or NAL unit streams containing information that the underlying stream decoder cannot interpret. This means that the HTML media element 304 retrieves data related to the underlying stream from the source buffer 302 in a default manner, where both enhanced and non-enhanced underlying streams are processed in a common way. The HTML media element 304 ignores SEI messages or NAL units carrying enhanced streams residing in the source buffer 302.

[0087] like Figure 3B As can be seen, the enhanced stream extracted from source buffer 302 and carrying the SEI message set is first passed to integration layer 312 before being passed to enhanced stream decoder 308 for decoding. Specifically, integration layer 312 extracts LCEVC data from the enhanced stream based on the type of source buffer. The extracted data can be NAL units with LCEVC data and Real-Time Transport Protocol (RTP) timestamps, or data extended from the media source and appended to the source buffer. The LCEVC data is indexed using PTS and stored for use when matching and combining the corresponding base stream with the LCEVC data using PTS. As before, this is done by comparing timing metadata with PTS, and then the matched LCEVC data can be combined with the base stream to generate the reconstructed video stream. Figure 3B As shown, the relevant LCEVC data, along with the offset generated based on the playback environment, is passed to the enhanced stream decoder 308.

[0088] The offset is calculated by offset calculation block 314 and can be accomplished using two methods. The first method involves calculating the offset during the appending of the source buffer, and the second method involves calculating the offset during the decoding of the enhanced stream.

[0089] Using the first method, there are cases where data with the same timestamp is appended to the source buffer multiple times during the append process. In this case, the frame time is calculated based on the FPS of the enhanced stream, and then this frame time is added to the provided timestamp according to the number of times the data is appended. For example, if the data is appended three times, the frame time will be added to the provided timestamp three times. This is then used to store LCEVC data used by the integration layer.

[0090] The second method calculates the offset based on one or more of the operating system, browser, player, and container format during decoding at the enhancement layer. This offset is then added to the timestamp provided by the video stream to retrieve the relevant LCEVC data from the stored LCEVC data.

[0091] from Figure 2A and Figure 2B As can be understood from the foregoing description of conventional systems 200 and 240, the rendering of video based on the first layer 202 and the second layer 204 is based on system 200's access to individual frames of the first layer 202 and the second layer 204. Similarly, in Figure 3A and Figure 3B In System 300, the rendering of the decoded underlying stream allows access to the underlying stream video data and specifically access to the decoded frames of the underlying encoded video.

[0092] However, the first-layer stream may be limited to individual frames of the first-layer stream and individual frames of the video rendered from the first-layer stream (such as for...). Figure 3A and Figure 3B Access to the decoded frames of the underlying encoded video is protected. These protections mean that it is difficult to apply enhancements to the first-layer stream using methods employed in conventional systems.

[0093] To address these issues, alternative methods are used in embodiments of the present invention.

[0094] Figure 4A This is a schematic diagram of an exemplary system 400 for decoding encoded multilayer video streams (such as encoded multilayer video streams using LCEVC encoding) according to an embodiment of the present invention. System 400 is similar in many respects to... Figure 3A and Figure 3B The system 300 shown is similar and includes the same features as... Figure 3A The elements shown correspond to the elements in the diagram. System 400 can also be implemented within a browser. In fact, in some embodiments, system 400 can also implement the conventional methods discussed above.

[0095] Similar to the conventional methods discussed above, one function of a browser is to transform documents written in markup scripting languages ​​(sometimes called markup languages) into a visual representation of a webpage. Markup scripting languages ​​are used to control the display of data in a rendered webpage. A markup language can contain markup video elements, which are then transformed into a video display area when processed by the browser. For example, a browser user might navigate to a webpage containing embedded video. When the browser renders the webpage, it receives data corresponding to the video. The browser may contain the resources necessary to decode and play the video so that it can be displayed to the user, for example, within the browser-rendered video display area on the client device's monitor. Examples of markup scripting languages ​​include any version of Hypertext Markup Language (HTML), such as HTML5 and Extensible Hypertext Markup Language (XHTML).

[0096] For example, the `<video>` tag indicates attributes associated with the display of a video on a webpage, such as the size of the video within the webpage and whether the video will play automatically when the webpage loads. For instance, the `<video>` tag also includes an indication of the video decoding format used to encode the video. This instructs the browser which decoders to use to decode the encoded video. The browser can then make at least one call to decoding functions within its own resources (which can be considered browser-native resources, on the browser's own machine) or to decoding functions implemented in the OS, as discussed further below.

[0097] and Figure 3A and Figure 3B The system is similar to 300. Figure 4A System 400 includes a source buffer 402 to receive encoded multi-layer video streams. In the example described, source buffer 402 may receive a first-layer stream and a second-layer stream, and is the source of both the first-layer and second-layer streams. In other examples, each stream may have a separate corresponding source buffer instead of a combined buffer for both streams. The source buffer may be part of memory, which may be accessible, for example, by a browser. The source buffer may be, for example, a Media Source Extension (MSE) application programming interface (API) SourceBuffer. The encoded multi-layer video stream may include an encoded base stream and an encoded enhancement stream, wherein the encoded base stream includes video content encoded by any base encoder (also known as a compressor), such as Advanced Video Decoder (AVC), High Efficiency Video Decoder (HEVC), VP9, ​​MPEG-5 Basic Video Decoder (EVC), or AOMedia Video 1 (AV1) encoder.

[0098] System 400 also includes an HTML media element 404 that implements a first-layer (e.g., a basic stream) video decoder, also known as a basic stream decompressor. The HTML media element 404 may indicate or otherwise implement the first-layer video decoder, which may be related to a reference... Figure 2A and Figure 2B The description is similar. In the example described, the encoded base stream is extracted from the source buffer 402 and decoded using an HTML media element 404. The HTML media element 404 can be a markup video element that provides an interface between the markup language and the decoding resource. For example, HTML5 includes markup video elements that can be used to embed video content in a webpage. Another example is a JavaScript library that builds a set of custom controls on top of the HTML5 video element, which can be considered as acting as a JavaScript player. It should be understood that markup video elements, such as the HTML5 video element, can be modified by adding additional resources that can be accessed by the markup video element, such as multi-layer video streaming decoding libraries, WebAssembly decoding libraries, and / or web worker functionality.

[0099] When using LCEVC, the enhancement stream can be carried within a set of Supplemental Enhancement Information (SEI) messages that accompany and are associated with the base stream, or within a separate Network Abstraction Layer (NAL) unit stream, for example, as Figure 1 As shown, this is carried in the PID stream. The basic stream decoders can be configured to ignore SEI messages or NAL unit streams if they contain information they cannot interpret, such as header information indicating a specific packet type. Therefore, in this case, the HTML media element 404 can retrieve data related to the basic stream from the source buffer 402 in a default manner, where both enhanced and non-enhanced basic streams are processed in a common way. In this case, the HTML media element 404 can ignore SEI messages or NAL units carrying the enhanced stream residing in the source buffer 402.

[0100] exist Figure 4A In some implementations of the exemplary system 400, the marked video element contains an indication of the video decoding format associated with the encoded underlying stream. When processed by a browser, the marked video element locates the appropriate underlying stream decoder associated with the video decoding format and decodes the encoded underlying stream. The HTML media element 404 can be implemented within a browser, utilizing the functionality of the OS of a client device including the browser, or leveraging the resources of both the browser and the OS. For example, the OS can utilize hardware acceleration to decode the encoded underlying stream, which reduces power consumption and the number of computations performed by the CPU compared to software decoding alone.

[0101] The decoded underlying stream is rendered in the first marked video display area 406. For example, the first marked video display area 406 corresponds to the area on a webpage where the video is expected to be displayed. The first marked video display area 406 may include markers. <video>Elements, such as HTML <video>Elements. However, despite the reference above... Figure 3A In the described example, rendering of the decoded underlying stream allows access to the underlying stream video data, such as decoded frames of the underlying encoded video, but... Figure 4A In System 400, access to the underlying stream video data may be restricted. This means that instead of combining the rendered, decoded underlying stream with the decoded enhanced stream to generate the reconstructed video stream, a different approach is taken.

[0102] Because the rendered base stream is not directly combined with the enhanced data, therefore... Figure 3A and Figure 3B Compared to System 300, the first marked video display area is not hidden. This means that the rendered video content corresponding to the base stream is visible to the webpage viewer before any enhancements are applied. However, compared to the reference above... Figure 3A and Figure 3B The system described is similar to 300. Figure 4A The system 400 can be used to decode and render video streams that are not encoded using multi-layer video decoding formats. For example, if a webpage includes a single-layer video stream lacking enhancement streams, then... Figure 4A The system 400 can be used to display decoded single-layer video streams.

[0103] System 400 further includes an enhanced stream decoder 408. The enhanced stream decoder 408 can implement the functions of one or more of the second-layer video decoder and decoding controller, each of which can be functionally similar to the reference [reference]. Figure 2A and Figure 2B The described scenarios are as follows. In the current case, the source buffer 402, which may include an MSE component, may issue a callback to indicate that data is ready for decoding. This callback may be received by the enhancement stream decoder 408 (e.g., the enhancement stream decoder 408 may register the callback as part of its initial configuration). When the callback is received, the encoded enhancement stream is retrieved from the source buffer 402 and decoded by the enhancement stream decoder 408. For example, the enhancement stream decoder 408 may retrieve the encoded enhancement stream from a portion of data stored within a set of SEI messages or PID streams in the source buffer 402. In some cases, the enhancement stream decoder 408 may obtain a first-layer stream and a second-layer stream (i.e., the base stream and the enhancement stream), demultiplex these two streams, and then discard the first-layer stream (e.g., because it is obtained and decoded via an HTML media element 404). At this stage, a timestamp can be obtained by retrieving the combined stream from the source buffer 402. For example, the PTS timestamp may be obtained from the base stream or from a source buffer callback. The timestamp allows identification of the playback time of the rendered base video, which can then be used to synchronize the enhancement with the base video.

[0104] When applying enhancements to a rendered base video, the enhancement stream decoder 408 receives the encoded enhancement stream, decodes it to obtain a decoded enhancement stream, and renders one or more enhancement overlay streams based on the decoded enhancement stream. (See above reference...) Figure 3A In the conventional approach described, enhancements are first combined with the underlying stream before being displayed, while... Figure 4A In system 400, one or more enhanced overlay streams are rendered in a second marked video display area 410, which overlaps with a first marked video display area 406, to generate an enhanced rendered video. For example, the first marked video display area 406 may include markers. <video>Elements, such as HTML <video>Elements, and the second marked video display area 410 can be a marked element. <canvas>Elements, such as HTML <canvas>Elements are used to overlay one or more enhancement overlay streams onto the base video displayed in the first marked video display area 406. As will be understood, the enhancement stream decoder 408 can also receive more than one encoded enhancement stream and decode them to obtain more than one decoded enhancement stream.

[0105] Therefore, although in Figure 3A In the example, the enhanced stream decoder 308 obtains the decoded base video data from the first marked video display area 306 and combines it with the enhanced data to generate an enhanced rendered video, but... Figure 4B In one embodiment, the enhanced stream decoder 408 renders one or more enhanced overlay streams in the second marked video display area 410, and obtains an enhanced rendered video as a result of the second marked video display area 410 coinciding with the first marked video display area 406. However, the enhanced stream decoder 408 can obtain metadata from the first marked video display area 406, such as the playback time of the rendered base video that can be used to synchronize one or more enhanced overlay streams with the rendered base stream.

[0106] One or more enhancement overlay streams can encode modifications to the luma and / or chroma components and can be at a higher resolution than the base video. For example, one or more enhancement overlay streams can encode positive or negative modifications to the luma component of the base video.

[0107] Positive modifications to the luminance component of the base video are used to selectively brighten pixels in the base video. This is achieved using an enhancement overlay stream, where each frame contains an array of values ​​between 0 and 1 corresponding to pixels in the enhanced rendered video. These values ​​represent the degree of opacity. When one or more enhancement overlay streams are rendered in the second marked video display area 410, the enhancement stream decoder 408 treats a value of 0 as indicating no modification to the underlying base video and a value of 1 as a completely opaque white pixel overlaid on the underlying base video. For values ​​between 0 and 1, the enhancement stream decoder 408 applies a white pixel overlay with an opacity equal to the corresponding value of the frame enhancement overlay stream. The enhancement stream decoder 408 may include shaders configured to perform positive luminance modifications in this rendering.

[0108] Negative modifications to the luminance component of the base video are used to selectively darken pixels in the base video. This is achieved using an enhancement overlay stream, where each frame contains an array of values ​​between 0 and 1 corresponding to pixels in the enhanced rendered video. However, while positive luminance modifications are treated as alpha values, for negative modifications, the enhancement stream decoder 408 renders the corresponding enhancement overlay stream by averaging the pixel values ​​of the enhancement overlay stream and the underlying luminance values ​​of the base video for each pixel of each frame of the enhancement overlay stream. The enhancement stream decoder 408 may include cascading style sheet (CSS) filters for this rendering of negative luminance modifications.

[0109] Typically, modifications to the luminance components of the base video are made to increase the image resolution; in this case, one or more enhancement overlay streams have a higher resolution than the base video. However, one or more enhancement overlay streams can also be used to correct errors introduced by encoding and decoding a reference video signal into the rendered base video; in this case, one or more enhancement overlay streams can have the same resolution as the rendered base video. In another example, one or more enhancement overlay streams encode modifications to the chrominance components of the rendered base video to increase the color space; in this case, one or more enhancement overlay streams can also have the same resolution as the rendered base video.

[0110] The enhancement stream decoder 408 may be a multi-layer video stream decoder plugin (DPI), such as an LCEVC decoder plugin, configured to decode LCEVC-encoded video streams. For example, the encoded enhancement stream may include one or more sets of encoded residuals, which are then decoded by the enhancement stream decoder 408 to obtain one or more sets of decoded residuals. The enhancement stream decoder 408 then uses one or more sets of decoded residuals to render one or more enhancement overlay streams. For example, when at least one set of decoded residuals includes both positive and negative residuals, the enhancement stream decoder 408 may render at least a first positive residual overlay stream and at least a first negative residual overlay stream.

[0111] The enhanced stream decoder 408 can provide the decoding capabilities of a second-layer decoder and the control capabilities of a decoding controller, such as reference... Figure 2A and Figure 2B Those described. Depending on the implementation, the enhanced stream decoder 408 may include a single component or two separate components with the same functional effect. One or more components of system 400 may be implemented in a browser. In one instance, a browser including the enhanced stream decoder 408 is provided.

[0112] Figure 4B This is a schematic diagram of an alternative exemplary system 400 for decoding encoded multi-layer video streams, such as encoded multi-layer video streams using LCEVC encoding. Figure 4A The common components and features function in the same way as previously described, and therefore will not be described in detail again.

[0113] As mentioned above, Figure 4B System 400 includes a source buffer 302 for receiving encoded multi-layer video streams, such as a first-layer stream and a second-layer stream. The encoded base stream is extracted from the source buffer 402 and decoded using an HTML media element 404. The decoded base stream is rendered in a first marked video display area 406.

[0114] The source buffer 402 issues a callback to indicate that data is ready for decoding, and the callback is received by the enhanced stream decoder 408. Upon receiving the callback, the encoded enhanced stream is extracted from the source buffer 402 and decoded by the enhanced stream decoder 408. The timestamp (e.g., a PTS timestamp) is obtained from the underlying stream or from the source buffer callback.

[0115] HTML media element 404 renders the base video signal in the first marked video display area 406, and enhancement stream decoder 408 overlays one or more enhancement streams in a second marked video display area 410 that overlaps with the first marked video display area 406 to generate a reconstructed video stream, as referenced above. Figure 4A The subject of discussion.

[0116] As discussed above, when using LCEVC, the encoded enhanced stream extracted from source buffer 402 is carried within the set of SEI messages that accompany and are associated with the base stream or within a separate NAL unit stream.

[0117] The underlying stream decoder is typically configured to ignore SEI messages or NAL unit streams if they contain information that the underlying stream decoder cannot interpret. This means that the HTML media element 404 retrieves data related to the underlying stream from the source buffer 402 in a default manner, where both enhanced and non-enhanced underlying streams are processed in a common way. The HTML media element 404 ignores SEI messages or NAL units carrying enhanced streams residing in the source buffer 402.

[0118] like Figure 4B As can be seen, the encoded augmented stream, extracted from source buffer 402 and carrying a set of SEI messages, is first passed to integration layer 412 before being passed to augmented stream decoder 408 for decoding. Specifically, integration layer 412 extracts LCEVC data from the augmented stream based on the type of source buffer. The extracted data can be NAL units with LCEVC data and Real-Time Transport Protocol (RTP) timestamps, or data appended to the source buffer from media source extensions. The LCEVC data is indexed using PTS and stored for use when one or more augmented overlay streams are superimposed on the rendered base video. This is done by comparing timing metadata (such as the playback time of the rendered base video) with the corresponding PTS that synchronizes one or more augmented overlay streams with the rendered base video signal. Figure 4B As shown, the relevant LCEVC data, along with the offset generated based on the playback environment, is passed to the enhanced stream decoder 408.

[0119] The offset is advantageously calculated by offset calculation block 414 using one of two methods. The first method involves calculating the offset during source buffer appending, and the second method involves calculating the offset during enhanced stream decoding, as referenced above. Figure 3B The offset calculation block 314 is described.

[0120] It is worth noting that while in conventional methods of rendering video, individual frames of the base video can be synchronized with corresponding frames of the enhanced stream, this may not be possible due to security protections applied to the base video. Therefore, Figure 4B The system 400 is configured to synchronize one or more enhancement overlays with the rendered base video using timing metadata of the rendered base video signal (preferably, the playback time of the rendered base video signal) and timestamps of frames of one or more enhancement overlays.

[0121] Now refer to Figure 5 An exemplary method 500 is described, which renders a representation of a reference video signal. Method 500 can utilize, as in the reference... Figure 4A or Figure 4B The described system 400 is performed, or can be implemented, for example, as a set of instructions executed by a processor. As in the example above, a multi-layer video stream encodes a video signal and includes at least a first layer and a second layer. The first layer is decoded using a first decoding method, and the second layer is decoded using a second decoding method. The first decoding method uses data that the second decoding method cannot access. For example, due to security or restricted internal data access, the PTS of the decoded frames within the first layer of the multi-layer video stream may be used by the first decoding method but not by the second decoding method.

[0122] Preferably, method 500 is implemented within a browser. As discussed above, the browser can be any browser that processes documents in markup scripting languages ​​to generate a visual representation of a webpage. The visual representation of the webpage can be made visible via a user interface associated with a client device associated with the browser.

[0123] At box 502, in the source buffer (such as in...) Figure 4A or Figure 4B The source buffer (402) receives the encoded multi-layer video stream. The encoded multi-layer video stream includes an encoded base stream and an encoded enhancement stream.

[0124] The encoded base stream can be a downsampled source signal encoded using a base encoder or codec, and can be decoded by a decoder such as a hardware-based decoder. The base encoder or codec can be any base encoder or codec, such as Advanced Video Decoder (AVC), High-Efficiency Video Decoder (HEVC), VP9, ​​MPEG-5 Basic Video Decoder (EVC), and AOMedia Video 1 (AV1) encoder and codec. Using existing base encoders and codecs as part of the encoding (and decoding) process ensures that systems unable to render multi-layer video content can still decode the base stream using existing base codecs. This means that no hardware updates are required to decode encoded multi-layer video streams using Method 500, and future base codecs can also be used without further hardware upgrades, provided the system hardware is upgraded to become compatible with future base codecs.

[0125] In some examples, the coded augmented stream includes a set of coded residuals that correct or enhance the underlying stream. Multiple levels of augmented data can exist within a hierarchical structure. The coded augmented stream can be encoded using a dedicated encoder configured to generate the coded augmented stream from uncompressed full-resolution video.

[0126] LCEVC Enhanced Stream is an example of this video stream encoded using a multi-layer decoding scheme. In this case, the video stream can be encoded by an LCEVC encoder. However, other examples are also envisioned.

[0127] exist Figure 5 At box 504, the encoded underlying stream is retrieved from the source buffer and decoded using tagged video elements. Tagged video elements can be elements of markup scripting languages ​​such as HTML and XHTML. For example, a tagged video element could be... Figure 4B The HTML media element shown is 404. The encoded underlying stream can then be decoded using a suitable decoder. The marked-video element indicates whether it receives or otherwise obtains the video decoding format of the encoded underlying stream. When processed by a browser, the marked-video element locates the decoder or codec associated with the video decoding format, causing the encoded underlying stream to be decoded by the decoder. For example, HTML5 includes markup languages... <video>The `<video>` element is used to embed video content in a webpage; it is an instance of the `<video>` tag. The underlying decoder is any decoder capable of decoding the encoded underlying stream.

[0128] A decoded base stream consists of multiple individual frames. For example, a frame corresponds to a still image or picture. In some instances, video consists of a series of frames. A frame may contain multiple pixels. Each frame includes data representing the properties of the video content. For example, a frame may include data defining the color of each pixel in the frame. This data can be used by markup video elements to form a visual representation of the video stream when rendered in the final webpage.

[0129] However, although the decoded underlying stream consists of multiple individual frames, the decoded underlying stream may be subject to security protections that restrict access to individual frames of the decoded underlying stream.

[0130] exist Figure 5 At box 506, the decoded underlying stream is rendered in the first marked video display area. The browser processes the marked video element corresponding to the marked video display area in the generated webpage, for example, by embedding a media player in the webpage and rendering the video content within the media player. The first video display area is not hidden, meaning that unenhanced video content may be displayed.

[0131] The scripting language can be used to define the first marked video display area.

[0132] At box 508, the encoded augmented stream is extracted from the source buffer and decoded. The encoded augmented stream includes augmentation data associated with one or more augmentation layers of a multi-layer video stream. The encoded augmented stream can be generated by the DPI of the multi-layer video stream (such as...). Figure 4A and Figure 4B The enhanced stream decoder 408 shown is used to decode and to obtain one or more enhanced overlay streams.

[0133] Finally, at box 510, one or more enhanced overlay streams are rendered in a second marked video display area superimposed on the first marked video display area. Preferably, a markup language is used. <canvas>Tags are used to render one or more enhancement overlay streams. In this way, enhancement data included in one or more enhancement overlay streams can be used to overlay enhancements onto the decoded base stream to generate an enhanced video signal without needing to access the data of the decoded base stream.

[0134] One or more enhancement overlay streams can encode modifications to the luma and / or chroma components and can be at a higher resolution than the base video. For example, one or more enhancement overlay streams can encode positive or negative modifications to the luma component of the base video.

[0135] Positive modifications to the luminance component of the base video are used to selectively brighten pixels in the base video. This is achieved using an enhancement overlay stream, where each frame contains an array of values ​​between 0 and 1 corresponding to pixels in the enhanced rendered video. These values ​​represent the degree of opacity. When one or more enhancement overlay streams are rendered in the second marked video display area, a value of 0 is considered to indicate no modification to the underlying base video, and a value of 1 is considered to be a completely opaque white pixel overlaid on the underlying base video. For values ​​between 0 and 1, the method involves applying a white pixel overlay with an opacity equal to the corresponding value of the frame enhancement overlay stream. This can be done using shaders.

[0136] Negative modifications to the luminance component of the base video are used to selectively darken pixels in the base video. This is achieved using an enhancement overlay stream, where each frame contains an array of values ​​between 0 and 1 corresponding to pixels in the enhanced rendered video. However, while positive luminance modifications are treated as alpha values, for negative modifications, the method involves rendering the corresponding enhancement overlay stream by averaging the pixel values ​​of the enhancement overlay stream and the underlying luminance values ​​of the base video for each pixel in each frame of the enhancement overlay stream. This can be done using Cascading Style Sheets (CSS) filters.

[0137] Typically, modifications to the luminance components of the base video are made to increase the image resolution; in this case, one or more enhancement overlay streams have a higher resolution than the base video. However, one or more enhancement overlay streams can also be used to correct errors introduced by encoding and decoding a reference video signal into the rendered base video; in this case, one or more enhancement overlay streams can have the same resolution as the rendered base video. In another example, one or more enhancement overlay streams encode modifications to the chrominance components of the rendered base video to increase the color space; in this case, one or more enhancement overlay streams can also have the same resolution as the rendered base video.

[0138] Figure 6 A spatially scalable decoding scheme is illustrated, which uses a downsampled source signal encoded with a base codec, adds first-level correction or enhancement data to the decoded output of the base codec to generate a corrected picture, and then adds another level of correction or enhancement data to an upsampled version of the corrected picture. Therefore, the spatially scalable decoding scheme can generate an enhancement stream with two spatial resolutions (higher and lower), which can be combined with the base stream at the lower spatial resolution.

[0139] In spatially scalable decoding schemes, methods and apparatus can be based on a holistic algorithm built upon existing encoding and / or decoding algorithms (e.g., MPEG standards such as AVC / H.264 and HEVC / H.265, and non-standard algorithms such as VP9 and AV1), which serve as a baseline for enhancement layers. Enhancement layers operate according to different encoding and / or decoding algorithms. In contrast to block-based approaches using MPEG family algorithms, the idea behind the holistic algorithm is to encode / decode video frames in a layered manner. Encoding frames in a layered manner involves generating residuals for the entire frame and then generating residuals for reduced or extracted frames, etc.

[0140] Figure 6 The system configuration of an example spatially scalable coding system 600 is shown. The coding process is divided into two halves, as indicated by the dashed lines. Each half can be implemented independently. Below the dashed lines is the base layer, and above the dashed lines is the enhancement layer, which can be usefully implemented in software. The coding system 600 can include enhancement-level processes only, or a combination of base-level and enhancement-level processes, as needed. The topology of the coding system 600 at a general layer is as follows. The coding system 600 includes an input I for receiving an input signal 601. Input I is connected to a downsampler 605D. The downsampler 605D outputs to a base encoder 620E at the base layer of the coding system 600. The downsampler 605D also outputs to a residual generator 610-S. The encoded base stream is created directly by the base encoder 620E and can be quantized and entropy-coded as needed according to the base coding scheme. The encoded base stream can be the base layer as described above, for example, the lowest layer in a multi-layer decoding scheme.

[0141] Above the dashed lines is a series of enhancement layer processes for generating the enhancement layer of a multi-layer decoding scheme. In this example, the enhancement layer consists of two sub-layers. In another example, one or more sub-layers may be provided. Figure 6 In order to generate the encoded sublayer 1 enhanced stream, the encoded base stream is decoded via a decoding operation applied at the base decoder 620D. In a preferred example, the base decoder 620D may be a decoding component that complements the encoding component within the base codec, which takes the form of the base encoder 620E. In other examples, the base decoding block 620D may alternatively be part of the enhancement layer. A difference is created between the decoded base stream output from the base decoder 620D and the downsampled input video via a residual generator 610-S (i.e., subtraction operations 610-S are applied to frames of the downsampled input video and frames of the decoded base stream to generate a first set of residuals). Here, the residual represents the error or difference between a reference signal or frame and a desired signal or frame. The residuals used in the first enhancement layer can be considered as correction signals because they are capable of 'correcting' future frames of the decoded base stream. This is useful because it can correct for odd patterns or other characteristics of the base codec. These features include, in particular, motion compensation algorithms applied by the underlying codec, quantization and entropy coding applied by the underlying codec, and block adjustment applied by the underlying codec.

[0142] exist Figure 6 In the first set of residuals, the residuals are transformed, quantized, and entropy-encoded to produce an encoded enhanced layer sublayer 1 stream. Figure 6 In this process, transform operation 610-1 is applied to the first set of residuals; quantization operation 620-1 is applied to the transformed set of residuals to generate a set of quantized residuals; and entropy coding operation 630-1 is applied to the quantized set of residuals to generate a coded enhancement layer sublayer 1 stream (e.g., at the first enhancement layer). However, it should be noted that in other examples, only quantization step 620-1 or only transform step 610-1 may be performed. Entropy coding may not be used, or it may optionally be used in addition to one or both of transform step 610-1 and quantization step 620-1. The entropy coding operation may be any suitable type of entropy coding, such as Huffman coding or run-length coding (RLE) operations, or a combination of both Huffman coding and RLE operations (e.g., RLE followed by Huffman or prefix coding).

[0143] To generate the encoded enhancement layer sublayer 2 stream, another level of enhancement information is created by generating and encoding another set of residuals via residual generator 600-S. The other set of residuals is the difference between the corrected and upsampled version (via upsampler 605U) of the decoded base stream (reference signal or frame) and the input signal 601 (desired signal or frame).

[0144] To achieve reconstruction of a corrected version of the decoded base stream that can be generated at the decoder, at least some of the sublayer 1 encoding operations are reversed to simulate the decoder process, taking into account at least some losses and specificities of the transform and quantization processes. For this purpose, the first set of residuals is processed by a decoding pipeline comprising an inverse quantization block 620-1i and an inverse transform block 610-1i. The quantized first set of residuals is inverse quantized at the inverse quantization block 620-1i and inverse transformed at the inverse transform block 610-1i in the encoding system 600 to regenerate a decoder-side version of the first set of residuals. The decoded base stream from decoder 620D is then combined with the decoder-side version of the first set of residuals (i.e., a summation operation 610-C is performed on the decoded base stream and the decoder-side version of the first set of residuals). The summation operation 610-C generates a reconstruction, such as a downsampled version of the input video likely to be generated at the decoder—i.e., the reconstructed base codec video). The reconstructed base codec video is then upsampled by upsampler 605U.

[0145] The processing in this example is typically performed frame-by-frame. Each color component of a frame can be processed in parallel or serially, as shown. It is worth noting that even if content protection restricts access to individual frames of the base stream at the decoder, the encoder can still generate the base and enhancement layers on a frame-by-frame basis. The structure of the base and enhancement streams is identical regardless of whether security protection is applied during decoding.

[0146] The upsampled signal (i.e., the reference signal or frame) is then compared with the input signal 601 (i.e., the desired signal or frame) to create another set of residuals (i.e., the difference operation is applied by the residual generator 600-S to the upsampled recreated frame to generate another set of residuals). This other set of residuals is then processed via an encoding pipeline that serves as a mirror image of the first set of residuals to become an encoded enhancement layer sublayer 2 stream (i.e., encoding operations are then applied to the other set of residuals to generate another encoded enhancement stream). Specifically, the other set of residuals is transformed (i.e., transformation operation 610-0 is performed on the other set of residuals to generate another transformed set of residuals). The transformed residuals are then quantized and entropy encoded in the manner described above with respect to the first set of residuals (i.e., quantization operation 620-0 is applied to the transformed set of residuals to generate another quantized set of residuals; and entropy encoding operation 630-0 is applied to the quantized set of residuals to generate an encoded enhancement layer sublayer 2 stream containing information of another enhancement level). In some cases, the operation can be controlled, for example, to allow only quantization step 620-1 to be performed, or only the transform and quantization steps to be performed. Entropy coding may optionally be used as a supplement. Preferably, the entropy coding operation may be a Huffman coding operation or a run-length encoding (RLE) operation or both (e.g., RLE followed by Huffman coding). The transform applied to both blocks 610-1 and 610-0 may be a Hadamard transform applied to 2×2 or 4×4 residual blocks.

[0147] Figure 6 The encoding operations in this approach do not introduce dependencies between local blocks of the input signal (e.g., compared to many known decoding schemes that apply inter-frame or intra-frame prediction to macroblocks and thus introduce macroblock dependencies). Therefore, Figure 6 The operations shown can be performed in parallel on 4×4 or 2×2 blocks, which greatly improves coding efficiency on multi-core central processing units (CPUs) or graphics processing units (GPUs).

[0148] like Figure 6 As shown, the output of the spatially scalable coding process is one or more enhancement streams for enhancement layers, which preferably include a first enhancement level and another enhancement level. This can then be combined with the base stream at the base level (e.g., via multiplexing or other means). The first enhancement level (sub-layer 1) can be viewed as implementing the corrected video at the base level, that is,, for example, correcting encoder glitches. The second enhancement level (sub-layer 2) can be viewed as another enhancement level used to convert the corrected video to the original input video or an approximation thereof. For example, the second enhancement level can add fine details lost during downsampling and / or help correct errors introduced by one or more of the transform operation 610-1 and quantization operation 620-1.< / canvas> < / video> < / canvas> < / canvas> < / video> < / video> < / video> < / video> < / canvas> < / canvas> < / video> < / video> < / video> < / canvas> < / video>

Claims

1. A method for rendering a representation of a reference video signal, the method comprising: Obtain a first rendered video signal representing the reference video signal at a first quality level; as well as One or more enhancement overlay streams are superimposed on the first rendered video signal to generate a second rendered video signal representing the reference video signal at a second quality level, which is a higher quality level than the first quality level.

2. The method according to claim 1, wherein obtaining the first rendered video signal comprises: Obtain the first decoded video stream; as well as The first rendered video signal is rendered in the first marked video display area based on the first decoded video stream.

3. The method according to claim 2, wherein obtaining the first decoded video stream comprises: Receive the first encoded video stream; as well as, In the first decoding process, the first encoded video stream is decoded to obtain the first decoded video stream.

4. The method according to claim 2 or claim 3, wherein the first marked video display area is HTML. <video> element.< / video> 5. The method according to any one of claims 2 to 4, wherein superimposing the one or more enhanced superimposed streams comprises: Obtain one or more decoded enhanced streams; as well as The one or more enhanced overlay streams are rendered in a second marked video display area based on the one or more decoded enhanced streams, the second marked video display area coinciding with the first marked video display area.

6. The method of claim 5, wherein obtaining the one or more decoded enhanced streams comprises: Receive one or more encoded enhanced streams; as well as, In the second decoding process, the one or more encoded enhanced streams are decoded to obtain the one or more decoded enhanced streams.

7. The method of claim 5 or claim 6, wherein the first rendered video signal includes data that is inaccessible to the second decoding method.

8. The method according to any one of claims 5 to 7, wherein the one or more encoded augmented streams comprise a set or more sets of encoded residuals, and decoding the one or more encoded augmented streams comprises decoding the set or more sets of encoded residuals to obtain a set or more sets of decoded residuals.

9. The method of claim 8, wherein the one or more encoded enhanced streams comprise a second set of encoded residuals in a second enhanced sublayer of at least a hierarchical coding scheme.

10. The method of claim 8 or claim 9, wherein at least one set of decoded residuals in the set or more sets of decoded residuals includes positive residuals and negative residuals, and decoding the one or more encoded enhanced streams comprises: One or more decoded positive residual streams are obtained from the positive residual; as well as One or more decoded negative residual streams are obtained from the negative residual.

11. The method of claim 10, wherein rendering the one or more enhanced overlay streams comprises: Based on the decoded positive residual stream, render one or more positive residual overlay streams in the second marked video display area; as well as Based on the decoded negative residual stream, one or more negative residual overlay streams are rendered in the second marked video display area.

12. The method of claim 11, wherein at least one of the one or more positive residual overlay streams encodes a modification of the luminance component of the rendered base video, the at least one positive residual overlay stream comprising an array of values ​​between 0 and 1, each of the values ​​corresponding to a pixel of the second rendered video signal, and rendering the at least one positive residual overlay stream in the second marked video display area comprises: For each pixel of the second rendered video signal, render a white pixel with an opacity equal to the corresponding value in the value array.

13. The method of claim 11 or claim 12, wherein at least one of the one or more negative residual overlay streams encodes a modification of the luminance component of the rendered base video, the at least one negative residual overlay stream comprising an array of values ​​between 0 and 1, each of the values ​​corresponding to a pixel of the second rendered video signal, and rendering the at least one negative residual overlay stream in the second marked video display area comprises, for each pixel of the second rendered video signal: averaging the corresponding value from the value array with the luminance value of the underlying pixel in the first rendered video signal, and rendering the pixel having the average luminance value.

14. The method according to any one of claims 5 to 13, wherein the second marked video display area is HTML. <canvas> element.< / canvas> 15. The method according to any one of claims 1 to 14, wherein superimposing the one or more enhanced overlay streams includes synchronizing the one or more enhanced overlay streams with the first rendered video signal based on the playback time of the first rendered video signal.

16. The method of claim 15, wherein each of the one or more enhanced overlay streams comprises a series of frames, the series of frames being indexed using a corresponding series of timestamps; as well as For each of the one or more enhanced overlay streams, synchronizing the enhanced overlay stream with the first rendered video signal includes: The current playback time of the first rendered video signal is compared with the timestamp of the frame indexed in the enhanced overlay stream; Based on the comparison, the frame of the enhanced overlay stream corresponding to the current playback time is identified; and The frame corresponding to the current playback time of the enhanced overlay stream is superimposed on the first rendered video signal.

17. The method of claim 16 and any one of claims 5 to 14, wherein the timestamp is derived from the one or more encoded enhanced streams.

18. The method of claim 16 or claim 17, wherein comparing the current playback time of the first rendered video signal with the timestamps indexed in the enhanced overlay stream comprises: Search for frames in the enhanced overlay stream that have timestamps falling within the range defined by the current playback time.

19. The method of claim 18, wherein the range is set based on a configurable drift offset.

20. The method of claim 19, wherein the configurable drift offset is based on the frame rate of the first rendered video signal.

21. The method of claim 18 or claim 19, wherein searching for frames in the enhanced overlay stream having timestamps falling within a range defined by reference to the current playback time comprises: Search for frames in the enhanced overlay stream, wherein the timestamp of the frame plus the configurable drift offset equals the current playback time.

22. The method according to any one of the preceding claims, wherein the second rendered video signal is at a higher resolution than the first rendered video signal.

23. The method according to any one of the preceding claims, wherein one or more of the one or more enhanced overlay streams encode modifications to the luminance and / or chrominance components of the rendered base video.

24. The method according to any one of the preceding claims, wherein the method is implemented within a browser.

25. A system for rendering a representation of a reference video signal, the system comprising: A video renderer configured to obtain a first rendered video signal representing the reference video signal at a first quality level; as well as An enhancement decoder is configured to overlay one or more enhancement overlay streams onto the first rendered video signal to generate a second rendered video signal representing the reference video signal at a second quality level, which is a higher quality level than the first quality level.

26. A computer-readable medium comprising instructions that, when executed, cause a processor to perform the method according to any one of claims 1 to 24.

Images