Pre-analysis for video coding

Abstract

A method for determining encoder parameters for encoding an input video comprising a sequence of video frames, the input video having a first resolution, the method comprising: acquiring a first and a second video frame of the input video, where the second video frame lies after the first video frame in the sequence of video frames; down-sampling the first and second frames to a second resolution to obtain first and second down-sampled video frames; generating a detail-aware metric based on the first and second downsampled video frames; encoder parameters for encoding the second video frame are determined based on the detail awareness metric, wherein the detail awareness metric includes an edge detection metric based on the second downsampled frame and a motion metric based on a difference between the first and second downsampled frames.

Description

Technical Field

[0001] The following disclosure relates to video coding. Specifically, the disclosure relates to efficient coding that improves compressibility while maintaining the perceived quality of reconstructed video by performing pre-analysis prior to encoding. The following disclosure is particularly applicable to pre-analysis prior to LCEVC (Low Complexity Enhancement Video Coding) encoding, although the described techniques can be used as pre-analysis prior to using other standardized encoding techniques. Background Art

[0002] When encoding and decoding video, there is always a balance to be struck between improving the compression of the encoded video and improving the fidelity of the decoded video to the original video.

[0003] In a simple case, encoder parameters such as the desired bit rate are selected when the codec is configured, and the selected encoder parameters are applied for encoding the entire video.

[0004] However, it is desirable to more intelligently configure the encoder to prioritize compression of portions of the video that contain less information, to prioritize fidelity of portions of the video that contain more information, or to ensure that the encoded video meets a set of requirements (e.g., a maximum bit rate) regardless of variations in the information density or other properties of the original video.

[0005] The coding techniques in the following specifications are particularly suitable for use with existing Low Complexity Enhancement Video Coding (LCEVC) techniques.

[0006] The standard specification for LCEVC is provided in the text of ISO / IEC 23094-2, version 1, Low Complexity Enhancement Video Coding, published in November 2021, and many possible implementation details of LCEVC are described in patent publications WO 2020 / 188273 and WO 2020 / 188229. Each of these earlier documents is incorporated herein by reference.

[0007] In broad terms, LCEVC enhances the fidelity of reproduction of decoded video after encoding and decoding using existing codecs. This is achieved by combining a base layer, which contains video encoded using an existing codec, with an enhancement layer, which indicates the residual difference between the original video and the expected decoded video produced by decoding the base layer using the existing codec. The enhancement layer can be combined with the decoded base layer to more accurately reproduce the original video.

[0008] In the context of LCEVC, the enhancement layer may also be encoded, for example, by downsampling it to a lower resolution or quantizing its pixel values.Thus, even within the standardized techniques of LCEVC, there are many encoder parameters that may be adapted to provide an improved compression-fidelity balance. Summary of the invention

[0009] According to a first aspect, a method for determining encoder parameters for encoding an input video comprising a sequence of video frames is provided, the input video having a first resolution, the method comprising: obtaining a first video frame of the input video; downsampling the first frame to a second resolution to obtain a first downsampled video frame; generating a detail perception metric based on the first downsampled video frame; determining encoder parameters for encoding the first video frame based on the detail perception metric, wherein the detail perception metric comprises an edge detection metric based on the first downsampled frame.

[0010] Optionally in the first aspect, the edge detection metric comprises a text detection metric.

[0011] Optionally in the first aspect, the edge detection metric is calculated by processing the first downsampled frame using directional decomposition to generate a set of directional components.

[0012] Optionally in the first aspect, the method includes generating a detail perception metric and determining encoder parameters for each of a plurality of local blocks of the second downsampled video frame.

[0013] Optionally in the first aspect, the encoder parameters include priorities for encoding resources.

[0014] Optionally in the first aspect, the encoder parameters are parameters for Low Complexity Enhancement Video Coding (LCCEVC).

[0015] Optionally in the first aspect, the encoder parameters include a residual mode selection for encoding a residual in an LCEVC enhancement layer when encoding the first frame.

[0016] Optionally in the first aspect, the encoder parameters include a decision whether to apply temporal prediction to an LCEVC enhancement layer when encoding a first frame.

[0017] Optionally in the first aspect, the encoder parameters include a quantization parameter for an LCEVC enhancement layer when encoding the first frame.

[0018] According to a second aspect, a method for encoding an input video comprising a sequence of video frames is provided, wherein the input video has a first resolution, the method comprising: obtaining a first video frame of the input video; performing a pre-analysis to determine encoder parameters for encoding the first video frame, the pre-analysis comprising the method according to the first aspect; instructing the encoder to encode the first video frame based on the encoder parameters.

[0019] Optionally in the second aspect, the encoder is an LCEVC encoder, and encoding the first video frame includes: downsampling the first video frame; encoding the downsampled first video frame using a base codec to obtain a base coding layer; decoding the base coding layer using the base codec to obtain a decoded reference video frame; calculating one or more residuals based on the difference between the first frame and the decoded reference video frame; and encoding the one or more residuals to obtain an enhancement layer, wherein the encoder parameters are parameters for calculating or encoding one or more of the residuals.

[0020] According to a third aspect, a method for determining encoder parameters for encoding an input video comprising a sequence of video frames is provided, the input video having a first resolution, the method comprising: obtaining a first and a second video frame of the input video, wherein the second video frame is located after the first video frame in the sequence of video frames; downsampling the first and the second frames to a second resolution to obtain a first and a second downsampled video frame; generating a detail perception metric based on the first and the second downsampled video frames; determining encoder parameters for encoding the second video frame based on the detail perception metric, wherein the detail perception metric comprises an edge detection metric based on the second downsampled frame and a motion metric based on the difference between the first and the second downsampled frames.

[0021] Optionally in the third aspect, the edge detection metric comprises a text detection metric.

[0022] Optionally in the third aspect, the edge detection metric is calculated by processing the second downsampled frame using directional decomposition to generate a set of directional components.

[0023] Optionally in the third aspect, the motion metric comprises a sum of absolute differences between the first downsampled frame and the second downsampled frame.

[0024] Optionally in the third aspect, the method includes generating a detail-perceptual metric and determining encoder parameters for each of a plurality of partial blocks of the second downsampled video frame.

[0025] Optionally in the third aspect, the encoder parameters include priorities for encoding resources.

[0026] Optionally in the third aspect, the encoder parameters are parameters for Low Complexity Enhancement Video Coding (LCCEVC).

[0027] Optionally in the third aspect, the encoder parameters include a residual mode selection for encoding a residual in an LCEVC enhancement layer when encoding the second frame.

[0028] Optionally in the third aspect, the encoder parameters include a decision whether to apply temporal prediction to the LCEVC enhancement layer when encoding the second frame.

[0029] Optionally in the third aspect, the encoder parameters include a quantization parameter for an LCEVC enhancement layer when encoding the second frame.

[0030] According to a fourth aspect, a method for encoding an input video comprising a sequence of video frames is provided, the input video having a first resolution, the method comprising: obtaining a first and a second video frame of the input video, wherein the second video frame is located after the first video frame in the sequence of video frames; performing a pre-analysis to determine encoder parameters for encoding the second video frame, the pre-analysis comprising the method according to the third aspect; instructing the encoder to encode the second video frame based on the encoder parameters.

[0031] Optionally in the fourth aspect, the encoder is an LCEVC encoder, and encoding the second video frame includes: downsampling the second video frame; encoding the downsampled second video frame using a base codec to obtain a base coding layer; decoding the base coding layer using the base codec to obtain a decoded reference video frame; calculating one or more residuals based on the difference between the second frame and the decoded reference video frame; and encoding the one or more residuals to obtain an enhancement layer, wherein the encoder parameters are parameters for calculating or encoding one or more of the residuals.

[0032] According to a fifth aspect, there is provided an apparatus comprising one or more processors and a memory, the memory storing instructions which, when executed by the processor, cause the processor to perform the method according to the first aspect or the third aspect.

[0033] According to a sixth aspect, there is provided an encoder configured to perform the method of the first aspect in parallel with the method of the second aspect.

[0034] According to a seventh aspect, there is provided an encoder configured to perform the method of the third aspect in parallel with the method of the fourth aspect.

[0035] According to an eighth aspect, there is provided a non-transitory computer-readable storage medium storing instructions which, when executed by one or more processors, cause the processors to perform the method according to the first aspect or the third aspect.

[0036] According to a ninth aspect, there is provided a non-transitory computer-readable storage medium storing instructions which, when executed by one or more processors, cause the processors to perform the method according to the first aspect in parallel with the method of the second aspect.

[0037] According to a tenth aspect, there is provided a non-transitory computer-readable storage medium storing instructions which, when executed by one or more processors, cause the processors to perform the method according to the third aspect in parallel with the method of the fourth aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1A high-level diagram showing the encoding process;

[0039] Figure 2 A high-level diagram showing the decoding process;

[0040] Figure 3 A high-level diagram showing the encoding process and specific encoding steps;

[0041] Figure 4 A high-level diagram showing the decoding process and specific decoding steps;

[0042] Figure 5 A high-level diagram showing the encoding process and residual processing;

[0043] Figure 6 A high-level diagram showing another decoding process;

[0044] Figure 7 A high-level schematic showing the encoding process and residual mode control;

[0045] Figure 8 Demonstrate classification and residual weighting;

[0046] Fig. 9 shows the decoding process using temporal prediction;

[0047] Figures 10a to 10c show the interaction between the pre-analysis module and the encoder;

[0048] Fig.11 is a block diagram showing details of a pre-analysis module in an embodiment;

[0049] Fig.12 is a flow chart schematically illustrating a method according to an embodiment;

[0050] Fig.13 is a flow chart schematically illustrating a method according to an embodiment. DETAILED DESCRIPTION

[0051] Before discussing the encoder parameter pre-analysis technique of the present invention, a discussion of encoding techniques with which the determined encoder parameters can be used is provided below. This also explains many of the terms used in the encoder parameter pre-analysis technique. Such encoding techniques have been previously described, for example, in WO 2020 / 188229.

[0052] The pre-analysis techniques discussed herein can be used with flexible, adaptable, efficient, and computationally inexpensive coding techniques and formats that combine a video coding format, a base codec (e.g., AVC, HEVC, or any other current or future codec) with coded data of enhancement layers coded using different techniques. The techniques use a downsampled source signal that is encoded using a base codec to form a base stream. An enhancement stream is formed using a coded set of residuals that correct or enhance the base stream, such as by increasing the resolution or by increasing the frame rate. There may be multiple enhancement layer data in a hierarchical structure. In a specific arrangement, the base stream may be decoded by a hardware decoder, while the enhancement stream may be suitable for software implementation.

[0053] It is critical that any optimizations used in the encoding techniques are tailored to the specific requirements or constraints of the enhancement stream and have low complexity. These requirements or constraints include: the potential reduction in computing power due to the need to software decode the enhancement stream; the need to combine the decoded residual set with the decoded frame; the possible structure of the residual data, i.e., a relatively high proportion of zero values ​​with highly variable data values ​​over a large range; subtle differences in quantized coefficient blocks; and the structure of the enhancement stream as a discrete set of residual frames divided into individual components. It should be noted that the constraints placed on the enhancement stream mean that a simple and fast entropy decoding operation is essential to enable the enhancement stream to effectively correct or enhance individual frames of the base decoded video. It should be noted that in some scenarios, the base stream is also decoded substantially simultaneously before combining, which puts a strain on resources.

[0054] In one case, the methods described herein may be applied to so-called data planes reflecting different color components of a video signal. For example, the methods described herein may be applied to different planes of YUV or RGB data reflecting different color channels. The different color channels may be processed in parallel. Thus, references to residual sets as described herein may include multiple residual sets, with each color component having a different residual set that forms part of a combined enhancement stream. The components of each stream may be aligned in any logical order, for example, each plane at the same level may be grouped together and sent together, or alternatively, residual sets of different levels in each plane may be sent together.

[0055] This document takes precedence over the requirements of the following ISO / IEC documents: "Call for Proposals for Low Complexity Video Coding Enhancements", ISO / IEC JTC1 / SC29 / WG11 N17944, Macau, China, October 2018; and "Requirements for Low Complexity Video Coding Enhancements", ISO / IEC JTC1 / SC29 / WG11 N18098, Macau, China, October 2018 (which are incorporated herein by reference). In addition, the methods described herein may be incorporated into products supplied by V-Nova International Ltd.

[0056] The general structure of a coding scheme in which the presently described techniques may be applied uses a downsampled source signal encoded with a base codec, adds a first level of correction data to the decoded output of the base codec to generate a corrected picture, and then adds another enhancement level of data to an upsampled version of the corrected picture. Thus, the streams are considered to be a base stream and an enhancement stream. This structure creates multiple degrees of freedom, allowing for great flexibility and adaptability to many situations, making the coding format suitable for many use cases, including over-the-top (OTT) transmission, live streaming, live ultra-high definition (UHD) broadcasting, etc. Although the decoded output of the base codec is not intended for viewing, it is a fully decoded video at a lower resolution, making the output compatible with existing decoders and also usable as a lower resolution output where deemed appropriate. In some cases, a base codec may be used to create a base stream. A base codec may include an independent codec controlled in a modular or "black box" manner. The methods described herein may be implemented with the aid of a computer program code that is executed by a processor and makes function calls on a hardware and / or software implemented base codec.

[0057] In general, the term "residual" as used herein refers to the difference between the value of a reference array or reference frame and the actual array or frame of data. The array can be a one-dimensional or two-dimensional array representing a coding unit. For example, a coding unit can be a 2x2 or 4x4 group of residual values ​​corresponding to a similarly sized area of ​​an input video frame. It should be noted that this generalized example is agnostic to the nature of the encoding operation performed and the input signal. Reference to "residual data" as used herein refers to data derived from a set of residuals, such as a set of residuals themselves or the output of a set of data processing operations performed on a set of residuals. Throughout this specification, in general, a set of residuals includes a plurality of residuals or residual elements, each of which corresponds to a signal element, i.e., an element of a signal or raw data. The signal can be an image or video. In these examples, the set of residuals corresponds to an image or frame of a video, wherein each residual is associated with a pixel of a signal, which is a signal element. The examples disclosed herein describe how these residuals can be modified (i.e., processed) to affect the encoding pipeline or the final decoded image while reducing the overall data size. The residuals or sets may be processed per residual element (or residual), or on a group basis, such as per tile or per coding unit, where a tile or coding unit is a contiguous subset of a set of residuals. In one case, a tile may comprise a group of smaller coding units. A tile may comprise a 16x16 set of pixels or residuals (e.g., an 8 by 8 set of 2x2 coding units, or a 4 by 4 set of 4x4 coding units). It should be noted that the processing may be performed on every video frame, or on only a set number of frames in a sequence.

[0058] In general, each enhancement stream or both enhancement streams may be encapsulated into one or more enhancement bitstreams using a set of Network Abstraction Layer Units (NALUs). A NALU is intended to encapsulate an enhancement bitstream in order to apply the enhancement to the correct base reconstructed frame. A NALU may, for example, contain a reference index to a NALU containing the base decoder reconstructed frame bitstream to which the enhancement must be applied. In this way, the enhancement may be synchronized to the base stream and the frames of each bitstream combined to produce the decoded output video (i.e., the residual of each frame of the enhancement level is combined with the frames of the base decoded stream). A group of pictures may represent multiple NALUs.

[0059] Returning to the initial process described above, where a base stream is provided along with two levels (or sub-levels) of enhancements within an enhancement stream, an example of a generalized encoding process is depicted in Figure 1, a block diagram of . An input full resolution video 100 is processed to generate various coded streams 101, 102, 103. A first coded stream (coded base stream) is produced by feeding a downsampled version of the input video to a base codec (e.g., AVC, HEVC, or any other codec). The coded base stream may be referred to as a base layer or base level. A second coded stream (coded layer 1 stream) is produced by processing a residual obtained by taking the difference between a reconstructed base codec video and a downsampled version of the input video. A third coded stream (coded layer 2 stream) is produced by processing a residual obtained by taking the difference between an upsampled version of a corrected version of the reconstructed base coded video and the input video. In some cases, Figure 1 The components of may provide a general low complexity encoder. In some cases, the enhancement stream may be generated by an encoding process that forms part of the low complexity encoder, and the low complexity encoder may be configured to control an independent base encoder and decoder (e.g., encapsulated as a base codec). In other cases, the base encoder and decoder may be provided as part of the low complexity encoder. In one case, Figure 1 The low complexity encoder may be viewed as a form of wrapper for the base codec, where the functionality of the base codec may be hidden from the entity implementing the low complexity encoder.

[0060] The downsampling operation illustrated by the downsampling component 105 may be applied to an input video to produce a downsampled video to be encoded by a base encoder 113 of a base codec. Downsampling may be performed in both vertical and horizontal directions, or alternatively only in the horizontal direction. The base encoder 113 and the base decoder 114 may be implemented by a base codec (e.g., as different functions of a common codec). The base codec and / or one or more of the base encoder 113 and the base decoder 114 may include appropriately configured electronic circuitry (e.g., a hardware encoder / decoder) and / or computer program code executed by a processor.

[0061] Each enhancement stream encoding process may not necessarily include an upsampling step. Figure 1 In , the first enhancement stream is conceptually the correction stream, while the second enhancement stream is upsampled to provide the enhancement level.

[0062] Referring to the process of generating the enhancement stream in more detail, to generate the encoded layer 1 stream, the encoded base stream is decoded by the base decoder 114 (i.e., a decoding operation is applied to the encoded base stream to generate a decoded base stream). The decoding may be performed by a decoding function or mode of the base codec. A difference is then created between the decoded base stream and the downsampled input video at the layer 1 comparator 110 (i.e., a subtraction operation is applied to the downsampled input video and the decoded base stream to generate a first set of residuals). The output of the comparator 110 may be referred to as a first set of residuals, such as a surface or frame of residual data, wherein a residual value is determined for each pixel at the resolution of the output of the base encoder 113, the base decoder 114, and the downsampling block 105.

[0063] The difference is then encoded by the first encoder 115 (ie, the level 1 encoder) to generate the encoded level 1 stream 102 (ie, the encoding operation is applied to the first set of residuals to generate the first enhancement stream).

[0064] As described above, the enhancement stream may include a first enhancement level 102 and a second enhancement level 103. The first enhancement level 102 may be considered a corrected stream, e.g., a stream that provides a correction level to a base encoded / decoded video signal at a lower resolution than the input video 100. The second enhancement level 103 may be considered to convert the corrected stream into another enhancement level of the original input video 100, e.g., it applies the enhancement level or correction to a signal reconstructed from the corrected stream.

[0065] exist Figure 1 In the example of , a second enhancement layer 103 is created by encoding another set of residuals. The other set of residuals is generated by a layer 2 comparator 119. The layer 2 comparator 119 determines the difference between an upsampled version of the decoded layer 1 stream, such as the output of the upsampling component 117, and the input video 100. The input to the upsampling component 117 is generated by applying the first decoder (i.e., the layer 1 decoder) to the output of the first encoder 115. This generates a set of decoded layer 1 residuals. These residuals are then combined with the output of the base decoder 114 at the summing component 120. This effectively applies the layer 1 residuals to the output of the base decoder 114. It allows the losses in the layer 1 encoding and decoding process to be corrected by the layer 2 residuals. The output of the summing component 120 can be regarded as an analog signal representing the output of applying the layer 1 processing to the encoded base stream 101 and the encoded layer 1 stream 102 at the decoder.

[0066] As mentioned, the upsampled stream is compared to the input video, which creates another set of residuals (i.e., a difference operation is applied to the upsampled recreated stream to generate another set of residuals). The other set of residuals is then encoded by the second encoder 121 (i.e., the level 2 encoder) as an encoded level 2 enhancement stream (i.e., the encoding operation is then applied to the other set of residuals to generate another encoded enhancement stream).

[0067] Therefore, if Figure 1 As shown in and described above, the output of the encoding process is a base stream 101 and one or more enhancement streams 102, 103, which preferably include a first enhancement layer and another enhancement layer. The three streams 101, 102 and 103 can be combined with or without additional information such as control headers to generate a combined stream representing the video coding framework structure of the input video 100. It should be noted that Figure 1 The components shown in FIG. 1 may operate on blocks or coding units of data, which correspond, for example, to 2x2 or 4x4 portions of a frame at a particular resolution level. The components operate without any inter-block dependencies, so they can be applied to multiple blocks or coding units within a frame in parallel. This is different from comparative video coding schemes, where there are dependencies (e.g., spatial dependencies or temporal dependencies) between blocks. The dependencies of comparative video coding schemes limit the level of parallelism and require much higher complexity.

[0068] Figure 1 A residual mode selection block 140 is shown. If a residual mode (RM) has been selected, the residual is processed (i.e., modified and / or graded and selected) in order to determine which residuals should be transformed and encoded, i.e., which residuals will be processed by the first and / or second encoders 115 and 121. Preferably, this processing is performed before entropy encoding. The residual mode selection 140 is an optional step that can configure or activate the processing or modification of the residual, i.e., the residual processing is performed according to the selected mode. For example, "residual mode (RM)" may correspond to a residual preprocessing mode, in which the residual of the enhancement layer is preprocessed before encoding. It should be noted that the residual preprocessing can be used independently of the encoder parameter preprocessing discussed below. This mode can be turned on and off depending on the requirements. For example, the residual mode can be configured via one or more control headers or fields, and the residual mode is an example of an encoder parameter that can be determined using encoder parameter preprocessing. In an alternative embodiment, the residual can always be modified (i.e., preprocessed), and therefore the selection of a mode is not required. In this case, the residual preprocessing can be hard-coded. Examples of residual processing are described in detail below.If selected, the residual mode may function to filter residuals within one or more of the level 1 and level 2 encoding operations, preferably at a stage prior to encoding the subcomponent.

[0069] exist Figure 2 The corresponding generalized decoding process is depicted in the block diagram of . It is said that Figure 2 Can show corresponding to Figure 1The low complexity decoder receives the three streams 101, 102, 103 generated by the low complexity encoder together with a header 204 containing additional decoding information. The header 204 may include encoder parameters determined using the encoder parameter pre-analysis described below. The encoded base stream 101 is decoded by a base decoder 210 corresponding to the base codec used in the low complexity encoder. The encoded layer 1 stream 102 is received by a first decoder 211 (i.e., a layer 1 decoder) which decodes the encoded layer 1 stream 102 as generated by the low complexity encoder. Figure 1 The first residual set encoded by the first encoder 115 is decoded. At the first summing component 212, the output of the base decoder 210 is combined with the decoded residual obtained from the first decoder 211. The combined video, which may be referred to as the layer 1 reconstructed video signal, is upsampled by the upsampling component 213. The encoded layer 2 stream 103 is received by the second decoder 214 (i.e., the layer 2 decoder). The second decoder 214 performs the decoding of the encoded layer 2 stream 103 as received by the second decoder 214. Figure 1 The second group of residuals encoded by the second encoder 121 are decoded. Although the header 204 is Figure 2 2 is shown as being used by the second decoder 214, but it may also be used by the first decoder 211 as well as the base decoder 210. The output of the second decoder 214 is a second set of decoded residuals. These may be at a higher resolution than the first set of residuals and the input to the upsampling component 213. At a second summing component 215, the second set of residuals from the second decoder 214 is combined with the output of the upsampling component 213 (i.e., the upsampled reconstructed level 1 signal) to reconstruct the decoded video 250.

[0070] According to the low complexity encoder, Figure 2 The low complexity decoder of the video signal can operate in parallel on different blocks or coding units of a given frame. In addition, the decoding performed by two or more of the base decoder 210, the first decoder 211 and the second decoder 214 can be performed in parallel. This is possible because there is no inter-block dependency.

[0071] During the decoding process, the decoder may parse the header 204 (which may contain global configuration information, picture or frame configuration information, and data block configuration information) and configure the low-complexity decoder based on those headers. In order to recreate the input video, the low-complexity decoder may decode each of the base stream, the first enhancement stream, and the other or second enhancement stream. The frames of the streams may be synchronized and then combined to derive the decoded video 250. Depending on the configuration of the low-complexity encoder and decoder, the decoded video 250 may be a lossy or lossless reconstruction of the original input video 100. In many cases, the decoded video 250 may be a lossy reconstruction of the original input video 100, where the loss has a reduced or minimal impact on the perception of the decoded video 250.

[0072] exist Figure 1 and 2 In each of the above, the layer 2 and layer 1 encoding operations may include the steps of transform, quantization, and entropy encoding (e.g., in the order described). Similarly, in the decoding stage, the residual may be passed through an entropy decoder, a dequantizer, and an inverse transform module (e.g., in the order described). Any suitable encoding and corresponding decoding operations may be used. However, preferably, the layer 2 and layer 1 encoding steps may be performed in software (e.g., as performed by one or more central or graphics processing units in the encoding device).

[0073] Transforms as described herein may use directional decomposition transforms, such as Hadamard-based transforms. Both may include a small kernel or matrix applied to the flattened coding units of the residual (i.e., 2x2 or 4x4 residual blocks). More details on the transforms may be found, for example, in patent applications PCT / EP2013 / 059847 or PCT / GB2017 / 052632, which are incorporated herein by reference. The encoder may select between different transforms to be used, for example, between the sizes of the kernels to be applied.

[0074] The transform may transform the residual information to four surfaces. For example, the transform may produce the following components: average, vertical, horizontal, and diagonal.

[0075] In summary, the methods and apparatus herein are based on an overall approach that builds upon existing encoding and / or decoding algorithms (e.g., MPEG standards such as AVC / H.264, HEVC / H.265, etc.; and non-standard algorithms such as VP9, ​​AV1, etc.), which are used as a baseline for enhancement layers for different encoding and / or decoding methods, respectively. The idea behind the overall approach of the example is to encode / decode video frames in a hierarchical manner, in contrast to using a block-based approach used in the MPEG family of algorithms. Encoding frames in a hierarchical manner includes generating a residual for a full frame, and then generating a residual for a decimated frame, and so on.

[0076] The video compression residual data of the full-size video frame may be referred to as LoQ-2 (e.g., 1920x1080 for an HD video frame), while the video compression residual data of the decimated frame may be referred to as LoQ-x, where x represents the number of corresponding hierarchical decimations. Figure 1 and 2In the described example of , the variable x may have values ​​1 and 2 representing the first and second enhancement streams. Thus, there are 2 hierarchical levels for which compressed residuals will be generated. Other naming schemes for the levels may also be applied without any functional changes (e.g., the level 1 and level 2 enhancement streams described herein may alternatively be referred to as level 1 and level 2 streams - counting down from the highest resolution).

[0077] Figure 3 A more detailed encoding process is depicted in the block diagram of . The encoding process is split into two halves as shown by the dotted lines. Below the dotted line is the base level of encoder 300, which can be usefully implemented in hardware or software. Above the dotted line is the enhancement level, which can be usefully implemented in software. Encoder 300 can include only the enhancement level process, or a combination of the base level process and the enhancement level process as desired. Encoder 300 can be usefully implemented in software, especially at the enhancement level. This arrangement allows, for example, a traditional hardware encoder that provides the base level to be upgraded using firmware (e.g., software) updates, where the firmware is configured to provide the enhancement level. In newer devices, both the base level and the enhancement level can be provided in hardware and / or a combination of hardware and software.

[0078] The encoder topology at a general level is as follows. The encoder 300 includes an input I for receiving an input signal 30. The input signal 30 may include an input video signal, where the encoder is applied frame by frame. The input I is connected to a downsampler 305D and a processing block 300-2. The downsampler 305D may correspond to Figure 1 The downsampling component 105, and the processing block 300-2 may correspond to Figure 1 The second encoder 121 of the present invention, the downsampler 305D is output to the base codec 320 at the base level of the encoder 300. The base codec 320 may implement Figure 1 The down sampler 305D is also output to the processing block 300-1. The processing block 300-1 may correspond to Figure 1 The processing block 300-1 passes the output to the upsampler 305U, which in turn outputs to the processing block 300-2. The upsampler 305U may correspond to Figure 1 The upsampling component 117 of the processing block 300-2 and 300-1 each includes one or more of the following modules: a transform block 310, a quantization block 320, an entropy coding block 330, and a residual processing block 350. The residual block 350 may occur before the transform block 310 and / or control the residual processing in the processing block 300. The order of processing may be as stated in the figure.

[0079] An input signal 30, such as full (or highest) resolution video in this example, is processed by an encoder 300 to generate various encoded streams. A base encoded stream is produced by feeding a downsampled version of the input video 30 to a base codec 320 (e.g., AVC, HEVC, or any other codec) at a base level using a downsampler 305D. The base encoded stream may include the output of a base encoder of the base codec 320. A first encoded stream (an encoded layer 1 stream) is created by reconstructing the encoded base stream to create a base reconstruction, and then taking the difference between the base reconstruction and the downsampled version of the input video 30. Reconstructing the encoded base stream may include receiving a decoded base stream from the base codec (i.e., the input to processing block 300-1 includes the following: Figure 1 300-1). The difference signal is then processed at block 300-1 to create an encoded layer 1 stream. Block 300-1 includes a transform block 310-1, a quantization block 320-1, and an entropy coding block 330-1. A second encoded stream (encoded layer 2 stream) is created by upsampling the corrected version of the base reconstruction using an upsampler 305U and taking the difference between the corrected version of the base reconstruction and the input signal 30. This difference signal is then processed at block 300-2 to create an encoded layer 2 stream. Block 300-2 includes a transform block 310-2, a quantization block 320-2, an entropy coding block 330-2, and a residual processing block 350-2. Following processing block 300-1, the blocks may be performed in the order shown in the figure (e.g., residual processing followed by transform followed by quantization followed by entropy coding).

[0080] Any known quantization scheme may be used to generate the residual signal into quanta so that certain variables may take only certain discrete values. In one case, quantization includes dividing by a predetermined step width. This may be applied at two levels (1 and 2). For example, quantization at block 320 may include dividing the transformed residual value by the step width. The step width may be predetermined, such as selected based on a desired quantization level. In one case, dividing by the step width may be converted to multiplying by an inverse step width, which may be more efficiently implemented in hardware. In this case, dequantization at block 320, for example, may include multiplying by the step width. Entropy encoding as described herein may include run length encoding (RLE), followed by processing the encoded output using a Huffman encoder. In some cases, when entropy encoding is desired, only one of these schemes may be used.

[0081] The encoded elementary stream may be referred to as a base layer stream.

[0082] Figure 3Residual processing blocks 350-2, 350-1 are shown located before the transform block 310. Although the residual processing is shown before the transform, optionally, this processing step can be arranged elsewhere, for example, later in the encoding process; however, when located before the transform step, the residual processing may have the greatest impact on the entire encoding pipeline because the efficiency propagates through the pipeline. For example, if the residual values ​​are filtered at an early stage (e.g., by setting to 0), this reduces the amount of calculations that need to be performed at subsequent stages within the processing block 300. The residual processing block 350 can be selected by the residual mode selection block 140 ( Figure 3 Not shown in Figure 1 ) is activated or configured. For example, if the residual mode is selected (e.g., turned on), the residual processing block 350 may be activated. The residual mode may be selected independently for the first and second enhancement streams (e.g., the residual processing blocks 350-2 and 350-1 may be activated and applied separately, one of which may be turned off while the other is turned on).

[0083] The residual processing block is configured to modify a set of residuals. Certain specific functionality of the residual processing block 310 is described in detail below, however, conceptually, the residual processing block 310 is used to modify the residuals. This can be viewed as a form of filtering or preprocessing. In some examples, the residuals may be graded or given priority as part of the filtering or preprocessing, whereby residuals with higher ratings or priorities are passed for further processing while residuals with lower ratings or priorities are not passed for further processing (e.g., set to 0 or a corresponding low value). In practice, the residual processing block is configured to 'eliminate' one or more residuals prior to the transform so that the transform is performed on a subset of the residuals.

[0084] The residual processing block 310 may be the same in both the L2 and L1 paths, or may be configured differently (or not included in a particular path) in order to reflect the different nature of those streams.

[0085] Specific examples may implement different residual processing modes. The residual mode selection block 140 may indicate whether the residual is to be processed, and in some embodiments, the type of processing to be performed. In general, an encoder (e.g., Figure 1 A low complexity encoder or Figure 3 The encoder 300 of FIG. 1 may include a residual mode control component 140 that selects and implements a residual mode, and a residual mode implementation component that implements processing for the selected residual mode with respect to the one or more enhancement streams. In other cases, only the residual processing block 350 may be provided within each enhancement level encoding without higher control functionality (e.g., within a higher level control component such as the control component 140). In the latter case, the functionality of the residual mode control component 140 is incorporated into the Figure 1 in the first and / or second encoders 115 and 121.

[0086] Examples of implementable residual modes include, but are not limited to, a mode in which no residual processing is performed, a binary mode whereby a particular residual is multiplied by 0 or 1, a weighted mode whereby the residual is multiplied by a weighting factor, a control mode whereby a particular block or coding unit is not processed (e.g., equivalent to setting all residual values ​​in a 2x2 or 4x4 coding unit to 0), a hierarchical or priority mode whereby residuals are graded or given priorities within a list and selected for further processing based on the rating or priority, a scoring mode whereby residuals are given a score for configuring residual encoding, and a classification mode whereby residuals and / or image elements are classified and the corresponding residuals are modified or filtered based on the classification.

[0087] As indicated herein, once the Figure 1 The residuals are calculated by comparing the image signal in its original form with the image signal in its reconstructed form. For example, in one case, the residual of the L-2 enhancement stream is determined by subtracting the upsampled output from the image signal in its original form (e.g., the input video as indicated in the figure). The input to the upsampling may be referred to as a reconstruction of the signal after analog decoding. In another case, the residual of the L-1 enhancement stream is determined by subtracting the image stream output by the base decoder from the downsampled form of the original image signal (e.g., the downsampled output).

[0088] In one residual mode, a decision may be made as to whether to encode and transmit a given set of residuals. For example, in one residual mode, a particular residual (and / or residual block - such as the 2x2 or 4x4 blocks described herein) may be selectively forwarded along the L-2 or L-1 enhancement processing pipeline by a staging component and / or a selection component. In other words, different residual modes may have Figure 1 Different residual handling in the L-2 and L-1 encoding components ...

[0089] In one residual mode, a binary weight of 0 or 1 may be applied to the residual, for example by the components discussed above. This may correspond to a mode in which selective residual processing is "on". In this mode, a weight of 0 may correspond to "ignoring" a particular residual, e.g., not forwarding the residual for further processing in the enhancement pipeline. In another residual mode, there may be no weighting (or the weights of all residuals may be set to 1); this may correspond to a mode in which selective residual processing is "off". In yet another residual mode, a normalized weight of 0 to 1 may be applied to a residual or a group of residuals. This may indicate an importance or "usefulness" weight for reconstructing a video signal at a decoder, e.g., where 1 indicates that the residual has normal use, and values ​​below 1 reduce the importance of the residual. In other cases, the normalized weight may be in another range, e.g., a range of 0 to 2 may give a particular residual importance with a weight greater than 1.

[0090] In the residual mode described above, the residuals and / or residual groups may be multiplied by assigned weights, where the weights may be assigned after a classification process applied to a set of corresponding elements and / or element groups. For example, in one case, each element or element group may be assigned a class represented by an integer value selected from a predefined integer set or range (e.g., 10 classes from 0 to 9). Each class may thus have a corresponding weight value (e.g., 0 for class 0, 0.1 for class 1, or some other nonlinear mapping). The relationship between the class and the weight value may be determined by analysis and / or experimentation, such as based on picture quality measurements at a decoder and / or within an encoder. The weights may then be used to multiply the corresponding residuals and / or residual groups, such as the residuals and / or residual groups corresponding to the elements and / or element groups. In one case, this correspondence may be spatial, such as calculating the residual based on a specific input element value, and applying the classification to the specific input element value to determine the weight of the residual. In other words, the classification may be performed on elements and / or groups of elements of an input image, which may be a frame of a video signal, but the co-located residuals and / or groups of residuals may then be weighted using weights determined from the classification rather than the elements and / or groups of elements. In this way, the characterization may be performed as a separate process from the encoding process and may therefore be calculated in parallel with the encoding of the residual process.

[0091] The above describes how the residuals may not be forwarded by setting the residual values ​​to 0 and / or by setting specific control flags for specific residuals or groups containing the residuals. In the latter case, a set of flags or binary identifiers may be used, each corresponding to an element or group of elements of the residual. Each residual may be compared to the set of flags, and each residual transformation may be prevented based on the flag. In this way, residual processing may not be destructive. Alternatively, residuals may be deleted based on flags. Furthermore, a set of flags is advantageous because it may be used repeatedly for residuals or groups of residuals without having to process each set or residual independently, and may be used as a reference. For example, each frame may have a binary bitmap that acts as a mask to indicate whether the residuals will be processed and encoded. Only residuals with a corresponding mask value of 1 may be encoded, and residuals with a corresponding mask value of 0 may be uniformly set to 0.

[0092] In the grading and filtering mode, a set of residuals may be assigned a priority or rating, which is then compared to a threshold to determine which residuals should be deselected or 'killed'. The threshold may be predetermined, or may be variable depending on the desired picture quality, transmission rate, or computational efficiency. For example, the priority or rating may be a value within a given range of values, such as a floating point value between 0 and 1 or an integer value between 0 and 255. The upper limit of the range (e.g., 1 or 255) may indicate the highest rating or priority. In this case, the threshold may be set to a value within the range. In the comparison, residuals whose corresponding rating or priority values ​​are below the threshold may be deselected (e.g., set to 0).

[0093] exist Figure 4 The block diagram depicts the corresponding Figure 3 A decoder 400 is shown that performs a decoding process of an encoder of the present invention. The decoding process is split into two halves as shown by the dotted lines. Below the dotted line is the base level of the decoder 400, which can be usefully implemented in hardware. Above the dotted line is the enhancement level, which can be usefully implemented in software. The decoder 400 can include only the enhancement level process, or a combination of the base level process and the enhancement level process as required. The decoder 400 can be usefully implemented in software, especially at the enhancement level, and can be suitably superior to traditional decoding techniques, especially traditional hardware techniques. Traditional techniques refer to early techniques that have been previously developed and sold, are already in the market, are inconvenient to replace and / or are expensive to replace, and can still be used for the purpose of decoding signals. In other cases, the base level may include any existing and / or future video encoding tools or techniques.

[0094] The decoder topology at a general level is as follows. Decoder 400 includes an input (not shown) for receiving one or more input signals including an encoded base stream, an encoded layer 1 stream, and an encoded layer 2 stream together with an optional header containing additional decoding information. Decoder 400 includes a base decoder 420 at the base layer, and processing blocks 400-1 and 400-2 at the enhancement layer. An upsampler 405U is also provided between processing blocks 400-1 and 400-2 to provide an upsampled version of the signal output by processing block 400-1 to processing block 400-2. Base decoder 420 may correspond to Figure 2 The basic decoder 210 of FIG. 4 may correspond to the processing block 400-1. Figure 2 The first decoder 211, the processing block 400-2 may correspond to Figure 2 The second decoder 214, and the upsampler 405U may correspond to Figure 2 Upsampler 213.

[0095] Decoder 400 receives the one or more input signals and directs the three streams generated by encoder 300. The encoded base stream is directed to and decoded by base decoder 420, which corresponds to base codec 420 used in encoder 300 and is used to reverse the encoding process at the base level. The encoded layer 1 stream is processed by block 400-1 of decoder 400 to recreate the first residual set created by encoder 300. Block 400-1 corresponds to processing block 300-1 in encoder 300 and is used to reverse or substantially reverse the processing of block 300-1 at the base level. The output of base decoder 420 is combined with the first residual set obtained from the encoded layer 1 stream. The combined signal is upsampled by upsampler 405U. The encoded layer 2 stream is processed by block 400-2 to recreate the additional residual created by encoder 300. Block 400-2 corresponds to processing block 300-2 of encoder 300 and is used to inverse or substantially inverse the processing of block 300-2 at the base level. The upsampled signal from upsampler 405U is combined with the additional residual obtained from the encoded level 2 stream to create a level 2 reconstruction of the input signal 30. The output of processing block 400-2 can be viewed as similar to Figure 2 The decoded video 250 of the decoded video.

[0096] As described above, the enhancement stream may include two streams, namely, an encoded layer 1 stream (first enhancement layer) and an encoded layer 2 stream (second enhancement layer). The encoded layer 1 stream provides a set of correction data that can be combined with a decoded version of the base stream to generate a corrected picture.

[0097] Figure 5 Show in more detail Figure 1320E. The encoded base stream is created directly by the base encoder 320E and may be quantized and entropy encoded as needed. In some cases, these latter processes may be performed as part of the encoding performed by the base encoder 320E. In order to generate the encoded layer 1 stream, the encoded base stream is decoded at the encoder 300 (i.e., the decoding operation is applied to the encoded base stream at the base decoding block 320D). The base decoding block 320D is shown as part of the base layer of the encoder 300 and is shown as being separated from the corresponding base encoding block 320E. For example, the base decoder 320D can be a decoding component that supplements the encoding component in the form of the base encoder 320E with a base codec. In other examples, the base decoding block 320D may actually be part of the enhancement layer, and more specifically may be part of the processing block 300-1.

[0098] Return to Figure 5 , creating a difference between the decoded base stream output from the base decoding block 320D and the downsampled input video (i.e., applying the subtraction operation 310-S to the downsampled input video and the decoded base stream to generate a first set of residuals). Here, the term residual is used in the same manner as is known in the art; that is, the residual represents the error or difference between a reference signal or frame and a desired signal or frame. Here, the reference signal or frame is the decoded base stream, and the desired signal or frame is the downsampled input video. Therefore, the residuals used in the first enhancement layer can be considered as correction signals because they are able to 'correct' the future decoded base stream to a closer approximation of the downsampled input video used in the base encoding operation. This is useful because this can correct for quirks or other characteristics of the base codec. These characteristics include, among other things, the motion compensation algorithm applied by the base codec, the quantization and entropy coding applied by the base codec, and the block adjustments applied by the base codec.

[0099] exist Figure 5 More details are shown in Figure 3 Specifically, the first residual set is transformed, quantized, and entropy encoded to produce an encoded level 1 stream. Figure 5, a transform operation 310-1 is applied to a first set of residuals; a quantization operation 320-1 is applied to the transformed set of residuals to generate a set of quantized residuals; and an entropy encoding operation 330-1 is applied to the quantized set of residuals to generate an encoded layer 1 stream at a first enhancement layer level. However, it should be noted that in other examples, only the quantization step 320-1, or only the transform step 310-1 may be performed. Entropy encoding may not be used, or may optionally be used in addition to one or both of the transform step 110-1 and the quantization step 320-1. The entropy encoding operation may be any suitable type of entropy encoding, such as a Huffman encoding operation or a run length encoding (RLE) operation, or a combination of both a Huffman encoding operation and an RLE operation. The residual processing operations 350-2, 350-1 may be provided before either or both of the transform operations 310-2, 310-1 in some embodiments. The residual processing operation 350 applies residual pre-processing as described herein, such as filtering the residuals received by the block so as to pass only a subset of the received residuals to the transform operation 310 (or in other words, to set certain residual values ​​to zero so that the original values ​​are not processed within subsequent operations in the pipeline).

[0100] As described above, the enhancement streams may include an encoded layer 1 stream (first enhancement layer) and an encoded layer 2 stream (second enhancement layer). The first enhancement layer may be considered to achieve corrected video at the base layer, that is, for example, correcting encoder and / or decoder artifacts. The second enhancement layer may be considered to be another enhancement layer that may be used to convert the corrected video to the original input video or a close approximation thereof (e.g., adding detail or sharpness). For example, the second enhancement layer may add fine details lost during downsampling and / or help correct errors introduced by one or more of the transform operation 310-1 and the quantization operation 320-1.

[0101] See also Figure 3 and Figure 5 To generate the encoded layer 2 stream, another enhancement level information is created at block 300-2 by generating and encoding another set of residuals. The other set of residuals is the difference between an upsampled version (via upsampler 305U) of the corrected version of the decoded base stream (reference signal or frame) and the input signal 30 (desired signal or frame).

[0102] In order to achieve reconstruction of a corrected version of the decoded base stream as would be generated at decoder 400, at least some of the processing steps of block 300-1 are reversed to emulate the process of decoder 200, and to account for at least some of the losses and quirks of the transform and quantization processes. To this end, block 300-1 includes an inverse quantization block 320-1i and an inverse transform block 310-1i. The quantized first residual set is inverse quantized at the inverse quantization block 320-1i in encoder 100 and inverse transformed at the inverse transform block 310-1i to regenerate a decoder-side version of the first residual set.

[0103] The decoded base stream from decoder 320D is combined with this improved decoder-side version of the first residual set (i.e., a summation operation 310-C is performed on the decoded base stream and the decoder-side version of the first residual set). The summation operation 310-C generates a reconstruction of a downsampled version of the input video as would likely be generated at the decoder - i.e., the reconstructed base codec video). Figure 3 and Figure 5 As shown in , the reconstructed base codec video is then upsampled by upsampler 305U.

[0104] The upsampled signal (i.e., reference signal or frame) is then compared to the input signal 30 (i.e., desired signal or frame) to create a second set of residuals (i.e., difference operation 300-S is applied to the upsampled recreated stream to generate another set of residuals). The second set of residuals is then processed at block 300-2 to become an encoded level 2 stream (i.e., encoding operations are then applied to another or second set of residuals to generate another or second encoded enhancement stream).

[0105] Specifically, the second residual set is transformed (i.e., the transform operation 310-2 is performed on the other residual set to generate another transformed residual set). The transformed residual is then quantized and entropy encoded in the manner described above in relation to the first residual set (i.e., the quantization operation 320-2 is applied to the transformed residual set to generate another quantized residual set; and the entropy encoding operation 320-2 is applied to the quantized other residual set to generate an encoded level 2 stream containing another enhancement information level). However, only the quantization step 20-1, or only the transformation and quantization steps may be performed. Entropy encoding may optionally be used as a supplement. Preferably, the entropy encoding operation may be a Huffman encoding operation or a run length encoding (RLE) operation or both. Similar to block 300-1, the residual processing operation 350-2 is used to pre-process (i.e., filter) the residual before the encoding operation of this block.

[0106] Therefore, if Figure 3 and 5As shown in and described above, the output of the encoding process is a base stream at the base layer, and one or more enhancement streams at the enhancement layer, preferably including enhancements of the first layer and enhancements of another layer. As discussed with reference to the previous example, Figure 5 The operations of can be applied in parallel to the coding units or blocks of color components of a frame because there are no inter-block dependencies. The encoding of each color component within a set of color components can also be performed in parallel (e.g., such that the number of copies is based on (number of frames)*(number of color components)*(number of coding units per frame)). Figure 5 It should also be noted that different color components may have different numbers of coding units per frame, for example, the luma (e.g., Y) component may be processed at a higher resolution than a set of chroma (e.g., U or V) components when human vision can detect greater changes in illumination than changes in color.

[0107] An encoded base stream and one or more enhancement streams are received at the decoder 400 . Figure 6 Show in more detail Figure 4 The decoder.

[0108] The encoded base stream is decoded at the base decoder 420 so as to produce a base reconstruction of the input signal 30 received at the encoder 300. This base reconstruction can be used in practice to provide a viewable reproduction of the signal 30 at a lower quality level. However, the main purpose of this base reconstruction signal is to provide a basis for a higher quality reproduction of the input signal 30. To this end, the decoded base stream is provided to the processing block 400-1. The processing block 400-1 also receives the encoded level 1 stream and reverses any encoding, quantization, and transformation that has been applied by the encoder 300. Block 400-1 includes an entropy decoding process 430-1, an inverse quantization process 420-1, and an inverse transformation process 410-1. Optionally, only one or more of these steps may be performed depending on the operations implemented at the corresponding block 300-1 at the encoder. By performing these corresponding steps, the decoded level 1 stream including the first residual set is made available at the decoder 400. The first residual set is combined with the decoded base stream from the base decoder 420 (i.e., a summation operation 410-C is performed on the decoded base stream and the decoded first residual set to generate a reconstruction of a downsampled version of the input video - i.e., the reconstructed base codec video). Figure 4 and Figure 6 As shown in , the reconstructed base codec video is then upsampled by upsampler 405U.

[0109] Additionally, and optionally in parallel, Figure 2The encoded layer 2 stream is processed at block 400-2 of the encoder 300 to produce another set of decoded residuals. Similar to processing block 300-2, processing block 400-2 includes an entropy decoding process 430-2, an inverse quantization process 420-2, and an inverse transform process 410-2. Of course, these operations will correspond to the operations performed at block 300-2 in encoder 300, and one or more of these steps may be omitted as desired. Block 400-2 produces a decoded layer 2 stream including another set of residuals, and these residuals are summed with the output from upsampler 405U at operation 400-C to create a layer 2 reconstruction of the input signal 30. The layer 2 reconstruction can be viewed as, for example Figure 2 In some examples, it is also possible to obtain and view the reconstructed video passed to the upsampler 405U - this will be of the first enhancement level but may be at a lower resolution than the level 2 reconstruction.

[0110] Therefore, as shown and described above, the output of the decoding process is (optional) base reconstruction, and the original signal reconstruction at a higher level. This example is particularly suitable for creating encoded and decoded videos at different frame resolutions. For example, the input signal 30 can be an HD video signal including a frame at a 1920x1080 resolution. In some cases, both base reconstruction and level 2 reconstruction can be used by a display device. For example, in the case of network services, level 2 streams can be interrupted more than level 1 streams and base streams (because they can contain up to 4x data volume, where downsampling reduces the dimension in each direction by 2). In this case, when the service occurs, the display device can recover to display the base reconstruction, and the level 2 stream is interrupted (for example, when the level 2 reconstruction is unavailable), and then recover to display the level 2 reconstruction when the network conditions improve. Similar methods can be applied when the decoding device suffers from resource constraints, such as the set-top box performing system updates can have an operating base decoder 220 to output the base reconstruction, but may not have the processing capacity to calculate the level 2 reconstruction.

[0111] The encoding apparatus also enables a video distributor to distribute video to a heterogeneous set of devices; those devices with only the base decoder 220 view the base reconstruction, while those devices with the enhancement layer can view the higher quality layer 2 reconstruction. In the comparative case, two complete video streams at separate resolutions are required to serve the two sets of devices. Since the layer 2 and layer 1 enhancement streams encode residual data, the layer 2 and layer 1 enhancement streams can be encoded more efficiently, e.g., the distribution of residual data is typically mostly around 0 quality (i.e., no difference exists) and typically takes a small range of values ​​around 0. This is especially true after quantization. In contrast, the complete video streams at different resolutions will have different distributions with non-zero means or medians, which require higher bit rates for transmission to the decoder.

[0112] like Figure 4 and 6 As seen in the examples of , the residual mode may be applied at the encoder, and the decoder may not require any additional residual processing. However, when residual processing is applied at the encoder, the layer-1 and / or layer-2 enhancement streams received at the decoder may be different from the comparison case where residual processing is not applied at the encoder. For example, when residual processing is applied, such as in accordance with any of the examples described herein, the layer-1 and / or layer-2 enhancement streams will typically contain a larger number of zero values ​​that can be more efficiently compressed by the entropy encoding stage.

[0113] Figure 7 An implementation example of the encoding process described and shown above is shown. As can be clearly recognized, the encoding and decoding steps of the stream are detailed.

[0114] The steps include a residual filtering mode step, a transform step, a quantization step, and an entropy encoding step. The encoding process identifies whether the residual filtering mode is selected. The residual filtering mode may include a form of residual grading. At the lowest level, the grading may be binary, for example, the residual is graded as 0 or 1, if the residual is graded as 0, it may not be selected for further processing; only the residual graded as 1 may be passed for further processing. In other cases, the grading may be based on a larger number of levels. If the residual mode is selected, a residual filtering step may be performed (for example, a residual grading operation may be performed on the first step of the residual to generate a graded residual set). The graded residual set may be filtered so that not all residuals are encoded into the first enhancement stream (or correction stream). In some cases, the grading and filtering steps may be combined into a single step, that is, some residual values ​​are filtered out while other residual values ​​are passed for encoding.

[0115] exist Figure 7 In an example of , if a residual mode is applied such that residual values ​​are processed prior to encoding within one or more enhancement layers, the result of the residual processing (e.g., a modified residual set) is then transformed, quantized, and entropy encoded to produce an encoded layer 1 or layer 2 stream. If a residual mode is not selected, the residual values ​​may be passed through a residual processing component for transform, quantization, and entropy encoding.

[0116] As mentioned above, it is generally preferred to 'kill' the residuals rather than the transformed coefficients. This is because processing the residuals at an earlier stage, for example by filtering the residuals based on a rating or other classification, means that the values ​​can be set to 0 to simplify calculations in later stages where the amount of computation is larger. In addition, in some cases, the residual mode can be set at the block or tile level. In this case, the residual preprocessing (i.e., the residual mode) can be selected for all residual values ​​corresponding to a particular coding unit or for a particular group of coding units. Because there is no inter-block dependency, it does not matter if a particular residual value is preprocessed while other residual values ​​are not. Being able to select the residual mode at the block or tile level enhances the flexibility of the proposed coding scheme.

[0117] Figure 7 A residual mode control block 360-1 is shown that can be used to apply a residual mode at one or more later levels in the encoding pipeline. Here, the residual mode control is shown as being only in the L1 path, but it can also be configured in the L2 path. The residual mode control block 360-1 is preferably located between the quantization 320-1 block and the entropy coding 330-1 block. In this case, the residual values ​​can be classified, graded and / or assigned scores at the residual mode selection block 350-1, but the modification of the residual values ​​can occur later than the residual mode selection block 350-1. Although not shown in the figure, the residual mode control block 360-1 can control one or more of the transform operation 310-1 and the quantization operation 320-1. In one case, the residual mode selection block 350-1 may set control flags for the residual elements (e.g., as described above), and these control flags may be used by the residual mode control block 360-1 to control one or more of the transform operation 310-1 and the quantization operation 320-1, or another operation after the quantization operation 320-1. In one case, all residual values ​​may be processed by the transform operation 310-1 and the quantization operation 320-1, but filtered, weighted, and / or set to zero via the residual mode control block 360-1. In another case, the quantization operation 320-1 may be configured to apply a coarser level of quantization based on a ranking or priority (including a binary ranking and priority) of the residual, such that the quantization operation 320-1 effectively sets a greater proportion of the residual values ​​to zero than if the residual mode is not activated.

[0118] The residual mode control block 360 - 1 optionally also provides a degree of feedback and analyzes the residual after the effect of the treatment to determine if the treatment had the appropriate effect or if it should be adjusted.

[0119] Figure 8An example 800 of applying a residual mode is shown. Example 800 relates to an example whereby classification (i.e., categorization) and weighting are applied. The concepts described with reference to the current example may also be partially applied to other residual modes. This example relates to an L-2 stream, but a similar set of components may be provided for an L-1 stream. The example is described with reference to a 2x2 coding unit, but other coding units and / or pixel groupings may be used. An input image element set 801 (shown as pixel values ​​i ij - For example, these pixel values ​​may be 16-bit or 8-bit integers representing a particular color component such as one of YUV or RGB, where i indicates an image row and j indicates an image column) are classified via a classification process 802 to generate a set of class indications 803 (e.g., in the range of integers from 0 to 4 representing 5 classes). The class may indicate contrast and / or texture levels. In other examples, a "class" may include a range of metrics such as contrast and / or texture metrics for a grouping of pixels or residuals.

[0120] exist Figure 8 , the class indication 803 is then used by a weight mapping component 804 to retrieve a set of weights 805 associated with the class indication 803. In this simple example, the weights are a set of values ​​between 0 and 1. Each class may have an associated weight that may be retrieved from a lookup table. In other cases, each weight may be a function of the class or metric value (e.g., as an example, Figure 8 The weight in is 1 / 10 of the class value, but the relationship between class value and weight can be any relationship set by the lookup table).

[0121] exist Figure 8 In parallel, the reconstructed upsampled element set 806 (shown as element u) is subtracted from the input image element 801 by a subtraction component 807. ij ) to generate an initial residual set 808 (shown as element r ij ). As shown in the figure, each coding unit or block of residual values ​​can be associated with a corresponding coding unit or block of pels and / or reconstructed pels at a particular resolution (for level 1 residuals, a similar process can be applied, but the pels can correspond to downsampled pixels). The residual 808 and the set of weights 805 are then input to a weight multiplication component 809, which multiplies the residual 808 by the set of weights 805 to output a modified set of residuals 810 (shown as r' ij As can be seen, a weight of 0 can be used to set a subset of the residuals to 0 (see 812). Thus, in Figure 8 In the example of 10Instead of being passed on for further processing, it is set to 0. The residual (e.g., 811) to which a non-zero weight is applied is passed on for further processing, but has been modified. In a simple case with binary weights (e.g., two classes), a weight of 1 may indicate that the residual value will be processed without modification. The non-zero weight may modify the residual in a way that modifies how the residual is encoded. For example, the classification at block 802 may include image classification, whereby the residual is modified based on the image classification of a particular pixel. In another case, the classification at block 802 may include assigning the image value 801 to a particular grouping based on one or more of brightness and contrast. In other examples, the classification at block 802 may select a single class and weight for the decoding unit of the four elements.

[0122] In some cases, the characterization may be performed at a location remote from the encoder and transmitted to the encoder. For example, a pre-recorded movie or television program may be processed once (e.g., by applying classification 802 and weight mapping 804) to determine a set of weights 805 for a set of residuals or groups of residuals. These weights may be transmitted to the encoder via a network, for example, which may include reference Fig. 9 A to 9C describe the residual mask, as will be described in more detail below. Alternatively, classification 802 or both classification 802 and weight mapping 804 may alternatively be performed as part of encoder parameter preprocessing described below.

[0123] In one case, as an alternative or in addition to weighting the residuals, the residuals may be compared against one or more thresholds derived from a classification process. For example, the classification process may determine a set of classes with a set of associated weights and thresholds, or only a set of associated thresholds. In this case, the residuals are compared to the determined thresholds, and residuals that fall below a particular threshold or thresholds are discarded and not encoded. For example, additional thresholding may be applied to the residuals from Figure 8 The modified residual of , and / or the weight mapping 804 and weight multiplication 809 stages can be replaced with threshold mapping and threshold application stages. In general, in both cases, for this example, the residual is modified for further processing based on a classification process, which can be applied to the corresponding image elements.

[0124] It should be noted that Figure 8 As shown in , for a particular embodiment, the local classification step may be optional (e.g., as indicated by the dashed line). In this case, one or more of the class indication 803 and the set of weights 805 may be obtained by a local process (e.g., from a remote location and / or from a stored file, and may be obtained from the encoder parameter preprocessing technique described below).

[0125] The method of residual mode processing described above may be applied at the encoder, but not at the decoder. Thus, this represents a form of asymmetric coding that may improve communication taking into account increased resources at the encoder. For example, the residuals may be weighted to reduce the size of the data transmitted between the encoder and decoder, thereby allowing quality improvement for a limited bit rate (e.g., where the discarded residuals have reduced detection capabilities at the decoder). Residual weighting may have a compound effect on transforms and quantization. Thus, residual weights may be applied in order to control transform and quantization operations, such as to optimize the bitstream given a particular available bandwidth.

[0126] In addition to the encoding and decoding techniques described above, in some examples described herein, information from two or more video frames associated with different time samples may be used, as described in WO 2020 / 188273. This may be described as a temporal pattern, for example, because it involves information from different times. Not all embodiments may utilize a temporal aspect.

[0127] The step of encoding one or more residual sets may utilize a temporal buffer arranged to store information related to a previous video frame. In one case, the step of encoding the residual set may include deriving a temporal coefficient set from the temporal buffer and modifying a current coefficient set using the retrieved temporal coefficient set. In these examples, the "coefficient" may include a transformed residual, such as defined by one or more coding units of a frame of a reference video stream - the method may be applied to both residuals and coefficients. In some cases, the modification may include subtracting the temporal coefficient set from the current coefficient set. This method may be applied to multiple coefficient sets, such as those involving a level 1 stream and those involving a level 2 stream. The modification of the current coefficient set may be selectively performed, for example, with reference to a coding unit within a video data frame.

[0128] refer to Figure 5 , temporal prediction may be applied between the transform 310-1, 310-2 and quantization 320-1, 320-2 steps of the coded level 1 and / or level 2 streams. Figure 6 , the temporal prediction may be applied after the inverse transform 410 - 1 , 410 - 2 step of decoding the level 1 and / or level 2 streams.

[0129] In some instances, there may be at least two time modes.

[0130] A first temporal mode that uses no temporal buffer or uses a temporal buffer with all zero values. The first temporal mode may be considered an intra mode, as it uses only information from within the current frame. In the first temporal mode, after any applied scaling and transform, coefficients may be quantized without modification based on information from one or more previous frames.

[0131] A second temporal mode utilizing a temporal buffer, e.g., using a temporal buffer with possibly non-zero values. The second temporal mode may be considered an inter-mode, as it uses information from outside the current frame, e.g., from multiple frames. In the second temporal mode, the dequantized coefficients of the previous frame may be subtracted from the coefficients to be quantized, after any applied residual prioritization and transform.

[0132] Temporal processing may be selectively applied at the encoder and / or decoder based on the indicated temporal mode. The temporal mode may be signaled for one or more of the two enhancement streams (e.g., at layer 2 and / or at layer 1). The temporal mode may be signaled independently for each enhancement layer. Each enhancement layer may use a different temporal buffer.

[0133] In addition, the temporal refresh parameter may signal when the temporal buffer needs to be refreshed, e.g., a first set of values ​​stored in the temporal buffer will be replaced by a second set of values. Temporal refresh may be applied at one or more of the encoder and decoder. For example, in an encoder, the temporal buffer may store dequantized coefficients of a previous frame loaded when a temporal refresh flag is set (e.g., equal to 1 indicating "refresh"). In this case, the dequantized coefficients are stored in the temporal buffer and used for temporal prediction of future frames (e.g., for subtraction) while the temporal refresh flag of the frame is not set (e.g., equal to 0 indicating "no refresh"). In this case, when a frame with an associated temporal refresh flag set to 1 is received, the contents of the temporal buffer are replaced. This may be performed on a per-frame basis and / or applied to portions of a frame such as tiles or coding units.

[0134] Fig. 9 A graphical representation 900 of a decoding process utilizing a temporal buffer as described in certain examples herein is shown. Fig. 9 The stages of the decoding process are shown from left to right. Fig. 9 The example shows how to apply additional upsampling operations after decoding the base image.

[0135] exist Fig. 9 , a decoded base picture 902 is shown. This may include the output of the base decoder described in the examples herein. In this example, optional upsampling (i.e., upscaling) is performed on the lower resolution decoded base picture 902. The lower resolution decoded base picture 902 can be considered as a level 0 or layer 0 signal. Upsampling of the decoded base picture can be applied based on the scaling factor sent with the signal.

[0136] Fig. 9 A first upsampling operation is shown to generate a preliminary intermediate picture 904. This can be viewed as the spatial resolution associated with a level 1 enhancement (eg, a level 1 signal). Fig. 9 , a preliminary intermediate picture 904 is added 906 to a first layer decoded residual 908 (e.g., produced by enhancement level 1) to generate a combined intermediate picture 910. The combined intermediate picture 910 may then be upsampled during a second upsampling operation to generate a preliminary output picture 912. The second upsampling operation may be selectively applied (e.g., may be omitted or performed on only one dimension instead of two dimensions) depending on a signaled scaling factor. The preliminary output picture 912 may be considered to be at a level 2 spatial resolution.

[0137] At stage 914, the preliminary output picture 912 is added to the second layer decoded residual 916 (e.g., produced by enhancement level 2). The second layer decoded residual 916 is shown with an added 918 contribution from the information stored in the temporal buffer 920. The information 920 may reduce the amount of information required to reconstruct the second layer residual 916. This may be beneficial because there is more data at the second level (level 2) due to the increase in spatial resolution (e.g., compared to the first level - level 1 - resolution). Fig. 9 , the last added output is the final combined output picture 922. This can be viewed as monochrome video, and / or the process can be repeated for multiple color components or planes to generate a color video output.

[0138] In addition to the above encoding and decoding techniques, in some implementations, one or more parameters of the quantization operation can be controlled to control the bit rate of one or more coded streams, as described in WO 2020 / 188273. In addition, different coded streams (e.g., an encoded layer 1 stream and an encoded layer 2 stream) can each be controlled to have a different bit rate.

[0139] In some cases, the quantization parameter may be set based on analysis of one or more of the base encoding and the enhancement stream encoding. The quantization parameter may be selected to provide a desired quality level, or to maximize the quality level within a set of predefined bit rate constraints.

[0140] For example, the encoder may include an output buffer configured to receive bits at a variable bit rate and output bits at a constant rate. The quantization parameter may be controlled by reading the state of the buffer and ensuring that the buffer does not overflow or become empty so that data is always available to be read at its output.

[0141] Such an output buffer may be arranged at the output of the coded layer-1 stream, the coded layer-2 stream or a combination of streams. Furthermore, each stream may have a separate buffer for output rate control.

[0142] In one case, the quantization parameter value is inversely proportional to the amount of data in the buffer. For example, to reduce the amount of encoded residual data, a low value of the quantization parameter can correspond to a larger quantization step value, which results in fewer quantization bins for a given range of residual values. Conversely, a high value of the quantization parameter can correspond to a smaller quantization step value, which increases the amount of encoded residual data, but can also increase the fidelity of the decoded video.

[0143] In some cases, quantization parameters may include:

[0144] a deadband parameter, such that residuals with values ​​within a certain range are set to zero (and effectively discarded),

[0145] The bin folding parameter, so that residuals with errors greater than the maximum are set to the maximum value,

[0146] quantization offset parameter (usually used together with the dead zone parameter) so that the quantized residual value associated with a quantization bin is offset from the corresponding residual value input used for quantization. This can reduce the number of bits required to represent the quantized residual value, and / or

[0147] Quantization matrix parameters. After the transform, the residual for a 2x2 or 4x4 coding unit may include coefficients such as those decomposed in the horizontal, vertical and diagonal directions. These may affect the perception of the decoded signal more seriously than other values ​​in the transform residual. Therefore, these values ​​with greater impact may be quantized with smaller quantization bins, while values ​​with less impact may be quantized with larger quantization bins. Different quantization parameters of different values ​​may be provided as quantization matrix parameters.

[0148] Additionally, the encoder's bitrate controller may be configured to identify "filler" bits of the encoded elementary stream and to discard or replace such "fillers".

[0149] Having described various implementations of encoders and decoders, the following description provides an example of a pre-analysis module that can be used with an encoder (e.g., an encoder as described above). In addition, the pre-analysis module can generate header data that can be merged as a stream with the encoded data generated by the encoder, wherein the header data is configured to be used by a corresponding decoder.

[0150] The encoder may receive one or more of encoder parameters and residual masks. The encoder parameters may include values ​​of one or more parameters that control the encoder 1000. In one case, the encoder parameters may include parameters of one or more of the base encoder, the processing components of the layer 1 stream, and the processing components of the layer 2 stream. The encoder parameters or residual masks may be applied at a per-residual or per-residual group (e.g., coding unit or block) level, or at a per-frame level or per-frame group level. The encoder parameters may be used to configure one or more of stream resolution, quantization, sequence processing, temporal prediction, bit rate, and codec for each stream. The residual mask may include, for example, a weighting from 0 to 1 to be applied to a set of residuals, such as a 2x2 or 4x4 grouping (i.e., block) of residuals. The residual mask may be similar to Figure 8 The class indication 803 and one of the weight sets 805 in . The residual mask may be supplied as a surface for each video frame (whereby there may be multiple surfaces for different color components). If the mask is applied at the group level, any received surface may be at a reduced resolution (e.g., for a 2x2 coding block, the mask may include video at half the resolution containing the residual weight values). The residual mask may indicate a priority for delivering the block to the decoder and / or for encoding. In another case, the residual mask may include a weighting that controls the processing of the block, such as a particular block may be visually enhanced or weighted. The weighting may be set based on a class (e.g., a label or a value) applied to one or more residual blocks. In some cases, the residual mask may be a binary mask (e.g., a binary bitmap) that indicates whether the residual value is encoded.

[0151] In some cases, the encoder may be adapted to perform encoding at multiple bit rates. In this case, encoder parameters applicable to each of the multiple bit rates may be provided. In some cases, the configuration data may be provided as one or more of global configuration data, per frame data, and per block data. In an example, the residual mask and time signaling may be provided in a per frame manner. For example, the multiple bit rates may be set based on the available capacity of the communication channel, such as measured bandwidth, and / or the desired use, such as using 2 Mbps of a 10 Mbps downlink channel.

[0152] In addition, the encoder can feed data back to the pre-analysis module. The data may include one or more of a base codec type, a set of required bit rates, and sequence information. The base codec type may indicate the type of base codec used for a set of current processing. In some cases, different base coders are available. In one case, the base coder may be selected based on the received base codec type parameter; in another case, the base codec type may be selected based on local processing within the encoder. The required bit rate set may indicate one or more bit rates to be used to encode the base stream and one or more of the two enhanced streams. Different streams may use different (or corresponding) bit rates. If available, the enhanced stream may use additional bandwidth; for example, if bandwidth is not available, the encoded base stream and the layer 1 stream may use bandwidth to provide a first quality level at a given bit rate; then, the encoded layer 2 stream may use a second bit rate to provide further improvement. This method may also be applied differentially to the base stream and the layer 2 stream, rather than the base stream and the layer 1 stream. The residual processing described herein may be used together with the bit rate parameter to control the bit rate of one or more of the enhanced streams.

[0153] In one case, the received encoder parameters may indicate one or more of the residual modes to be applied by the encoder. Again, the residual mode may be set at a per-frame, per-tile, and / or per-block or coding unit level. The encoder parameters may indicate the mode for each stream separately, or indicate a common mode for two enhancement streams. The residual mode parameters may be received by the residual mode selection component described herein. In some cases, the residual mode selection component may be omitted, and the residual mode parameters may be received directly by other components of the encoder, such as the components of the examples herein may receive the residual mode parameters from the cloud interface of the encoder. In some cases, each residual mode may be indicated by an integer value. The residual mode may indicate what form of residual (pre) processing will be applied.

[0154] The pre-analysis module can be located locally on the encoder and can be configured to operate simultaneously with the encoder. For example, the pre-analysis module and the encoder can be separate software or hardware modules of a single device. Alternatively, the pre-analysis module can be configured to operate asynchronously with the encoder.

[0155] In some embodiments, Fig. 10A As shown, the pre-analysis module can be a remote device configured to communicate with the encoder 1000 (or encoding process). Fig. 10A The encoder 1000 is shown communicating across a network 1010 (represented by clouds in the figure). The encoder may include an implementation of any of the encoders from the previous figures. In one embodiment, the encoder 1000 may receive configuration data across the network 1010 and / or transmit configuration data across the network 1010.

[0156] In one case, the encoder 1000 may have different configuration settings for remote or cloud configuration. In one mode, which may be a "default" mode, the encoder 1000 may be configured to make a remote procedure call across a network to retrieve initial configuration parameters to perform encoding as described herein. In another mode, which may be a "custom" mode, the encoder 1000 may retrieve local parameter values ​​that indicate a specific user configuration, such as a specific set of tools used by the encoder 1000 and / or the configuration of those tools. In one case, the encoder 1000 may have different modes that indicate which parameters are to be retrieved from a remote device and which parameters are to be retrieved from a local storage device.

[0157] Fig. 10B The encoder 1000 is shown to send configuration data to and / or receive configuration data from a remote control server 1020 via the network 1010. The control server 1020 may include a server computing device that implements a pre-analysis module and an application programming interface for receiving or sending data. For example, the control server 1020 may implement a RESTful interface, whereby data may be transmitted via (secure) hypertext transfer protocol (HTTP) requests and responses. In another case, a side channel implemented using a specific communication protocol (e.g., at the transport or application layer) may be used for communication between the control server 1020 and the encoder 1000 via the network 1010. The network 1010 may include one or more wired and / or wireless networks, including local area networks and wide area networks. In one case, the network 1010 may include the Internet.

[0158] Fig. 10C The encoder 1000 is shown as including a configuration interface 1030 configured to communicate, for example, with a remote control server 1020 via the network 1010. The configuration interface 1030 may include hardware interfaces such as Ethernet and / or wireless adapters and / or software to provide a communication stack to communicate via one or more communication networks. Fig. 10C , configuration parameters and settings 1032 used and / or stored by the encoder 1000 are transmitted via the network using a configuration interface 1030. Encoder configuration parameters, which may be stored in one or more memories or registers, are received 1034 from the configuration interface. In one case, the encoder configuration parameters may control one or more of the downsampling, base encoder, and base decoder components within the encoder, such as shown in the figure. The configuration interface also transmits L-1 control data 1036 and L-2 control data 1038 data to each of the L-1 and L-2 stream control components. These components may configure the use of tools on each enhancement stream. In one case, the L-1 and L-2 stream control components control one or more of the residual mode selection, transform, quantization, residual mode control, and entropy coding components (e.g., as shown in the figure and described herein).

[0159] Using a cloud configuration as described herein may provide implementation advantages. For example, the encoder 1000 may be remotely controlled, e.g., based on a network control system and measurements. The encoder 1000 may also be upgraded to provide new functionality, e.g., by upgrading firmware that provides additional data to enhance processing, e.g., based on measurements or pre-processing supplied by one or more remote data sources or control servers. This provides a flexible way to upgrade and control legacy hardware devices.

[0160] Fig.11 A pre-analysis module 1100 is schematically shown, which is configured to determine one or more encoder parameters and may be implemented as computer software or hardware.

[0161] The pre-analysis module 1100 is configured to receive an input video 100, which may be the same input video 100 that is later received by the encoder 1000 as part of an encoding process, such as Figure 1 , Figure 3 , Figure 5 and Figure 7 The encoding process is shown.

[0162] The pre-analysis module 1100 outputs the encoder parameters to the encoder 1000, which uses the encoder parameters as described above. For example, the pre-analysis module 1100 may be located at Fig. 10B The control server 1020 shown in FIG. Fig. 10C Interface 1030 is shown in communication with encoder 1000 .

[0163] The pre-analysis module 1100 comprises a perceptual metric generator 1110. As a first stage of pre-analysis, the perceptual metric generator 1110 generates detail perceptual metrics based on one or more frames of the input video 100.

[0164] A detail perception metric is a metric about how noticeable the details shown in one or more frames are expected to be and / or how noticeable changes in those details are expected to be. The detail perception metric may be generated for individual coding units or blocks of a frame or for an entire frame. In addition, the detail perception metric may be generated for different data planes, such as different color components of a video signal. Additionally or alternatively, the detail perception metric may be calculated by comparing two or more frames.

[0165] In some examples, the detail perception metric may include an edge detection metric.A user may be more likely to notice a loss of detail in the edges of an object depicted in a frame when compared to a loss of detail in the body of the object.

[0166] The edge detection metric can be implemented using a transformation. The transformation may be similar to Figure 3The method used in transforming the residuals in elements 310-1 and 310-2, but applied to a frame (or coding unit or block thereof). The transform as described herein may use a directional decomposition transform, such as a Hadamard-based transform. The transform may include a small kernel or matrix applied to the flattened coding units (i.e., 2x2 or 4x4 pixel blocks) of the frame. The pre-analysis module may select between different transforms to be used, for example, between the sizes of the kernels to be applied.

[0167] The transformation may transform pixel information to a surface. For example, the transformation may produce the following components: vertical tilt ("V", corresponding to the vertical difference of the pixel, which is the sum of the top pixels minus the sum of the bottom pixels), horizontal tilt ("H", corresponding to the horizontal difference of the pixel, which is the sum of the left pixels minus the sum of the right pixels), and diagonal tilt ("D", corresponding to the residual difference, which is not explained by a simple directional conversion in the pixel block). For reference, PCT / EP2013 / 059847 Figure 3 Examples of directional decomposition are shown in A and 3B.

[0168] The edge detection metric may alternatively include a binary selection, or a selection from a set of discrete options, such as: no edge, few edges, a large number of edges. Such a selection may be based on comparing one or more elements of the directional decomposition transform to one or more corresponding thresholds. The thresholds may in turn depend on high-level parameters used for encoding, such as the desired bit rate. For example, when a low bit rate is desired, the threshold for determining the presence of an edge may be relatively high in order to more compactly encode a large portion of the input video.

[0169] Additionally, edge detection metrics may include text detection metrics. Text features are often defined by edges, and users are particularly likely to notice loss of detail in text depicted in a frame.

[0170] In some examples, the detail perception metric may include a motion metric based on comparing two or more frames. A user may be more likely to notice detail loss in directional motion than detail loss in other types of motion. In addition, when a frame or portion of a frame is static, a viewer may be more likely to notice tiny details, and therefore retaining residual information may be important, such as a particular static residual element may be prioritized over a set of contrasting instantaneous residual elements. Also, noise sources in the original video recording at a higher resolution (e.g., an L-2 enhancement stream) may result in many small but instantaneous residual values ​​(e.g., normally distributed values ​​of -2 or -1 or 1 or 2) - these residual values ​​may be given a lower priority and / or set to 0 before residual processing in the enhancement level encoder.

[0171] The motion metric may comprise the sum of absolute differences (SAD) between a pair of frames.The motion metric may be evaluated in this manner per frame pair, per block pair, or per coding unit pair.

[0172] For example, the motion metric between frame m and frame n may be based on J o =Sum(abs(I x,y,n –I x,y,m )), where I x,y,n is the value of the coding unit (x, y) of frame n, and I x,y,m is the value of coding unit (x,y) of frame m.

[0173] Furthermore, when the motion metric is based on comparing more than two frames, the motion metric may include a weighted sum of SAD values. For example, a detail perception metric for frame n may be calculated by comparing frame n to each of the previous frames k and m, and the motion metric may be based on:

[0174] J o =Sum(abs(I x,y,n –I x,y,m ))+Sum(abs(I x,y,n –I x,y,k )),or

[0175] J o =w m ·Sum(abs(I x,y,n –I x,y,m ))+w k ·Sum(abs(I x,y,n –I x,y,k )),

[0176] where w m and w k is the weight factor.

[0177] The motion metric may alternatively include a binary selection, or a selection from a set of discrete options, such as: no motion, low motion, high motion. Such a selection may be based on comparing the sum of absolute differences to one or more thresholds. The thresholds may in turn depend on high-level parameters for encoding, such as the desired bit rate. For example, when a low bit rate is desired, the threshold for determining the presence of motion may be relatively high, so that most of the input video is encoded more compactly.

[0178] The first frame and the second frame used to generate the motion metric may be consecutive frames of the input video 100, or the motion metric may be generated at a reduced frequency as the context requires (e.g., comparing motion between two frames separated by N≥1 intermediate frames in the input video, comparing motion between randomly sampled frames in the input video, etc.). The frequency of generating the motion metric may depend on the motion metric (e.g., reducing the motion metric generation frequency after generating detail-aware metrics for a series of frames exhibiting low motion).

[0179] The number of times a motion metric is calculated can be reduced by reusing the same calculation for forward and backward motion. In other words, when a motion metric is calculated by comparing frame m and frame n, the motion metric can be used when generating the detail-aware metric for frame m as well as when generating the detail-aware metric for frame n. For example, adjacent frames can be paired and the motion metric calculated once for each pair of frames (i.e., a motion metric is calculated for frame 1 and frame 2, for frame 3 and frame 4, for frame 5 and frame 6, etc.).

[0180] The detail-aware metric may include a combination of metrics. For example, the detail-aware metric may include an edge detection metric based on the second frame and a motion metric based on a difference between the first frame and the second frame.

[0181] Reference again Fig.11 , the pre-analysis module 1100 further comprises a downsampler 1105 configured to downsample frames of the input video 100 before the frames are received by the perceptual metric generator 1110. The downsampler 1105 may be similar to Figure 1 The downsampling component 105 is configured to generate a detail perception metric based on one or more downsampled video frames. As a result, detail perception metrics are generated based on one or more downsampled video frames. This downsampling reduces the processing resources required to generate detail perception metrics and determine encoder parameters, but, as discovered by the inventors, it has minimal impact on the perceived quality of video encoded and decoded according to the encoder parameters determined in this manner.

[0182] When the pre-analysis module 1100 is connected with Figure 1 When used with the encoder shown, the downsampler 1105 can be configured to downsample the frame to the same resolution as the downsampling component 105. The advantage of this is that the encoder performs level 1 encoding at the same resolution as the pre-analysis module determines the encoder parameters. Alternatively, the downsampler 1105 can be configured independently of the encoder 1000.

[0183] Reference again Fig.11, the pre-analysis module 1100 may also include a feature extractor 1120. As part of the first stage of pre-analysis, the feature extractor 1120 may extract additional metrics and statistics for determining one or more encoder parameters. For each block or coding unit of a frame, the extracted features may include: histogram; average; minimum; maximum. Based on the extracted features, the feature extractor may classify each block or coding unit within the frame, for example, by providing a perceptual rating relative to neighboring blocks or coding units. In some embodiments, the feature extractor 1120 may be omitted.

[0184] In some embodiments, the pre-analysis module 1100 includes at least one encoder parameter determination unit. As a second stage of pre-analysis, the encoder parameter determination unit determines encoder parameters based on detail perception metrics. In some embodiments, the encoder parameter determination unit may also use features extracted by the feature extractor 1120 and detail perception metrics to determine encoder parameters. Encoder parameters may be determined for each frame, for each group of one or more frames, or for a portion of one or more frames (e.g., a patch, block, or coding unit). In some embodiments, the encoder parameter determination unit is omitted, and the detail perception metrics themselves are used as encoder parameters that may be provided to the encoder 1000.

[0185] In many embodiments, the encoder parameter indicates a priority associated with a frame, block, or coding unit for which the detail perception metric is calculated. A higher priority means that more resources are allocated at the encoder (e.g., more bits in the encoded bitstream to be able to decode the video with lower loss, or more time-consuming processing at the encoder). On the other hand, a lower priority means that lower resources are allocated at the encoder (e.g., more lossy compression or more truncation processing is accepted at the encoder).

[0186] In a specific example, the pre-analysis module includes a residual mode selector 1130 as an implementation of the encoder parameter determination unit. This type of encoder parameter is particularly suitable for pre-analysis of the LCEVC encoder. Figure 1 Examples of residual modes and their applications are discussed in relation to residual mode selection block 140 .

[0187] The residual mode may be determined by classifying the frame, block or coding unit to be encoded. The classification may be based on, for example, certain spatial and / or temporal characteristics of the input image, such as a detail perception metric and optionally also on features extracted by the feature extractor 1120. For example, the residual mode may be selected by comparing the detail perception metric to one or more thresholds.

[0188] In another specific example, the pre-analysis module includes a time prediction controller 1140 as an implementation of the encoder parameter determination unit. The time prediction controller is configured to determine whether to apply time prediction. This type of encoder parameter is also particularly suitable for pre-analysis of the LCEVC encoder. Figure 5 and Fig. 9 Temporal prediction is explained. In some embodiments, the temporal prediction controller 1140 may be omitted.

[0189] The detail perception metric may be used to estimate the cost of temporal prediction on a per-frame and / or per-part (e.g., per-tile and / or per-coding-unit) basis. If temporal prediction is expected to result in a loss in perceptual quality, the cost of temporal prediction may be increased. On the other hand, the cost of temporal prediction may be reduced based on the expected improvement in compression in frames encoded using temporal prediction of residuals.

[0190] In one case, the cost for determining whether to apply temporal prediction is controllable, such as by setting a parameter in a configuration file. The cost can be evaluated with or without temporal prediction, and temporal prediction can be used when temporal prediction has a lower cost than not using temporal prediction.

[0191] In some cases, the encoding parameters may include a mapping indicating whether temporal prediction is applied for a frame or a group of portions of frames of the video.

[0192] In one example, the cost function may simply be the motion metric generated by the perceptual metric generator 1110 .

[0193] The temporal prediction controller 1140 may also be configured to control whether a temporal refresh is performed on a frame.

[0194] In another specific example, the pre-analysis module includes a rate controller 1150. The rate controller is configured to manage encoding to achieve a desired bit rate, as described above with reference to the output buffer feature present in some encoders.

[0195] For example, the rate controller may be configured to determine one or more quantization parameters. The determined quantization parameters may include any one of a quantization bin size, a dead zone parameter, a bin folding parameter, a quantization offset parameter, and a quantization matrix parameter.

[0196] The rate controller may be configured to determine encoding parameters based on the detail perception metric generated by the perception metric generator 1110 and optionally also based on the features extracted by the feature extractor 1120. As an example, the detail perception metric may indicate a high perception of detail in a particular portion (e.g., a patch or block) of one or more frames. This may be due to, for example, an edge or motion. At the same time, the features extracted by the feature extractor 1120 may indicate that the pixel values ​​in the particular portion fall within a small portion of the total possible value range. In response, the quantization bin size parameter may be reduced and the size of the dead zone may be increased. This may have the effect of increasing the level of detail without increasing the number of bits required for the residual.

[0197] Where multiple encoder parameters are determined, then the determination of one parameter may be used as input to the determination of another parameter.

[0198] For example, if a quantization parameter determined for a frame, block, or coding unit would result in at least one corresponding residual at the encoder being not quantized or quantized to zero, a residual mode may be determined to prevent transformation or quantization of the residual at the encoder. This avoids performing unnecessary transformations on the residual at the encoder and thereby saves encoder processing resources.

[0199] The encoder parameter determination units 1130, 1140, 1150 may be configured to deliver the encoder parameters to the encoder 1000 in real time. Alternatively, the pre-analysis module 1100 may store the determined encoder parameters for subsequent use in the encoder 1000.

[0200] In other embodiments, the pre-analysis module 1100 may simply generate the detail perceptual metric using the perceptual metric generator 1110 and pass the detail perceptual metric to the encoder 1000. The encoder parameter determination unit may alternatively be arranged as a part of the encoder 1000.

[0201] Fig.12 The method that may be performed by the pre-analysis module in one embodiment is schematically illustrated.

[0202] refer to Fig.12 , at step S1210, the pre-analysis module obtains the first video frame of the input video 100. The first frame is any frame that is expected to be subsequently encoded by the encoder 1000, and "first" does not indicate any specific position in the frame sequence of the input video.

[0203] At step S1220, the pre-analysis module downsamples the first video frame to obtain a first downsampled video frame. This can be implemented using the downsampler 1105 as described above.

[0204] At step S1230, the pre-analysis module generates a detail perceptual metric based on the first down-sampled video frame. This can be implemented using the perceptual metric generator 1110 as described above.

[0205] At step S1240, the pre-analysis module determines encoder parameters for encoding the first video frame based on the detail perception metric. This step can be implemented using an encoder parameter determination unit as described above (such as the residual mode selector 1130, the temporal prediction controller 1140 or the rate controller 1150).

[0206] Fig.13 A method that may be performed by a pre-analysis module in another embodiment is schematically illustrated.

[0207] refer to Fig.13 , at step S1310, the pre-analysis module obtains two video frames (a first video frame and a second video frame) of the input video 100. The first and second video frames are any two different frames that are expected to be subsequently encoded by the encoder 1000, and "first" and "second" do not indicate any specific position in the frame sequence of the input video, although the second video frame is located after the first video frame in the video frame sequence (i.e., at least one frame appears after it). As an example, in the frame sequence of the input video 100, the second frame can be a frame after the first frame (i.e., the frame immediately following it), or two frames after the first frame.

[0208] At step S1320, the pre-analysis module downsamples the first and second video frames to obtain first and second downsampled video frames. This can be implemented using the downsampler 1105 as described above.

[0209] At step S1330, the pre-analysis module generates a detail perceptual metric based on the first and second downsampled video frames. This may be implemented using the perceptual metric generator 1110 as described above.

[0210] At step S1340, the pre-analysis module determines encoder parameters for encoding the second video frame based on the detail perception metric. This step can be implemented using an encoder parameter determination unit as described above (such as the residual mode selector 1130, the temporal prediction controller 1140 or the rate controller 1150).

[0211] Fig.13 Methods and Fig.12 The method of is different in that a plurality of frames are used to determine the detail-aware metric at step S1330, and thus the detail-aware metric may include a motion metric.

Claims

1. A method for determining encoder parameters for encoding an input video comprising a sequence of video frames, the input video having a first resolution, the method comprising: Acquire first and second video frames of the input video, wherein the second video frame is located after the first video frame in the video frame sequence; downsampling the first and second frames to a second resolution to obtain first and second downsampled video frames; generating a detail perception metric based on the first and second downsampled video frames; determining encoder parameters for encoding the second video frame based on the detail perception metric, Wherein the detail-aware metric comprises an edge detection metric based on the second downsampled frame and a motion metric based on a difference between the first and second downsampled frames.

2. The method of claim 1, wherein the edge detection metric comprises a text detection metric.

3. A method according to any preceding claim, wherein the edge detection metric is calculated by processing the second downsampled frame using directional decomposition to generate a set of directional components.

4. A method according to any preceding claim, wherein the motion metric comprises a sum of absolute differences between the first downsampled frame and the second downsampled frame.

5. A method according to any preceding claim, comprising generating the detail-perceptual metric and determining encoder parameters for each of a plurality of local blocks of the second downsampled video frame.

6. A method according to any preceding claim, wherein the encoder parameters comprise priorities for encoding resources.

7. The method according to any preceding claim, wherein the encoder parameters are parameters for Low Complexity Enhancement Video Coding (LCCEVC).

8. The method of claim 7, wherein the encoder parameters include a residual mode selection for encoding a residual in an LCEVC enhancement layer when encoding the second frame.

9. The method of claim 7 or claim 8, wherein the encoder parameters include a decision whether to apply temporal prediction to an LCEVC enhancement layer when encoding the second frame.

10. The method according to any one of claims 7 to 9, wherein the encoder parameters include quantization parameters for an LCEVC enhancement layer when encoding the second frame.

11. A method for encoding an input video comprising a sequence of video frames, the input video having a first resolution, the method comprising: Acquire first and second video frames of the input video, wherein the second video frame is located after the first video frame in the video frame sequence; performing a pre-analysis to determine encoder parameters for encoding the second video frame, the pre-analysis comprising the method according to any one of claims 1 to 10; An encoder is instructing to encode the second video frame based on the encoder parameters.

12. The method of claim 11, wherein the encoder is an LCEVC encoder, and encoding the second video frame comprises: downsampling the second video frame; encoding the downsampled second video frame using a base codec to obtain a base coding layer; decoding the base coding layer using the base codec to obtain a decoded reference video frame; calculating one or more residuals based on a difference between the second frame and the decoded reference video frame; as well as encoding the one or more residuals to obtain an enhancement layer, The encoder parameters are parameters used to calculate or encode one or more of the residuals.

13. An apparatus comprising one or more processors and a memory, the memory storing instructions which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 10.

14. An encoder configured to perform the method according to any one of claims 1 to 10 in parallel with the method according to claim 11 or claim 12.

15. A non-transitory computer-readable storage medium storing instructions which, when executed by one or more processors, cause the processors to perform the method according to any one of claims 1 to 10.

16. A non-transitory computer-readable storage medium storing instructions which, when executed by one or more processors, cause the processors to perform the method according to any one of claims 1 to 10 in parallel with the method according to claim 11 or claim 12.

Images