Loop filter with multiple regions

Abstract

A method and apparatus for performing loop filtering in an encoder or decoder provides a region using a common set of filter parameters. An index may be sent from the encoder to the decoder, the index indicating which set of filter parameters to use for the particular region. The loop filter may be a sample adaptive offset, an adaptive loop filter, or any other such filter. An encoder classifies regions of a picture according to blocks using a common set of filter parameters. The classification may be in the form of a graph. The filtering block uses a common filter parameter set for the region. The decoder parses the bitstream to obtain a set of filter parameters and an index representing the set of filter parameters of the decoded region.

Description

[0001] This application is a divisional application of the Chinese invention patent application with application date of July 10, 2019 and application number 201980045556.3. Technical Field

[0002] At least one of the present embodiments generally relates to a method or apparatus for video encoding or decoding. Background Art

[0003] To achieve high compression efficiency, image and video coding schemes typically employ prediction, including motion vector prediction and transforms, to balance spatial and temporal redundancy in video content. Typically, intra-frame or inter-frame prediction is used to exploit correlations between frames within or between frames. The difference between the original image and the predicted image (often expressed as a prediction error or residual) is then transformed, quantized, and entropy encoded. To reconstruct the video, the compressed data is decoded through the inverse processes of entropy coding, quantization, transform, and prediction.

[0004] Loop filters allow post-filtering of the reconstructed image to reduce coding artifacts. For example, the Sample Adaptive Offset (SAO) filter allows adding an offset to certain categories (or classes) of reconstructed samples to reduce coding artifacts. Another example is the Adaptive Loop Filter (ALF), which implements a Wiener linear post-filter on the reconstructed samples. Another example is the Deblocking Filter (DBF), which reduces blocking artifacts by smoothing block boundaries. Summary of the Invention

[0005] The shortcomings and disadvantages of the prior art are addressed by the general aspects described herein, which point in the direction of block shape adaptive intra prediction in decoding and encoding.

[0006] According to a first aspect, a method is provided. The method comprises the steps of determining a region of a picture in which at least one reconstructed block in the picture is to be filtered using a common filter parameter set; obtaining a plurality of filter parameter sets; filtering the region of the picture including the at least one reconstructed block using the common filter parameter set for blocks within the region; and encoding information including syntax indicating the filter parameter set used to filter the region and an encoded version of the region in a bitstream.

[0007] According to another aspect, a second method is provided. The method includes the steps of: decoding syntax from a bitstream, the syntax indicating a plurality of filter parameter sets for filtering a region of a picture; determining a region of the picture from the bitstream using a common filter parameter set for filtering at least one reconstructed block of the picture; filtering the at least one reconstructed block using the filter parameter set associated with the region containing the at least one reconstructed block; and decoding the filtered reconstructed block of the picture.

[0008] According to another aspect, a device is provided. The device includes a processor. The processor can be configured to encode a video block or decode a bitstream by executing any of the above methods.

[0009] According to another general aspect of at least one embodiment, there is provided an apparatus comprising: the apparatus of any decoding embodiment; and at least one of: (i) an antenna configured to receive a signal, the signal comprising the video block, (ii) a band limiter configured to limit the received signal to a frequency band comprising the video block, and (iii) a display configured to display an output representing the video block.

[0010] According to another general aspect of at least one embodiment, a non-transitory computer-readable medium is provided that includes data content generated according to any of the described encoding embodiments or variations.

[0011] According to another general aspect of at least one embodiment, there is provided a signal comprising video data generated according to any of the encoding embodiments or variations described.

[0012] According to another general aspect of at least one embodiment, a bitstream is formed that includes data content generated according to any of the described encoding embodiments or variations.

[0013] According to another general aspect of at least one embodiment, there is provided a computer program product comprising instructions that, when the program is executed by a computer, causes the computer to perform any one of the described decoding embodiments or variations.

[0014] These and other aspects, features and advantages of these general aspects will become apparent from the following detailed description of exemplary embodiments which is to be read in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A standard generic video decoder (left) and encoder (right) are shown.

[0016] Figure 2The determination of the reconstruction sample class in the case of EO mode is shown.

[0017] Figure 3 It is shown that in the case of the BO mode, the pixel range from 0 to 255 (8 bits) is evenly divided into 32 bands.

[0018] Figure 4 Picture-based SAO filtering (left) and "SAO filtering process" (right) called "SAO filtering process" for each sample group are shown.

[0019] Figure 5 Flowchart showing encoder decisions for ALF (left) and examples of ALF filter shapes (right).

[0020] Figure 6 A CTU is shown that references a list of previously coded (new) SAO parameters.

[0021] Figure 7 An example of the decoding and reconstruction process of the proposed SAO filter is shown.

[0022] Figure 8 An example of SAO block size defined relative to CTU size is shown.

[0023] Figure 9 It shows that the current SAO block can inherit parameters from its neighbor on the left or above.

[0024] Figure 10 It shows that the filter area can be rectangular or a row / column of filter blocks.

[0025] Figure 11 An example of encoding sao_palette_index and new_flag is shown.

[0026] Figure 12 It shows that the current SAO block can inherit parameters from the left or above neighbor outside the SAO area.

[0027] Figure 13 Shown are (left) parsing filter parameters in raster scan order for all blocks and filtering per region, and (right) both parsing and filtering on a region basis.

[0028] Figure 14 It is shown that several post filters can share the same region.

[0029] Figure 15 A common standard encoding scheme is shown.

[0030] Figure 16 A common standard decoding scheme is shown.

[0031] Figure 17 A typical processor arrangement is shown in which the described embodiments may be implemented.

[0032] Figure 18 An embodiment of a method of encoding using loop filtering with multiple regions is shown.

[0033] Figure 19 An embodiment of another method for decoding using loop filtering with multiple regions is shown.

[0034] Figure 20 An embodiment of an apparatus for encoding or decoding using loop filtering with multiple regions is shown. DETAILED DESCRIPTION

[0035] The general aspects described herein are in the field of video compression. The general aspects relate to loop filtering, such as using Sample Adaptive Offset (SAO) (also known as SAO palette) using "advanced merging" techniques, as described in the following commonly used EP applications, the teachings of which are expressly incorporated herein by reference:

[0036] (1) EP application number 17305626.8, entitled “A method and apparatus for picture encoding and decoding” (attorney’s docket number: PF170034),

[0037] (2) EP application No. 18305736.3, entitled “Advanced Merger Parallelizable SAO” (Agent File No. PF180072),

[0038] (3) EP application number 17305033.7, entitled “A method and apparatus for image encoding and decoding” (attorney docket number: PF160213), and

[0039] (4) EP application number 17305627.6, entitled “A method and apparatus for picture encoding and decoding” (agent file number: PF170089).

[0040] The loop filter allows post-filtering of the reconstructed picture to reduce coding artifacts (see Figure 1 For example, SAO allows adding offsets to certain categories (or classes) of reconstructed samples to reduce coding artifacts. Another example is the adaptive loop filter (ALF), which implements Wiener linear post-filtering of the reconstructed samples. Another example is the deblocking filter (DBF), which reduces blocking artifacts by smoothing block boundaries.

[0041] Typically, the loop filter (k) process (decoder side) consists of the following steps:

[0042] 1) Analyze C(k) filter parameter sets P(c); c = 0...C(k)

[0043] 2) Classify the reconstructed points into C(k) classes

[0044] 3) Use parameter P(c) to filter the reconstructed samples belonging to class c

[0045] Typically, the loop filter (k) process (encoder side) consists of the following steps:

[0046] 1) Classify the reconstructed points into C(k) classes

[0047] 2) Derive C(k) filter parameter sets P(c); c = 0...C(k)

[0048] 3) Use parameter P(c) to filter the reconstructed samples belonging to class c

[0049] Aims of the aspects described herein are to improve loop filter performance by using region-based post-filtering.

[0050] In HEVC (High Efficiency Video Coding), when enabled, a Coding Tree Unit (CTU) can be coded with three SAO modes (SaoTypeIdx): inactive (OFF), edge offset (EO), or band offset (BO). In EO or BO mode, a set of parameters is coded per channel (Y, U, V), possibly shared with neighboring CTUs (see SAO merge flag). The SAO mode is the same for Cb and Cr components.

[0051] In the case of EO, each reconstructed sample is divided into NC=5 classes (sao_eo_class) depending on the local gradient, as Figure 2 As shown, (NC-1) offset values are encoded, one for each category (the offset of a category is equal to zero).

[0052] In the case of BO, the pixel range of values (eg, from 0 to 255, 8 bits) is evenly divided into 32 bands, and the sample values belonging to (NC-1)=4 consecutive bands are modified by adding an offset off(n). Figure 3 An example of 4 consecutive bands is shown. (NC-1) offset values are encoded, one for each of the (NC-1) bands (the remaining band offsets are equal to zero).

[0053] In the case of EO or BO, the offset may not be encoded but copied from the adjacent upper or left CTU (merge mode). Figure 4 Depicted is how SAO is processed on a picture (left) and the SAO filter process itself for each CTU (right).

[0054] In EP application No. 17305627.6 entitled “Method and apparatus for picture encoding and decoding”, it is proposed to collect all samples of a reconstructed picture by using or sharing the same SAO parameters to derive an optimal SAO parameter set.

[0055] ALF filter

[0056] In the JEM software, each 2×2 block is classified into 25 classes based on its directionality and its activity using local gradients. Next, the ALF filter coefficients for each class are derived for the entire image.

[0057] For the luma samples of each CTU (filter block), the encoder decides whether to apply ALF and includes the appropriate signaling flag in the slice header. For chroma samples, the decision to apply the filter is made at the picture level rather than the CTU level.

[0058] ALF filter parameter sets can be signaled in the first CTU or slice header. Up to 25 luma filter coefficient sets can be signaled. To reduce overhead, filter coefficients from different classes can be merged. Furthermore, the ALF coefficients of the reference picture are stored, allowing them to be reused as ALF coefficients for the current picture (ALF temporal prediction).

[0059] To support ALF temporal prediction, a candidate list of ALF filter sets is maintained. When decoding a new sequence, the candidate list is empty. After decoding a picture, the corresponding filter set can be added to the candidate list. Temporal prediction of ALF coefficients improves coding efficiency for inter-coded frames. To improve coding efficiency when temporal prediction is not available (intra-frames), a set of 16 fixed filters is also assigned to each class.

[0060] Advanced Merge SAO and other features

[0061] Feature 1:

[0062] In EP application No. 17305626.8 entitled “A method and apparatus for picture encoding and decoding”, all SAO parameters (SAO parameter candidate lists) are first encoded (e.g. in a slice header or in the first CTU), and then an SAO block is encoded, said SAO block including a (merged / candidate) index that refers to the previously defined and encoded SAO parameter list (new candidate), e.g. Figure 6 shown.

[0063] The number of SAO candidates (nb_SAO_cand) and the SAO parameter list are encoded in the same order as they are used. At the encoder, the SAO parameter candidate list is reordered after encoding each candidate index, placing the most recently used parameters at the top of the list. More precisely, the candidate list is reordered so that the spatially most recently used candidates are sorted first. This can be achieved by constructing a map of the last used candidates.

[0064] OFF (all offsets of all components are zero) candidates are implicitly placed in the list, rather than explicitly encoded, at a position not too far from the top (eg, position <= 2).

[0065] Feature 2:

[0066] In EP application No. 18305736.3 entitled “Advanced Merging Parallelizable SAO”, the principle of EP application No. 17305626.8 entitled “Method and Apparatus for Picture Coding and Decoding” is extended, so that the area in which the current SAO block can inherit parameters from other SAO blocks is limited to:

[0067] -In the wavefront parallel causal region,

[0068] - and / or make the number of candidates in the reordered SAO candidate list lower than a predefined value (see "list_reordered_size" in PF180072),

[0069] - and / or limit the number of candidates by the maximum distance (dist_max) of the candidates to the current SAO block.

[0070] Feature 3:

[0071] In EP application No. 17305033.7 entitled “Method and apparatus for image encoding and decoding” and EP application No. 17305626.8 entitled “Method and apparatus for picture encoding and decoding”, the size of the SAO block (i.e., the size to which the SAO parameters are applied) is encoded in the slice header. In EP application No. 17305626.8 entitled “Method and apparatus for image encoding and decoding”, the width and / or height of the SAO block is N times the size of the CTU, where N=1, 2, or 1 / 2 (see Figure 8 ).

[0072] In the paper entitled “Description of the Proposal for SDR, HDR and 360° Video Coding by Qualcomm and Technicolor - Low and High Complexity Versions” (JVET-J0021, 10th JVET Conference, San Diego, CA, USA, April 2018) and in the paper entitled “CE2: Testing of the SAO Design from JVET-J0021 (CE2.3.2)” by A. Gadde, D. Rusanovskyy, M. Karczevicz (11th JVET Conference, Ljubljana, SI, July 10-18, 2018), it was proposed to encode two flags (merge_left_flag and merge_above_flag) to indicate whether the current SAO block is inherited from the left, above or other neighbor (see Figure 9 During the parsing phase, the SAO parameters on the left and above are checked to see if they are the same. If not, the flag above is not parsed. In other cases, the current SAO block is marked as "active" and the SAO parameters are not inherited but encoded / decoded.

[0073] All information is decoded at the beginning of a slice or picture. After parsing the merge flags of all SAO blocks, the SAO parameters of the SAO blocks marked as "active" are decoded.

[0074] In the article entitled “CE2-3.3SAO_Palette Results and Discussion” by P. Bordes, F. Racapé (JVET-K0192, 11th JVET Conference, July 10-18, 2018, SI, Louisiana), it is reported that by using the JVET reference software (VTM1.0) and using common test conditions (CTC), the combination of the above techniques provided a BD rate gain of 0.17% in AI, 0.38% in RA, and 0.52% in LDB, respectively.

[0075] In JVET-K0324, under the same conditions, the BD rate gains of AI were reported to be 0.11%, RA was 0.30%, and LDB was 0.43%.

[0076] In at least some embodiments described herein, the general aspects described are intended to:

[0077] - Combining these two approaches to equalize the coding gains of the two techniques.

[0078] - Avoids the problem of parsing all filter parameters at the start of a slice / picture, which introduces a single frame delay.

[0079] - Potentially sharing a group (region) of reconstruction samples for several (more than one) loop filters for post-filter parameter derivation.

[0080] In at least one embodiment, the general aspects of post-loop filters such as SAO or ALF described in EP Application No. 17305626.8, entitled "Method and Apparatus for Image Coding and Decoding" (see features 1, 2, and 3: first signaling the SAO or ALF parameter sets and using an index to reference them, enabling / disabling the filter for the current block, etc.) are extended to multiple regions within a slice or picture, respectively. These features can be combined with adapting the post-filter block size to each slice or picture. Several different post-filters can share the same region. Another concept in the encoder is, for example, to calculate the post-filter parameters for each region.

[0081] In a first embodiment, a filter region is defined that is associated with each filter block. A filter block is of the same size as its filter parameter set. Each filter block then belongs to a filter region. A filter region is a sub-portion of a slice or picture (a subset of a filter block). A filter region can be Figure 10 The rectangle shown (e.g., a slice) can also be a row / column of filter blocks. The filter region can also correspond to an entire slice or picture. The size or shape of the filter region (e.g., encoded as a map) is typically encoded in the slice, picture, or sequence header.

[0082] For the SAO filter, a current filter block belonging to a region may inherit SAO parameters from candidate SAO parameters corresponding to an SAO block within the same region.

[0083] In the second embodiment, for the SAO filter, the syntax changes as follows:

[0084] -First parse sao_palette_index for each SAO chunk

[0085] - If sao_palette_index is equal to the value idx_new (eg idx_new=3), the SAO block is marked as NEW (new_flag=true), which means that this is the first time that the SAO parameters are used in the current filter region.

[0086] In the parsing phase, if sao_palette_index=idx_new, the SAO parameters are parsed after parsing sao_palette_index and made available to other SAO blocks of the region in the filtering phase. In the filtering phase, the SAO parameters are added to the merged (inherited) SAO parameter list of other candidates for the current SAO region.

[0087] In a variation of this second embodiment, the value of idx_new may vary from slice to slice or region to region. It may be encoded in the slice header or with the first SAO block, or may be derived from other parameters, such as a function of the quantization parameter (QP).

[0088] In another variation of the second embodiment, for the ALF filter, the ALF filter parameter set can be signaled in the first filter block of the region, or in the region header (for example, if the region is a slice defined in HEVC) or slice / picture header. The ALF filter parameter set remains unchanged for all filter blocks in the region.

[0089] In another variant, when the filter supports time-domain prediction (eg, ALF time-domain prediction), a list of filter parameter sets is maintained per region, and the filter blocks of the current region may use filter parameters corresponding to the co-located region in the reference picture.

[0090] In another variation of the second embodiment, the number of SAO blocks in a region marked NEW is encoded at the beginning of the region (eg, together with the first SAO block of the region).

[0091] In the third embodiment, the value of sao_palette_index is encoded with n1+1+n2 bits, and new_flag is the n1th bit (idx_new_bit), as shown in Figure 11 As shown, n1 and n2 are redefined parameters, or parameters that change adaptively and conditionally according to the context.

[0092] In a variant of this embodiment, n1<=2 and the two n1 bits are merge_left_flags and merge_above_flag. Advantageously, the encoding of merge_above_flag differs from JVET-J0021 and JVET-K0324 if the above and left parameters are the same.

[0093] In the fourth embodiment, for the SAO block in the first column of the SAO area (or the SAO block in the first row of the SAO area), the SAO parameter list available for merging further includes parameters of a left (or upper) SAO block outside the SAO area.

[0094] Advantageously, when decoding the first SAO block of a region, the left column and upper row of SAO block parameters outside the current region are added to the list of the current SAO region.

[0095] In a fifth embodiment, the filter block size can be varied per region. The filter block size is encoded for each region based on a predefined table or derivation rule, or can be inferred from other encoding parameters such as the quantization parameter (QP).

[0096] For example, the basic block size is defined in the SPS, PPS or slice header (e.g. 128x128), and a QP table is used to indicate the scaling factor to be applied to the filter block width and height. An example of such a table is given below.

[0097] QP range 1-25 26-35 36-45 46-51 Zoom x0.5 x1 x2 x2

[0098] In a sixth embodiment, the filter block parameters can be resolved in a classic raster scan order in stripes. Next, the filtering stage is performed region by region (see Figure 13 The SAO filtering stage groups the SAO candidate list to reorder the association of SAO parameters with each SAO block, the class of samples, and the application of SAO offsets to correct the reconstructed samples.

[0099] The ALF filtering stage groups the classes of samples and the filters that reconstruct the samples.

[0100] Alternatively, the filter block parameters can be parsed region by region (typically using a raster scan of the filter block across the filter regions), and then the filtering stage is performed region by region (see Figure 13 right).

[0101] In the seventh embodiment, several different loop filters k (k=0, ..., N) can share the same filter region, so that multiple filters can perform parsing and filtering based on the region. The order of parsing / classification / filtering can be interleaved between filters within a region, such as Figure 14 shown.

[0102] In the eighth embodiment, several different post-filters can advantageously share the same classification process, such that c(k1)=c(k2), where k1 is different from k2. This means that the set of samples belonging to a class of filter k1 is the same as the set of samples belonging to a class of filter k2. In this case, the class is classified once. Other variations of different filters sharing the same classification process can also be used.

[0103] The proposed techniques allow for improvements to the entire video compression process. These techniques are lightweight in terms of memory access. They improve the post-filtering process by grouping the different filter stages on a region-by-region basis and making the post-filtering process parallelizable. This is achieved by improving the in-loop filtering.

[0104] The proposed modifications to the state-of-the-art SAO filter (existing standardized HEVC) or ALF reuse most of the traditional SAO or ALF block-level logic / operations. Therefore, the existing design of the HEVC or JEM codec using the post-filter can be reused to the greatest extent possible, thereby reducing the implementation cost of the proposed technique.

[0105] This document describes various aspects, including tools, features, embodiments, models, methods, and the like. Many of these aspects are described in specific terms, and often in a way that sounds limiting, at least to illustrate individual characteristics. However, this is for clarity and does not limit the application or scope of these aspects. In fact, all of the different aspects can be combined and interchanged to provide more aspects. Furthermore, these aspects can also be combined and interchanged with aspects described in previous documents.

[0106] The aspects described and considered in this document can be implemented in many different forms. Figure 15 、 16 and 17 provide some embodiments, but other embodiments are contemplated, and Figure 15 、 16 The discussion in 17 and 18 does not limit the breadth of implementation. At least one aspect generally relates to video encoding and decoding, and at least one other aspect generally relates to transmitting a generated or encoded bitstream. These and other aspects can be implemented as methods, apparatus, computer-readable storage media having stored thereon instructions for encoding or decoding video data according to any of the described methods, and / or computer-readable storage media having stored thereon a bitstream generated according to any of the described methods.

[0107] In this application, the terms "reconstruction" and "decoding" are used interchangeably, the terms "pixel" and "sample" are used interchangeably, and the terms "image," "picture," and "frame" are used interchangeably. Typically, but not necessarily, the term "reconstruction" is used on the encoder side, while "decoding" is used on the decoder side.

[0108] Various methods are described herein, and each method includes one or more steps or actions for achieving the method. Unless a specific order of steps or actions is required for proper operation of the method, the order and / or use of specific steps and / or actions can be modified or combined.

[0109] Various methods and other aspects described herein may be used to modify modules, e.g. Figure 15 and Figure 16Intra-frame prediction, entropy coding and / or decoding modules (160, 360, 145, 330) of the video encoder 100 and decoder 200 are shown. Furthermore, the present aspects are not limited to VVC or HEVC and can be applied to other standards and proposals (whether pre-existing or developed in the future), as well as extensions of any such standards and proposals (including VVC and HEVC). Unless otherwise specified or technically excluded, the aspects described in this document can be used alone or in combination.

[0110] Various numerical values are used in this document, for example, {{1,0}, {3,1}, {1,1}}. The specific values are for example purposes, and the described aspects are not limited to these specific values.

[0111] Figure 15 An encoder 100 is shown. Variations of this encoder 100 are contemplated, but for clarity, the encoder 100 is described below without describing all contemplated variations.

[0112] Before being encoded, the video sequence may undergo pre-encoding processing (101), for example, applying a color transform to the input color picture (e.g., from RGB 4:4:4 to YCbCr 4:2:0), or performing a remapping of the input picture components to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization of one color component). Metadata may be associated with the pre-processing and appended to the bitstream.

[0113] In encoder 100, a picture is encoded by encoder elements as described below. The picture to be encoded is segmented (102) and processed in units, such as CUs. Each unit is encoded using, for example, intra or inter mode. When encoding a unit in intra mode, it performs intra prediction (160). In inter mode, motion estimation (175) and compensation (170) are performed. The encoder decides (105) which of intra mode or inter mode to use for encoding the unit and indicates the intra / inter decision, for example, by a prediction mode flag. The prediction residual is calculated, for example, by subtracting (110) the predicted block from the original image block.

[0114] The prediction residual is then transformed (125) and quantized (130). The quantized transform coefficients, along with motion vectors and other syntax elements, are entropy encoded (145) to output a bitstream. The encoder can skip the transform and directly quantize the untransformed residual signal. The encoder can bypass the transform and quantization, that is, encode the residual directly without applying the transform or quantization process.

[0115] The encoder decodes the coded block to provide a reference for further prediction. The quantized transform coefficients are dequantized (140) and inverse transformed (150) to decode the prediction residual. The decoded prediction residual is combined with the prediction block (155) to reconstruct the image block. A loop filter (165) is applied to the reconstructed picture to perform, for example, deblocking / SAO (sample adaptive offset) filtering to reduce coding artifacts. The filtered image is stored in a reference picture buffer (180).

[0116] Figure 16 1 shows a block diagram of a video decoder 200. In the decoder 200, the bitstream is decoded by the decoder elements described below. The video decoder 200 generally performs the same operations as described above. Figure 15 The encoding path is the reverse of the decoding path. The encoder 100 typically also performs video decoding as part of encoding the video data.

[0117] Specifically, the input to the decoder includes a video bitstream, which may be generated by the video encoder 100. The bitstream is first entropy decoded (230) to obtain transform coefficients, motion vectors, and other encoding information. Picture segmentation information indicates how the picture is segmented. Thus, the decoder can segment (235) the picture based on the decoded picture segmentation information. The transform coefficients are inverse quantized (240) and inverse transformed (250) to decode the prediction residual. The decoded prediction residual is combined (255) with the prediction block to reconstruct the image block. The prediction block can be obtained (270) from intra-frame prediction (260) or motion compensated prediction (i.e., inter-frame prediction) (275). A loop filter (265) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (280).

[0118] The decoded picture may be further processed by post-decoding processing (285), such as an inverse color transform (e.g., conversion from YCbCr 4:2:0 to RGB 4:4:4) or inverse remapping that performs the inverse of the remapping process performed in the pre-encoding process (101). The post-decoding processing may use metadata derived in the pre-encoding process and signaled in the bitstream.

[0119] Figure 17A block diagram of an example of a system in which various aspects and embodiments are implemented is shown. System 1000 can be embodied as a device including the various components described below, and is configured to perform one or more aspects described herein. Examples of such devices include, but are not limited to, various electronic devices, such as personal computers, laptops, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. The elements of system 1000 (alone or in combination) can be implemented in a single integrated circuit, multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 1000 are distributed across multiple ICs and / or discrete components. In various embodiments, system 1000 is communicatively coupled to other similar systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports. In various embodiments, system 1000 is configured to implement one or more aspects described herein.

[0120] The system 1000 includes at least one processor 1010 configured to execute instructions loaded therein to implement, for example, the various aspects described in this document. The processor 1010 may include embedded memory, input / output interfaces, and various other circuits known in the art. The system 1000 includes at least one memory 1020 (e.g., a volatile memory device and / or a non-volatile memory device). The system 1000 includes a storage device 1040, which may include non-volatile memory and / or volatile memory, including but not limited to EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash memory, magnetic disk drive, and / or optical disk drive. As non-limiting examples, the storage device 1040 may include an internal storage device, a connected storage device, and / or a network accessible storage device.

[0121] System 1000 includes an encoder / decoder module 1030, which is configured to process data to provide encoded video or decoded video, and the encoder / decoder module 1030 may include its own processor and memory. The encoder / decoder module 1030 represents a module that may be included in a device to perform encoding and / or decoding functions. As is well known, a device may include one or both encoding and decoding modules. In addition, as is known to those skilled in the art, the encoder / decoder module 1030 may be implemented as a separate component of system 1000, or may be incorporated into the processor 1010 as a combination of hardware and software.

[0122] Program code to be loaded onto the processor 1010 or the encoder / decoder 1030 to perform various aspects described herein may be stored in the storage device 1040 and subsequently loaded into the memory 1020 for execution by the processor 1010. According to various embodiments, one or more of the processor 1010, the memory 1020, the storage device 1040, and the encoder / decoder module 1030 may store one or more of various items during the execution of the processes described herein. Such stored items may include, but are not limited to, input video, decoded video or a portion of decoded video, bitstreams, matrices, variables, and intermediate or final results from processing equations, formulas, operations, and operational logic.

[0123] In several embodiments, memory internal to the processor 1010 and / or encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device can be the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. The external memory can be memory 1020 and / or storage device 1040, such as dynamic volatile memory and / or non-volatile flash memory. In several embodiments, external non-volatile flash memory is used to store the operating system of the television. In at least one embodiment, fast external dynamic volatile memory such as RAM is used as working memory for video encoding and decoding operations, such as for MPEG-2, HEVC, or VVC (Versatile Video Coding).

[0124] As shown in block 1130, input to the elements of system 1000 may be provided through various input devices. Such input devices include, but are not limited to: (i) an RF section that receives an RF signal transmitted over the air, such as by broadcast, (ii) a composite input terminal, (iii) a USB input terminal, and / or (iv) an HDMI input terminal.

[0125] In various embodiments, the input devices of block 1130 have associated input processing elements as are known in the art. For example, the RF section may be associated with elements required to: (i) select a desired frequency (also referred to as selecting a signal, or band-limiting a signal to a frequency band), (ii) down-convert the selected signal, (iii) in some embodiments, again band-limit the frequency band to a narrower frequency band to select a signal frequency band, which may be referred to as a channel (for example), (iv) demodulate the down-converted and band-limited signal, (v) perform error correction, and (vi) demultiplex to select the desired data packet stream. The RF section of various embodiments includes one or more elements for performing these functions, such as a frequency selector, a signal selector, a frequency band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or near-baseband frequency) or to baseband. In a set-top box embodiment, the RF part and its associated input processing element receive the RF signal that sends by wired (for example, cable) medium, and by filtering, down-conversion and filtering to the desired frequency band again to perform frequency selection.Various embodiments rearrange the order of above-mentioned (and other) elements, remove some of them, and / or add other elements that perform similar or different functions.Adding element can comprise and insert element between existing element, for example, insert amplifier and analog to digital converter.In various embodiments, the RF part comprises antenna.

[0126] In addition, the USB and / or HDMI terminals may include corresponding interface processors for connecting the system 1000 to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of input processing, such as Reed-Solomon error correction, may be implemented, for example, in a separate input processing IC or in the processor 1010 as desired. Similarly, aspects of USB or HDMI interface processing may be implemented, as desired, in a separate interface IC or in the processor 1010. The demodulated, error-corrected, and demultiplexed streams are provided to various processing elements, including, for example, the processor 1010 and the encoder / decoder 1030, which are combined with memory and storage elements to process the data streams required for presentation on the output device.

[0127] The various components of system 1000 can be disposed within an integrated housing. Within the integrated housing, the various components can be interconnected and transmit data between them using a suitable connection arrangement 1140 (e.g., an internal bus known in the art), including an I2C bus, wiring, and a printed circuit board.

[0128] System 1000 includes a communication interface 1050 capable of communicating with other devices via a communication channel 1060. Communication interface 1050 may include, but is not limited to, a transceiver configured to send and receive data through communication channel 1060. Communication interface 1050 may include, but is not limited to, a modem or a network card, and communication channel 1060 may be implemented, for example, in a wired and / or wireless medium.

[0129] In various embodiments, a wireless network such as IEEE 802.11 is used to transmit data streams to system 1000. Wireless signals for these embodiments are received via communication channel 1060, for example, suitable for Wi-Fi communication, and communication interface 1050. Communication channel 1060 for these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other top-level communications. Other embodiments use a set-top box that transmits data via an HDMI connection to input block 1130 to provide streaming data to system 1000. Still other embodiments use an RF connection to input block 1130 to provide streaming data to system 1000.

[0130] System 1000 can provide output signals to various output devices including display 1100, speakers 1110, and other peripherals 1120. In various examples of embodiments, other peripherals 1120 include one or more of a stand-alone DVR, a disk player, a stereo system, a lighting system, and other devices that provide functionality based on the output of system 1000. In various embodiments, a communication protocol such as AV.Link, CEC, or other communication protocols is used between system 1000 and display 1100, speakers 1110, or other peripherals 1120, allowing device-to-device control with or without user intervention. Output devices can be communicatively coupled to system 1000 via dedicated connections via respective interfaces 1070, 1080, and 1090. Alternatively, output devices can be connected to system 1000 via communication interface 1050 using communication channel 1060. Display 1100 and speakers 1110 can be integrated into a single unit with other components of system 1000 in an electronic device (e.g., a television). In various embodiments, the display interface 1070 includes a display driver, such as a timing controller (T Con) chip.

[0131] For example, if the RF portion of input 1130 is part of a separate set-top box, the display 1100 and speaker 1110 may alternatively be separate from one or more other components. In various embodiments where the display 1100 and speaker 1110 are external components, the output signals may be provided through dedicated output connections, including, for example, an HDMI port, a USB port, or a COMP output.

[0132] The embodiments may be executed by computer software implemented by the processor 1010 or by hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments may be implemented by one or more integrated circuits. As a non-limiting example, the memory 1020 may be of any type suitable for the technical environment and may be implemented using any appropriate data storage technology, such as optical storage devices, magnetic storage devices, semiconductor-based storage devices, fixed memory, and removable memory. As a non-limiting example, the processor 1010 may be of any type suitable for the technical environment and may include one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0133] Various implementations involve decoding. As used herein, "decoding" may include, for example, all or part of the processing performed on a received coded sequence to produce a final output suitable for display. In various embodiments, such processing includes one or more processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such processing also includes or alternatively includes processing performed by the decoder of the various implementations described herein, such as extracting weight indices to be used for various intra-frame prediction reference arrays.

[0134] As a further example, in one embodiment, "decoding" refers only to entropy decoding, in another embodiment, "decoding" refers only to differential decoding, and in another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" is intended to refer specifically to a subset of operations or generally to a broader decoding process will be clear based on the context of the specific description and is believed to be well understood by those skilled in the art.

[0135] Various implementations involve encoding. Similar to the discussion above regarding "decoding," "encoding," as used in this application, may include, for example, all or part of the processing performed on an input video sequence to produce an encoded bitstream. In various embodiments, such processing includes one or more processes typically performed by an encoder, such as segmentation, differential coding, transforms, quantization, and entropy coding. In various embodiments, such processing also includes or alternatively includes processing performed by the encoder of the various implementations described in this application, such as weighting of an intra-frame prediction reference array.

[0136] As a further example, in one embodiment, "encoding" refers only to entropy encoding, in another embodiment, "encoding" refers only to differential encoding, and in another embodiment, "encoding" refers to a combination of differential encoding and entropy encoding. Whether the phrase "encoding process" is intended to refer specifically to a subset of operations or generally to a broader encoding process will be clear based on the context of the specific description and is believed to be well understood by those skilled in the art.

[0137] Note that the syntax elements used here are descriptive terms. Therefore, they do not exclude the use of other syntax element names.

[0138] When a diagram is represented as a flow chart, it should be understood that it also provides a block diagram of the corresponding apparatus. Similarly, when a diagram is represented as a block diagram, it should be understood that it also provides a flow chart of the corresponding method / process.

[0139] Various embodiments refer to rate-distortion calculation or rate-distortion optimization. In particular, in the coding process, a balance or trade-off between rate and distortion is often considered, typically given a computational complexity constraint. Rate-distortion optimization is often described as minimizing a rate-distortion function, which is a weighted sum of rate and distortion. There are different approaches to solving the rate-distortion optimization problem. For example, these approaches can be based on extensive testing of all coding options (including all considered modes or coding parameter values) and a complete evaluation of their coding costs and the associated distortion of the reconstructed signal after encoding and decoding. Faster approaches can also be used to save coding complexity, particularly by computing an approximate distortion based on a prediction or prediction residual signal rather than the reconstructed signal. A hybrid of these two approaches can also be used, for example using an approximate distortion for only some possible coding options and the full distortion for others. Other approaches only evaluate a subset of possible coding options. More generally, many approaches use any of a variety of techniques to perform optimization, but optimization does not necessarily require a comprehensive evaluation of coding costs and associated distortion.

[0140] The implementations and aspects described herein can be implemented, for example, in methods or processes, devices, software programs, data streams, or signals. Even if discussed only in the context of a single implementation (e.g., discussed only as a method), the implementation of the features discussed can also be implemented in other forms (e.g., devices or programs). Devices can be implemented, for example, in appropriate hardware, software, and firmware. The methods can be implemented, for example, in a processor, which generally refers to a processing device, which includes, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, for example, computers, mobile phones, portable / personal digital assistants ("PDAs") and other devices that facilitate information communication between end users.

[0141] Reference to "one embodiment" or "an embodiment" or "an implementation" or "an implementation" and other variations thereof means that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "in one implementation" or "in an implementation" and any other variations thereof in various places herein are not necessarily all referring to the same embodiment.

[0142] Furthermore, reference may be made herein to "determining" various information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from a memory.

[0143] Additionally, reference may be made herein to "accessing" various information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0144] Furthermore, this document may refer to "receiving" various information. Like "accessing," receiving is a broad term. Receiving information can include one or more of the following, for example, accessing information or retrieving information (e.g., from a memory device). Furthermore, during operations such as storing information, processing information, sending information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information, "receiving" is often involved in one way or another.

[0145] It should be understood that, for example, in the context of "A / B," "A and / or B," and "at least one of A and B," the use of any of " / ," "and / or," and "at least one of" is intended to include selecting only the first listed option (A), or only the second listed option (B), or both options (A and B). As another example, in the context of "A, B, and / or C" and "at least one of A, B, and C," such language is intended to include selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A, B, and C). As will be appreciated by persons of ordinary skill in the art and related arts, this can be extended to any number of listed items.

[0146] Furthermore, as used herein, the term "signaling" means, among other things, indicating something to a corresponding decoder. For example, in some embodiments, an encoder signals a specific one of a plurality of weights to be used for an intra-frame prediction reference array. Thus, in embodiments, the same parameters are used on both the encoder and decoder sides. Thus, for example, an encoder may send (explicitly signal) specific parameters to a decoder so that the decoder can use the same specific parameters. Conversely, if the decoder already has the specific parameters along with other parameters, signaling may be used instead (implicitly signaling), simply allowing the decoder to know and select the specific parameters. By avoiding the transmission of any actual function, bit savings are achieved in various embodiments. It should be understood that signaling can be implemented in a variety of ways. For example, in various embodiments, one or more syntax elements, flags, etc. are used to signal information to a corresponding decoder. While previously mentioned as a verb, the term "signaling" can also be used here as a noun.

[0147] As will be apparent to those skilled in the art, implementations may generate various signals formatted to carry information that can be stored or transmitted. This information may include, for example, instructions for performing a method, or data generated by one of the described implementations. For example, a signal may be formatted to carry a bitstream of the described embodiments. Such a signal may, for example, be formatted as an electromagnetic wave (e.g., using a radio frequency portion of a spectrum) or a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is well known, the signal may be transmitted over a variety of different wired or wireless links. The signal may be stored on a processor-readable medium.

[0148] Embodiments may include one or more of the following features or entities, alone or in combination, across various claim categories and types:

[0149] • Modifying the loop filter processing applied in the decoder and / or encoder.

[0150] Enable multiple filter parameter sets and regions in the decoder and / or encoder.

[0151] • Inserting a syntax element in the signaling that enables the decoder to identify the region where the filter parameter set is to be applied for loop filtering.

[0152] • Based on these syntax elements, a set of filter parameters is selected to be applied to the decoder.

[0153] Apply loop filtering at the decoder, such as adaptive loop filtering and sample adaptive offset filtering.

[0154] • According to any of the embodiments discussed, loop filtering is performed at the encoder.

[0155] • A bitstream or signal comprising one or more of the described syntax elements or variants thereof.

[0156] • Creating and / or sending and / or receiving and / or decoding a bitstream or signal comprising one or more of said syntax elements or variants thereof.

[0157] • A television, set-top box, mobile phone, tablet or other electronic device that performs loop filtering according to any of the described embodiments.

[0158] • A television, set-top box, mobile phone, tablet, or other electronic device that performs loop filtering according to any of the described embodiments and displays (e.g., using a monitor, screen, or other type of display) the resulting image.

[0159] • A television, set-top box, mobile phone, tablet, or other electronic device that tunes (e.g., using a tuner) to a channel to receive a signal including an encoded image and performs loop filtering according to any of the described embodiments.

[0160] • A television, set-top box, mobile phone, tablet, or other electronic device that receives over-the-air (e.g., using an antenna) a signal comprising an encoded image and performs loop filtering according to any of the described embodiments.

[0161] Various other generalized and specific inventions and claims are also supported and contemplated throughout this disclosure.

[0162] Figure 18 One embodiment of a method 1800 for encoding a block of video data using the general aspects described herein is shown. The method begins at a start block 1801, and control proceeds to a function block 1810 to determine a region of a picture in which at least one reconstructed block of the picture is filtered using a common set of filter parameters. Control then proceeds from block 1810 to block 1818 to obtain a plurality of filter parameters. Control then proceeds from block 1818 to block 1830 to filter the region of the picture including the at least one reconstructed block using a common set of filter parameters for the blocks within the region. Control then proceeds from block 1830 to block 1840 to encode information and an encoded version of the region in a bitstream, the information including syntax indicating the set of filter parameters used to filter the region.

[0163] Figure 19One embodiment of a method 1900 for decoding a block of video data using the general aspects described herein is shown. The method begins at a start block 1901, and control proceeds to a function block 1910 to decode syntax from a bitstream, the syntax indicating a plurality of filter parameter sets for filtering a region of a picture. Control then proceeds from block 1910 to block 1920 to determine a region of the picture from the bitstream using a common filter parameter set for filtering at least one reconstructed block of the picture. Control then proceeds from block 1920 to block 1930 to filter the at least one reconstructed block using the filter parameter set associated with the region containing the at least one reconstructed block. Control then proceeds from block 1930 to block 1940 to decode the filtered reconstructed block of the picture.

[0164] Figure 20 One embodiment of an apparatus 2000 for encoding or decoding a block of video data is shown. The apparatus includes a processor 2010 and may be interconnected to a memory 2020 via at least one port. The processor 2010 and the memory 2020 may also have one or more additional interconnects to connect to the outside world.

[0165] Processor 2010 is configured to encode or decode video data by forming a plurality of reference arrays from reconstructed samples of a block of video data, predicting a target pixel of the block of video data by applying weight sets selected from a plurality of weight sets to one or more of the plurality of reference arrays, respectively, calculating a final prediction of the target pixel of the video block as a function of the predictions from the one or more reference arrays, respectively, and encoding or decoding the video block using the final prediction.

Claims

1. A method comprising: determining a region of the picture in which at least one reconstructed block in the picture is to be filtered using a common set of filter parameters; obtaining multiple filter parameter sets; filtering a region of the picture including the at least one reconstructed block using a common set of filter parameters for blocks within the region; as well as Information and an encoded version of the region are encoded in a bitstream, the information including syntax indicating a set of filter parameters for filtering the region.

2. A device comprising: A processor configured to: determining a region of the picture in which at least one reconstructed block in the picture is to be filtered using a common set of filter parameters; obtaining multiple filter parameter sets; filtering a region of the picture including the at least one reconstructed block using a common set of filter parameters for blocks within the region; as well as Information and an encoded version of the region are encoded in a bitstream, the information including syntax indicating a set of filter parameters for filtering the region.

3. A method comprising: decoding syntax from a bitstream, the syntax indicating a plurality of filter parameter sets for filtering a region of a picture; determining, from the bitstream, a region of the picture using a common set of filter parameters for filtering at least one reconstructed block of the picture; filtering the at least one reconstructed block using the set of filter parameters associated with the region containing the at least one reconstructed block; and The filtered reconstructed block of the picture is decoded.

4. A device comprising: A processor configured to: decoding syntax from a bitstream, the syntax indicating a plurality of filter parameter sets for filtering a region of a picture; determining, from the bitstream, a region of the picture using a common set of filter parameters for filtering at least one reconstructed block of the picture; filtering the at least one reconstructed block using the set of filter parameters associated with the region containing the at least one reconstructed block; and The filtered reconstructed block of the picture is decoded.

5. The method according to claim 1 or 3, or the apparatus according to claim 2 or 4, wherein the syntax is an index indicating the filter parameter set.

6. The method according to claim 1 or 3, or the apparatus according to claim 2 or 4, wherein filtering is performed for a sample adaptive offset filter and the index is applied to SAO blocks.

7. The method according to claim 1 or 3, or the apparatus according to claim 2 or 4, wherein the filter parameter set of the SAO filter can be used to filter other blocks in the SAO area.

8. The method according to claim 1 or 3, or the apparatus according to claim 2 or 4, wherein the set of filter parameters of the adaptive loop filter is signaled in the first filter block of the region.

9. The method according to claim 1 or 3, or the device according to claim 2 or 4, wherein: For adaptive loop filter time-domain prediction, the filter block of the current region uses the filter parameter set corresponding to the co-located region in the reference picture.

10. The method according to claim 1 or 3, or the device according to claim 2 or 4, wherein: For an SAO block in the first column or the first row of an SAO area, sample adaptive offset filter parameters available for merging include filter parameters of a left or upper SAO block outside the SAO area.

Images