Content adaptive alf band classifier

Abstract

Methods and apparatus are provided for classifying signals within a video compression / decompression loop for filtering. In at least one embodiment, a reconstructed video signal is classified according to amplitude values. The classification covers a plurality of frequency bands, which can be uniformly or non-uniformly spread across the signal range. In another embodiment, a reconstructed residual signal is similarly classified. The classification is used to implement different filtering parameters based on the classification.

Description

[0001] Cross-references to related applications This application claims the benefit of European serial number 24305435.0, filed on 25 March 2024, which is incorporated herein by reference in its entirety. Technical Field

[0002] At least one embodiment in the present embodiments generally relates to a method or apparatus for video compression and processing. Background Technology

[0003] The general aspects described herein relate to methods for filtering signals within a video compression / decompression loop. Summary of the Invention

[0004] At least one embodiment in the present embodiments generally relates to a method or apparatus for content-adapted loop filter band classification of video samples.

[0005] According to a first aspect, a method is provided. The method includes the steps of: determining at least one classifier for video samples in a video compression loop; filtering reconstructed video samples based on said at least one classifier; and encoding said video samples using the filtered reconstructed video samples.

[0006] According to the second aspect, another method is provided. This method includes the steps of: determining at least one classifier for video samples in a video compression loop; filtering the reconstructed video samples based on the at least one classifier; and encoding the video samples using the filtered reconstructed video samples.

[0007] According to another aspect, an apparatus is provided. The apparatus includes a processor and a memory. The processor can be configured to operate on digital video data according to the aforementioned method.

[0008] According to another general aspect of at least one embodiment, an apparatus is provided comprising means according to any decoding embodiment; and at least one of the following: (i) an antenna configured to receive a signal including a video block, (ii) a band limiter configured to limit the received signal to a band including the video block, or (iii) a display configured to display an output representing the video block.

[0009] According to another general aspect of at least one embodiment, a non-transitory computer-readable medium is provided that contains data content generated according to any of the described encoding embodiments or variations.

[0010] According to another general aspect of at least one embodiment, a signal comprising video data generated according to any of the described encoding embodiments or variations is provided.

[0011] According to another general aspect of at least one embodiment, video data or bitstreams are formatted to include data content generated according to any of the described encoding embodiments or variations.

[0012] According to another general aspect of at least one embodiment, a computer program product including instructions is provided that, when executed by a computer, cause the computer to perform any of the described decoding embodiments or variations.

[0013] These and other aspects, features, and advantages of the general aspects will become apparent from the following detailed description of exemplary embodiments, taken in conjunction with the accompanying drawings. Attached Figure Description

[0014] Figure 1 The diagram illustrates a standard, common video compression scheme.

[0015] Figure 2 The diagram illustrates a standard, universal video decompression scheme.

[0016] Figure 3 The illustration shows an example of a pixel range from 0 to 255 using 8 bits and evenly divided into 32 frequency bands in the band offset (B0) mode.

[0017] Figure 4 The illustration shows an example of candidate brightness sample locations used in the CCSAO BO classifier.

[0018] Figure 5 The illustration shows an example of a classifier with adaptive parameters derived from the entire image, as described in the general description.

[0019] Figure 6 The diagram shows a uniform 3D histogram (left) and a 3D histogram adapted to the (minimum, maximum) values ​​of the components (right).

[0020] Figure 7 The illustration shows an example of a non-uniform 3D histogram of the brightness component.

[0021] Figure 8 The illustration shows an embodiment of the first method according to the described aspects.

[0022] Figure 9 The illustration shows an embodiment of the second method according to the described aspects.

[0023] Figure 10 The illustration shows one embodiment of the apparatus according to the described aspects.

[0024] Figure 11 The diagram illustrates a processor-based system for encoding / decoding, according to a generally described aspect. Detailed Implementation

[0025] The embodiments described herein belong to the field of in-loop filtering. Several different types of filtering can occur in compressed video encoders / decoders.

[0026] Post-filter within the loop.

[0027] In recent video codecs, post-filtering can be used to enhance reconstructed images. The post-filtering stage is called in-loop filtering when the filtered image is stored in the decoded picture buffer (DPB) for further use in inter-frame prediction.

[0028] Some post-filters use additional data (such as filter parameters (e.g., linear filter coefficients, sample offsets, etc.)), which can be implicit (predefined values ​​assumed to be known on the decoder side), also known as fixed filters, or explicit (values ​​encoded in the bitstream), also known as non-fixed or adaptive filters or sample adaptive offset (SAO) filters.

[0029] Image samples can be classified into N categories Cl i=0，..N-1 And a set of N filter parameters P i=0，..N-1 They can be selected (implicit case) or encoded in the bitstream (explicit case). In one variant, a subset of the filter parameters is implicit, while the rest are explicit.

[0030] For each sample, if the sample is classified as Cl k Then use filter parameter P k To filter the samples.

[0031] SAO and CCSAO filters.

[0032] SAO (Sample Adaptive Shift) band-shifting (BO) mode uses a band classifier where the sample range of component 'c' values ​​(e.g., 0..255 in 8 bits) is uniformly divided into N(c) = 32 bands, and sample values ​​belonging to (NC) = 4 consecutive bands are added to the bitstream (P). i=0…3 The offset explicitly encoded in =off(i)) Let's make the changes. Figure 3 An example of four consecutive frequency bands is shown. The offset values ​​of (NC) band symbols and the starting frequency band are encoded, with each of the (NC) frequency bands corresponding to an offset value (the remaining frequency bands have an offset equal to zero).

[0033] Similar to SAO, CCSAO (Cross Component Sample Adaptive Offset) classifies reconstructed samples into different categories, appropriately deriving an offset for each category and adding the offset to the reconstructed samples in that category. However, unlike SAO, which uses only a single luminance / chrominance component of the current sample as input, CCSAO utilizes all three components to classify the current sample into different categories.

[0034] For example, in CCSAO BO mode, for a given luminance / chrominance sample, three candidate samples are selected to classify the given sample into different categories: a collocated Y sample, a collocated U sample, and a collocated V sample. Then, the sample values ​​of these three selected samples are classified into three different bands. Y band U band V The frequency band is determined, and the category of a given color sample is indicated by the joint index "i". An offset is signaled and added to the reconstructed sample belonging to that category, which can be formulated as: (1) In equation (1), {Y col U col V col} represents the three corresponding samples used to classify the current sample; {N} Y N U N V} are respectively applied to {Y col U col V col The number of equally divided frequency bands across the entire range; BD is the internal coding bit depth; C rec and C' rec These are the reconstructed samples before and after applying CCSAO; σ CCSAO [i] is the CCSAO offset value applied to the i-th BO category. In the current design, luminance samples at the same position can be selected from 9 candidate positions, while the positions of the chrominance samples at the same position are fixed, such as... Figure 4 Depicted.

[0035] Similar to SAO, different classifiers can be applied to different local regions to further enhance the overall image quality. The parameters of each classifier (i.e., Y...) col N Y N U N V The position and offset of the classifier are signaled at the frame level, and explicitly signaled at the CTB (Code Tree Block) level which classifier to use and switch between. Some constraints may apply. For example, for each classifier, {N}Y N U N V The maximum value of} is set to {16, 4, 4}, and the offset is constrained to the range [-15, 15]. The maximum number of classifiers per frame is constrained to 4.

[0036] ALF and CC-ALF filters.

[0037] In ECM, the Adaptive Loop Filter (ALF) can classify reconstructed samples (luminance and / or chrominance) into different categories. Then, a category index (Cl...) k Each reconstructed sample is associated with a set of filter parameters P. k Associated with each category index. For example, P k It can be a set of linear filter coefficients: (2), Among them, P k ={c i} i=-n…+n These are filter coefficients, where xi is a sample neighbor of x, x = x0 and F k (x) is the filtered value of x.

[0038] Several classifiers can be used, and the type of classifier can be indicated by a signal in the bitstream. For example, an ALF non-fixed filter can use a Laplace or band classifier. In one variant, for example, other reconstructed sample-based data (such as residual samples) can be used for classification. A band classifier classifies brightness reconstructed samples into equally spaced bands between 0 and (1 << bit depth), where the bit depth is an integer bit depth representation of the reconstructed sample data.

[0039] Adaptive limiting.

[0040] Minimum and maximum sample values ​​can be encoded in the bitstream (e.g., slice header). Capping operations are applied to filtering, sampling, interpolation, weighted prediction, weighted combination, reconstruction, and other stages to ensure that the generated predicted and / or reconstructed samples remain within a defined dynamic range. At the encoder, the minimum / maximum value can be obtained by scanning the sample values ​​in the original image. If a motion-compensated temporal filter (MCTF) is applied to the image, the minimum / maximum value can be obtained by scanning the sample values ​​in the MCTF-pre-filtered image.

[0041] Existing ALF classifiers (e.g., ALF band classifiers) do not consider the occurrence of sample values ​​(the use of fixed bands), so some bands may contain a relatively small number of samples, which jeopardizes the efficiency of classification.

[0042] ALF classification uses the luminance component. In some cases, the type of artifact encoded can be related to some object in the image, and this can be better tracked using color classification.

[0043] It is proposed to modify (one or more) the (in-loop) post-filter classifier or add (one or more) other classifiers that are better correlated with image semantics (e.g., color, sample value statistics). The parameters of this adaptive classifier can be derived from an image and stored for use with the current and / or some subsequent (decoding order) images.

[0044] Adaptive classifiers can use amplitude limiting information to derive frequency band division classification; and use histograms to divide unequal frequency bands or equal frequency bands between minimum and maximum values.

[0045] The adaptive classifier can also derive color band segmentation classification using 1D or 3D histograms, derive (color) bands or unequal color band segmentation classification, and can derive (color) band classifiers using the last I-frame or a reference image from co-localization.

[0046] Example 0 – Exporting classifier parameters using a complete image.

[0047] In existing techniques, ALF uses fixed parameters (410) to classify reconstructed sample data. The parameters are fixed, meaning they are the same for all images in the sequence and are not adapted to the current image characteristics. These fixed parameters include the maximum number of classes (e.g., 25), the brightness band size (band classifier), the process for deriving the orientation from the fixed Laplacian weights, etc.

[0048] exist Figure 5 In the embodiment shown, to better correlate the classification with image semantics (e.g., color, sample value statistics), this embodiment proposes that the (semantic) classifier parameters are no longer fixed (410), but can be adapted to the statistics of some (reconstructed) sample values ​​in a region larger than one CTU, or derived from them. For example, a slice or a complete reconstructed image can be used to derive the classifier parameters (405).

[0049] The adaptive classifier parameters (410) can be used for the current image (to classify its samples) and / or stored for some subsequent (decoding order) images (to classify its samples).

[0050] For some implementations, it may be undesirable to use new parameters derived from the current image (410) or a region larger than one coding tree unit (CTU) (405) to classify samples of the current image, as this requires a post-filtering process to wait for the entire image (or region) to be reconstructed. In this case, the newly derived adaptive classifier parameters can be used only for some subsequent (decoding order) images. In a variant, if the current image is an inter-frame image (e.g., a P or B slice type), the adaptive classifier parameters can be derived using a co-located reference image or the last I image (or any other image stored in the decoded image buffer DPB), since it is available at the start of decoding the current frame.

[0051] For the remainder, once the filter parameters are determined, the filtering process is the same as the regular process. For each sample in CTU (415), the class identifier allows selection of filter coefficients (430) that can be read in the bitstream (460), and the filter (440) is applied.

[0052] In one variant, all or some of the adaptive classifier parameters are not derived from a reconstructed image, but are read from a bitstream. For example, adaptive clipping information from the bitstream can be used to adapt the classifier parameters.

[0053] Example 1 – Classifier parameters derived from adaptive throttling data.

[0054] According to Example 0, the adaptive parameters of the classifier may include the minimum and / or maximum sample values ​​of a component (e.g., the luminance component). For example, the band classifier process can be adapted to these values ​​by reducing the number and / or band size to remain within the [min;max] sample range. For example, a uniform (equally spaced) band can be defined as shown in Equation (3): (3).

[0055] in band_size It is a uniform bandwidth size. nb_class It is the number of categories (e.g., 25), and Cl(x) This is the frequency band index of sample x. The classifier's adaptation parameters include... band_size and (min; max) values.

[0056] The values ​​of (min;max) can be derived from the analysis of a reconstructed image (or slice, or region in an image) (410), or they can be derived from adaptive clipping information encoded in the bitstream.

[0057] Example 2 – Classification of Non-Uniform Frequency Bands.

[0058] According to Example 0 or Example 1, the filter band classifier is non-uniform: the bands can have different sizes. The band classifier can derive the bands from the histogram of the reconstructed image. For example, the classifier can use histogram equalization so that each band can contain approximately the same number of samples.

[0059] Advantageously, the classifier can build a lookup table (LUT), which allows storing the mapping between sample values ​​and classification indices. In this case, the adaptive parameters of the classifier include the LUT.

[0060] Example 3 - Color frequency band division and classification.

[0061] According to Embodiment 0, Embodiment 1, or Embodiment 2, the band classifier can use more than one component to perform band partitioning. For example, the band classifier can use a 3D (uniform or non-uniform) histogram corresponding to three components (e.g., RGB or YCbCr). In this case, the adaptive parameters of the classifier can be a 3D LUT, which associates the classifier's index with each value of the reconstructed color sample triplet (RGB or YUV).

[0062] Similar to Example 1, the bandwidth can be adapted to the (min; max) values ​​of some component samples. Figure 6 -Right side). The values ​​of (min;max) can be derived from the analysis of the reconstructed image (or slice, or region in the image) (410), or they can be derived from the adaptive clipping information encoded in the bitstream.

[0063] Non-uniform 3D histograms can be used. For example, in Figure 7 The image shows such a non-uniform histogram. Non-uniform 3D histograms can be constructed such that each 3D band contains approximately the same number of samples. For example, one can begin by constructing a histogram larger than the target... nb_ class (Significantly) more frequency bands in the 3D histogram. Next, the smallest frequency bands can be iteratively merged / fused until a 3D histogram is obtained. nb_class A frequency band can contain approximately the same number of samples.

[0064] Example 4 – Overview.

[0065] The filter can be ALF, CC-ALF, SAO, or CCSAO.

[0066] Figure 8An embodiment of a method 800 according to the general aspects described herein is illustrated. The method begins at a start box 801, and control continues to a box 810 for determining at least one classifier for video samples in a video compression loop. Control continues from box 810 to a box 820 for filtering the reconstructed video samples based on at least one classifier. Control continues from box 820 to a box 830 for encoding video samples using the filtered reconstructed video samples.

[0067] Figure 9 Another embodiment of the method 900 according to the general aspects described herein is shown. The method begins at a start box 901, and control continues to a box 910 for determining at least one classifier for video samples in a video decompression loop. Control continues from box 910 to a box 920 for filtering the reconstructed video samples based on at least one classifier. Control continues from box 920 to a box 930 for decoding the video samples using the filtered reconstructed video samples.

[0068] Figure 10 An embodiment of an apparatus 1000 for encoding, decoding, compressing, decompressing, or filtering video data using the aforementioned methods is shown. The apparatus includes a processor 1015 and can be interconnected with a memory 1025 via at least one port. Both the processor 1015 and the memory 1025 may also have one or more additional interconnects to external connections.

[0069] The processor 1015 is also configured to insert or receive information in the bitstream and to compress, encode, or decode using any of the described aspects.

[0070] The embodiments described herein encompass a wide variety of aspects, including tools, features, embodiments, models, methods, etc. Many of these aspects are described specifically and are generally described in a seemingly limiting manner, at least to illustrate individual characteristics. However, this is for the purpose of clarity and does not limit the application or scope of those aspects. In fact, all the different aspects can be combined and interchanged to provide further aspects. Furthermore, these aspects can also be combined and interchanged with those described in earlier applications.

[0071] The aspects described and envisioned in this application can be realized in many different forms. Figure 1 , Figure 2 and Figure 11 Some embodiments have been provided, but other embodiments are contemplated, and Figure 1 , Figure 2 and Figure 11The discussion does not limit the breadth of implementation options. At least one aspect typically relates to video encoding and decoding, and at least one other aspect typically relates to transmitting a generated or encoded bitstream. These and other aspects can be implemented as methods, apparatus, computer-readable storage media having instructions stored thereon for encoding or decoding video data according to any of the methods, and / or computer-readable storage media having a bitstream generated according to any of the described methods stored thereon.

[0072] In this application, the terms "reconstructed" and "decoded" are used interchangeably, as are the terms "pixel" and "sample," and the terms "image," "picture," and "frame." Generally, but not necessarily, the term "reconstructed" is used on the encoder side, while "decoded" is used on the decoder side.

[0073] This document describes various methods, and each method includes one or more steps or actions for implementing the described method. Unless the correct operation of the method requires a specific order of steps or actions, the order and / or use of specific steps and / or actions may be modified or combined.

[0074] The various methods and other aspects described in this application can be used to modify, for example... Figure 1 and Figure 2 The modules of the video encoder 100 and decoder 200 shown include, for example, intra-frame prediction, entropy coding, and / or decoding modules (160, 260, 145, 230). Furthermore, this aspect is not limited to VVC or HEVC and can be applied to, for example, other standards and recommendations (whether pre-existing or future-developed), and any extensions to such standards and recommendations (including VVC and HEVC). Unless otherwise stated or technically excluded, the aspects described in this application may be used individually or in combination.

[0075] Various numerical values ​​are used in this application. Specific values ​​are for illustrative purposes only, and the aspects described are not limited to these specific values.

[0076] Figure 1 The encoder 100 is illustrated. Variations of this encoder 100 are envisioned, but for clarity, encoder 100 is described below without describing all anticipated variations.

[0077] Before being encoded, the video sequence may undergo pre-coding (101), for example, applying color transformations to the input color image (e.g., a conversion from RGB 4:4:4 to YCbCr 4:2:0), or performing remapping of the input image components to obtain a signal distribution more resilient to compression (e.g., using histogram equalization with one of the color components). Metadata may be associated with pre-processing and attached to the bitstream.

[0078] In encoder 100, as described below, the image is encoded by encoder elements. The image to be encoded is segmented (102) and processed in units, for example, CUs. Each unit is encoded using, for example, intra-mode or inter-mode. When a unit is encoded in intra-mode, it performs intra-prediction (160). In inter-mode, motion estimation (175) and compensation (170) are performed. The encoder determines (105) which of the intra-mode or inter-mode is used to encode the unit and indicates the intra / inter-mode decision by, for example, a prediction mode flag. For example, the prediction residual is calculated by subtracting (110) the prediction block from the original image block.

[0079] The predicted residual is then transformed (125) and quantized (130). The quantized transform coefficients, along with the motion vector and other syntax elements, are entropy encoded (145) to output a bitstream. The encoder can skip the transform and apply quantization directly to the untransformed residual signal. The encoder can bypass both the transform and quantization, i.e., directly encode the residual without applying either the transform or quantization process.

[0080] The encoder decodes the encoded blocks to provide a reference for further prediction. The quantized transform coefficients are dequantized (140) and inversely transformed (150) to decode the prediction residuals. The image blocks are reconstructed by combining (155) the decoded prediction residuals and the prediction blocks. An in-loop filter (165) is applied to the reconstructed image to perform, for example, deblocking / SAO (sample adaptive offset) filtering to reduce coding artifacts. The filtered image is stored at the reference image buffer (180).

[0081] Figure 2 The diagram illustrates a block diagram of a video decoder 300. In decoder 200, as described below, the bitstream is decoded by decoder elements. Video decoder 200 typically performs functions similar to... Figure 1 The decoder traversal is the inverse of the encoding traversal described in [the document]. Encoder 100 typically also performs video decoding as part of the encoding of video data.

[0082] Specifically, the input to the decoder includes a video bitstream, which can be generated by the video encoder 100. The bitstream is first entropy decoded (230) to obtain transform coefficients, motion vectors, and other encoded information. Image segmentation information indicates how the image is segmented. Therefore, the decoder can segment (335) the image based on the decoded image segmentation information. The transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residuals. By combining (255) the decoded prediction residuals and prediction blocks, image blocks are reconstructed. Prediction blocks can be obtained (270) from intra-frame prediction (260) or motion-compensated prediction (i.e., inter-frame prediction) (275). An in-loop filter (265) is applied to the reconstructed image. The filtered image is stored at a reference image buffer (280).

[0083] The decoded image can undergo further post-decoding processing (285), such as inverse color transformation (e.g., conversion from YcbCr4:2:0 to RGB 4:4:4) or inverse remapping of the remapping process performed in pre-encoding processing (101). Post-decoding processing can utilize metadata derived in pre-encoding processing and signaled in the bitstream.

[0084] Figure 11 This is a block diagram illustrating examples of systems implementing various aspects and embodiments therein. System 1100 may be embodied as a device including the various components described below and configured to perform one or more of the aspects described in this document. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. Elements of system 1100 may be embodied individually or in combination in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 1100 are distributed across multiple ICs and / or discrete components. In various embodiments, system 1100 is communicatively coupled to one or more other systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports. In various embodiments, system 1100 is configured to implement one or more of the aspects described in this document.

[0085] System 1100 includes at least one processor 1010 configured to execute instructions loaded thereon for implementing, for example, the various aspects described herein. Processor 1010 may include embedded memory, input / output interfaces, and various other circuitry as known in the art. System 1100 includes at least one memory 1020 (e.g., a volatile memory device and / or a non-volatile memory device). System 1100 may include a storage device 1040, which may include non-volatile memory and / or volatile memory, including but not limited to electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, disk drives, and / or optical disk drives. As a non-limiting example, storage device 1040 may include internal storage devices, attached storage devices (including removable and non-removable storage devices), and / or network-accessible storage devices.

[0086] System 1100 includes an encoder / decoder module 1030 configured to, for example, process data to provide encoded or decoded video, and the encoder / decoder module 1030 may include its own processor and memory. The encoder / decoder module 1030 represents one or more modules that can be included in a device to perform encoding and / or decoding functions. As is known, a device may include one or both of an encoding module and a decoding module. Furthermore, as is known to those skilled in the art, the encoder / decoder module 1030 may be implemented as a separate element of system 1100, or it may be incorporated within processor 1010 as a combination of hardware and software.

[0087] Program code to be loaded onto processor 1010 or encoder / decoder 1030 to execute the various aspects described in this document may be stored in storage device 1040 and subsequently loaded onto memory 1020 for execution by processor 1010. According to various embodiments, one or more of processor 1010, memory 1020, storage device 1040, and encoder / decoder module 1030 may store one or more of various items during the execution of the processes described in this document. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0088] In some embodiments, the memory within the processor 1010 and / or encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, external memory (e.g., the processing device may be the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. The external memory may be memory 1020 and / or storage device 1040, such as dynamically volatile memory and / or non-volatile flash memory. In several embodiments, external non-volatile flash memory is used to store, for example, the operating system of a television. In at least one embodiment, a fast external dynamic volatile memory, such as RAM, is used as working memory for video encoding and decoding operations, such as for MPEG-2 (MPEG stands for Moving Picture Experts Group; MPEG-2 is also known as ISO / IEC 13818, and 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC stands for High Efficiency Video Coding; also known as H.265 and MPEG H Part 2), or VVC (Multi-Functional Video Coding, a new standard being developed by the Joint Video Experts Group JVET).

[0089] Inputs to the components of system 1100 can be provided by various input devices, as indicated in block 1130. Such input devices include, but are not limited to: (i) an RF section that receives radio frequency (RF) signals, for example, transmitted over the air by a broadcasting device; (ii) a component (COMP) input terminal (or a set of COMP input terminals); (iii) a universal serial bus (USB) input terminal; and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Figure 11 Other examples not shown include composite videos.

[0090] In various embodiments, as known in the art, the input device of block 1130 has associated respective input processing elements. For example, the RF section may be associated with elements suitable for: (i) selecting a desired frequency (also known as selecting a signal, or limiting a signal to a frequency band), (ii) down-converting the selected signal, (iii) further limiting the signal to a narrower frequency band to select, for example, a signal band that may be referred to as a channel in some embodiments, (iv) demodulating the down-converted and limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired data packet stream. The RF section in various embodiments includes one or more elements to perform these functions, such as frequency selectors, signal selectors, frequency limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF section may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., intermediate frequency or near-baseband frequency) or baseband. In one set-top box embodiment, the RF section and its associated input processing elements receive RF signals transmitted via a wired (e.g., cable) medium and perform frequency selection by filtering, down-converting, and re-filtering to a desired frequency band. Various embodiments rearrange the order of the above-described (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions. Adding elements may include inserting elements between existing elements, such as, for example, inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0091] Furthermore, the USB and / or HDMI terminals may include their respective interface processors for connecting system 1100 to other electronic devices across USB and / or HDMI connections. It should be understood that various aspects of input processing (e.g., Reed-Solomon error correction) can be implemented as needed, for example, within a separate input processing IC or within processor 1010. Similarly, aspects of USB or HDMI interface processing can be implemented as needed, either within a separate interface IC or within processor 1010. Demodulated, error-corrected, and demultiplexed streams are provided to various processing elements, including, for example, processor 1010 and encoder / decoder 1030, to operate in conjunction with memory and storage elements to process the data streams as needed for presentation on the output device.

[0092] Various components of system 1100 can be housed within an integrated housing. Within the integrated housing, various components can be interconnected and transmit data between them using suitable connection arrangements (e.g., internal buses as known in the art, including inter-IC (I2C) buses, wiring, and printed circuit boards).

[0093] System 1100 includes a communication interface 1050 that enables communication with other devices via a communication channel 1060. The communication interface 1050 may include, but is not limited to, a transceiver configured to transmit and receive data via the communication channel 1060. The communication interface 1050 may include, but is not limited to, a modem or network interface card (NIC), and the communication channel 1060 may be implemented, for example, in a wired and / or wireless medium.

[0094] In various embodiments, data is streamed to or otherwise provided to system 1100 using a wireless network such as a Wi-Fi network, for example, IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). In these embodiments, the Wi-Fi signal is received via a communication channel 1060 and a communication interface 1050 suitable for Wi-Fi communication. The communication channel 1060 in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other over-the-top communications. Other embodiments use a set-top box that delivers data via an HDMI connection of input block 1130 to provide streaming data to system 1100. Still other embodiments use an RF connection of input block 1130 to provide streaming data to system 1100. As indicated above, various embodiments provide data in a non-streaming manner. Furthermore, various embodiments use wireless networks instead of Wi-Fi, such as cellular networks or Bluetooth networks.

[0095] System 1100 can provide output signals to various output devices, including display 1105, speaker 1110, and other peripheral devices 1120. Display 1105 in various embodiments includes one or more of, for example, a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. Display 1105 can be used in a television, tablet computer, laptop, cellular phone (mobile phone), or another device. Display 1105 can also be integrated with other components (e.g., as in a smartphone) or separate (e.g., an external monitor for a laptop). In various examples of embodiments, other peripheral devices 1120 include one or more of a standalone digital video disc (or digital multifunction disc) (DVR for both terms), a disc player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 1120 that provide functionality based on the output of system 1100. For example, a disc player performs the function of playing the output of system 1100.

[0096] In various embodiments, signaling such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols that enable inter-device control with or without user intervention is used to communicate control signals between system 1100 and display 1105, speaker 1110, or other peripheral devices 1120. Output devices may be communicatively coupled to system 1100 via dedicated connections through their respective interfaces 1070, 1080, and 1090. Alternatively, output devices may be connected to system 1100 via communication interface 1050 using communication channel 1060. Display 1105 and speaker 1110 may be integrated into a single unit with other components of system 1100 in electronic devices such as, for example, a television set. In various embodiments, display interface 1070 includes a display driver, such as, for example, a timing controller (TCon) chip.

[0097] For example, if the RF portion of input 1130 is part of a standalone set-top box, then display 1105 and speaker 1110 may alternatively be separate from one or more other components. In various embodiments where display 1105 and speaker 1110 are external components, the output signal may be provided via a dedicated output connection, including, for example, an HDMI port, a USB port, or a COMP output.

[0098] The embodiments can be implemented by computer software implemented by processor 1010, or by hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments can be implemented by one or more integrated circuits. As a non-limiting example, memory 1020 can be of any type suitable for the technical environment and can be implemented using any suitable data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. As a non-limiting example, processor 1010 can be of any type suitable for the technical environment and can encompass one or more of microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures.

[0099] Various implementations involve decoding. As used herein, "decoding" can encompass all or part of a process performed, for example, on a received encoded sequence to produce a final output suitable for display. In various embodiments, such a process includes one or more processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such a process also includes, or alternatively includes, processes performed by the decoders of the various implementations described herein.

[0100] As a further example, in one embodiment, "decoding" refers only to entropy decoding; in another embodiment, "decoding" refers only to differential decoding; and in yet another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. It will be clear, and is considered well understood by those skilled in the art, whether the phrase "decoding process" is intended to specifically refer to a subset of operations or generally to a broader decoding process, based on the context of the specific description.

[0101] Various implementations involve encoding. In a manner similar to the discussion above regarding “decoding,” the term “encoding,” as used herein, can encompass all or part of a process performed, for example, on an input video sequence to produce an encoded bitstream. In various embodiments, such a process includes one or more processes typically performed by an encoder, such as segmentation, differential coding, transform, quantization, and entropy coding. In various embodiments, such a process also includes, or alternatively includes, processes performed by the encoders of the various implementations described herein.

[0102] As a further example, in one embodiment, "encoding" refers only to entropy encoding; in another embodiment, "encoding" refers only to differential encoding; and in yet another embodiment, "encoding" refers to a combination of differential and entropy encoding. It will be clear, and is considered well understood by those skilled in the art, whether the phrase "encoding process" is intended to specifically refer to a subset of operations or generally to a broader encoding process, based on the context of the specific description.

[0103] Note that the grammatical elements used in this article are descriptive terms. Therefore, the use of other grammatical element names is not excluded.

[0104] When a diagram is presented as a flowchart, it should be understood that it also provides a block diagram of the corresponding apparatus. Similarly, when a diagram is presented as a block diagram, it should be understood that it also provides a flowchart of the corresponding method / process.

[0105] Various implementations can refer to parametric models or rate-distortion optimization. Specifically, during the encoding process, a balance or trade-off between bit rate and distortion is typically considered, often within constraints of computational complexity. This can be measured by rate-distortion optimization (RDO), or by least mean square (LMS), mean absolute error (MAE), or other such measures. Rate-distortion optimization is typically formulated as minimizing a rate-distortion function, which is a weighted sum of bit rate and distortion. Different approaches exist to address the rate-distortion optimization problem. For example, these approaches can be based on extensive testing of all encoding options, including all considered modes or encoding parameter values, and a complete evaluation of their encoding costs and the associated distortion of the reconstructed signal after encoding and decoding. Faster methods can also be used to save encoding complexity, particularly by calculating approximate distortion based on predicting or predicting the residual signal rather than the reconstructed signal. A hybrid of these two approaches can also be used, such as by using approximate distortion only for some possible encoding options and full distortion for others. Other methods evaluate only a subset of possible encoding options. More generally, many methods employ any of a variety of techniques to perform optimization, but optimization is not necessarily a complete assessment of coding costs and associated distortions.

[0106] The implementations and aspects described herein can be implemented, for example, in methods or processes, apparatuses, software programs, data flows, or signals. Even if discussed only in the context of a single implementation form (e.g., discussed only as a method), the features discussed can be implemented in other forms (e.g., apparatuses or programs). Apparatuses can be implemented, for example, in suitable hardware, software, and firmware. Methods can be implemented, for example, in a processor, which generally refers to a processing device, including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices, such as, for example, computers, cellular phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate communication of information between end-users.

[0107] The references to "an embodiment" or "an embodiment" or "an implementation" or "an implementation," and other variations thereof, mean that a particular feature, structure, characteristic, etc., described in connection with that embodiment is included in at least one embodiment. Therefore, the appearance of the phrases "in an embodiment" or "in an embodiment" or "in an implementation," and any other variations appearing throughout this application, do not necessarily refer to the same embodiment.

[0108] Furthermore, this application may refer to "determining" each piece of information. Determining information may include, for example, one or more of the following: estimated information, calculated information, predicted information, or information retrieved from memory.

[0109] Furthermore, this application may refer to "accessing" individual pieces of information. Accessing information may include, for example, receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information, or one or more of these.

[0110] Furthermore, this application may refer to "receiving" individual pieces of information. Like "access," "receiving" is intended to be a broad term. Receiving information may include, for example, accessing information or retrieving information (e.g., from memory) one or more. Moreover, "receiving" is generally referred to in one manner or another during operations such as, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0111] To be understood, for example, in the cases of “A / B,” “A and / or B,” and “at least one of A and B,” the use of any of the following “ / ,” “and / or,” and “at least one” is intended to cover selecting only the first listed option (A), or only the second listed option (B), or both options (A and B). As another example, in the cases of “A, B, and / or C” and “at least one of A, B, and C,” such wording is intended to cover selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A, B, and C). As will be apparent to those skilled in the art and related fields, this can be extended to as many listed items as possible.

[0112] Furthermore, as used herein, the term "notify by signaling" specifically refers to instructing the corresponding decoder to do something. For example, in some embodiments, the encoder signals a particular one of multiple transforms, encoding modes, or flags. In this way, in one embodiment, the same transform, parameter, or mode is used at both the encoder and decoder sides. Thus, for example, the encoder can transmit (explicit signaling) a specific parameter to the decoder so that the decoder can use the same specific parameter. Conversely, if the decoder already has the specific parameter as well as other parameters, signaling can be used without transmitting (implicit signaling) to simply allow the decoder to know and select the specific parameter. Bit savings are achieved in various embodiments by avoiding the transmission of any actual functionality. It should be understood that notification by signaling can be done in various ways. For example, in various embodiments, one or more syntax elements, flags, etc., are used to signal information to the corresponding decoder. Although the verb form of the term "notify by signaling" has been referred to above, the term "signal" can also be used as a noun herein.

[0113] It will be apparent to those skilled in the art that implementations can generate various signals that are formatted to carry, for example, information that can be stored or transmitted. This information may include, for example, instructions for performing a method, or data generated by one of the described implementations. For example, the signal may be formatted to carry a bitstream of the described embodiment. Such a signal may be formatted as, for example, electromagnetic waves (e.g., using the radio frequency portion of the spectrum) or baseband signals. Formatting may include, for example, encoding the data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. It is well known that signals can be transmitted via various wired or wireless links. The signal may be stored on a processor-readable medium.

[0114] The preceding sections describe numerous embodiments across various claim classes and types. Features of these embodiments may be provided individually or in any combination. Furthermore, embodiments may include one or more of the following features, devices, or aspects, individually or in any combination, across various claim classes and types: At least one embodiment includes classifying signals in a compression / decompression system based on signal characteristics.

[0115] At least one embodiment includes the foregoing embodiments, and further includes filtering the signal within the loop according to the classification.

[0116] At least one embodiment includes any encoding or decoding operation based on the above operations.

[0117] At least one embodiment includes a bitstream or signal that includes one or more of the described syntax elements or variations thereof.

[0118] At least one embodiment includes a bitstream or signal that includes a syntax for conveying information generated according to any of the described embodiments.

[0119] At least one embodiment includes creating and / or transmitting and / or receiving and / or decoding according to any of the described embodiments.

[0120] At least one embodiment includes a method, process, apparatus, medium for storing instructions, medium for storing data, or signal according to any of the described embodiments.

[0121] At least one embodiment includes inserting in the signaling that enables the decoder to determine the syntax elements of the decoded information in a manner corresponding to that used by the encoder.

[0122] At least one embodiment includes creating and / or transmitting and / or receiving and / or decoding a bitstream or signal comprising one or more of the described syntax elements or variations thereof.

[0123] At least one embodiment includes a TV, set-top box, mobile phone, tablet computer or other electronic device that performs one or more transformation methods according to any one of the described embodiments.

[0124] At least one embodiment includes a TV, set-top box, mobile phone, tablet computer or other electronic device that performs one or more transformation methods according to any of the described embodiments to determine and display (e.g., using a monitor, screen or other type of display) the resulting image.

[0125] At least one embodiment includes a TV, set-top box, mobile phone, tablet computer or other electronic device that selects, band-limits or tunes (e.g., using a tuner) a channel to receive a signal including an encoded image and performs one or more transformation methods according to any one of the described embodiments.

[0126] At least one embodiment includes a TV, set-top box, mobile phone, tablet computer or other electronic device that receives signals including encoded images over the air (e.g., using an antenna) and performs one or more transformation methods.

Claims

1. A method comprising: Determine at least one classifier for video samples in a video compression loop; The reconstructed video samples are filtered based on the at least one classifier; as well as, The video samples are encoded using filtered, reconstructed video samples.

2. An apparatus comprising: Memory; as well as The processor is configured as follows: Determine at least one classifier for video samples in a video compression loop; The reconstructed video samples are filtered based on the at least one classifier; and... The video samples are encoded using filtered, reconstructed video samples.

3. A method comprising: Determine at least one classifier for video samples in the video decompression loop; The reconstructed video samples are filtered based on the at least one classifier; as well as The video samples are decoded using filtered reconstructed video samples.

4. An apparatus comprising: Memory; as well as The processor is configured as follows: Determine at least one classifier for video samples in the video decompression loop; The reconstructed video samples are filtered based on the at least one classifier; as well as The video samples are decoded using filtered reconstructed video samples.

5. The method according to any one of claims 1 or 3, or the apparatus according to any one of claims 2 or 4, wherein, The determination of at least one classifier is based on information from video sample regions larger than the coding tree unit.

6. The method of any one of claims 1 or 3, or the apparatus of any one of claims 2 or 4, wherein, The determination of at least one classifier is based on information from frames of video samples.

7. The method of any one of claims 1 or 3, or the apparatus of any one of claims 2 or 4, wherein, The determination of at least one classifier is based on information in the bitstream.

8. The method of any one of claims 1, 3, 5, or 6, or the apparatus of any one of claims 2, 4, 5, or 6, wherein, The determination of at least one classifier is based on the analysis of reconstructed video samples.

9. The method of any one of claims 1, 3, 5, 6, or 8, or the apparatus of any one of claims 2, 4, 5, 6, or 8, wherein, The adaptive parameters of the at least one classifier include at least one of the following: bandwidth size, minimum and maximum values, lookup table information, and amplitude limiting information.

10. The method of any one of claims 1, 3, 5, 6, 8, or 9, or the apparatus of any one of claims 2, 4, 5, 6, 8, or 9, wherein, Separate classifiers were used for different video components.

11. The method of any one of claims 1, 3, 5, 6, 8, 9, or 10, or the apparatus of any one of claims 2, 4, 5, 6, 8, 9, or 10, wherein, Filtering is performed by filters including adaptive loop filters, cross-component adaptive loop filters, sample adaptive offset filters, or cross-component sample adaptive offset filters.

12. The method of any one of claims 1, 3, 5, 6, or 8-11, or the apparatus of any one of claims 2, 4, 5, 6, or 8-11, wherein, Sample-based data, including reconstructions of residual samples, were used for classification.

13. An apparatus comprising: The apparatus according to claim 2; as well as At least one of the following: (i) an antenna configured to receive a signal including a video block, (ii) a band limiter configured to limit the received signal to a band including the video block, and (iii) a display configured to display an output representing the video block.

14. A non-transitory computer-readable medium for playback using a processor, comprising data content generated by the method of claim 1.

15. A non-transitory computer-readable storage medium for playback using a processor, comprising program code that, when executed by a computer, causes the computer to perform the method of claim 3.

Images