Joint adaptive loop and output filter

Abstract

A method and apparatus for encoding or decoding a video are provided, in which a filtering stage is applied to reconstructed samples of a picture of the video. The filtering is based on a joint set of adaptive filters comprising at least one in-loop filter and at least one output filter, wherein the at least one in-loop filter is used to provide a first version of the picture to be used as a reference picture to predict a subsequent picture, and the at least one output filter is used to provide a second version of the picture to be used as an output picture by a decoder.

Description

Cross-reference to related applications

[0001] This application claims priority to European Patent Application No. 23306917.8, filed on 6 November 2023, the entire disclosure of which is incorporated herein by reference. Technical Field

[0002] This embodiment generally relates to video compression. Specifically, it relates to a method and apparatus for encoding or decoding images or videos. More specifically, this embodiment relates to reconstructed samples in an improved video compression system. Background Technology

[0003] To achieve high compression efficiency, image and video coding schemes typically utilize prediction and transform to take advantage of spatial and temporal redundancy in video content. Intra-frame or inter-frame prediction is usually used to fully utilize intra-frame or inter-frame correlations, and then the difference between the original block and the predicted block (often called prediction error or prediction residual) is transformed, quantized, and entropy-coded. In inter-frame prediction, motion vectors used for motion compensation are typically predicted from a motion vector predictor. To reconstruct the video, the compressed data is decoded through the inverse processes corresponding to entropy coding, quantization, transform, and prediction. Summary of the Invention

[0004] According to one aspect, a method for encoding video is provided. The method includes obtaining reconstructed samples of images of the video, and determining, based on the reconstructed samples, a joint adaptive filter set comprising at least one loop filter and at least one output filter, wherein the at least one loop filter is used to provide a first version of the image that will be used as a reference image, and the at least one output filter is used to provide a second version of the image that will be used as an output image by a decoder.

[0005] According to another aspect, an apparatus for encoding video is provided. The apparatus includes one or more processors operable to acquire reconstructed samples of images of the video and, based on the reconstructed samples, determine a joint adaptive filter set including at least one loop filter and at least one output filter, wherein the at least one loop filter is used to provide a first version of the image that will be used as a reference image, and the at least one output filter is used to provide a second version of the image that will be used as an output image by a decoder.

[0006] According to one aspect, a method for decoding video is provided. The method includes decoding encoded data representing an image to obtain reconstructed samples of the image, using one or more loop filters to obtain a reference image from the reconstructed samples, and using one or more output filters to obtain an output image from at least the reconstructed samples, wherein the one or more loop filters and the one or more output filters are obtained from a joint adaptive filter set.

[0007] According to another aspect, an apparatus for decoding video is provided. The apparatus includes one or more processors operable to decode encoded data representing an image to obtain reconstructed samples of the image, use one or more loop filters to obtain a reference image from the reconstructed samples, and use one or more output filters to obtain an output image from at least the reconstructed samples, wherein the one or more loop filters and the one or more output filters are obtained from a joint adaptive filter set.

[0008] This document describes other embodiments that can be used alone or in combination.

[0009] One or more embodiments also provide a computer program including instructions that, when executed by one or more processors, cause the one or more processors to perform any of the methods for encoding or decoding video according to any embodiment described herein. One or more embodiments also provide a non-transitory computer-readable medium and / or a computer-readable storage medium having instructions stored thereon for encoding or decoding video according to the methods described herein.

[0010] One or more embodiments also provide a computer-readable storage medium storing a bit stream generated according to the method described herein. One or more embodiments also provide a method and apparatus for transmitting or receiving a bit stream generated according to the described method. Attached Figure Description

[0011] Figure 1 A block diagram of a system that can implement aspects of this embodiment is shown.

[0012] Figure 2A A block diagram of an embodiment of a video encoder that can implement aspects of this embodiment is shown.

[0013] Figure 2B A block diagram of another embodiment of a video encoder that can implement aspects of this embodiment is shown.

[0014] Figure 3A A block diagram of an embodiment of a video decoder that can implement aspects of this embodiment is shown.

[0015] Figure 3BA block diagram of another embodiment of a video decoder that can implement aspects of this embodiment is shown.

[0016] Figure 4 An example of a loop filter in a VVC is shown. Figure 5 Examples of a luminance ALF 7×7 diamond filter (left) and a chrominance ALF 5×5 diamond filter (right) are shown. Figure 6 An example of the filter shape for signal-informed luminance ALF in ECM-9 is shown, taken from JVET-AD2025. Figure 7 An example of a joint loop and output filter according to an embodiment is shown. Figure 8 Examples of combined loop and output filters according to other embodiments are shown: combined loop and display filter (top) and combined loop and output filter for a machine (bottom). Figure 9 An example flowchart for encoding video according to an embodiment is shown.

[0017] Figure 10 An example flowchart for decoding video is shown according to an embodiment.

[0018] Figure 11 An example of a two-step joint loop and output filter optimization process according to an embodiment is shown.

[0019] Figure 12 An example of a two-step joint loop and output filter optimization process according to another embodiment is shown.

[0020] Figure 13 An example of constrained optimization of a second adaptive filter set according to an embodiment is shown.

[0021] Figure 14 An example flowchart for decoding video according to another embodiment is shown.

[0022] Figure 15 An example of constrained optimization of an adaptive output filter set according to an embodiment is shown.

[0023] Figure 16 An example of constrained optimization of an adaptive loop filter set according to an embodiment is shown.

[0024] Figure 17 An example of joint adaptive loop and output filtering based on the output of the loop filter is shown.

[0025] Figure 18An example of joint adaptive loop and output filter optimization based on the loop filter output is shown on the encoder side.

[0026] Figure 19 An example of a 37-tap adaptive output filter according to an embodiment is shown.

[0027] Figure 20 An example of a 40-tap adaptive output filter according to an embodiment is shown.

[0028] Figure 21 An example of joint adaptive loop and output filter optimization based on MCTF-based sample and original sample is shown according to an embodiment.

[0029] Figure 22 An example of joint adaptive loop and output filter optimization based on BIF-processed samples and original samples according to an embodiment is shown.

[0030] Figure 23 An example of a neural network-based loop filter is shown.

[0031] Figure 24 An example of a multi-output neural network according to an embodiment is shown.

[0032] Figure 25 An example of a multi-output neural network according to another embodiment is shown.

[0033] Figure 26 An example of the temporal hierarchical decomposition of frames in a group of images (GOP) is shown.

[0034] Figure 27 A block diagram of a system that can implement aspects of this embodiment according to another embodiment is shown.

[0035] Figure 28 An example of two remote devices communicating on a communication network is shown, based on this principle.

[0036] Figure 29 The syntax of an example signal based on this principle is shown. Detailed Implementation

[0037] This application describes various aspects, including tools, features, embodiments, models, solutions, etc. Many of these aspects are described in detail and, at least to show individual features, are generally described in a manner that may sound restrictive. However, this is for clarity of description and does not limit the application or scope of these aspects. In fact, all the different aspects can be combined and interchanged to provide further aspects. Furthermore, these aspects can be combined and interchanged with aspects described in previous applications.

[0038] The aspects described and envisioned in this application can be implemented in many different forms. The following... Figure 1 Figures 2 and 3 provide some embodiments, but other embodiments are also contemplated, and... Figure 1 The discussion in Figures 2 and 3 does not limit the breadth of implementations. At least one of these aspects generally relates to video encoding and decoding, and at least another aspect generally relates to the transmission of generated or encoded bitstreams. These and other aspects can be implemented as a method, an apparatus, a computer-readable storage medium storing instructions for encoding or decoding video data according to any of the described methods, and / or a computer-readable storage medium storing bitstreams generated according to any of the described methods.

[0039] In this application, the terms “reconstruction” and “decoding” are used interchangeably, the terms “pixel” and “sample” are used interchangeably, and the terms “image”, “picture” and “frame” are used interchangeably.

[0040] This document describes various methods, and each method includes one or more steps or actions for implementing the described method. Unless the correct operation of the method requires a specific order of steps or actions, the order and / or use of specific steps and / or actions can be modified or combined. Furthermore, terms such as "first," "second," etc., can be used in various embodiments to modify elements, components, steps, operations, etc., for example, "first decoding" and "second decoding." The use of such terms does not imply an ordering of the modified operations unless specifically required. Therefore, in this example, the first decoding does not need to be performed before the second decoding and can occur, for example, before, during, or within a time period overlapping with the second decoding.

[0041] This aspect is not limited to VVC or HEVC, and can be applied to, for example, other standards and recommendations (whether pre-existing or developed in the future) and any extensions of such standards and recommendations (including VVC and HEVC). Unless otherwise stated or technically excluded, the aspects described in this application may be used alone or in combination.

[0042] Figure 1A block diagram illustrating examples of systems that can implement various aspects and embodiments is shown. System 100 can be implemented as a device including a variety of components and configured to perform one or more aspects described in this application. Examples of such devices include, but are not limited to, a variety of electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, networked home appliances, and servers. Elements of system 100 (individually or in combination) can be embodied in a single integrated circuit, multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 100 are distributed across multiple ICs and / or discrete components. In various embodiments, system 100 is communicatively coupled to other systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports. In various embodiments, system 100 is configured to implement one or more aspects described in this application.

[0043] System 100 includes at least one processor 110 configured to execute instructions loaded thereon for implementing various aspects, such as those described in this application. Processor 110 may include embedded memory, input / output interfaces, and various other circuitry known in the art. System 100 includes at least one memory 120 (e.g., a volatile memory device and / or a non-volatile memory device). System 100 includes a storage device 140, which may include non-volatile memory and / or volatile memory, including but not limited to EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash memory, disk drives, and / or optical disk drives. Storage device 140 may include internal storage devices, attached storage devices, and / or network-accessible storage devices, as non-limiting examples.

[0044] System 100 includes an encoder / decoder module 130 configured to, for example, process data to provide encoded or decoded video, and the encoder / decoder module 130 may include its own processor and memory. The encoder / decoder module 130 represents a module that may be included in a device to perform encoding and / or decoding functions. As is known, a device may include one or both encoding and decoding modules. Furthermore, the encoder / decoder module 130 may be implemented as a separate element of system 100, or it may be incorporated within processor 110 as a combination of hardware and software (as known to those skilled in the art).

[0045] Program code to be loaded onto processor 110 or encoder / decoder 130 to execute the various aspects described in this application may be stored in storage device 140 and subsequently loaded onto memory 120 for execution by processor 110. According to various embodiments, one or more of processor 110, memory 120, storage device 140, and encoder / decoder module 130 may store one or more items of a variety during execution of the processes described in this application. Such stored items may include, but are not limited to, input video, decoded video or portions thereof, bitstreams, matrices, variables, and intermediate or final results from processing equations, formulas, operations, and operational logic.

[0046] In some embodiments, memory within processor 110 and / or encoder / decoder module 130 is used to store instructions and provide working memory for processing during encoding or decoding. However, in other embodiments, external memory (e.g., the processing device may be processor 110 or encoder / decoder module 130) is used for one or more of these functions. External memory may be memory 120 and / or storage device 140, such as dynamic volatile memory and / or non-volatile flash memory. In several embodiments, external non-volatile flash memory is used to store the television's operating system. In at least one embodiment, fast external dynamic volatile memory (such as RAM) is used as working memory for video encoding and decoding operations (e.g., for MPEG-2, HEVC (HEVC stands for High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Multi-Functional Video Coding, also known as H.266, a standard developed by the Joint Video Experts Group JVET)).

[0047] Inputs to the components of system 100 may be provided by a variety of input devices as indicated in box 105. Such input devices include, but are not limited to, (i) a radio frequency (RF) section that receives, for example, RF signals transmitted over the air by a broadcaster, (ii) component (COMP) input terminals (or a set of COMP input terminals), (iii) universal serial bus (USB) input terminals, and / or (iv) high-definition multimedia interface (HDMI) input terminals. Figure 1 Other examples not shown include composite video.

[0048] In various embodiments, the input device of block 105 has corresponding input processing elements known in the art. For example, the RF section may be associated with elements suitable for: (i) selecting a desired frequency (also known as selecting a signal, or band-limiting a signal to a frequency band); (ii) down-converting the selected signal; (iii) band-limiting it again to a narrower frequency band to select, for example, a signal band that may be referred to as a channel in some embodiments; (iv) demodulating the down-converted and band-limited signal; (v) performing error correction; and (vi) demultiplexing to select the desired data packet stream. The RF section in various embodiments includes one or more elements for performing these functions, such as frequency selectors, signal selectors, band limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF section may include a tuner that performs a variety of these functions, such as down-converting a received signal to a lower frequency (e.g., intermediate frequency or near-baseband frequency) or to baseband. In one set-top box embodiment, the RF section and its associated input processing elements receive RF signals transmitted via a wired (e.g., cable) medium and perform frequency selection by filtering, down-converting, and re-filtering to the desired frequency band. Various embodiments rearrange the order of the above (and other) components, remove some of these components, and / or add other components that perform similar or different functions. Adding components may include inserting components between existing components, such as inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0049] Furthermore, the USB and / or HDMI terminals may include corresponding interface processors for connecting system 100 to other electronic devices across USB and / or HDMI connections. It should be understood that multiple aspects of input processing (e.g., Reed-Solomon error correction) may be implemented as needed, for example, within a separate input processing IC or within processor 110. Similarly, aspects of USB or HDMI interface processing may be implemented as needed, either within a separate interface IC or within processor 110. The demodulated, error-corrected, and demultiplexed streams are provided to various processing elements, including processor 110 and encoder / decoder 130, which operate in combination with memory and storage elements to process the data streams for presentation on an output device.

[0050] Various components of system 100 can be provided within an integrated housing. Within the integrated housing, various components can be interconnected and transmit data therebetween using suitable connection means 115 (e.g., internal buses known in the art, including I2C buses, wiring, and printed circuit boards).

[0051] System 100 includes a communication interface 150 that enables communication with other devices via a communication channel 190. The communication interface 150 may include, but is not limited to, a transceiver configured to send and receive data via the communication channel 190. The communication interface 150 may include, but is not limited to, a modem or network interface card (NIC), and the communication channel 190 may be implemented, for example, within a cabling and / or wireless medium.

[0052] In various embodiments, data is streamed to system 100 using a Wi-Fi network such as IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). The Wi-Fi signals in these embodiments are received via communication channel 190 and communication interface 150, which are adapted for Wi-Fi communication. Communication channel 190 in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other over-the-top services to communicate. Other embodiments use a set-top box to provide streaming data to system 100, transmitting data via an HDMI connection to input box 105. Still other embodiments use an RF connection to input box 105 to provide streaming data to system 100. As described above, various embodiments provide data in a non-streaming manner. Furthermore, various embodiments use wireless networks other than Wi-Fi, such as cellular networks or Bluetooth networks.

[0053] System 100 can provide output signals to a variety of output devices, including a display 165, a speaker 175, and other peripheral devices 185. The display 165 in various embodiments includes, for example, one or more of a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. The display 165 can be used in a television, tablet computer, laptop computer, mobile phone, or other device. The display 165 can also be integrated with other components (e.g., in a smartphone) or separate (e.g., an external monitor for a laptop computer). In examples of other embodiments, the other peripheral devices 185 include one or more of a stand-alone digital video disc (or digital multifunction disc) (DVR, both terms), a disk player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 185 that provide functionality based on the output of system 100. For example, a disk player performs the function of playing the output of system 100.

[0054] In various embodiments, signaling such as AV.Link, CEC, or other communication protocols is used to transmit control signals between system 100 and display 165, speaker 175, or other peripheral devices 185. These protocols enable device-to-device control with or without user intervention. Output devices can be communicatively coupled to system 100 via dedicated connections through corresponding interfaces 160, 170, and 180. Alternatively, output devices can be connected to system 100 via communication interface 150 using communication channel 190. Display 165 and speaker 175 can be integrated into a single unit within an electronic device (e.g., a television) along with other components of system 100. In various embodiments, display interface 160 includes a display driver, such as a timing controller (TCon) chip.

[0055] Display 165 and speaker 175 may alternatively be separated from one or more other components, for example, if the RF portion of input 105 is part of a separate set-top box. In various embodiments, where display 165 and speaker 175 are external components, output signals may be provided via dedicated output connections, such as HDMI ports, USB ports, or COMP outputs.

[0056] The embodiments may be executed by computer software, hardware, or a combination of hardware and software, which are executed by processor 110. As a non-limiting example, the embodiments may be implemented by one or more integrated circuits. Memory 120 may be of any type suitable for the technical environment and may be implemented using any suitable data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory, as a non-limiting example. Processor 110 may be of any type suitable for the technical environment and may include one or more of microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures, as a non-limiting example.

[0057] Figure 2A An example of a block-based hybrid video encoder 200 is shown. Variations of this encoder 200 are envisioned, but for clarity, the encoder 200 is described below without describing all anticipated variations.

[0058] In some embodiments, Figure 2A An encoder is also shown that improves upon the HEVC or VVC standard (Multi-Functional Video Coding, standard ITU-T H.266, ISO / IEC 23090-3, 2020) or encoders using technologies similar to HEVC or VVC (such as the encoder ECM (Enhanced Compression Model) being developed by JVET (Joint Video Exploration Group)).

[0059] Before being encoded, the video sequence may undergo pre-coding processes (201), such as applying color transformations to the input color image (e.g., converting from RGB 4:4:4 to YCbCr 4:2:0), performing remapping of the input image components to obtain a signal distribution more robust to compression (e.g., using histogram equalization of the color components), or resizing the image (e.g., downsizing). Metadata may be associated with the preprocessing and appended to the bitstream.

[0060] In encoder 200, the image is encoded by encoder elements as described below. The image to be encoded is divided (202) and processed, for example, in units of CUs (coding units) or blocks. In this disclosure, different expressions may be used to refer to such units or blocks resulting from image division. Such terms may be coding unit or CU, coding block or CB, luminance CB or block. A CTU (coding tree unit) refers to a set of blocks or a set of units or a set of coding units (CUs). In some embodiments, a CTU may be considered as a block or itself as a unit.

[0061] Each unit is encoded using, for example, an intra-frame or inter-frame mode. When a unit is encoded in intra-frame mode, it performs intra-frame prediction (260). In inter-frame mode, motion estimation (275) and compensation (270) are performed. The encoder decides (205) whether to use intra-frame or inter-frame mode to encode the unit and indicates the intra-frame / inter-frame decision by, for example, a prediction mode flag. The encoder can also mix (263) intra-frame prediction results and inter-frame prediction results, or mix results from different intra-frame / inter-frame prediction methods. The prediction residual is calculated by, for example, subtracting (210) the predicted block from the original image block.

[0062] The motion refinement module (272) refines the motion field of a block using an already available reference image, without referencing the original block. The motion field for a region can be considered as the set of motion vectors for all pixels within that region. If the motion vectors are based on sub-blocks, the motion field can also be represented as the set of motion vectors for all sub-blocks within that region (all pixels within a sub-block have the same motion vector, and the motion vector can vary with the sub-block). If a single motion vector is used for the entire region, the motion field for that region can also be represented by a single motion vector (the motion vector is the same for all pixels within that region).

[0063] The predicted residual is then transformed (225) and quantized (230). The quantized transform coefficients, along with the motion vector and other syntax elements, are entropy encoded (245) to output a bitstream. The encoder can skip the transform and apply quantization directly to the untransformed residual signal. The encoder can bypass both the transform and quantization, i.e., encode the residual directly without applying the transform or quantization process.

[0064] The encoder decodes (reconstructs) the encoded blocks to provide a reference for further prediction. The quantized transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residuals. The image blocks are reconstructed by combining (255) the decoded prediction residuals and the predicted blocks. A loop filter (265) is applied to the reconstructed image to perform one or more of, such as deblocking filtering, SAO (Sample Adaptive Shift) filtering, or ALF (Adaptive Loop Filter) filtering, to reduce coding artifacts. The filtered image is stored in the reference image buffer (280). Such a filtered image is also referred to hereinafter as the reference image.

[0065] Figure 3A A block diagram of a video decoder 300 is shown. In decoder 300, the bitstream is decoded by decoder elements as described below. Video decoder 300 typically performs operations similar to... Figure 2A The encoding process described herein is the opposite of the decoding process (pass). Encoder 200 typically also performs video decoding as part of the video data encoding.

[0066] Specifically, the decoder's input includes a video bitstream that can be generated by the video encoder 200. First, the bitstream is entropy-decoded (330) to obtain transform coefficients, motion vectors, and other encoded information. Image segmentation information indicates how the image is segmented. Therefore, the decoder can segment (335) the image based on the decoded image segmentation information. The transform coefficients are dequantized (340) and inverse transformed (350) to decode the prediction residuals. By combining (355) the decoded prediction residuals and the predicted blocks, image blocks are reconstructed.

[0067] The predicted blocks can be obtained from intra-frame prediction (360) or motion-compensated prediction (i.e., inter-frame prediction) (375). The decoder can mix (373) the intra-frame prediction results and the inter-frame prediction results, or mix the results from multiple intra-frame / inter-frame prediction methods. Before motion compensation, the motion field can be refined (372) by using an already available reference image. A loop filter (365) is applied to the reconstructed image. The filtered image is stored in the reference image buffer (380). Note that for a given image, the contents of the reference image buffer 380 on the decoder 300 side are the same as the contents of the reference image buffer 280 on the encoder 200 side for the same image.

[0068] The decoded image can undergo further post-decoding processing (385), such as inverse color transformation (e.g., from YCbCr 4:2:0 to RGB 4:4:4) or inverse remapping of the remapping process performed in pre-encoding processing (201), or resizing of the reconstructed image (e.g., enlargement). Post-decoding processing can utilize metadata derived in pre-encoding processing and signaled in the bitstream.

[0069] Some embodiments described herein relate to loop filters and out-of-loop filters. The embodiments provided herein are intended to improve reconstructed samples of images from a video being encoded or decoded. More specifically, the embodiments aim to jointly improve the quality of predictions provided by a reference image and the image output by the decoder, both obtained from the same reconstructed sample input.

[0070] In some embodiments, the filters provided herein relate to adaptive loop filters (ALF), but are extended to any other linear Wiener-based or nonlinear filters.

[0071] Any of the embodiments described herein can be implemented, for example, in the loop filter module of a video encoder or in the loop filter module and post-decoding processing module of a video decoder. For example, the embodiments described herein can be implemented in... Figure 2A It is implemented in the loop module 265 of the video encoder 200 or in the loop filter module 365 and / or the post-decoding processing module 385 of the video decoder 300.

[0072] In another variation, such as Figure 2B and 3B As shown, joint adaptive filter modules (290, 390) are added to encoder 200 and decoder 300, so that the embodiments provided herein are implemented in these joint adaptive filter modules (290, 390). Figure 2B The encoder and Figure 3B The decoders shown are respectively similar to Figure 2A encoder and Figure 3A The decoder has a loop filter module (265, 365), but the output of the loop filter module is provided as input to a joint adaptive filter, which performs further loop filtering to provide a reference image to be stored in a reference image buffer (280, 380), and performs output filtering to provide the decoder output image. On the decoder side, the output image can be provided to the post-decoding processing module (385).

[0073] Block-based intra / inter-frame prediction and transform coding, along with quantization, can introduce various artifacts at low to medium bit rates. To reduce these artifacts, video coding standards such as VVC and video coding schemes such as ECM have implemented loop filters, including Deblocking Filter (DBF), Bilateral Filtering (BIF), Sample Adaptive Offset (SAO), and Adaptive Loop Filtering (ALF).

[0074] Deblocking filtering (DBF) aims to smooth out discontinuities that may appear along block boundaries. Sample adaptive offset (SAO) is designed to attenuate artifacts appearing around edges and uses the offset signaled in the bitstream to correct for local average intensity variations (aka banding artifacts). Bilateral filter (BIF) is designed to further denoise the reconstructed image to remove artifacts caused by quantization in the transform domain. Finally, adaptive loop filter (ALF) is determined as the optimal filter on the encoder side according to the rate-distortion criterion to enhance the reconstructed image. The ALF filter is transmitted with the bitstream and then retrieved and used on the decoder side.

[0075] The VVC standard implements three types of loop filters: Deblocking Filter (DBF), Sample Adaptive Offset (SAO), and Adaptive Loop Filter (ALF) (e.g., in...). Marta Karczewicz et al., "VVC In-Loop Filters" Device)", IEEE Transactions on Circuits and Systems for Video Technology (IEEE Video Technical Circuits and Systems Bulletin, Vol. 31, No. 10, October 2021 (as defined in [the original text]). Deblocking filters aim to reduce block discontinuities. Sample adaptive offset primarily aims to reduce artifacts caused by transform coefficient quantization. Adaptive loop filters and cross-component adaptive loop filters are adaptive filters that enable the enhancement of reconstructed signals, for example, using Wiener filter coding schemes.

[0076] Figure 4 An example of the workflow of a VVC loop filter is described in the figure. [JVET-T2002], Algorithm Description for Versatile Video Coding and Test Model 11 (VTM 11) (Algorithm description of code and test model 11) (§3.7). For each block of the image, reconstructed samples of the image are obtained from the decoded residuals and predictions. Then, if the local deblocking condition is met, the luminance and chrominance reconstructed samples located at the block boundaries are first filtered using a deblocking filter (DBF). Then, based on the classification using a band-based classifier or an edge classifier, an offset is locally added using Sample Adaptive Offset (SAO). Finally, before storing the resulting sample values ​​in the reference image buffer, an Adaptive Loop Filter (ALF) and its variant, the Cross-Component Adaptive Loop Filter (CCALF), are applied.

[0077] ALF is an adaptive filter that is applied to reduce the mean square error (MSE) between the original sample of an image and the reconstructed sample of the image (obtained after encoding and decoding the image and the application of a loop filter), using, for example, a Wiener filter coding scheme. The ALF filter can be transmitted in the bitstream and decoded on the decoder side, or derived from parameters previously signaled or from predefined parameters, and then applied to the reconstructed sample.

[0078] In VVC, the ALF filter is point-symmetric, DC-neutral, and has integer coefficients. Marta Karczewicz et al., "VVC In-Loop filters," IEEE Transactions on Circuits and Systems for Video Technology, Vol. 31, No. 10, October 2021 In VVC, the ALF filter uses a 7×7 diamond filter for luminance and a 5×5 diamond filter for chrominance, such as... Figure 5 The filtering operation is as described. make For the reconstructed sample, and let This represents the value after ALF filtering. In the linear implementation of ALF, The calculation is as follows: (1) in, Represents the filter coefficients. and: (2) in Indicates the relationship with the first coefficients The coordinate offsets of the associated reconstructed samples.

[0079] In the nonlinear implementation of ALF, equation (1) becomes: (3) in: (4) in It is related to the coefficient The associated limiting parameter, which is determined by the limiting index. Confirmed. Limiting parameters. Export as follows: (5) in Indicates the sample bit depth, and It can be 0, 1, 2 or 3.

[0080] The ALF for luminance exhibits local adaptive properties, with its classification based on local gradients. Therefore, the ALF for luminance relies on a filter set, i.e., multiple filters with an accompanying mapping list, where each category of the classification is associated with a specific filter in that filter set. This mapping list may also be referred to as the classification graph in this paper. Based on the classification, geometric filter transformations, such as 90-degree rotation, diagonal flip, or vertical flip, are applied.

[0081] There are many pre-trained luminance filters and filter sets available on the encoder and decoder sides. In VVC, there are 64 pre-trained luminance filters and 16 pre-trained filter sets.

[0082] Luminance is classified at the 4×4 sub-block level based on its directionality and Laplace activity.

[0083] First, calculate the sample gradient values ​​for the horizontal, vertical, and two diagonal directions:

[0084] Calculate the sub-block horizontal gradient based on the sample gradient. Vertical gradient and the gradients of the two diagonals and :

[0085] Where index and This refers to the coordinates of the top-left sample in the 4×4 luminance sub-block.

[0086] Second, in order to allocate directionality The ratio of the maximum to the minimum horizontal and vertical gradients of the sub-block. , (8) And the ratio of the maximum to the minimum of the diagonal gradients of the two sub-blocks. , (9) Compare with each other and with a set of thresholds and The blocks are compared and categorized as follows: Step 1 If both conditions are met and Then Set to 0 (the block is categorized as "texture") Step 2 if Then, in step 3, the directionality is calculated. Otherwise, calculate in step 4.

[0087] Step 3 if Then Set to 2 (block is classified as "strong horizontal / vertical"); otherwise... Set to 1 (the block is classified as "weak horizontal / vertical").

[0088] Step 4 if Then Set to 4 (block is classified as "strong diagonal"); otherwise... Set to 3 (the block is classified as "weak diagonal").

[0089] Third, activity value The calculation is as follows: (10) Activity Value It is further mapped to the range of 0 to 4 (inclusive), and the quantized value is denoted as .

[0090] Finally, each 4x4 luminance block was categorized into one of 25 categories: (11) Each category is assigned a specific filter.

[0091] The filter set contains up to 25 filters, and if multiple filters exist, there is also a list of 25 indices. This list indicates which filter should be used for each category. Depending on the category, geometric transformations are applied to the filters, such as 90-degree rotation, diagonal flip, or vertical flip.

[0092] In VVC, the ALF used for chroma does not implement local classification. Instead, it features region-adaptive characteristics. Up to eight ALF chroma filters are available simultaneously on both the encoder and decoder sides, and each chroma CTB is notified by a signal which filter to use.

[0093] The ALF filter coefficients and clipping index are determined on the encoder side, for example, by solving the Wiener-Hopp equation to minimize the MSE between the reconstructed sample and its original value. In VVC, this optimization can be performed at the frame / strip level.

[0094] For luminance, the encoder first calculates an optimal filter set containing 25 filters, where 25 is the number of luminance categories. Then, the encoder iteratively selects the category pairs that exhibit the lowest distortion when combined, and recalculates the corresponding optimal luminance filter set. Ultimately, this produces 25 distinct luminance filter sets: one with 25 filters, another with 24 filters, and so on, up to a filter set containing a single filter. Finally, the optimal luminance filter set can be selected based on a rate-distortion criterion that takes into account the corresponding signaling cost of the filter set.

[0095] Once frame / strip-level optimization is complete, the encoder, based on the actual number of CTUs used to optimize the current frame / strip for the current luminance filter set, selects whether it is worthwhile to transmit the new luminance filter set, or whether it is beneficial or preferred to use a pre-trained luminance filter set or a filter set that has been signaled instead. This encoder decision is based on the rate-distortion conditions evaluated at the frame / strip level.

[0096] If the rate-distortion condition on luminance is met, the luminance filter set is signaled in the ALF Adaptive Parameter Set (APS), i.e., for each luminance filter: each filter coefficient, and if applicable, the corresponding limiting index, and then, if there is more than one luminance filter, a category-to-filter mapping or classification graph.

[0097] If the rate-distortion condition in chromaticity is met, the chromaticity filter is notified by a signal in the ALF APS.

[0098] In VVC, an ALF APS can contain a set of luminance filters and up to eight chrominance filters.

[0099] At the CTU level, the encoder signals whether to perform ALF. If ALF is enabled, one index signals which luma filter set should be used in the pre-training and signaled filter sets. Another index signals which chroma filter should be used.

[0100] ALF in Enhanced Compression Model (ECM) software has seen further development. JVET-AD2025, Algorithm Description of Enhanced Compression Model 9 (ECM 9), § 3.4 Its main features are summarized below.

[0101] The size of the chromatic ALF diamond filter has been increased to 9×9.

[0102] A fixed ALF filter was provided for luminance. Three different classifiers were used to filter the luminance samples. , and The classifier uses three distinct filter sets (F0, F1, and F2). Sets F0 and F1 contain fixed filters with coefficients specific to the classifier. and Pre-training. Fixed filters from F0 and F1 correspond to, for example... Figure 6 The first and second fixed filters are shown. The filter coefficients in set F2 are communicated with the signal. For example, the filter from F2 corresponds to... Figure 6 The spatial filter (spatial tap) is shown. Which filter from set Fi is used for a given sample depends on the classifier used. The category Ci assigned to the sample is determined. In addition to the two fixed-brightness filters (F0, F1), a third fixed-brightness filter is used, which has no classification. For example, the third fixed-brightness filter corresponds to... Figure 6 The third fixed filter is shown.

[0103] Furthermore, luminance classification is extended using an additional surrogate classifier. For a luminance filter set notified by a signal, a signal flag indicates whether a surrogate classifier is applied. Geometric transformations are not applied to the surrogate band classifier. When applying a band-based classifier, the sum of sample values ​​for a 2×2 luminance block is first calculated. Then, the category index is calculated as follows: class_index = (sum 25)>>(sample bit depth + 2) (12) The output of the fixed filter, the sample before deblocking, and the residual sample are used as additional inputs to the ALF filter, which is then signaled. The final filtered sample value is calculated as follows: (13) in, It is the neighboring sample and the current sample The difference in the limiting values ​​between them The intermediate samples generated from the first fixed filter and the current sample The difference in the limiting values ​​between them The intermediate samples generated from the third fixed filter and the current samples The difference in the limiting values ​​between them The intermediate samples generated from the first fixed filter and the current sample The difference in the limiting values ​​between them The intermediate samples generated from the second fixed filter and the current sample The difference in the limiting values ​​between them The intermediate samples generated from the third fixed filter and the current sample The difference in the limiting values ​​between them It is the neighboring samples before DBF and the current sample The difference in the limiting values ​​between them It is the co-sampling samples before DBF and the current sample The difference in the limit between them, and where These are adjacent residual sample values ​​after amplitude limiting. The current sample is filtered by one of the fixed filters. The amplitude-limited residual samples. For the residual samples, the fixed filter is reused as the offline fixed filter trained after SAO reconstruction.

[0104] Figure 6 An example of the complete filter shape of an ALF using residual samples as additional input is shown in the figure.

[0105] Some embodiments described herein provide a novel adaptive filter, namely a joint filter, also referred to below as a joint adaptive loop and output filter. In some variations, adaptive filtering may involve sample classification. In the following, loop filtering or intra-loop filtering refers to the generation of the image used in the encoder-decoder loop. The generated image is used in the same manner on both the encoder and decoder sides. The terms loop filtering or intra-loop filtering are used interchangeably.

[0106] In contrast, the output image produced by the output filter is the output provided by the decoder for a given purpose (display, machine task, etc.).

[0107] In some embodiments, the joint adaptive filter provided herein is intended to replace the ALF in the encoding / decoding workflow.

[0108] In this embodiment, the adaptive loop filter and the adaptive output filter are jointly derived on the encoder side and transmitted in the bitstream.

[0109] In another embodiment, the adaptive loop filter and the adaptive output filter are implemented using a neural network and jointly derived or learned.

[0110] On the decoder side, the joint filter framework presents two outputs: the first output is fed into the reference image buffer within the prediction loop, and the second output is used as the decoder's output, such as... Figure 7 As shown in the embodiments.

[0111] exist Figure 7 In this embodiment, the preceding loop filter applied to the reconstructed samples before the joint adaptive filter can include DBF, SAO, and BIF. In this embodiment, the joint adaptive filter replaces the ALF, compared to traditional encoding / decoding schemes such as VVC and ECM. In other embodiments, the joint adaptive filter may replace other filters.

[0112] Second output ( Figure 7 The decoder output (in the output) can be used for, for example, display purposes. However, the principles described herein are not strictly limited to joint loops and display filters. In another embodiment, the second output can also be dedicated to view composition. In a further embodiment, the second output may not be intended for human viewing and evaluation, but rather for use by machine vision algorithms such as object detection, instance segmentation, or object tracking. Figure 8 An example of such an embodiment is shown.

[0113] Some embodiments involve a novel adaptive filter framework, namely a joint loop filter and an output filter.

[0114] In this embodiment, these joint filters are adaptive: their parameters (i.e., filter coefficients and potential clipping indices) are adaptively determined on the encoder side based on sample values ​​and local or non-local features. Adaptive filtering may also involve sample classification.

[0115] The loop filter feeds into the reference image buffer, which is the image that will be used as a reference for prediction. The output filter produces the image output by the decoder for display or other purposes.

[0116] For convenience, in the following text, any filter will be referred to as the first or second filter, and the corresponding output will be referred to as the first or second output.

[0117] In one embodiment, the loop filter and the output filter are jointly determined on the encoder side and transmitted along with the bit stream, for example in a dedicated adaptive parameter set (APS).

[0118] In another embodiment, both the loop filter and the output filter are pre-trained offline, i.e., before encoding. In this embodiment, the joint adaptive loop and output filter, as well as the set of joint adaptive loop and output filters, are hard-coded on both the encoder and decoder sides.

[0119] In another embodiment, both the loop filter and the output filter are implemented using a neural network that shares a common processing path and are jointly trained.

[0120] In variations of these embodiments, the parameters of the at least one loop filter and the at least one output filter are jointly determined using reconstructed samples, predicted target samples for the at least one loop filter, and output target samples for the image used in the at least one output filter. As will be further described below, the predicted target samples can be obtained from a denoised version of the original image, and the output target samples can be obtained from the original image or another version whose acquisition process depends on the output purpose of the decoder.

[0121] The following sections introduce various joint filtering implementations and optimization methods.

[0122] Figure 9 An example of a method 900 for encoding video according to an embodiment is shown. At 910, the current image of the video is encoded to generate encoded data representing the image and stored in a bitstream, for example using a combination of... Figure 2A Or one or more of the coding modules described in 2B. At 920, a reconstructed sample of the image is obtained. For example, the reconstructed sample is used as a reconstruction module ( Figure 2A The output of (or 255 in 2B) or as a loop filter ( Figure 2A The output of (or 265 in 2B) is obtained. At 930, a joint adaptive filter set is determined using the reconstructed samples. This joint adaptive filter set includes at least one loop filter and at least one output filter. The at least one loop filter is used to provide a first version of the reconstructed image intended to be used as a reference image, while the at least one output filter is used to provide a second version of the reconstructed image intended to be used as an output image by the decoder.

[0123] In the variant, at 940, it is checked whether the joint adaptive filter set satisfies the rate-distortion criterion. For this purpose, a first rate-distortion cost is determined for the image using the joint adaptive filter set used for that image and taking into account the bit rate used to transmit the joint adaptive filter set. For the image, a second rate-distortion cost is determined using the previously transmitted joint adaptive filter set used for that image or any other loop and output filters. At 940, it is checked whether the first rate-distortion cost is better than the second rate-distortion cost. If so, at 950, the joint adaptive filter set is encoded in the bitstream, and at 960, the loop filters of the joint adaptive filter set are used to obtain a reference image. Otherwise, at 970, other loop filters (e.g., previously transmitted loop filters) are used to obtain the reference image. At 980, the reference image is stored in a reference image buffer.

[0124] Figure 10 An example of a method 1000 for decoding video according to an embodiment is shown.

[0125] At position 1010, encoded data representing the current image of the video is obtained from the bitstream, and the current image is decoded, for example, using a combination of... Figure 3A Or one or more of the decoding modules described in 3B. At 1020, a reconstructed sample of the image is obtained. For example, the reconstructed sample is used as a reconstruction module ( Figure 3A The output of (or 355 in 3B) or as a loop filter ( Figure 3AThe output of (or 365 in 3B) is obtained. At 1030, the joint adaptive filter set is obtained. In some variations, the joint adaptive filter set is decoded from the bitstream. It can correspond to the joint adaptive filter set determined for the current image being decoded or a previously transmitted joint adaptive filter set. In other variations, the joint adaptive filter set is hard-coded and retrieved from the decoder's memory. The joint adaptive filter set includes at least one loop filter and at least one output filter.

[0126] At 1040, the loop filter of the joint adaptive filter set obtained at 1030 is used to obtain the reference image, which is then stored in the reference image buffer. At 1050, the output filter of the joint adaptive filter set obtained at 1030 is used to obtain the output image, which is then output by the decoder.

[0127] Determining the joint adaptive filter set for the image can be accomplished using rate-distortion optimization as described below. Figure 11 As shown in Figure 12, the adaptive filter can be jointly determined in two processing steps.

[0128] In the first step, multiple adaptive filter set candidates (1110, 1120, 1210, 1220) are computed. Each filter set candidate is computed to minimize the distortion between the filtered image and the target image under a given configuration (e.g., given the total number of adaptive filters to be transmitted in the bitstream).

[0129] For the loop filter, adaptive filter optimizations (1110, 1210) use the target samples for prediction and the reconstructed samples, and output a candidate set of loop filters. For the output filter, adaptive filter optimizations (1120, 1220) use the target samples for the output (in... Figure 11 (or other target samples in 12) and reconstructed samples, and output the candidate output filter set.

[0130] Then, in the second step, the candidates for the loop filter and the output filter are tested against each other under the rate-distortion optimization (RDO) criterion, which can be written as: (14) in This represents the rate required to signal the total number of adaptive filters. This represents the distortion between the filtered image and the target image, such as mean squared error (MSE). It is the Lagrange parameter that balances the rate and the distortion.

[0131] In the variant, such as Figure 11The loop filter and the output filter are both determined using the same RDO criterion (1130), as described in equation (14). The values ​​are the same.

[0132] In this variant, RDO can be implemented using a single global criterion without differentiation: (15) in and These represent the rate cost of notifying the loop filter and the output filter using signals, respectively. This indicates the distortion between the image after loop filtering and its corresponding target image. This indicates the distortion between the filtered output image and its corresponding target image. Alternatively, RDO (1130) can be achieved using two criteria that need to be minimized separately: (16) (17) In another variation, such as Figure 12 As described, the loop filter and the output filter use different specific Lagrangian parameters (i.e., respectively). and To determine.

[0133] In this variant, RDO(1230) can also be implemented indiscriminately using a single global criterion: (18) Alternatively, two criteria that need to be minimized separately can be used: (19) (20) In this variant example, In another example, .

[0134] In the example, It depends on the quantization parameter QP_slice of the current slice of the image being encoded.

[0135] In the example, It depends on the so-called base quantization parameter QP_base. In VVC and ECM (and HEVC), the base quantization parameter QP_base is the quantization parameter value used to derive the strip QP. For example, according to JVET-T2002 for random access configuration, the QP of each inter-frame coded image is derived by adding an offset to the base QP.

[0136] In the variant, global optimization with a separation process can be used to determine the loop filter and output filter of the joint adaptive filter set.

[0137] In this variant, the loop filter and the output filter can be as follows: Figure 11 and 12 The equations (15), (16) & (17), (18), or (19) & (20) are used to optimize the results without any difference.

[0138] In this variant, the joint adaptive filter set consists of a combination of an adaptive loop filter set and an output filter set, namely: one or more loop filters, and if there are more than one loop filter, a category-to-filter mapping for the loop filters, plus one or more output filters, and if there are more than one output filter, a category-to-filter mapping for the output filters.

[0139] In this variant example, the joint adaptive filter is determined in a two-step process. In the first step, multiple loop filter set candidates and multiple output filter set candidates are computed independently, where each filter set candidate is computed to minimize the distortion between the filtered image and the target image for a given number of adaptive filters to be transmitted in the bitstream.

[0140] make This indicates the number of categories in the adaptive filtering classification. If each category has its own specific filter, the adaptive filter set can contain... A filter. If two categories are merged to share the same filter, it can also contain only one filter. A filter is used, and so on, until each category is merged and shares the same filter, containing a single filter. This produces A number of potential filter set candidates. Figure 11 and 12 As shown, the number of filters This serves as input for adaptive filter optimization, which determines the loop or output set of adaptive filter candidates.

[0141] At the end of the first step, therefore, exists A set of candidate loop filters and A set of candidate output filters. Then in the second step (e.g.) Figure 11 1130 or Figure 12 The final joint filter set is selected during the period of 1230.

[0142] In one variation, rate-distortion criteria such as equations (16) & (17) or (19) & (20) are used to independently select the final set of loop filters and the final set of output filters. In another variation, rate-distortion criteria such as equation (15) or (18) are used to select the final set of output filters. The final joint filter set is selected from the candidate joint filter sets.

[0143] In another variation, combinatorial optimization can be used to jointly determine the loop filter and output filter of the joint adaptive filter set. In this variation, the determination of one of the loop filter or output filter of the joint adaptive filter set is constrained by the determination of the other of the loop filter or output filter of the joint adaptive filter set.

[0144] In this variant, the filter optimization is sequential: first, adaptive filter optimization (1310) is performed to provide a first set of candidate filters, then RDO (1330) is performed to obtain the final first set of filters, and the final second set of filters, which is a joint adaptive filter set, is obtained by adaptive filter optimization (1320) constrained by the first determined final first set of filters, as follows. Figure 13 The description in the text refers to general situations.

[0145] In this variant, the optimization of the second filter is constrained by the optimization result of the first filter. This constraint can advantageously be twofold and involves both the number of filters in the filter set and the classification achieved through the filter set, i.e., the mapping from category to filter.

[0146] In this variant, the encoder derives a first filter set and a second filter set, which advantageously represent the same number of filters and use the same category to filter mapping list (or classification graph).

[0147] In this variant, joint adaptive filtering relies on signaling of a joint adaptive filter set. The joint adaptive filter set includes one or more adaptive loop filters, the same number of adaptive output filters, and, if more than one type (loop or output) adaptive filter exists, a common category-to-filter mapping list.

[0148] If the rate distortion criterion is met (e.g.) Figure 9 In step 940), the encoder then signals the joint adaptive filter set in, for example, the APS.

[0149] In a further variation, dependent coding can be used for the second filter. In this further variation, the filter coefficients and limiting indexes of the second filter set are encoded based on the coefficients and limiting indexes of the first filter set.

[0150] In the example, the dependency encoding involves coefficient prediction and amplitude limit index prediction.

[0151] In such an example, the filter coefficients of the second filter set can be differentially encoded relative to the corresponding coefficients of the first filter set; that is, the signed difference between each second filter coefficient and its corresponding first filter coefficient is signaled in the APS.

[0152] In the same manner, the limiting index of the second filter set can be differentially encoded relative to the corresponding limiting index of the first filter set, that is, the signed difference between each second filter limiting index and the corresponding first filter limiting index is signaled in the APS.

[0153] In the example of this sequential optimization variant, such as Figure 15 The method first optimizes the loop filter, and then optimizes the output filter under some constraints derived from the optimal adaptive loop filter set.

[0154] In another example of this sequential optimization variant, such as Figure 16 As shown, the output filter can be optimized first, and then the loop filter can be optimized under some constraints derived from the optimal adaptive loop filter set.

[0155] In these examples of this sequential optimization variant, the second adaptive filter optimization takes as input the number of filters determined for the final first filter set and the categories to be merged. The second adaptive filter optimization provides as output a final second filter set, which includes the same number of filters as the final first filter set.

[0156] In another variation, the second adaptive filter is based on the first adaptive filter, wherein the second adaptive filter includes at least one tap on the output of the first adaptive filter (i.e., at least one filter coefficient and at least one potential limiting index).

[0157] In this variant example, at least one tap of the output filter is applied to the output of the loop filter.

[0158] In this example, the joint adaptive loop and output filtering occur on both the encoder and decoder sides, as follows: Figure 17 As outlined. In this example, the reconstructed sample is filtered by the adaptive loop filter of the joint adaptive filter set (1710), and then by the adaptive output filter of the joint adaptive filter set (1720), which also takes the output of the adaptive loop filter as input to provide the final sample output by the decoder.

[0159] On the encoder side, the joint filter optimization is as follows: Figure 18The combination is as described. The final set of loop filters is obtained as the output of loop filter optimization (1810) and used to filter the reconstructed samples (1820). The filtered reconstructed samples, along with the unfiltered reconstructed samples and the target output samples (here, the original samples, but other versions of the original samples may also be used) are fed as input to output filter optimization (1830) to provide the final set of output filters.

[0160] Figure 19 and 20 An example of the adaptive output filter shape is provided. In both images, the gray squares represent the current sample to be filtered.

[0161] exist Figure 19 In the diagram, taps #0 to #9 are applied to adjacent reconstructed samples, taps #10 to #27 and #30 are applied to the output of the first pre-trained (hard-coded) filter, tap #31 is applied to the co-position output of the second pre-trained filter, taps #28, #29 and #33 are applied to the samples before deblocking, tap #32 is applied to the co-position residual, tap #34 is applied to the residual filtered by the pre-trained filter, tap #35 is applied to the co-position output of the third pre-trained filter, and tap #36 is applied to the output of the joint loop filter.

[0162] exist Figure 20 In the process, taps #0 to #9 are applied to adjacent reconstructed samples, taps #10 to #27 and #34 are applied to the output of the first pre-trained filter, tap #35 is applied to the co-position output of the second pre-trained filter, taps #28 to #31 and #38 are applied to the output of the third pre-trained filter, taps #32, #33 and #36 are applied to the samples before deblocking, tap #37 is applied to the co-position residual, and tap #39 is applied to the output of the joint loop filter.

[0163] In another example of this variation, a loop filter is applied to the output of the output filter.

[0164] In the above embodiments, in some variations, the joint adaptive filter set may include different numbers of loop filters and output filters, while in other variations, the joint adaptive filter set includes the same number of loop filters and output filters. Loop filtering is applied on the decoder side depending on whether the loop filters and output filters have the same number of filters. Figure 10 1040 in the middle) and output filter ( Figure 10 (1050 in the middle) must be classified in the same way or in a different way before. Figure 14An example of method 1400 is shown, which may be part of a method for decoding video according to the embodiments described above. At 1410, it is determined whether the loop filter and the output filter in the joint adaptive filter set have the same number of filters. If not, at 1420, classification of the reconstructed samples is performed based on the number of categories defined for the loop filter in the joint adaptive filter set, and at 1430, classification of the reconstructed samples is performed based on the number of categories defined for the output filter in the joint adaptive filter set. Otherwise, at 1440, the same classification of the reconstructed samples is performed for both the loop filter and the output filter. At 1450, a reference image is obtained and stored by filtering the reconstructed samples using the loop filter and the classification obtained at 1420 or 1440. At 1460, an output image is obtained by filtering the reconstructed samples using the output filter and the classification obtained at 1430 or 1440.

[0165] In other variations, even if the joint adaptive filter set includes the same number of filters for both loop and output filters, different classifications can be performed at 1420 and 1430, for example, using different classification methods or criteria. In this variation, it can be determined at 1410 whether the loop and output filters of the joint adaptive filter set use the same classification. This can be determined, for example, based on parameters transmitted with the joint adaptive filter set or hard-coded at the decoder. In any case, the same classification must be performed for each type of filter (loop, output) at both the encoder and decoder.

[0166] Further embodiments are provided below, in which the loop filter and the output filter are determined using the same optimization algorithm on the encoder side but with different inputs. For example, the loop filter is adaptively derived from the denoised image, while the output filter is adaptively derived from the original image. It is indeed beneficial to present a reference image with lower noise levels, as noise levels typically affect the efficiency and quality of predictions.

[0167] In this variant, the denoised image is a temporally denoised image. As a first example of this variant, the loop filter is computed using the temporally denoised image, such as using motion-compensated temporal filtering (MCTF), while the output filter is computed using the original samples, such as... Figure 21 The description.

[0168] In another example of this variation, the output filter is computed using an image that has already been processed by MCTF, while the loop filter is computed using an image processed by a stronger MCTF.

[0169] In another variation, the denoised image is a spatially denoised image. In the first example of this variation, the loop filter is computed using the spatially denoised image, such as using a bilateral filter (BIF), while the output filter is computed using the original samples, such as... Figure 22 The description.

[0170] In another example of this variation, the output filter is computed using an image that has already been processed by BIF, while the loop filter is computed using an image processed by a stronger BIF.

[0171] In a further variation, combined temporal and spatial denoising is used to provide the denoised image. In the first example of this variation, the loop filter is computed using the temporally and spatially denoised image, for example, using a combined step of MCTF first and BIF later, while the output filter is computed using the original samples.

[0172] In another example of this variation, the output filter is computed using an image that has already been processed by MCTF and BIF, while the loop filter is computed using an image processed by a stronger MCTF and BIF.

[0173] In some of the embodiments described above, the joint filters of the joint adaptive filter set are signaled to the decoder. In embodiments where the decoding of loop filters and output filters is independent, they can be signaled in separate APSs.

[0174] In a variant, the loop filter set is signaled in the APS, and the output filter set is signaled in the SEI (Supplemental Enhancement Information) message. In another embodiment, the joint loop and output filters can be signaled in the same APS.

[0175] In other embodiments, a joint adaptive filter providing the following output can be implemented using deep learning-based filtering: a first filtered image is used as a reference image, and a second filtered image is used as the output of the decoder. The same method as in the embodiments described above can be applied to loop filtering based on neural networks (NNs).

[0176] Figure 23 An example of NN-based loop filtering is shown. Figure 23 As shown, the data is input into the model, for example: - Reconstruction stripes (Rec) after (or before) DBF - Predicted Image (Pred) - QP Information (QP) - Block type (the type of block, such as intra-frame, inter-frame, bidirectional prediction, unidirectional prediction, etc.) Other inputs are possible (segmentation information, DBF information, etc.).

[0177] The neural network (NN) model can operate on a block-by-block basis or on the entire image. In the block-by-block case, the input described above corresponds to the input of the current block to be filtered. When operating on the entire image, the input described above corresponds to the input of the entire image. The NN model outputs either the filtered block or the filtered image. Figure 23 (Refrigerated / filtered).

[0178] This model typically consists of the following parts: - The head, where each input is processed independently, typically consists of convolutions and activation sets. For example, a typical head branch consists of a 3×3 convolution followed by ReLU (or similar).

[0179] - A merging layer, which aggregates the outputs of each header, is typically a concatenation layer followed by a convolution. For example, a typical merging layer consists of a concatenation layer followed by a 1×1 convolution to reduce depth, ReLU, and a 3×3 convolution with ReLU.

[0180] - The backbone, consisting of a series of layers, typically a series of convolutions and activations, such as with skip connections. In JVET- AE2019(Algorithm description for Neural Network-based Video Coding NNVC-6.0 (Algorithm Description of Neural Network-Based Video Coding NNVC-6.0), 31st JVET Conference, Geneva, Switzerland, July 2023 From the 11th to the 19th of the month An example of a trunk block is described in ().

[0181] - The tail layer, which outputs the results at the correct resolution and channels, typically consists of 3-channel convolutions and pixel shuffles to output each filtered component Y, U, and V.

[0182] In an embodiment, Figure 23 The NN filtering model shown is adapted to provide a joint adaptive filter based on the principles described in this paper.

[0183] In this embodiment, the neural network is trained to produce two outputs: one output of the neural network optimizes the reconstruction toward the original image or toward the output target image, and the other output of the neural network optimizes the reconstruction toward the predicted target image (e.g., a denoised image (e.g., the original image filtered using MCTF)), which is designed to optimize the prediction of frames to be further decoded.

[0184] Figure 24 An example of a neural network for filtering according to this embodiment is shown. Figure 24 As shown, a common path is used from the input to the head layer and then to the backbone layer. The model then splits into two branches: A branch outputs an optimized frame relative to the original frame: the filtered output path Y0 One branch outputs a frame optimized relative to the denoised frame, which is stored in the decoded picture buffer (DPB): the filtered DPB path Y1 The tail portion consists of a set of layers dedicated to each given output. The specialization of each tail portion is done during the training phase of the model.

[0185] During the training phase, calculate the loss: (twenty one) Y0 and Y1 are the outputs of the NN, Y org It is the original frame, Y denoised These are denoised frames. Factor alpha ( This allows for balancing the quality between each output.

[0186] In other variations, the denoised image Y denoised The above can be combined Figure 21 or Figure 22 Any variation of the described method is obtained. Additionally, the original frame Y used during the training phase... org It is a version of the original image, which is the target output image designed for decoder purposes, such as if the decoder is defined for machine tasks or other purposes.

[0187] In combination Figure 24 In variations of the described embodiment, the balance between each output Y0 and Y1 is also an input to the model, such as... Figure 25 As shown. During the training phase, the factor alpha ( The factor alpha (Falpha) is determined by the inter-frame balance within a Group of Pictures (GOP). For example, a frame with a low temporal ID will be heavily used for prediction within the GOP, thus the factor alpha used for that frame will be lower. ) is set to a factor alpha that is used for frames with higher time IDs. A larger value. For high time IDs, i.e., when the frame is not used as a reference image, the factor alpha ( ) is set to zero.

[0188] The frame's time ID is the time level of frames within a GOP when the GOP is decomposed along the time axis. For example, Figure 26 The temporal hierarchical decomposition of the GOP (Group of Pictures) for five images is shown. Figure 26In the example, frame I0 is encoded, then frame I4 is encoded using data from frame I0, then frame I2 is encoded using data from the past (frame I0) and the future (frame I4), and so on. For example, frames I0 and I4 have low time IDs (e.g., 0), frames I1 and I3 have high time IDs (e.g., 2), and frame I2 has a time ID in the middle (e.g., 1). In this example, for simplicity, we assume a GOP size of 5, but this principle applies to GOPs of any size.

[0189] In other variations, equilibrium (factor alpha) This can also be determined in the same way as the QP / lambda adjustment within the GOP. For example, QP / lambda is a hybrid of theoretical relationships and manual adjustments. This relationship is typically set in a configuration file.

[0190] like Figure 25 As shown, during inference, the balance (factor alpha) The relative importance of frames in the GOP (e.g., using the time IDs described above) is used as input into the model.

[0191] The above-mentioned neural network-based filtering processes can be implemented in, for example... Figure 2A , Figure 2B and Figure 3A , Figure 3B The encoder and decoder shown are used as additional modules for loop filtering, or as alternatives to the loop filtering modules.

[0192] Figure 27 A block diagram of a system that can implement aspects of this embodiment according to another embodiment is shown. Figure 27 The following is illustrated for use in conjunction with this document. Figures 4 to 26 One embodiment of the apparatus 2700 for encoding or decoding video is described. The apparatus includes a processor 2710 and can be interconnected to a memory 2720 via at least one port. The processor 2710 and memory 2720 may also have one or more additional interconnects to external connections.

[0193] According to any embodiment described herein, processor 2710 is configured to obtain reconstructed samples of images of a video and, based on the reconstructed samples, determine a joint adaptive filter set comprising at least one loop filter and at least one output filter, wherein the at least one loop filter is used to provide a first version of the image that will be used as a reference image, and the at least one output filter is used to provide a second version of the image that will be used as an output image by a decoder.

[0194] For example, processor 2710 uses a computer program product that includes code instructions that implement any of the embodiments described herein.

[0195] In another embodiment, according to any of the embodiments described herein, processor 2710 is configured to decode encoded data representing an image to obtain reconstructed samples of the image, use one or more loop filters to obtain a reference image from the reconstructed samples, and use one or more output filters to obtain an output image from at least the reconstructed samples, wherein the one or more loop filters and the one or more output filters are obtained from a joint adaptive filter set. For example, processor 2710 uses a computer program product including code instructions that implement any of the embodiments described herein.

[0196] In an embodiment, such as Figure 28 As shown, in the context of a transmission between two remote devices A and B via a communication network NET, device A includes a processor associated with RAM and ROM memories, the processor being configured to implement, as combined with Figures 1 to 27 The described method for encoding video, wherein device B includes a processor associated with RAM and ROM memories, the processor being configured to implement, as in combination with... Figures 1 to 27 The method described is for decoding video. As an example, the network is a broadcast network, suitable for broadcasting / transmitting encoded video from device A to decoding devices including device B.

[0197] Figure 29 An example of the syntax for signals transmitted via a packet-based transport protocol is shown. Each transmitted packet P includes a header H and a payload PAYLOAD. In some embodiments, the payload PAYLOAD may include video data encoded according to any of the embodiments described above. The payload may also include any signaling as described above, such as any signaling related to a joint adaptive filter set.

[0198] Various implementations involve decoding. As used herein, “decoding” can encompass all or part of a process performed on, for example, a received encoded sequence to produce a final output suitable for display. In various embodiments, such a process includes one or more of the processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such a process also or alternatively includes processes performed by a decoder of various embodiments described herein, such as entropy decoding of a sequence of binary symbols to reconstruct image or video data.

[0199] As a further example, in one embodiment, "decoding" refers only to entropy decoding; in another embodiment, "decoding" refers only to differential decoding; in yet another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding; and in yet another embodiment, "decoding" refers to the entire image reconstruction process including entropy decoding. Whether the phrase "decoding process" is intended to specifically refer to a subset of operations or to refer to the broader decoding process will be clear from the context of the specific description and is believed to be well known to those skilled in the art.

[0200] Various implementations involve encoding. In a manner similar to the discussion above regarding “decoding,” the term “encoding,” as used herein, can encompass all or part of a process performed on, for example, an input video sequence to produce an encoded bitstream. In various embodiments, such processes include one or more of the processes typically performed by an encoder, such as partitioning, differential coding, transform, quantization, and entropy coding. In various embodiments, such processes also, or alternatively, include processes performed by an encoder according to various embodiments described herein, such as determining resampling filter coefficients and resampling the decoded image.

[0201] As a further example, in one embodiment, “encoding” refers only to entropy encoding; in another embodiment, “encoding” refers only to differential encoding; and in yet another embodiment, “encoding” refers to a combination of differential and entropy encoding. Whether the phrase “encoding process” is intended to specifically refer to a subset of operations or to refer to a broader encoding process will be clear from the context of the specific description and is believed to be well known to those skilled in the art.

[0202] Note that the grammatical elements used in this article are descriptive terms. Therefore, the use of other grammatical element names is not excluded.

[0203] This disclosure has described various types of information that can be transmitted or stored, such as syntax. This information can be packaged or arranged in various ways, including those common in video standards, such as placing the information in SPS, PPS, NAL units, headers (e.g., NAL unit headers, picture headers, or stripe headers), or SEI messages. Other methods may also be used, including those common in system-level or application-level standards, such as placing the information in one or more of the following: a. SDP (Session Description Protocol), a format for describing multimedia communication sessions for session announcement and session invitation purposes, such as as described in RFC and used in conjunction with RTP (Real-Time Transport Protocol) transmission.

[0204] b. DASH MPD (Media Presentation Description) descriptors, such as those used in DASH and transmitted via HTTP, are associated with a representation or set of representations to provide additional characteristics to the content representation.

[0205] c. RTP header extensions, for example, used during RTP streaming.

[0206] d. ISO basic media file formats, such as those used in OMAF, use boxes, which are object-oriented building blocks defined by unique type identifiers and lengths, also known as "atoms" in some specifications.

[0207] e. An HLS (HTTP Live Stream) manifest transmitted via HTTP. The manifest can be associated, for example, with a version or set of versions of the content to provide characteristics of that version or set of versions.

[0208] When a diagram is presented as a flowchart, it should be understood that it also provides a block diagram of the corresponding apparatus. Similarly, when a diagram is presented as a block diagram, it should be understood that it also provides a flowchart of the corresponding method / process.

[0209] Some embodiments mention rate-distortion optimization. Specifically, during the encoding process, a balance or trade-off between rate and distortion is typically considered, usually with constraints on computational complexity. Rate-distortion optimization is generally formulated as minimizing a rate-distortion function, which is a weighted sum of rate and distortion. Different approaches exist to address the rate-distortion optimization problem. For example, these approaches can be based on extensive testing of all encoding options, including all considered modes or encoding parameter values, and a complete evaluation of their encoding costs and the associated distortion of the reconstructed signal after encoding and decoding. Faster approaches can also be used to save encoding complexity, particularly by calculating approximate distortion based on predicting or predicting the residual signal rather than the reconstructed signal. A hybrid of these approaches can also be used, such as using approximate distortion only for some possible encoding options and full distortion for others. Other approaches evaluate only a subset of possible encoding options. More generally, many approaches employ multiple techniques to perform optimization, but optimization is not necessarily a complete evaluation of both encoding costs and associated distortion.

[0210] The implementations and aspects described herein can be implemented, for example, in methods or processes, apparatuses, software programs, data streams, or signals. Even if discussed only in the context of a single implementation (e.g., discussed only as a method), implementations of the discussed features can also be implemented in other forms (e.g., apparatuses or programs). Apparatuses can be implemented, for example, in suitable hardware, software, and firmware. Methods can be implemented, for example, in a processor, where processor generally refers to a processing device, including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices, such as, for example, computers, mobile phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate information communication between end users.

[0211] References to "an embodiment," "an example," "an implementation," or "an implementation," as well as other variations, mean that a particular feature, structure, characteristic, etc., described in connection with that embodiment is included in at least one embodiment. Therefore, the phrases "in one embodiment," "in an embodiment," "in one implementation," or "in one implementation," and any other variations appearing in various places in this application do not necessarily refer to the same embodiment.

[0212] Furthermore, this application may refer to "determining" various types of information. Determining information may include, for example, one or more of the following: estimated information, calculated information, predicted information, or information retrieved from memory.

[0213] Furthermore, this application may refer to "accessing" various types of information. Accessing information may include, for example, receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information, or one or more of these.

[0214] Furthermore, this application may refer to "receiving" various types of information. Like "accessing," "receiving" is intended to be a broad term. Receiving information may, for example, include accessing information or retrieving information (e.g., from memory) or one or more of it. Further, "receiving" generally refers in some way to operations such as, for example, storing information, processing information, sending information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0215] It should be understood that the use of any " / ", "and / or", and "...at least one of", such as in "A / B", "A and / or B", and "at least one of A and B", is intended to cover selecting only the first listed option (A), or only the second listed option (B), or selecting both options (A and B). As a further example, in the cases of "A, B, and / or C" and "at least one of A, B, and C", such wording is intended to cover selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or selecting all three options (A, B, and C). As will be apparent to those skilled in the art and related fields, this can be extended to as many items as possible listed.

[0216] Furthermore, as used herein, the term "signal" refers, among other things, to instructing the corresponding decoder to do something. In this way, in embodiments, the same parameters are used on both the encoder and decoder sides. Thus, for example, the encoder can send (explicit signaling) a specific parameter to the decoder so that the decoder can use the same specific parameter. Conversely, if the decoder already has that specific parameter as well as other parameters, signaling can be used without transmission (implicit signaling) to simply allow the decoder to know and select that specific parameter. Bit savings are achieved in various embodiments by avoiding the transmission of any actual functionality. It should be understood that signaling can be implemented in various ways. For example, in various embodiments, one or more syntax elements, flags, etc., are used to signal information to the corresponding decoder. While the foregoing refers to the verb form of the term "signal," the term "signal" can also be used as a noun herein.

[0217] As will be apparent to those skilled in the art, implementations can generate signals of various formats to carry information that can, for example, be stored or transmitted. This information may include, for example, instructions for performing a method, or data generated by one of the described implementations. For example, the signal may be formatted to carry a bitstream of the described embodiments. Such a signal may be formatted, for example, as an electromagnetic wave (e.g., using a portion of the radio frequency spectrum) or as a baseband signal. Formatting may, for example, include encoding the data stream and modulating a carrier wave using the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is known, the signal can be transmitted via a variety of different wired or wireless links. The signal may be stored on a processor-readable medium.

[0218] Several embodiments have been described above. The features of these embodiments may be provided individually or in any combination across various claim classes and types.

Claims

1. A method comprising: Obtain reconstructed samples of images from the video. Based on the reconstructed samples, a joint adaptive filter set comprising at least one loop filter and at least one output filter is determined. The at least one loop filter is used to provide a first version of the image that will be used as a reference image, and the at least one output filter is used to provide a second version of the image that will be used as an output image by the decoder.

2. An apparatus comprising one or more processors, said one or more processors being operable to: Obtain reconstructed samples of images from the video. Based on the reconstructed samples, a joint adaptive filter set comprising at least one loop filter and at least one output filter is determined. The at least one loop filter is used to provide a first version of the image that will be used as a reference image, and the at least one output filter is used to provide a second version of the image that will be used as an output image by the decoder.

3. The method of claim 1, or the apparatus of claim 2, wherein the parameters of the at least one loop filter and the at least one output filter are jointly determined using the reconstructed samples and using the predicted target samples for the at least one loop filter and the output target samples of the image for the at least one output filter.

4. The method of claim 1 or 3, or the apparatus of claim 2 or 3, wherein at least a portion of the determined joint adaptive filter set is transmitted in one of the dedicated adaptive parameter set and the SEI message.

5. The method or apparatus of claim 3, wherein determining the joint adaptive filter set comprises: Determine at least one candidate set of loop filters. Determine at least one candidate set of output filters. The joint adaptive filter set is determined by selecting a first candidate set from the at least one loop filter candidate set and a second candidate set from the at least one output filter candidate set.

6. The method or apparatus of claim 5, wherein the selection of the first candidate set and the second candidate set is performed based on the same rate distortion criterion using the same Lagrange parameters, or based on different rate distortion criteria using the same Lagrange parameters, or based on the same rate distortion criterion using different Lagrange parameters, or based on different rate distortion criteria using different Lagrange parameters.

7. The method or apparatus of claim 6, wherein the Lagrange parameter used in the rate-distortion criterion for selecting the first candidate set from the at least one loop filter candidate set depends on the quantization parameter used for the image.

8. The method or apparatus of claim 6 or 7, wherein the Lagrange parameter used in the rate-distortion criterion for selecting the second candidate set from the at least one output filter candidate set depends on the underlying quantization parameter.

9. The method or apparatus of any one of claims 5 to 8, wherein the at least one loop filter candidate set and the at least one output filter candidate set are determined independently.

10. The method or apparatus of claim 9, wherein in response to determining that the first candidate set includes more than one loop filter, the first candidate set being associated with a first classification graph; and in response to determining that the second candidate set includes more than one output filter, the second candidate set being associated with a second classification graph, the first classification graph and the second classification graph being different.

11. The method or apparatus of claim 5, wherein determining one of the at least one loop filter candidate set and the at least one output filter candidate set is constrained by a candidate set selected from the other of the at least one loop filter candidate set and the at least one output filter candidate set.

12. The method or apparatus of claim 11, wherein the second candidate set comprises the same number of output filters as the number of loop filters in the first candidate set, and in response to determining that the first candidate set or the second candidate set comprises more than one loop filter, the same classification map is associated with the first candidate set and the second candidate set.

13. The method or apparatus of any one of claims 5 to 12, wherein the joint adaptive filter set is signaled in the bitstream in response to determining that the joint adaptive filter set satisfies the rate distortion criterion.

14. The method or apparatus of claim 4 or 13, wherein at least one of the loop filter and the output filter of the joint adaptive filter set is encoded by prediction from the other of the output filter and the loop filter of the joint adaptive filter set.

15. The method or apparatus of claim 14, wherein the predicted encoding includes at least one of coefficient prediction and amplitude limiting index prediction.

16. The method or apparatus of any one of claims 3 to 15, wherein the predicted target sample is obtained from a denoised image, said denoised image being obtained from at least one of motion-compensated temporal filtering of the original image and spatial denoising of said original image.

17. The method or apparatus of any one of claims 3 to 16, wherein the output target sample is obtained from an original image or from another denoised image, said other denoised image being obtained from at least one of motion-compensated temporal filtering of the original image and spatial denoising of the original image.

18. The method or apparatus of any one of claims 11 to 17, wherein at least one tap of the output filter of the joint adaptive filter set uses the output of the loop filter of the joint adaptive filter set.

19. The method or apparatus of claim 18, wherein the at least one output filter set is determined to use samples obtained from the output of filtering the reconstructed samples with the loop filter of the joint adaptive filter set.

20. The method of any one of claims 1 or 3 to 19, further comprising, or the means of any one of claims 2 to 17, wherein the one or more processors are further configured to encode data representing the image and store a reference image obtained from the reconstructed sample using a loop filter from a determined set of joint adaptive filters in a buffer.

21. A method comprising: Decoding the encoded data representing the image yields a reconstructed sample of the image. One or more loop filters are used to obtain a reference image from the reconstructed samples. One or more output filters are used to obtain the output image from at least the reconstructed samples. The one or more loop filters and the one or more output filters are obtained from a joint adaptive filter set.

22. An apparatus comprising one or more processors, said one or more processors being operable to: Decoding the encoded data representing the image yields a reconstructed sample of the image. One or more loop filters are used to obtain a reference image from the reconstructed samples. One or more output filters are used to obtain the output image from at least the reconstructed samples. The one or more loop filters and the one or more output filters are obtained from a joint adaptive filter set.

23. The method of claim 21, or the apparatus of claim 22, wherein the joint adaptive filter set is decoded from the bitstream.

24. The method or apparatus of claim 23, wherein one of the at least one loop filter in the joint adaptive filter set and the at least one output filter in the joint adaptive filter set is decoded using a prediction from the other of the at least one loop filter and the at least one output filter in the joint adaptive filter set.

25. The method or apparatus of claim 24, wherein the prediction includes at least one of coefficient prediction and amplitude limiting index prediction.

26. The method of any one of claims 21 or 23 to 25, or the apparatus of any one of claims 22 to 25, wherein the joint adaptive filter set comprises a first number of the one or more loop filters and a second number of the one or more output filters, and in response to determining that the first number and the second number are different, the method further comprises, or the one or more processors are further configured to, determine a first classification map associated with the one or more loop filters and a second classification map associated with the one or more output filters, the first classification map and the second classification map being different.

27. The method of any one of claims 21 or 23 to 25, or the apparatus of any one of claims 22 to 25, wherein in response to determining that the joint adaptive filter set comprises the same number of loop filters and output filters, the method further comprises, or the one or more processors are further configured to determine the same classification map associated with the one or more loop filters and the one or more output filters.

28. The method of any one of claims 21 or 23 to 27, or the apparatus of any one of claims 22 to 27, wherein at least one tap of at least one of the one or more output filters uses the output of one or more loop filters of the joint adaptive filter set.

29. The method of any one of claims 21 or 23 to 28, or the apparatus of any one of claims 22 to 28, wherein for at least one sample to be filtered, the method includes or the one or more processors are configured to obtain one of the one or more output filters, wherein the obtained output filter uses a reconstructed sample adjacent to the sample to be filtered and at least one of the following: one or more samples output by a pre-trained filter applied to the reconstructed sample, a sample obtained before the application of a deblocking filter, a decoding residual, and a filtered version of the decoding residual.

30. The method of claim 1, 21 or 28, or the apparatus of claim 2, 22 or 28, wherein the reconstructed sample is obtained as the output of one of deblocking filtering and sample adaptive offset filtering.

31. The method of claim 1 or 21, or the apparatus of claim 2 or 22, wherein the joint adaptive filter set is provided as a neural network, the neural network taking at least the reconstructed samples as input and providing the reference image as a first output and the output image as a second output.

32. The method of claim 1, 21, or 31, or the apparatus of claim 2, 22, or 31, wherein the neural network further takes at least one of a prediction sample, quantization parameter information, and a type of block for encoding the image as input.

33. The method of any one of claims 1, 21, or 31 to 32, or the apparatus of any one of claims 2, 22, or 31 to 32, wherein the neural network comprises the same processing path and two output branches, wherein one of the two output branches is a first tail portion comprising one or more layers and outputting the reference image, and the other of the two output branches is a second tail portion comprising one or more layers and outputting the output image.

34. The method or apparatus of claim 33, wherein the neural network comprises: One or more head portions, each of which independently processes one input to the neural network. Furthermore, the common processing path includes a merging layer that aggregates the outputs of one or more head portions and a backbone portion that processes the outputs of the merging layer, the outputs of which are provided as inputs to the first tail portion and the second tail portion.

35. The method of any one of claims 1, 21, or 31 to 34, or the apparatus of any one of claims 2, 22, or 31 to 34, wherein the neural network is trained using a combination of a first loss based on original samples of the image and samples of the output image, and a second loss based on denoised samples of the image and samples of the reference image.

36. The method or apparatus of claim 35, wherein the combination uses weights applied to the second loss, and wherein the weights are provided as input to the neural network.

37. The method or apparatus of claim 36, wherein the weight depends on the temporal level of the image.

38. A signal comprising encoded data representing a block of video, wherein the signal further comprises data representing a joint adaptive filter set, the joint adaptive filter set comprising one or more loop filters and one or more output filters.

39. A non-transitory computer-readable medium storing encoded data representing blocks of video, wherein the signal further includes data representing a joint adaptive filter set, the joint adaptive filter set including one or more loop filters and one or more output filters.

40. The signal of claim 38, or the non-transitory computer-readable medium of claim 39, wherein at least a portion of the joint adaptive filter set is transmitted in one of a dedicated adaptive parameter set and an SEI message.

41. A computer program product comprising instructions for causing one or more processors to perform the method as described in any one of claims 1 to 21 or 23 to 37.

42. A non-transitory computer-readable medium storing executable program instructions to cause a computer executing the program instructions to perform the method according to any one of claims 1 to 3 to 21 or 23 to 37.

43. An apparatus comprising: The apparatus according to any one of claims 22 to 37; as well as At least one of the following: (i) an antenna configured to receive or transmit a signal including data representing video; (ii) a band limiter configured to limit the signal to a frequency band including data representing video; and (iii) a display configured to display the video.

44. The device of claim 43, wherein the device includes at least one of a television, a mobile phone, a tablet computer, and a set-top box.

Images