Neural Network-Based Filters in Video Coding

Abstract

In one implementation, only a single offset parameter is signaled in the bitstream to perform in-loop filtering of the reconstructed sample versions of the block. A neural network is used to generate pixel-wise weight masks based on the reconstructed sample versions. Because the neural network parameters are known in both the encoder and the decoder, these parameters do not need to be signaled in the bitstream. A single offset parameter scaled by the weighted mask is used to adjust the samples in the block. Thus, even if only a single offset parameter is used, the samples are adjusted by a pixel-wise offset. The neural network may also take other parameters, such as a quantization parameter and a picture type, as input. Furthermore, there may be multiple neural networks that generate different weight masks, where different offsets are signaled and one or more of the neural networks are selected for filtering.

Description

[Technical Field]

[0001] The present embodiments generally relate to methods and apparatus for filtering in video encoding or decoding. [Background technology]

[0002] To achieve high compression efficiency, image and video coding schemes typically employ prediction and transformation to exploit spatial and temporal redundancy within video content. Generally, intra- or inter-prediction is used to exploit intra- or inter-picture correlation, and then the difference between the original block and the predicted block, often called the prediction error or prediction residual, is transformed, quantized, and entropy coded. To reconstruct the video, the compressed data is decoded by the inverse process corresponding to entropy coding, quantization, transformation, and prediction. Summary of the Invention

[0003] According to one embodiment, there is provided a method of video encoding or decoding, the method comprising: accessing a version of reconstructed samples of a region of a picture; generating a weight for each sample of a plurality of samples in the region based on the reconstructed version of samples of the region using a neural network; encoding or decoding a single offset parameter for the region; and filtering the region by adjusting the plurality of samples in the region, wherein the samples in the region are adjusted in response to the weight for the sample and the offset for the region.

[0004] According to another embodiment, there is provided an apparatus for video encoding or decoding, comprising one or more processors configured to: access a version of reconstructed samples of a region of a picture; generate a weight for each sample of a plurality of samples in the region based on the version of reconstructed samples of the region using a neural network; encode or decode a single offset parameter for the region; and filter the region by adjusting the plurality of samples in the region, wherein the samples in the region are adjusted in response to the weight for the sample and the offset for the region.

[0005] According to another embodiment, there is provided an apparatus for video encoding or decoding comprising: means for accessing versions of reconstructed samples of a region of a picture; means for generating weights for each sample of a plurality of samples in the region based on the reconstructed version of samples of the region using a neural network; means for encoding or decoding a single offset parameter for the region; and means for filtering the region by adjusting the plurality of samples in the region, wherein the samples in the region are adjusted in response to the weights for the samples and the offset for the region.

[0006] According to another embodiment, there is provided an apparatus for video encoding or decoding comprising: means for accessing versions of reconstructed samples of a region of a picture; means for generating, using a plurality of neural networks, a plurality of weights for one sample of a plurality of samples in the region based on the reconstructed version of samples of the region; means for encoding or decoding a plurality of offset parameters for the region; and means for filtering the region by adjusting the plurality of samples in the region, wherein the samples in the region are adjusted in response to the plurality of weights for the sample and the plurality of offsets for the region.

[0007] One or more embodiments also provide a computer program comprising instructions that, when executed by one or more processors, cause the one or more processors to perform an encoding or decoding method according to any of the above-described embodiments. One or more of the present embodiments also provide a computer-readable storage medium having stored thereon instructions for encoding or decoding video data according to the above-described methods. One or more embodiments also provide a computer-readable storage medium having stored thereon a bitstream generated according to the above-described methods. One or more embodiments also provide methods and apparatus for transmitting or receiving a bitstream generated according to the above-described methods. [Brief explanation of the drawings]

[0008] [Figure 1] 1 shows a block diagram of a system in which aspects of the present embodiments may be implemented; [Figure 2] 1 shows a block diagram of one embodiment of a video encoder. [Figure 3] 1 shows a block diagram of one embodiment of a video decoder. [Figure 4] 1 shows an example of continuous loop filtering. [Figure 5]1 is a diagrammatic example showing four 1-D directional patterns for EO (Edge Offset) sample classification. [Figure 6] 1 is a diagrammatic example showing that positive offsets are used for categories 1 and 2 of the EO classification, and negative offsets are used for categories 3 and 4. [Figure 7] 1 is a graphical example showing a BO (Band Offset) with associated starting band position and offsets of four consecutive bands. [Figure 8] 1 illustrates an exemplary method for decoding a bitstream using SAO. [Figure 9] Here we present an example of using a Convolutional Neural Network (CNN) to restore an image after reconstruction. [Figure 10] 1 illustrates an encoder architecture according to one embodiment. [Figure 11] Shown is a portion of the picture to be filtered, the weight mask from the NN, and the filter result. [Figure 12] 1 illustrates an example of a NN used in a filter, according to one embodiment. [Figure 13] 1 illustrates a decoder architecture according to one embodiment. [Figure 14] 1 illustrates an encoder architecture with pre-classification as input to a neural network, according to one embodiment. [Figure 15] 1 illustrates an encoder architecture having multiple neural network-based filters, according to one embodiment. [Figure 16] 10 illustrates an encoder architecture having multiple neural network-based filters according to another embodiment. [Figure 17] 10 illustrates an encoder architecture having multiple neural network-based filters according to yet another embodiment. [Figure 18]1 illustrates a decoder architecture for selecting one CNN among multiple CNNs, according to one embodiment. [Figure 19] 10 illustrates a decoder architecture for selecting CNNs from among multiple CNNs according to another embodiment. [Figure 20] 1 illustrates an encoding process using multiple NNs for correction, according to one embodiment. [Figure 21] An example of a linear combination of NN outputs is shown below. [Figure 22] 1 illustrates a decoding process using multiple NNs for correction, according to one embodiment. [Figure 23] 1 illustrates a method for selecting K NNs to be combined, according to one embodiment. [Figure 24] Examples of spatial division with K=2 and K=4 are shown, where different patterns are associated with different NN filters. [Figure 25] 1 shows that the number of NNs actually used depends on the partitioning shape, according to one embodiment. [Figure 26] Here is an example of applying a 3x4 convolutional layer in only one direction. [Figure 27] 1 illustrates an example of training a NN based on a dataset with different coding mode features, according to one embodiment. [Figure 28] Two examples of activation functions are shown: ReLU and leaky ReLU. DETAILED DESCRIPTION OF THE INVENTION

[0009] FIG. 1 illustrates a block diagram of an example system in which various aspects and embodiments may be implemented. System 100 may be embodied as a device including various components described below and configured to perform one or more of the aspects described herein. Examples of such devices include, but are not limited to, various electronic devices, such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. Elements of system 100, singly or in combination, may be embodied in a single integrated circuit, multiple ICs, and / or separate components. For example, in at least one embodiment, the processing elements and encoder / decoder elements of system 100 are distributed across multiple ICs and / or separate components. In various embodiments, system 100 is communicatively coupled to other systems or other electronic devices, for example, via a communication bus or through dedicated input and / or output ports. In various embodiments, system 100 is configured to implement one or more of the aspects described herein.

[0010] System 100 includes at least one processor 110 configured to execute instructions loaded therein, for example, to implement various aspects described herein. Processor 110 may include embedded memory, input / output interfaces, and various other circuits as known in the art. System 100 includes at least one memory 120 (e.g., a volatile memory device and / or a non-volatile memory device). System 100 includes storage device 140, which may include non-volatile and / or volatile memory, including, but not limited to, EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic disk drives, and / or optical disk drives. Storage device 140 may include, by way of non-limiting example, an internal storage device, an attached storage device, and / or a network-accessible storage device.

[0011] System 100 includes, for example, an encoder / decoder module 130 configured to process data to provide encoded or decoded video, which may include its own processor and memory. Encoder / decoder module 130 represents a module that may be included within a device to perform encoding and / or decoding functions. As is known, a device may include one or both of an encoding and a decoding module. Additionally, encoder / decoder module 130 may be implemented as a separate element of system 100 or may be incorporated within processor 110 as a combination of hardware and software, as known to those skilled in the art.

[0012] Program code loaded onto the processor 110 or the encoder / decoder 130 to perform various aspects described herein may be stored in the storage device 140 and then loaded onto the memory 120 for execution by the processor 110. According to various embodiments, one or more of the processor 110, the memory 120, the storage device 140, and the encoder / decoder module 130 may store one or more of various items during the execution of the processes described herein. Such stored items may include, but are not limited to, input video, decoded video, or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, expressions, operations, and operational logic.

[0013] In some embodiments, memory internal to the processor 110 and / or the encoder / decoder module 130 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be either the processor 110 or the encoder / decoder module 130) is used for one or more of these functions. The external memory may be memory 120 and / or storage device 140, such as dynamic volatile memory and / or non-volatile flash memory. In some embodiments, external non-volatile flash memory is used to store the television's operating system. In at least one embodiment, high-speed external dynamic volatile memory, such as RAM, is used as working memory for video coding and decoding operations, such as MPEG-2, HEVC, or VVC.

[0014] Input to the elements of system 100 may be provided through various input devices, as shown in block 105. Such input devices may include, but are not limited to, (i) an RF section that receives RF signals transmitted over the air by, for example, a broadcast station, (ii) a composite input, (iii) a USB input, and / or (iv) an HDMI input.

[0015] In various embodiments, the input devices of block 105 have associated respective input processing elements, as known in the art. For example, the RF section may be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal or band-limiting a signal to a frequency band), (ii) downconverting the selected signal, (iii) band-limiting again to a narrower frequency band to select a signal frequency band, which in particular embodiments may be referred to (for example) as a channel, (iv) demodulating the downconverted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired stream of data packets. The RF section of various embodiments includes one or more elements that perform these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a downconverter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner that performs these various functions, including, for example, downconverting received signals to a lower frequency (e.g., an intermediate frequency or a near-baseband frequency) or to baseband. In one embodiment of a set-top box, the RF section and its associated input processing elements receive RF signals transmitted over a wired (e.g., cable) medium and perform frequency selection by filtering, downconverting, and re-filtering to a desired frequency band. Various embodiments rearrange the order of the above-described (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions. Adding elements may include inserting elements between existing elements, for example, inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0016] Additionally, the USB and / or HDMI terminals may include respective interface processors for connecting system 100 to other electronic devices over USB and / or HDMI connections. It should be understood that various aspects of the input processing, e.g., Reed-Solomon error correction, may be implemented, for example, within a separate input processing IC or within processor 110, as desired. Similarly, aspects of the USB or HDMI interface processing may be implemented, as desired, within a separate interface IC or within processor 110. The demodulated, error corrected, and demultiplexed stream is provided to various processing elements, including, for example, processor 110 and encoder / decoder 130, which operate in combination with memory and storage elements to process the data stream as desired for presentation on an output device.

[0017] The various elements of system 100 may be provided within an integrated housing in which the various elements may be interconnected and transmit data between each other using suitable connection arrangements 115, e.g., internal buses known in the art, including I2C buses, wiring, and printed circuit boards.

[0018] System 100 includes a communication interface 150 that enables communication with other devices over a communication channel 190. Communication interface 150 may include, but is not limited to, a transceiver configured to transmit and receive data over communication channel 190. Communication interface 150 may include, but is not limited to, a modem or a network card, and communication channel 190 may be implemented in a wired and / or wireless medium, for example.

[0019] In various embodiments, data is streamed to system 100 using a Wi-Fi network, such as IEEE 802.11. The Wi-Fi signal in these embodiments is received via communication channel 190 and communication interface 150 adapted for Wi-Fi communication. Communication channel 190 in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to enable streaming applications and other over-the-top communications. In other embodiments, streaming data is provided to system 100 using a set-top box that delivers data via an HDMI connection in input block 105. In yet other embodiments, streaming data is provided to system 100 using an RF connection in input block 105.

[0020] System 100 may provide output signals to various output devices, including display 165, speakers 175, and other peripheral devices 185. Other peripheral devices 185, in various example embodiments, include one or more of a standalone DVR, a disc player, a stereo system, a lighting system, and other devices that provide functionality based on the output of system 100. In various embodiments, control signals are communicated between system 100 and display 165, speakers 175, or other peripheral devices 185 using signaling such as AV.Link, CEC, or other communication protocols that enable inter-device control with or without user intervention. Output devices may be communicatively coupled to system 100 via dedicated connections through respective interfaces 160, 170, and 180. Alternatively, output devices may be connected to system 100 via communication interface 150 using communication channel 190. Display 165 and speakers 175 may be integrated into a single unit with other components of system 100, for example, in an electronic device such as a television. In various embodiments, display interface 160 includes a display driver, for example, a timing controller (TCon) chip.

[0021] Display 165 and speakers 175 may alternatively be separate from one or more of the other components, for example, if the RF portion of input 105 is part of a separate set-top box. In various embodiments in which display 165 and speakers 175 are external components, the output signal may be provided via a dedicated output connection, including, for example, an HDMI port, a USB port, or a COMP output.

[0022] 2 shows an example video encoder 200, such as a High Efficiency Video Coding (HEVC) encoder. FIG. 2 may also show an encoder that improves on the HEVC standard or employs technology similar to HEVC, such as the Versatile Video Coding (VVC) encoder under development by the Joint Video Exploration Team (JVET).

[0023] In this application, the terms "reconstructed" and "decoded" may be used interchangeably, the terms "encoded" and "coded" may be used interchangeably, the terms "pixel" or "sample" may be used interchangeably, and the terms "image," "picture," and "frame" may be used interchangeably. Usually, but not necessarily, the term "reconstructed" is used on the encoder side, and the term "decoded" is used on the decoder side.

[0024] Before being encoded, a video sequence may undergo encoding pre-processing (201), such as applying a color transformation to the input color picture (e.g., converting from RGB 4:4:4 to YCbCr 4:2:0) or performing a remapping of the input picture components to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization of one of the color components). Metadata may be associated with the pre-processing and attached to the bitstream.

[0025] In the encoder 200, a picture is coded by the encoder elements, as described below. The picture to be coded is divided (202) into units, e.g., CUs, and processed. Each unit is coded, e.g., using either intra mode or inter mode. When a unit is coded in intra mode, it performs intra prediction (260). In inter mode, motion estimation (275) and motion compensation (270) are performed. The encoder determines (205) whether to use intra mode or inter mode to code the unit, and indicates the intra / inter decision, e.g., via a prediction mode flag. The encoder may also mix (263) intra and inter prediction results, or mix results from different intra / inter prediction methods.

[0026] The prediction residual is calculated (210), for example, by subtracting the predicted block from the original image block. The motion refinement module (272) uses already available reference pictures to refine the motion field of a block without referring to the original block. The motion field for a region can be viewed as the collection of motion vectors for all pixels comprising that region. If the motion vectors are subblock-based, the motion field can also be represented as the collection of all subblock motion vectors within the region (all pixels within a subblock have the same motion vector, and the motion vector can be different for each subblock). If a single motion vector is used for a region, the motion field for the region can also be represented by a single motion vector (the same motion vector for all pixels within the region).

[0027] The prediction residual is then transformed (225) and quantized (230). The quantized transform coefficients, as well as motion vectors and other syntax elements, are entropy coded (245) to output a bitstream. The encoder can skip the transform and apply quantization directly to the untransformed residual signal. The encoder can bypass both the transform and quantization, i.e., the residual is coded directly without applying the transform or quantization processes.

[0028] The encoder decodes the coded block to provide a reference for further prediction. The quantized transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residual. The decoded prediction residual is combined with the predicted block (255) to reconstruct an image block. An in-loop filter (265) is applied to the reconstructed picture to perform, for example, deblocking / sample adaptive offset (SAO) filtering to reduce coding artifacts. The filtered image is stored in a reference picture buffer (280).

[0029] 3 shows a block diagram of an exemplary video decoder 300. In the decoder 300, the bitstream is decoded by decoder elements as described below. The video decoder 300 generally performs a decoding pass that is the inverse of the encoding pass, as described in FIG. 2. The encoder 200 also generally performs video decoding as part of encoding the video data.

[0030] In particular, the decoder's input includes a video bitstream, which may be generated by video encoder 200. The bitstream is first entropy decoded (330) to obtain transform coefficients, motion vectors, and other coding information. Picture partition information indicates how the picture is partitioned. The decoder may then partition the picture according to the decoded picture partition information (335). The transform coefficients are dequantized (340) and inverse transformed (350) to decode the prediction residual. The decoded prediction residual and the predicted block are combined (355) to reconstruct an image block.

[0031] The predicted block may be obtained (370) from intra prediction (360) or motion-compensated prediction (i.e., inter prediction) (375). The decoder may blend (373) intra and inter prediction results, or may blend results from multiple intra / inter prediction methods. Before motion compensation, the motion field may be refined (372) by using already available reference pictures. An in-loop filter (365) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (380).

[0032] The decoded picture may further undergo post-decoding processing (385), such as an inverse color conversion (e.g., YCbCr 4:2:0 to RGB 4:4:4 conversion) or an inverse remapping that performs the inverse of the remapping process performed in the pre-encoding processing (201). The post-decoding processing may use metadata derived in the pre-encoding processing and signaled in the bitstream.

[0033] As explained above, in modern video codecs such as HEVC or VVC, several filters are applied to the reconstructed samples of a video picture with the aim of reducing coding artifacts and reducing distortion of the original picture. For example, in HEVC, a deblocking filter (DBF) and a sample adaptive offset (SAO) filter are applied successively to the reconstructed samples. In VVC, another filter called an adaptive loop filter (ALF) is applied at the very end of the process. During the development stage of VVC, several other complementary block-based filters were considered: a bilateral filter (BF), a Hadamard filter, and a diffusion filter.

[0034] Figure 4 shows an example of successive loop filtering. In this example, four successive filters are applied: a bilateral filter (450), a DBF (460), a SAO (470), and an ALF (480). The output is the reconstructed picture samples. For simplicity of notation, we refer to the input to the encoder as S and the input to the in-loop filtering as

number

number

number

[0035] These different filters generally perform some or all of the following functions: 1. Pixel classification. 2. Encoder only: Determining filter parameters (e.g. DBF, SAO, ALF but not BF). 3. Encoding / decoding filter parameters (e.g., DBF, SAO, ALF, but not BF). 4. Class-dependent filtering.

[0036] SAO filter

[0037] In HEVC and VVC, SAO is a sample-based filtering operation on a CTU (Coding Tree Unit) basis, which allows adding offsets to some categories of reconstructed samples to reduce coding artifacts. A CTU consists of one CTB per component. SAO can be activated or deactivated for each CTB. Two SAO modes are specified: Edge Offset (EO) and Band Offset (BO). For EO, ​​sample classification is based on the local directional structure in the picture being filtered. For BO, sample classification is based on the sample value. Parameters for EO or BO can be explicitly coded or derived from a neighborhood. SAO can be applied to the luma and chroma components, where the SAO mode is the same for the Cb and Cr components. SAO parameters are configured separately for each color component.

[0038] Edge Offset

[0039] EO uses four 1-D orientation patterns, namely horizontal, vertical, 135° diagonal, and 45° diagonal, as shown in Figure 5, for sample classification, where the label “p c " represents the current sample, and the labels "p0" and "p1" represent the two adjacent samples. Four EO classes are specified based on the direction, and each EO class corresponds to one direction. The selected EO class is signaled as side information in the bitstream. [Table 1]

[0040] For a given EO class, each sample in the CTB is classified into one of five (NC=5) categories based on its local gradient. cThe current sample value labeled "」" is compared with its two adjacent values along the selected 1-D direction. The classification rules for the samples are summarized in Table 1. Also, as shown in FIG. 6, categories 1 and 4 are respectively associated with local valleys and local mountains along the selected 1-D pattern, and categories 2 and 3 are respectively associated with concave corners and convex corners along the selected 1-D pattern. If the current sample value does not belong to categories 1 to 4 (i.e., the current sample value is the same as its two adjacent values, or p0 < p c <p1 or p0 > p c > p1), the current sample value is in category 0 and SAO is not applied (i.e., the offset is 0). For categories 1 to 4, the offset is encoded.

[0041] Band offset

[0042] In the case of BO, the range of sample values (e.g., 0 to 255 in 8 bits) is evenly divided into 32 bands, and the sample values belonging to (NC - 1) = 4 consecutive bands are corrected by adding the offsets off(n), n = 1,..., 4, as shown in FIG. 7. Only the offsets and the start band positions of the 4 consecutive bands are signaled to the decoder. The start band position indicates the position of the first band having the encoded offset. One offset is coded for each of the (NC - 1) bands, and the remaining bands have offsets equal to 0.

[0043] In the case of EO or BO, the offset is not coded, but can be copied from the adjacent CTU above or to the left (merge mode). In one example, FIG. 8 shows a method 800 for decoding a bitstream using SAO. After the bitstream is decoded (810), the reconstructed samples are further filtered using SAO. Note that the terms "decoded" or "reconstructed" may mean that the bitstream is partially "decoded," e.g., the signal obtained after deblocking filtering but before SAO filtering, and that the reconstructed samples may differ from the final decoded output used for display.

[0044] If SAO is activated for a slice, the decoder checks whether SAO is activated for the current CTB (820). If SAO is activated, the decoder classifies the samples in the CTB into N = 5 categories (830). An offset value off(n) for each category can be obtained (840). In step 850, for sample s with category c(s), the sample can be adjusted to s = s + off(c(s)). As explained above, the sample offset can be 0, and the corresponding sample is not adjusted. If there are other in-loop filters after the SAO filter, the filtered samples undergo further filtering. The filtered reconstructed samples are used as the final output of the decoder.

[0045] An encoder can perform a process similar to method 800. On the encoder side, step 810 is implemented to obtain reconstructed samples, for example, an SAO filtering process uses the reconstructed samples from the deblocking filter as input. The offset value is coded in the bitstream. The filtered reconstructed samples can be used as references for other pictures.

[0046] In the encoding stage, for each category c of each class, the offset can be determined by collecting the sum of the differences between the original (target) sample values ​​and the reconstructed sample values ​​diff(c). Denoting N(c) as the number of samples in the current block that belong to category c, the average offset off(c) added to the reconstructed samples can be derived as follows: off(c)=diff(c) / N(c) (Equation 1)

[0047] Deep Learning Filter

[0048] Figure 9 shows an example of using a convolutional neural network (CNN) to restore an image after reconstruction. An image S is first

number

number

[0049] In practice, a CNN usually consists of several convolutional layers followed by an activation layer (e.g., a sigmoid function, a ReLU function, or a leaky ReLU function), and the loss function may also consider several regularization terms to stabilize the CNN training.

[0050] Several techniques have been developed for filtering reconstructed pictures with CNNs. The main difference between CNNs and fully connected neural networks is that CNNs share weights along the spatial dimensions of their input and their intermediate representations. Given the local statistics of a natural image, CNNs typically require far fewer parameters than fully connected neural networks to achieve comparable performance, for example, in image restoration. Furthermore, weight sharing makes the number of parameters in a CNN independent of its input size, meaning that a trained CNN can restore images of various sizes.

[0051] In one technique, a three-layer CNN is shared by the luma and chroma components. The CNN parameters are trained and then coded into the bitstream along with the first I picture of each random-access segment (RAS). Training uses only pictures of temporal levels 0 and 1. Multi-level on / off control is applied at the picture, coding tree block (CTB), and 32x32 block levels for each color component. In a variant, three different two-layer CNNs for luma and three different two-layer CNNs for chroma are coded. The index of the best CNN to use for each of luma and chroma is signaled per CTB. The CNNs are compressed to 6 bits per weight. This on / off control is done per tile.

[0052] These methods encode all neural network parameters in the bitstream, creating signaling overhead. Furthermore, the neural network must be trained per group of pictures (RAS), which can lead to complexity / latency that is unacceptable in practice.

[0053] In another technique, to overcome the aforementioned limitations, a larger but fixed set of neural network parameters is trained once, one for each QP. The three input components (Y, U, V) are concatenated to be processed together by the CNN. The input sample block is padded with pixels of a specific size corresponding to the total padding size of the CNN during training. The number of parameters can be further reduced by repeating several layers.

[0054] This application proposes an in-loop filter based on a neural network (NN) that can replace one or several existing in-loop filters or be added to existing in-loop filters. Because the proposed filter adjusts samples using adaptive offsets, as in the case of an HEVC or VVC SAO filter, the proposed filter is called an NN-based SAO (sample adaptive offset) filter. In one embodiment, the NN-based filter adjusts reconstructed samples by offsets, as is performed in an SAO filter. In particular, the NN filter determines a weight mask. A weight in this mask corresponds to either whether a sample of a reconstructed block is corrected or the strength of the correction of this sample, depending on the value of the weight. The NN filter is controlled by a small number of parameters (offsets) to control the strength of the filter. These parameters are coded in the bitstream.

[0055] NN-based in-loop filter architecture

[0056] Figure 10 shows an encoder architecture (1000) according to one embodiment, where W represents a weight mask. If the weights are binary, i.e., either 0 or 1, then the weight mask for index i is

number

number

[0057] When the first layer of a NN is a convolutional layer, the set of data

number

number

number

[0058] For a block S to be coded by the video encoder module (1010), the corresponding (locally) reconstructed block is

number

[0059] The filter control parameter "offset" is as follows:

number

number

[0060] In a variant, the filter control parameter "offset" is calculated using a mean square error minimization method as follows:

number

number

[0061] The value of "offset" is coded in the bitstream for each block (1050). In a variant, the value "offset" is quantized before coding. The operations "X" (1060) and "+" (1070) correspond to the multiplication and term-wise summation of all terms of W with the scalar value "offset", respectively. Since "offset" is used to scale the weight mask W, the offset may also be considered as a scaling parameter. In a variant, the value of W is clipped, for example between -1 and 1.

[0062] Figures 11(a), 11(b), and 11(c) show a portion of the initial reconstructed picture to be filtered, the corresponding weight mask from the NN, and the filtered result, respectively. Here, a six-layer CNN with ReLU activation and one final clipping layer is used, as shown in Figure 12. In Figure 11(b), different shading corresponds to different weight values. Alternatively, other activation functions can be used, such as leaky ReLU as shown in Figure 28 with an alpha parameter equal to 0.1. The leaky ReLU activation function has two advantages: first, it facilitates the convergence of the backpropagation algorithm and thus the NN training stage; and second, it allows negative weight mask values. A variant uses leaky ReLU in the inner layers and ReLU only in the final layer.

[0063] Using a block as an example, the pixel values ​​are as follows:

number

[0064] The weight mask is as follows:

number

[0065] The offset for the block calculated in (Equation 2) is 6. Therefore,

number

number

[0066] When using (Equation 2b), the offset to the block is 8. Therefore,

number

number

[0067] Figure 13 shows a decoder architecture (1300) according to one embodiment. The decoder's input includes a video bitstream, e.g., generated by the video encoder 1000. The video decoder module (1310) may correspond to the decoder 300, except for the in-loop filter (365), which is augmented or replaced by the proposed filter (1340). The NN (1320) should be the same as that used in the corresponding encoder to properly decode the bitstream. The input to the NN (1320) is the reconstructed block to be filtered, and the output is a weight mask W. The output of the NN filter (1320) may be a scaled offset (residual) to correct one or more components (one channel), e.g., luma and chroma residual samples (three channels) or two chroma residual samples (two channels), possibly along with other information.

[0068] Note that the parameters of the NN are known at the decoder and are not conveyed in the bitstream.

[0069] A filter control parameter "offset" is decoded from the bitstream for the block (1310). The control parameter is then multiplied by a weight mask (1360). That is, the control parameter is scaled by the weight for each sample in the block to generate a scaled offset for each sample. The scaled offset is then added to the corresponding sample in the initial reconstructed block (1370).

[0070] By generating a sample-by-sample weight mask W, the product W.offset provides the adjustment offset for each sample in the block. Note that since the parameters of the NN are not transmitted in the bitstream, only a single control parameter needs to be communicated for the block for the filtering process. Thus, with very little signaling overhead, the proposed filter achieves sample-by-sample adjustment in filtering, which can improve compression efficiency.

[0071] NN-based filter architecture with pre-classification

[0072] In this embodiment, the NN module has additional inputs, such as a quantization step (QP), an image type (e.g., type I, P, or B), and reconstructed residual samples or reconstructed samples from another component. In a variant, the additional input is a classification module (1420) that classifies the samples of the block as shown in Figure 14. In Figure 14, the module (1420) is shown with dashed lines to indicate that it is optional. Although a decoder is shown in Figure 14, the corresponding encoder can be modified accordingly. This classification (1420) can be based on local gradients or other semantic classification. For example, the classifier is the same as the classifier used in existing in-loop filters, such as the HEVC / VVC SAO, ALF classifier, or deblocking filter classifier. The classifier:

number

[0073] One advantage of using a pre-classifier input is that the number of layers in the NN (1430) can be reduced, since the purpose of the first layer is generally to perform classification. However, the use of an explicit a priori classifier can reduce the ability of the training to learn optimal classifications.

[0074] In-loop filter architecture with multiple neural networks

[0075] In this embodiment, several NN filters (1530, 1540) are used, as shown in Figure 15. In the encoder, k filters (k is set to 2 in Figure 15) are applied to

number

number

number

[0076] In another embodiment, as shown in FIG. 16, a single classifier is used (1620), whereby:

number

number

number

number

[0077] In the variant shown in FIG. 17 where k=2, a single offset is calculated for all NN filters.

[0078] Figure 18 shows a decoder architecture (1800) with multiple CNNs, according to one embodiment. The decoder's input includes a bitstream. The video decoder module (1810) may correspond to the decoder 300, except for the in-loop filter (365), which is augmented or replaced with a CNN-based filter (1870).

[0079] A filter control parameter "offset" and a filter index "b" are decoded from the bitstream for the block (1810). The filter index "b" controls which of the K CNNs (1820, 1830) should be used to generate the weight mask W (1840). The control parameter "offset" is then multiplied with the weight mask (1850). The scaled offset is then added to the corresponding sample in the initial reconstructed block (1860).

[0080] In Figure 18, the CNN is used without pre-classification. In other embodiments, a pre-classification module as illustrated for the encoder, for example in Figures 15-16, can be applied.

[0081] In Figure 18, the selection of the CNN to be used for a block and the control parameter "offset" allows the filtering process to be adapted to the local characteristics of the current block. However, the selection of a single CNN may not be optimal, as it may occur that two or more CNNs can both provide good correction, in which case it may be preferable to somehow accumulate the benefits of these two or more CNNs. This may also depend on how the CNNs were trained.

[0082] Below, we describe several methods for leveraging the power of CNNs by combining their outputs rather than selecting the best one. In one example, assume there are several (N) CNNs that may be trained separately using different datasets with different characteristics and / or encoded using different parameters (e.g., different QP values, different picture sizes, or sample resolutions). A dataset consisting of reconstructed samples from a normal bitstream can also be used to classify CTUs using the decoded SAO type.

[0083] Mutual connection of NN

[0084] Neural network ensembles are a learning paradigm in which multiple neural networks are used together to solve a problem. In this embodiment, we apply this general principle to NN-based filters via a weighted linear combination of several NN outputs.

[0085] Figure 19 shows a decoding process (1900) using multiple NNs for correction, according to one embodiment. The decoder uses K filter indices {i0,...i k-1} and K offsets {offset 0,...offset k-1} and}. In this example, K=2, which corresponds to NN(1920, 1930). The inputs to the K NNs are the reconstructed blocks

number

number

number

[0086] FIG. 20 shows an encoding process (2000) using multiple NNs for compensation, according to one embodiment. The encoder selects K NNs from N available NNs. In this example, K=2. More generally, this process can be applied when there are two or more NNs (2020, 2025) used in the filters. The inputs to the K NNs are the reconstructed block and possibly additional information such as QP, coding mode, reconstructed residuals or reconstructed samples of other components (2015). Note that the value of K can be different for luma or chroma NN-based filters, for example, if the luma and chroma do not share the same filter.

[0087] Given a reconstructed block

number

number

number

[0088] In one embodiment, the scaling value (offset k ) can be derived on the encoder side using a least squares minimization (LSM) of the mean squared error (MSE) as follows:

number

[0089] In the specific case of using two CNNs, K=0 or 1, the LSM method allows deriving the values ​​of offset0 and offset1 as follows:

number

number

number

number

[0090] In the following, we use K=2 as an example, but can easily extend to values ​​of K>2. Figure 22 shows a decoding process (2200) using multiple NNs for correction, according to one embodiment. Similar to method 2000, the initial reconstruction samples

number

[0091] Figure 23 shows a method (2300) for selecting K NNs to be combined for K = 2, given the current reconstruction block. Initially, bestCost is set to a large value. For each possible pair {k0, k1} (2320), if k0 = k1, NN(k0) is applied (2310) and one single offset0 is derived (2345). If k0 ≠ k1, NN(k1) is also applied (2310) and scaling parameters {offset0, offset1} are derived (2340). Correction coefficients are calculated (2350, 2355) to produce a corrected reconstruction block Rec'

number

[0092] Syntax Example

[0093] Table 2 provides an example of a syntax for coding syntax elements related to the various embodiments described above. In particular, Table 2 provides an example of a syntax for coding the index of a NN and the scaling parameters used to correct one reconstructed block using the mutual combination of the NN. In this example, the current block position in the picture is (rx, ry), where K=2 for the luma component and K=1 for the chroma component. [Table 2]

[0094] The meaning of these syntax elements is as follows: filter_luma_flag, filter_chroma_flag specify whether the luma or chroma sample block is corrected (using an NN filter), respectively. idx_filter_off_val_luma[k] is the K indices {k i} i=0,1 For example, k i =idx_filter_off_val_luma[i]. In one embodiment, k i The value of k i+1 ≧k i Before coding, they are ordered (e.g., in ascending order) so that k i+1 =k i By allowing the case where the number of NN filters actually used (cpt_scale_off) is less than K, it is possible to deal with the case where less than K scaling parameters are coded. The variable cpt_scale_off is a non-zero scaling parameter offset k corresponds to the number of In one example, the identifier indicates how many NNs are combined, e.g., if K=2, then k0=k1, allowing to manage the case where only one single NN is used (no mutual coupling) and only one scaling parameter is coded. In a variant, K indices {k i} i=0,1 is derived as follows:

number

number

number

number

[0095] Table 3a shows the results of using the proposed NN cross-coupling method for the Luma NN filter compared to an NN-based filter that does not combine the NN outputs. The proposed combination of NN outputs results in a bitrate reduction of approximately 0.89%, compared to a 0.57% bitrate reduction for the uncoupling method. Table 3b shows the results obtained with the proposed cross-coupling of NNs with another set of NNs. The results in Tables 3a and 3b (left) were obtained using a NN trained with a ReLU activation function. The results in Table 3b (right) were obtained using a NN trained with a leaky ReLU activation function. In this example, a bitrate reduction of approximately 2.45% is obtained using the leaky ReLU activation function, compared to a 1.60% bitrate reduction with the ReLU activation function. [Table 3] [Table 4]

[0096] Spatial coupling of NN

[0097] In another embodiment, several NNs are combined to correct the current reconstructed block using a spatial segmentation of the block into several (K) regions, where different NNs can be used for different block divisions. Figure 24 shows some examples of such spatial divisions with K=2 and K=4, where different patterns are associated with different NN filters. In these examples, one scaling parameter (offset) is used per division / region of the block. k ) is coded.

[0098] In a variant, several partitions within a block use the same NN, but possibly different scaling parameter values, as shown in Figure 24(f). At the decoding stage, this has the advantage of inferring one single NN, but adapting the scaling to each partition.

[0099] Table 4 provides an example of syntax elements related to this embodiment. In particular, Table 4 provides an example of syntax for coding the index of the NN, the partition shape (dir_split), and the scaling parameters used to correct one reconstructed block using the mutual combination of the NNs. In this example, the current block position in the picture is (rx, ry), and K=1 for the chroma components. For the luma component, the number of NNs actually used (cpt_scale_off) depends on the partition shape as shown in Figure 25. The meaning of the syntax elements is the same as in Table 2. The index or scaling parameter predictor may be the value of a previously decoded partition. [Table 5]

[0100] Table 5 shows the results of using the proposed spatial combination method of NNs compared to a NN-based filter that does not combine the NN outputs. [Table 6]

[0101] In one embodiment, how many NNs can be combined (K) may be signaled in the bitstream (e.g., slice header or picture header). Additionally, a set of N NNs may be signaled from a larger set of M NNs, where M>N. In a variant, the N NNs may be inferred from other parameters in the bitstream, such as the quantization parameter (QP), picture size, or the nature of the video (e.g., sports, games, movies). For example, M NNs may be signaled from M different QP values ​​{qp0,...,qp M-1 For example, if the decoder has been trained on a video sequence encoded with {, the decoder may infer N NNs from the current QP. For example, the subset may be created from NNs trained with some (e.g., two) QP values ​​below the current QP value and some (e.g., two) QP values ​​above the current QP value.

[0102] The previous embodiment can be extended to other correction models (formula for Corr(x)) and other models for applying the correction (formula for Rec'(x)). For example, the correction can be based on the following model: Correction model: - Linear correction model. The parameters are offset k , bias}

number

number

number

number

[0103] As mentioned before, the scaling parameter {offset k ,bias} is derived by, for example, setting the variable {offset} equal to 0. k This can be done by the encoder using the LSM method, which involves a system of K+1 equations obtained by partial differentiation of (Eq. 3) with respect to {k, bias}.

number

[0104] Region-based neural network filters (e.g., deblocking filters)

[0105] If the NN filter is a region-based filter (e.g., a deblocking filter) specialized in correcting some spatially located artifacts of known positions, the correction may be limited to these known positions (e.g., reconstructed CU edges). In this case, the NN filter inference may be a unidirectional scanning convolution as shown in FIG. 26 in the direction of the spatially known artifact positions (e.g., horizontal or vertical CU edges).

[0106] training

[0107] Training a NN (setting the NN parameters) can be done in a traditional way based on supervised learning, where the output of the NN matches the desired output (original signal) by trying to minimize a loss function, such as the difference between the NN output and the desired output. For example, backpropagation methods can be used, where the calculation of the gradient of the loss function with respect to the network weights allows iterative adjustment of the NN weights and biases, such as gradient descent or stochastic gradient descent.

[0108] In one embodiment, training of the NN parameters is performed by minimizing a loss using a gradient descent algorithm. The loss is called an "offset k " or "Offset k and b k " Distortion in coding costs

number

[0109] In the training phase, the dataset is a set of possible pairs of block patches.

number

number

[0110] Classification can be performed using a dataset created from blocks coded with a range of QPs using coding modes, or from blocks selected from only I-pictures, or only P- or B-pictures, as shown in Figure 27. That is, several NNs can be trained based on datasets with different coding mode characteristics. Training can also be performed in at least two passes. In a first pass, a set of NNs (NN-1) is trained (2740) using patches extracted (2730) from intra-pictures of the decoded bitstream B1 (2710, 2720). Then, a set of video sequences is coded (2750) using these NN-1 filters enabled only on intra-pictures to generate bitstream B2. In a second pass, patches can be extracted (2770) from inter-pictures of the decoded bitstream B2 (2760) to train another set of NNs (NN-2) dedicated to inter-picture filtering (2780).

[0111] If the purpose of the NN filter is to replace an existing filter (e.g., SAO, ALF...) used in the bitstream, the classification must be present in the bitstream.

number

number

[0112] However, the classification in the bitstream can be biased by the choice of encoder, which may be based on rate-distortion and other contextual considerations. For example, the coding cost (rate) depends on the CABAC context, which depends on the history of the CABAC encoder. For training purposes, it may be preferable to consider only distortion, rather than rate-cost. We consider CTUs coded only in mode NEW (discarding merge and OFF modes).

number

[0113] Preferably without using SAO parameters in the bitstream,

number

[0114] For a scheme with multiple NNs, fine-tuning can be performed iteratively as follows. 1) Step 1: The main dataset MD is divided into k datasets D b Divide into (b=1,...,k). 2) Train k NN filters separately. 3) Step 2: k datasets D bRegenerate (b = 1, ..., k) as follows: For each data "I" in MD, select the NN filter "t" that minimizes the loss, and add "i" to D t Next, retrain the k filters with D b (b = 1, ..., k). 4) Repeat 3).

[0115] However, it may happen that the number of data in one data set "x" becomes very small. To avoid this problem, the minimum number N of data in each data set can be determined and the following steps can be taken. 1) Step 1: Divide the main data set MD into k data sets D b (b = 1, ..., k). 2) Train the k NN filters separately. 3) Step 2: ○ For each data "i" in MD: ● Initialize the list of data set candidates for "i" to C(i) = {0, ...(k - 1)} ● Calculate the loss Lb(i) obtained with each filter b. ● "i" is marked as "not placed" ○ For each data "i" in MD: ○ While (the data "i" is marked as not placed in one data set): ● t = argmin k {L k (i)} (where k = t is the filter that minimizes L k (i) for k ∈ C(i)) ● If the cardinality {Dt < N}, add the data "i" to Dt ● Otherwise, if one data "d" in Dt has a loss Lt(d) > Lt(i), replace "d" with "i" in Dt, "d" becomes "not placed" (and is marked as such), and "i" is marked as "placed". ● Otherwise, remove "t" from the list C(i). 4) Retrain the k filters with Db Retrain with (b=1,...,k). 5) Repeat step 3).

[0116] In the above, the NN filtering process is performed block by block, since current video standards are usually block-based. However, this embodiment can be applied to regions with shapes other than rectangular or square, because the NN can be trained and implemented for other shapes or can be a fully convolutional network, and therefore is independent of region shape or size.

[0117] Legacy filters such as SAO or ALF in HEVC and VVC use a closed-loop approach in which an "optimal" correction is calculated, but the classification of pixels into filters is usually very simple. Previous CNN-based methods, on the other hand, use sophisticated classification but open-loop correction, where the correction is not necessarily beneficial. According to this embodiment, a deep neural network (NN) is provided to replace or complement the SAO filter and restore images after reconstruction by a video codec. In particular, the proposed filter leverages the power of CNNs to classify pixels to be corrected, while maintaining a correction "closed loop" by calculating the optimal correction to apply in the encoder. To generalize binary classification, the CNN can also calculate the amount of correction to apply to a particular pixel. This leverages the benefits of encoding parameters to control the filter operation, while requiring only a small amount of data to be coded in the bitstream.

[0118] In the proposed filters as shown in Figures 12 and 13, only a single offset is transmitted for a block, or as shown in Figure 14, more than one offset is transmitted for a block. That is, a very small amount of data is encoded to signal the offset. However, since the NN can generate a pixel-wise weight mask (the values ​​in the mask can be different for each pixel), the actual offset (weight * offset) to be applied to adjust the pixels in the block can be different for each pixel, thus achieving finer granularity than the SAO filters in HEVC and VVC at a lower signaling cost. The NN can also generate weights on a sub-block basis (the same weights within a sub-block, but the weights can be different for each sub-block within a block).

[0119] Also, the proposed NN filter may be applied only at some specific locations within a picture, for example, it may be used to specifically correct blocking artifacts that occur mainly near block boundaries and / or only at transform boundaries or only at prediction units.

[0120] Furthermore, the method is not limited to NN-based filters, but can be applied to any other filter or conventional filter in which a correction term is added to the reconstructed picture to improve image quality and reduce coding artifacts. Although in-loop filtering is described in the above examples, the proposed filtering method can also be performed outside the coding loop, for example as a post-processing step applied outside the decoder.

[0121] Various methods are described herein, each of which includes one or more steps or actions for achieving the described method. Unless a specific order of steps or actions is required for proper operation of the method, the order and / or use of specific steps and / or actions may be modified or combined. Additionally, terms such as “first,” “second,” etc. may be used in various embodiments to modify elements, components, steps, operations, etc., e.g., “first decoding” and “second decoding.” The use of such terms does not imply a modified ordering of operations unless specifically required. Thus, in this example, the first decoding need not be performed before the second decoding, but may occur, for example, before, during, or during an overlapping time with the second decoding.

[0122] Various methods and other aspects described herein can be used to modify modules of the video encoder 200 and video decoder 300, such as the motion refinement and motion compensation modules (270, 272, 372, 375), as shown in Figures 2 and 3. Furthermore, the aspects are not limited to VVC or HEVC, but can be applied, for example, to other standards and recommendations, and extensions of any such standards and recommendations. Unless otherwise indicated or technically excluded, the aspects described herein can be used individually or in combination.

[0123] Various numerical values ​​are used in this application. The specific values ​​are for illustrative purposes and the described aspects are not limited to these specific values.

[0124] Various implementations include decoding. As used herein, "decoding" may encompass all or a portion of the processes performed on a received encoded sequence to generate a final output suitable for, for example, a display. In various embodiments, such processes include one or more of the processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. Whether the phrase "decoding process" is intended to refer specifically to a subset of operations or generally to a broader decoding process will be clear based on the context of the particular description and is believed to be well understood by those skilled in the art.

[0125] Various implementations include encoding. Similar to the above discussion of "decoding," "encoding," as used in this application, may encompass all or part of the processes performed on an input video sequence to, for example, generate an encoded bitstream.

[0126] Implementations and aspects described herein may be implemented in, for example, a method or process, an apparatus, a software program, a data stream, or a signal. Even if discussed only in the context of a single type of implementation (e.g., discussed only as a method), the implementation of the discussed features may also be implemented in other forms (e.g., an apparatus or a program). An apparatus may be implemented in, for example, appropriate hardware, software, and firmware. The method may be implemented in an apparatus such as a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include, for example, communication devices such as computers, mobile phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate communication of information between end users.

[0127] References to "one embodiment" or "embodiment" or "one implementation" or "implementation," as well as other variations thereof, mean that a particular feature, structure, characteristic, etc. described in connection with an embodiment is included in at least one embodiment. Thus, appearances of the phrases "in one embodiment" or "in an embodiment" or "in one implementation" or "in an implementation" appearing in various places throughout this specification, as well as any other variations thereof, do not necessarily all refer to the same embodiment.

[0128] Additionally, the application may refer to "determining" various information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from memory.

[0129] Additionally, the application may refer to "accessing" various information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0130] Additionally, the application may refer to "receiving" various information. Receiving, like "accessing," is intended to be a broad term. Receiving information may include, for example, one or more of accessing information or retrieving information (e.g., from memory). Furthermore, "receiving" typically involves some form of operation, such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0131] For example, in the case of "A / B," "A and / or B," and "at least one of A and B," it should be understood that the use of any of the following " / ," "and / or," and "at least one of" is intended to encompass selection of only the first listed alternative (A), or selection of only the second listed alternative (B), or selection of both alternatives (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such phrases are intended to encompass selection of only the first listed alternative (A), or selection of only the second listed alternative (B), or selection of only the third listed alternative (C), or selection of only the first and second listed alternatives (A and B), or selection of only the first and third listed alternatives (A and C), or selection of only the second and third listed alternatives (B and C), or selection of all three alternatives (A, B, and C). This can be expanded as many times as the number of listed items, as would be apparent to one of ordinary skill in this and related arts.

[0132] Also, as used herein, the term "signaling" specifically refers to indicating something to a corresponding decoder. For example, in certain embodiments, an encoder signals a quantization matrix for dequantization. Thus, in embodiments, the same parameters are used on both the encoder and decoder sides. Thus, for example, an encoder can transmit a specific parameter to a decoder (explicit signaling) so that the decoder can use the same specific parameter. Conversely, if the decoder already has that specific parameter and other parameters, it can use signaling without transmission (implicit signaling) to simply allow the decoder to recognize and select that specific parameter. By avoiding transmitting any actual function, bit savings are realized in various embodiments. It should be understood that signaling can be achieved in various manners. For example, one or more syntax elements, flags, etc. are used to signal information to a corresponding decoder in various embodiments. Although the above refers to the verb form of the word "signal," the word "signal" can also be used as a noun herein.

[0133] As will be apparent to those skilled in the art, implementations may generate various signals formatted to carry information that may be, for example, stored or transmitted. The information may include, for example, instructions for performing a method or data generated by one of the described implementations. For example, a signal may be formatted to carry a bitstream of the described embodiments. Such a signal may be formatted, for example, as an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or as a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The signal it carries may be, for example, analog or digital information. The signal may be transmitted over a variety of different wired or wireless links, as is known. The signal may be stored on a processor-readable medium.

Claims

1. accessing a reconstructed sample version of a region of the picture; generating a weight for each sample of a plurality of samples in the region based on a version of the reconstructed sample of the region using a neural network; encoding, in a bitstream, offset parameters corresponding to the neural network for the region, wherein neural network parameters of the neural network are not encoded in the bitstream; filtering the plurality of samples within the region, wherein a sample of the plurality of samples within the region is adjusted based on a weight for the sample and the offset parameter for the region, the offset parameter for the region being scaled by the weight for the sample, and the scaled offset being added to the sample to adjust the sample; A method comprising:

2. The method of claim 1 , wherein multiple neural networks are used, each network of the multiple neural networks corresponding to a classification of the region.

3. generating, using a separate neural network, a separate weight for each sample of the plurality of samples in the region based on a version of the reconstructed samples of the region; encoding in the bitstream another offset parameter corresponding to the other neural network for the region, wherein neural network parameters of the other neural network are not encoded in the bitstream; Further comprising:

2. The method of claim 1, wherein the samples within the region are adjusted further based on the further weight for the sample and the further offset parameter for the region, the further offset parameter for the region being scaled by the further weight for the sample, and the scaled further offset being further added to the sample to adjust the sample.

4. The method described in claim 3, wherein the sum of the scaled offset and the scaled further offset is added to the sample in the region to filter the sample.

5. An apparatus comprising one or more processors, wherein the one or more processors: Access the reconstructed sample version of a region of the picture, generating a weight for each sample of a plurality of samples in the region based on a version of the reconstructed sample of the region using a neural network; encoding offset parameters for the regions in the bitstream, and neural network parameters of the neural network are not encoded in the bitstream; filtering the plurality of samples within the region, wherein a sample of the plurality of samples within the region is adjusted based on a weight for the sample and the offset parameter for the region, the offset parameter for the region being scaled by the weight for the sample, and the scaled offset being added to the sample to adjust the sample; The apparatus is configured to:

6. The apparatus of claim 5 , wherein a plurality of neural networks are used, each network of the plurality of neural networks corresponding to a classification of the region.

7. the one or more processors: using another neural network to generate another weight for each sample of the plurality of samples in the region based on a version of the reconstructed sample of the region; encoding in the bitstream another offset parameter corresponding to the other neural network for the region, wherein neural network parameters of the other neural network are not encoded in the bitstream; It is further structured as follows:

6. The apparatus of claim 5, wherein the samples within the region are adjusted further based on the further weight for the sample and the further offset parameter for the region, the further offset parameter for the region being scaled by the further weight for the sample, and the scaled further offset being further added to the sample to adjust the sample.

8. The apparatus described in claim 7, wherein the sum of the scaled offset and the further scaled offset is added to the samples in the region to filter the samples.

9. accessing a reconstructed sample version of a region of the picture; generating a weight for each sample of a plurality of samples in the region based on a version of the reconstructed sample of the region using a neural network; decoding, from a bitstream, offset parameters corresponding to the neural network for the region, wherein neural network parameters of the neural network are not transmitted in the bitstream; filtering the plurality of samples within the region, wherein a sample of the plurality of samples within the region is adjusted based on a weight for the sample and the offset parameter for the region, the offset parameter for the region being scaled by the weight for the sample, and the scaled offset being added to the sample to adjust the sample; A method comprising:

10. The method of claim 9 , wherein a plurality of neural networks are used, each network of the plurality of neural networks corresponding to a classification of the region.

11. generating, using a separate neural network, a separate weight for each sample of the plurality of samples in the region based on a version of the reconstructed samples of the region; decoding from the bitstream another offset parameter corresponding to the other neural network for the region, wherein neural network parameters of the other neural network are not transmitted in the bitstream; and Further comprising:

10. The method of claim 9, wherein the samples within the region are adjusted further based on the further weight for the sample and the further offset parameter for the region, the further offset parameter for the region being scaled by the further weight for the sample, and the scaled further offset being further added to the sample to adjust the sample.

12. The method described in claim 11, wherein the sum of the scaled offset and the scaled further offset is added to the sample in the region to filter the sample.

13. An apparatus comprising one or more processors, the one or more processors: Access the reconstructed sample version of a region of the picture, generating a weight for each sample of a plurality of samples in the region based on a version of the reconstructed sample of the region using a neural network; decoding offset parameters for the region from the bitstream, where neural network parameters for the neural network are not transmitted in the bitstream; filtering the plurality of samples within the region, wherein a sample of the plurality of samples within the region is adjusted based on a weight for the sample and the offset parameter for the region, the offset parameter for the region being scaled by the weight for the sample, and the scaled offset being added to the sample to adjust the sample; The apparatus is configured to:

14. The apparatus of claim 13 , wherein a plurality of neural networks are used, each network of the plurality of neural networks corresponding to a classification of the region.

15. the one or more processors: using another neural network to generate another weight for each sample of the plurality of samples in the region based on a version of the reconstructed sample of the region; decoding from the bitstream another offset parameter corresponding to the other neural network for the region, the neural network parameters of the other neural network not being transmitted in the bitstream; It is further structured as follows:

14. The apparatus of claim 13, wherein the samples within the region are adjusted further based on the further weight for the sample and the further offset parameter for the region, the further offset parameter for the region being scaled by the further weight for the sample, and the scaled further offset being further added to the sample to adjust the sample.

16. The apparatus of claim 15, wherein the sum of the scaled offset and the scaled further offset is added to the samples in the region to filter the samples.

Images