Encoder, decoder, and method for encoding image using soft classification

By using CNN-based adaptive loop filters in video encoding, the encoding artifact problem in the video encoding standard is solved, the bit rate improvement and complexity reduction are achieved, and the compression performance is significantly improved.

CN119013972APending Publication Date: 2024-11-22FRAUNHOFER GESELLSCHAFT ZUR FORDERUNG DER ANGEWANDTEN FORSCHUNG EV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380027079.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-02-15
Filing Date
2023-02-14
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

Modern video encoding standards have encoding artifact problems when compressing images, such as block effects, ringing effects, or blur artifacts, which make it difficult to trade off between low size of the image and low distortion of the reconstructed image.

Method used

Adaptive loop filter based on convolutional neural network (CNN) is adopted to replace hard classification by soft classification, calculate the weight of the FIR filtered version of the reconstructed frame, and introduce a second filtering step to improve compression efficiency.

Benefits of technology

The bit rate of video encoding is significantly improved, the codec complexity is reduced, and the bit rate savings of 3.85%/4.75% is achieved in the experimental results, while reducing the number of multiplications per sample and improving compression performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119013972A_ABST
    Figure CN119013972A_ABST
Patent Text Reader

Abstract

An apparatus for decoding a picture from a binary representation of an image, the apparatus configured to reconstruct samples of the image based on the binary representation; and classifying the reconstructed samples using soft classification, such as a convolutional neural network (CNN), for adaptive loop filtering (ALF).
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Embodiments of the present invention relate to an encoder for encoding an image into a binary representation of the image. Other embodiments of the present invention relate to an apparatus and a decoder for decoding an image from the binary representation of the image. Further embodiments relate to a method for encoding an image and a method for decoding an image, for example using the encoder, apparatus, decoder, and bitstream, respectively.

[0002] Some embodiments of the present invention relate to an adaptive loop filter based on convolutional neural network (CNN) classification. Background Art

[0003] Modern video coding standards, such as H.264 / AVC [1,2], H.265 / HEVC [3,4] or the recently finalized H.26 / VVC [5,6], initially exhibit coding artifacts due to quantization and block-based prediction and transform. Such artifacts may appear as artificial discontinuities along block boundaries, distortion of high-frequency information at sharp transitions, or over-smoothing of edges, known as blocking, ringing or blurring artifacts, respectively. To reduce these artifacts, loop filtering has become a key tool. Starting with AVC, the deblocking filter (DBF) [7] was introduced. In HEVC, sample adaptive offset (SAO) [8] was also included. Finally, the adaptive loop filter (ALF) [9,10,11] was applied to VVC as the third loop filter after DBF and SAO. Experimental results show that the ALF contributes significantly to the overall compression performance of VVC

[12] . Summary of the invention

[0004] In video and image coding, there is a common need to strike a trade-off between low size of the compressed image and low distortion of the reconstructed image, such distortion being recognizable as artifacts.

[0005] The embodiments provided by the independent claims provide a coding concept with an improved bit rate compared to known concepts.

[0006] The dependent claims describe advantageous embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings, in which:

[0008] Figure 1a A schematic block diagram of a decoding device according to an embodiment is shown;

[0009] Figure 1b is a schematic diagram of an encoder or encoding device according to at least one embodiment described herein:

[0010] Figure 2 According to one embodiment, Figure 1a Schematic block diagram of a possible implementation of a classifier.

[0011] Figure 3 is an example table representing possible parameter settings for a base set of layers of a convolutional neural network according to one embodiment;

[0012] Figure 4 shows a schematic diagram of a CNN loop filter according to one embodiment;

[0013] Figure 5-6 Schematic results of BD rate savings and codec complexity of a specific approach using the VVC reference software VTM 13.0 under AI and RA configurations are shown, respectively. DETAILED DESCRIPTION

[0014] The embodiments will be discussed in detail below, however, it should be understood that the embodiments provide many applicable concepts that can be embodied in various image compressions, such as video and still image encoding. The specific embodiments discussed only illustrate specific ways to implement and use the present concepts and do not limit the scope of the embodiments. In the following description, a number of details are set forth to more fully explain the embodiments of the present disclosure. However, those skilled in the art will appreciate that other embodiments can be implemented without these specific details. In other cases, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring the examples described herein. In addition, unless otherwise specifically stated, the features of the different embodiments described herein can be combined with each other.

[0015] In the following description of the embodiments, the same or similar parts or parts having the same function have the same reference mark or name, and the repeated description of the parts having the same reference mark or name is generally omitted. Therefore, the description of the parts having the same or similar reference marks or names can be interchangeable or can be applied to each other in different embodiments.

[0016] At least some embodiments are based on the recognition that the basic idea of ​​ALF is to classify the reconstructed samples into multiple categories based on local activity and directionality. The samples in each category are then filtered using a class-dependent FIR filter. The filter coefficients of the FIR filter are either represented by a signal or belong to a set of predetermined filter coefficients. The FIR filtering process can be supplemented by a nonlinear shearing operation.

[0017] Classifying different sets of FIR filters based on sample values ​​may be crucial for the compression effect of ALF. On the other hand, the inventors found that convolutional neural networks (CNNs) include powerful tools for solving various classification tasks. Therefore, it was studied whether CNN can also be used in the classification step of ALF to further improve its compression performance. For this method, the hard classification of ALF can be replaced by soft classification. Therefore, the implementation scheme proposes a loop filter that calculates the weighted sum of the FIR filtered version of the reconstructed frame, where the weight factor is sample adaptive and determined by CNN. In order to further improve the compression efficiency, a second filtering step can be added. The method defined in the implementation scheme may contribute to the general emerging field of CNN-based loop filters [13, 14, 15, 16].

[0018] This paper briefly reviews the ALF algorithm and then describes the CNN-based loop filter according to the implementation scheme. In addition, the complexity of the proposed method is analyzed and simulation results are presented.

[0019] The following is a description of the details related to adaptive loop filtering.

[0020] ALF divides the reconstructed samples into L = 25 categories C k Each class has an associated FIR filter f k If y is the input frame to be filtered by ALF, the ALF process can be summarized as

[0021]

[0022] Here, χC k It is C k The characteristic function of is defined by:

[0023]

[0024] where I is the set of all sample positions of y.

[0025] Filter f k The coefficients of can be signaled, in which case they are derived at the encoder to achieve the target frame and The mean square error between them is minimized, while the data rate of the transmission coefficient is kept small enough. k Classes are computed based on the local Laplacian and directional activity.

[0026] Figure 1aA schematic block diagram of a decoding device 10 according to an embodiment is shown. The decoding device 10 can be configured to receive a bitstream 12 containing information representing a binary or digital representation of an image (e.g., a still image or a video image). The decoding device 10 is configured to reconstruct a plurality of samples 141 to 144 of the image based on the binary representation. The device includes a classifier 16 configured to classify the reconstructed samples 141 to 144 using soft classification for use in an adaptive loop filter ALF. Possible soft classifications can be obtained from a convolutional neural network CNN. The device 10 can be configured to provide a reconstructed, filtered image 18 based on the ALF. The device can operate according to a known ALF, wherein the known hard classification is replaced by a classifier 16 capable of soft classification.

[0027] In the following, specific details of a convolutional neural network (CNN) are described in relation to the embodiments described herein.

[0028] The proposed CNN-based loop filter, such as the operation of classifier 16, can be defined as follows:

[0029]

[0030] Here, Φ1,…,Φ L can represent the classification output of a trained CNN-based classifier with training parameters Φ, f k represents the FIR filter also determined during the training process. This can be seen as an extension of formula (1) where the ALF classifier is replaced by a CNN classifier.

[0031] According to one embodiment, the apparatus is configured to enhance the functionality of an H.266 or VVC decoder, or is a decoder with enhanced functionality.

[0032] According to one embodiment, the soft classification 16 is configured to classify samples into a plurality of classes and apply an associated filter to each class, wherein each sample is associated with at least one class.

[0033] According to one embodiment, the associated filter is at least partially implemented as a finite impulse response FIR filter.

[0034] According to one embodiment, ALF is implemented according to formula (1), that is,

[0035]

[0036] in is the result of adaptive loop filtering; y is the reconstructed frame before loop filtering; L is the number of classification categories; χC k is the characteristic function of class C with index k, and fk is a FIR filter associated with class k; and wherein soft classification is implemented to provide a substitute for the characteristic function.

[0037] According to one embodiment, the soft classification is at least partially implemented by a CNN, which includes a convolutional layer and a first to seventh base layer group, such as Figure 2 shown.

[0038] According to one embodiment, the CNN includes exactly one convolutional layer 22 and exactly 7 base layer groups 241 to 247.

[0039] Figure 1b 1 is a schematic diagram of an encoder or encoding device 15 according to at least one embodiment described herein. The encoder 15 may include a classifier 16' that may operate according to the decoding device. The encoder 15 may additionally or alternatively operate in a different manner as described herein, for example generating a bitstream 12 from an image or frame 17 using a corresponding filter structure and / or processing unit.

[0040] Below, an example classifier based on CNN will be described in more detail.

[0041] 1. CNN Classifier

[0042] Figure 2 According to one embodiment, Figure 1a A schematic block diagram of a possible implementation of the classifier 16 is shown. Figure 2 As shown, the CNN classifier in (2) may include a single convolutional layer 22 and a plurality of base layer groups 241 to 247, in which the convolutional layer (Conv) 22, batch normalization (BN)

[17] , and rectified linear (ReLU) activation function

[18] are applied, respectively. Figure 3 The table shown describes the parameter settings for each basic layer group 241 to 247. Example values ​​are shown for the corresponding layer (column layer), the corresponding size of the convolution kernel and input / output channels (column size), the convolution type (column type) and the number of multiplications (column # (mul)).

[0043] That is, according to one embodiment, the decoding device is configured to implement soft classification by implementing a rectified linear (ReLU) activation function through convolution and batch normalization.

[0044] in other words, Figure 3An example architecture of a CNN-based classifier used in the embodiments described herein is shown. In the second column, (_; _; _; _) denotes the size of the convolution kernel (the first two parameters) and the number of input / output channels (the second two parameters). In the third column, the type of convolution is given (NS: non-separable / DS: depth-wise separable). The last column gives the number of multiplications per sample of the layer, where the scaling factor 1 / 4 (in rows 5-8) is given by Figure 2 Zhongyou ↓ The downsampling operator 26 indicated by 2 is applied to the 3rd to 6th base layer groups 243 to 246. Figure 3 This structure of the CNN shown may be associated with dimensions (a, b, c, d), which refer to the convolution kernel sizes a and b, the number of input channels c, and the number of output channels d; wherein the type of the layer indicates the type of convolution as non-separable NS; or depth-wise separable DS; and wherein #(mul) represents the number of multiplications of the layer.

[0045] According to one embodiment, the ALF includes a CNN to implement at least a portion of the soft classification, the CNN is based on formula (2), that is,

[0046]

[0047] in is the result of adaptive loop filtering; y is the reconstructed frame before loop filtering; L is the number of categories; Φ1,…,Φ L Define the classification output, and f k is the FIR filter associated with class k, and Θ defines the weights of at least one layer, some layers, or all layers in the CNN.

[0048] 1.1 Additional input data

[0049] In addition to the input frame 28 (y), the quantization parameter plane (QP parameter plane) QP, the reconstructed frame y before deblocking DBF The prediction signal Pred is sent as input 32 to the first base layer group 241. The aforementioned QP parameter plane is defined as a constant input plane filled with normalized QP values. Adding such a plane as input allows a unified model to be used for all QP values. In addition, the input y DBF and Pred can help the classifier identify compression artifacts more accurately. In summary, Figure 2 The first basic layer group in may include or have 8 input channels, and the 8 input channels are composed of 4 output channels from the first convolutional layer 241, the reconstructed input frame and the three above-mentioned additional input planes of input 32.

[0050] That is, according to one embodiment, the apparatus is configured to perform soft classification by using a CNN adapted to classify the reconstructed samples of a frame using at least one of:

[0051] Quantization parameter QP information, for example, QP parameter plane;

[0052] a reconstructed version of the frame before the deblocking filter DBF; and

[0053] • Prediction signal based on inter-frame or intra-frame prediction.

[0054] According to one embodiment, the first basic layer group of the soft-classified CNN is adapted to receive 8 input channels, preferably exactly 8 input channels. According to a preferred embodiment, the 8 input channels include:

[0055] Quantization parameter QP information, for example, QP parameter plane;

[0056] a reconstructed version of the frame before the deblocking filter DBF; and

[0057] Prediction signal based on inter-frame or intra-frame prediction;

[0058] The four output channels of the convolutional layer connected to the first base layer; and

[0059] Reconstructed input frame.

[0060] 1.2 Subsampling

[0061] For example, one of the main purposes of the classifier 16 represented in equation (2) may be to identify dominant features around a sample location. These features are likely to be shared by all sample locations within a sufficiently small local neighborhood. In fact, the ALF classifier in equation (1) performs 4×4 block-based classification, providing the same class index for each 4×4 block, which significantly reduces its complexity. The same principle can be applied to the embedded CNN classifier 16 and can be implemented by the subsampling operator 26. More precisely, Figure 2 The output channels of the second base layer group 242 in FIG. 5 may apply a max pooling operator 34 with a 3×3 window, followed by a 2DN downsampling 26 with a factor of 2. In the last layer of the CNN, the downsampling step may be recovered by upsampling 36 with a trained upsampling filter.

[0062] That is, according to one embodiment, the apparatus 10 may be adapted for soft classification to identify dominant features around the sample location.

[0063] According to one embodiment, the soft classification comprises a subsampler for providing a subsampling operator.

[0064] According to one embodiment, in order to implement the subsampling operator, the soft classification includes a CNN comprising a maximum pooling operator 34 with a 3x3 window, followed by a 2DN downsampling 26 with a factor of 2 applied to the output channels of the second basic layer group of the CNN; wherein in the last layer of the CNN, the downsampling step is restored by upsampling 36 using a trained upsampling filter.

[0065] 1.3 Depthwise Separable Convolution

[0066] The complexity of a convolutional layer with regular convolution can be given by k1·k2·n per sample in ·n out The multiplication is given by, where k1×k2 is the size of each convolution kernel, and n in / out is the number of input and output channels. Using depthwise separable convolutions

[19] , this complexity can be significantly reduced by splitting the filtering process into two parts. First, a 2D convolution with a k1×k2 kernel can be performed independently on each input channel. Subsequently, a full convolution with a 1×1 kernel can be applied across all channels. Overall, this may require k1·k2·n per sample. in +n in ·n out multiplication.

[0067] According to one embodiment, the soft classification of the apparatus 10 is configured for depthwise separable convolution DS.

[0068] For example, according to one embodiment, a depthwise separable convolution comprises a two-part filtering process; wherein the first part comprises a 2D convolution with a k1 x k2 kernel performed independently on each input channel of the soft classification; wherein the second part comprises a full convolution but with a 1 x 1 kernel applied on all input channels. Such a depthwise separable convolution is described in

[19] .

[0069] 1.4 Soft Classification

[0070] The last or even final step of a CNN classifier is to apply a softmax function as described in

[20] 38. Therefore, if ψ l is the output channel of the 7th basic layer group, then the classification output Φ in equation (2) k for

[0071]

[0072] It is worth noting that Φ k (i) can be interpreted as the estimated probability of the corresponding sample position i∈I being associated with the class index k. Different from the classification process of ALF, the classification output Φ of the CNN classifier k The partitioning of i is not necessarily provided.

[0073] According to one embodiment, the soft classification is adapted to apply a softmax function 38 to the output channels of the last (eg seventh) base layer group 247 of the soft classification.

[0074] According to one embodiment, the softmax function 38 comprises a structure based on the following formula:

[0075]

[0076] where Φ k (i) is interpreted as the estimated probability that the corresponding sample position i∈I is associated with the class with index k; Φ k is the classification output; and ψ l is the output channel of the last base layer group.

[0077] 2. Filtering process

[0078] A possible second step of the CNN loop filter according to equation (2) is to apply multiple 2D filters f corresponding to different classes k The embodiment introduces some additional features of the CNN loop filter to make this simple filtering process more efficient. That is, according to one embodiment, the ALF is adapted to apply multiple 2D filters (f k ).

[0079] 2.1 Shear Operator

[0080] In

[21] , a nonlinear modification of the final design of the ALF in VVC was proposed. The basic idea is to use a clipping function (see layer 22) to reduce the impact of adjacent sample values ​​when they differ too much from the current sample being filtered. Therefore, at sample position x, filtering the input signal y with a 2D filter f can be modified to:

[0081]

[0082] Here, Clip is a clipping function defined by Clip(d, b) = min(b; max(-b, d)), and ρ(i) is some parameter signaled in the bitstream. Figure 2 As shown, the filtering process y*f of formula (2) k The same nonlinear operation with trained shear parameter ρ(i) is also applied to the first convolutional layer of the classifier. For simplicity of notation, the 2D convolution including shear is still represented by y*f k Here, during training, the clipping operation can be replaced by a hyperbolic tangent to avoid gradient vanishing.

[0083] Therefore, according to an embodiment, the ALF is adapted to filter the classified samples using a clipping function to reduce the impact when the values ​​of adjacent samples differ too much from the current sample value being filtered.

[0084] According to one embodiment, the shear function is based on the determination rule of Formula 4, that is,

[0085]

[0086] Modify the filtering of the input signal y at sample position x with a 2D filter f, where "Clip" is the clipping function defined by Clip(d;b)=min(b;max(-b;d)) and ρ(i) is the number of pixels used in the filtering process y* * f k and the training clipping parameters of the first convolutional layer of the soft classification CNN.

[0087] 2.2 Second filtering process

[0088] Some or all of the parameters in equation (2) can be stored and used independently of the input data y to be filtered. In order to better adapt to specific signal characteristics, according to the implementation scheme, an additional filtering process can be introduced, now using an adaptive filter transmitted in the bitstream and optimized at the encoder For example, for each input frame, this additional second filtering step can be performed using f k The filtering is performed and can be defined as:

[0089]

[0090] filter It can be symmetrical, for example, in the shape of a 7×7 rhombus. Therefore, for k=1, ... each L It may be necessary to signal 13 coefficients The coefficient can be calculated To achieve the minimum mean square error between the target frame and the filtered reconstructed frame. In order to save signal cost, similar to ALF, a merging algorithm can be applied to reduce the filter It should be noted that the introduction of the second-stage filter and the transmission coefficient It seems that the transmission filter f k The coefficient of is more beneficial, although the latter is more in line with the design of ALF.

[0091] That is, according to one embodiment, the filter coefficients of the ALF are received as part of the bitstream.

[0092] According to one embodiment, the filter of the ALF comprises a 2D 7 x 7 diamond shape.

[0093] According to one embodiment, the ALF comprises a CNN for implementing at least part of the soft classification, the CNN being adapted for two-stage filtering based on formula (5), i.e.,

[0094]

[0095]

[0096] in is the result of adaptive loop filtering; y is the reconstructed frame before loop filtering; L is the number of categories; Φ1,…,Φ L Define the classification output, f k is the first-order FIR filter associated with class k, is the second-order FIR filter associated with class k, and Θ defines the weights of at least one layer, some layers, or all layers in the CNN.

[0097] According to one embodiment, the decoding device 10 may be configured to receive parameters of the FIR filter, such as the weights of the filter and / or its size or shape, as part of the bitstream.

[0098] 3. Complexity Analysis

[0099] In this section, the complexity of the CNN loop filter in equation (5), which consists of the CNN classifier and the filtering process, is analyzed by the number of multiplications per sample. Figure 3 As can be seen from the table, according to the proposed implementation, the total number of multiplications per sample of the CNN classifier can be 11302. The filtering process may include or consist of two parts, the first part using f k For filtering, the second part uses For the first 2D filter f k , 7×7 filters can be selected, while the second filtering process uses a 7×7 symmetric diamond filter with 13 label filter coefficients This may require 25·(7 2 +13) = 1550 multiplications. Therefore, the total number of multiplications per sample across all convolutional layers can be given by:

[0100] 11302+1550=12852

[0101] The total number of training parameters may be equal to 29873, where each parameter may be stored with 32-bit precision.

[0102] This corresponds to a memory requirement of 129.036KB and a worst-case complexity of 12.852kMACs per luma pixel, the latter being achieved when every sample in the sequence is filtered using the proposed loop filter.

[0103] According to one embodiment, the soft classification is adapted to perform a maximum of 15,000 multiplications per total number of samples, such as 11,302, to classify the reconstructed samples. According to one embodiment, the soft classification is adapted to perform a maximum of 2,000 multiplications per total number of samples, such as 1,550 multiplications, for implementing the first stage filtering and the second stage filtering. According to one embodiment, the soft classification is adapted to provide a maximum of 35,000 training parameters, such as 29,873 training parameters.

[0104] According to one embodiment, the decoding device 10 is a VVC decoder.

[0105] Figure 4 Shows the use of y k =(y*f k ) is an example of a schematic diagram of the CNN loop filter in equation (2), where * is the convolution.

[0106] Experimental Results

[0107] In this section, simulation results of the proposed loop filter are given as examples. To this end, two CNN loop filter models are trained on the BVI-DVC dataset

[22] , where only the luminance component of the signal is used for training. The original video data is compressed by VVC VTM 13.0 in RA configuration, and the ALF is switched with a sequence QP from the set {22; 27; 32; 37; 42}. The first model is trained on I frames, while the second one is trained on B frames. During training, the Adam optimization algorithm

[23] is used with the mean squared error loss function. In addition, the batch size is set to 64, where each batch consists of a randomly selected 64×64 block from the original sequence and the corresponding block from the reconstructed frame, the latter being expanded to 80×80 size to mitigate padding effects. After training, the CNN loop filter is integrated into the VVC test model version VTM 13.0 so that the first and second models can be applied to I and B frames, respectively. Which model to apply or whether to apply the original ALF is signaled at the frame level and decided by the RD decision at the encoder.

[0108] If a CNN model is applied, it can be turned on and off at the CTU level, which sends a switch signal. In addition, a signal is also sent at the CTU level to indicate whether the second-level filtering in equation (5) is to be additionally applied. For the second-level filtering, The filter coefficients of are signaled per frame. They are determined at the encoder by performing an RD search, which is similar to the method used to determine the filter coefficients in the ALF encoder of VTM. The CNN-based loop filter proposed in this paper is only applied to the luma component. For the chroma components, the loop filter of VVC can remain unchanged, including the ALF for chroma and the cross-component ALF (CCAF)

[11] .

[0109] Figure 5 and Figure 6 The table in Figure 2 shows the BD rate savings and codec complexity of the proposed method relative to the VVC reference software VTM13.0 under AI and RA configurations, respectively. Figure 5 The table in shows the rate-distortion performance comparison with VTM 13.0 in AI, while Figure 6 The table in shows the rate-distortion performance comparison with VTM 13.0 in RA. The average encoder and decoder runtime ratios for all sequences between test and anchor points are provided. All experimental results are performed using CTC in JVET

[24] for AI and RA configurations with two sets of QP values, namely {22; 27; 32; 37} (low QP) and {27; 32; 37; 42} (high QP).

[0110] In conjunction with the implementation scheme, a new ALF data-driven extension is proposed and implemented. The final loop filtering process is still performed by a classification step and subsequent FIR filtering. For the classification step, a method such as CNN is used. Experimental results show that an average bit rate reduction (for brightness) of 3.85% / 4.75% (low / high QP) and 4.39% / 4.33% (low / high QP) can be obtained under AI and RA configurations, respectively. Compared with other methods of CNN-based loop filtering, the number of multiplications required for each sample can be reduced. For example, other existing CNN loop filtering methods (such as [13, 14, 15, 16]) all achieve significant bitrate reductions over VVC, namely 6.54% / 2.81% (AI / RA, compared to VVC VTM 4.0

[13] ), 2.9% / 3.6% (AI / RA, compared to HEVC+ALF

[14] ), 8.9% (AI, compared to HEVC

[15] ) and 4.6% (RA, compared to VVC VTM 7.0

[16] ). However, they all require higher complexity compared to the specific methods - the number of multiplications per sample is about 1.0x 10 5 , 3.6x 10 5 , 1.7x 10 5 and 7.0x 1.0 6Finally, one might think that the apparent high decoder runtime seen in the specific results is due to a non-optimized implementation that does not use any custom libraries for fast CNN execution. Such libraries could be used to speed up the decoder runtime.

[0111] The described implementation allows for a new data-driven generalization of the adaptive loop filter (ALF) for Versatile Video Coding (VVC). The underlying description shows how the traditional ALF classification process and FIR filtering process can be generalized to define a natural model architecture for a convolutional neural network (CNN) based loop filter. Experimental results show that the average bitrate savings for all intra and random access configurations are 3.85% / 4.75% and 4.39% / 4.33% with VVC at low and high QP settings, respectively. The complexity, measured in terms of the number of multiplications per sample, is significantly reduced compared to other CNN based loop filters.

[0112] Encoder

[0113] For the above features, a suitable encoder is also an embodiment of the present invention. The device for encoding an image into a binary representation of the image is configured to receive a frame associated with the digital representation of the image; derive samples from the frame; classify the reconstructed samples using a convolutional neural network CNN for adaptive loop filtering ALF; and provide a bit stream allowing the reconstructed samples, such as bit stream 12.

[0114] According to one embodiment, the adaptive loop filtering is based on an adaptive filter Therein the encoder is adapted to transmit filter coefficients of the adaptive filter in the bitstream.

[0115] According to one embodiment, the apparatus is configured to determine coefficients of the adaptive filter such that a mean square error between the reconstructed frame and the filtered reconstructed frame is minimized.

[0116] According to one embodiment, the filter of the ALF comprises a 2D 7 x 7 diamond shape.

[0117] The encoder and decoder may include respective features and functionality related to each other. This may include similar features of each, for example, regarding the filters being used, but may also include inverse functionality related to encoding / decoding.

[0118] Decoder

[0119] It is further advantageous to implement if the basic invention also relates to the decoder side.

[0120] According to one embodiment, a device for decoding an image from a binary representation of a frame is provided, wherein the device is configured to reconstruct samples of the frame based on the binary representation; and classify the reconstructed samples using soft classification to obtain classified samples. The device is configured to filter the classified samples using an FIR filter.

[0121] Reference will now be made to the basic architecture of a loop filter used in a decoder according to one embodiment.

[0122] According to a first embodiment, finite impulse response (FIR) filtering with a soft classifier may be implemented in the decoder.

[0123] In such a decoder, a mapping can be applied that assigns to each sample position i in the reconstructed frame y and optionally other inputs a value _Φ i (i; y), ..., Φ N (i; y), so that for a sample position i0, at least two values ​​Φ k (i0;y),Φ l (i0; y), 1≤k, l≤N satisfies Φ k (i0;y)≠0 and Φ l (i0; y) ≠ 0, and the decoder generates the final reconstructed frame The sample value at its sample position i Based on or given by the formula:

[0124]

[0125] Each f j They are all FIR filters, and * indicates convolution.

[0126] According to one embodiment, the decoder can be adjusted such that the filter f j The coefficients of are decoded from the bitstream frame by frame or sequence by sequence or other units. According to an embodiment that can be implemented additionally or alternatively, the decoder can be adapted so that the parameters of the function Φ are decoded from the bitstream frame by frame or sequence by other units.

[0127] According to a second embodiment, a finite impulse response (FIR) filter with a soft classifier and an additional second stage filtering may be implemented in the decoder.

[0128] In such a decoder, a mapping can be applied which, in the reconstructed frame y and optionally some other input, assigns to each sample position i in the reconstructed frame the value Φ i (i; y), ..., Φ N (i; y), FIR filter f j, j = 1; ...; N, so that for j = 1, ..., N, the decoder uses the classification output Φ1, ..., Φ N Generate intermediate reconstructed frame z j , and take the sample value at each sample position i.

[0129] z j (i) = Φ j (i)·(f j *y)(i) (7)

[0130] FIR filter g j , j = 1, ..., N, the final reconstructed frame y^ is generated by the decoder, and the sample value of the decoder at the sample position i is Based on or given according to the following rules:

[0131]

[0132] Among them, f j is the FIR filter for the jth class, and * denotes convolution. The parameter N described herein may correspond to the parameter L.

[0133] According to one embodiment, such a decoder may be configured as a filter g j The coefficients of are decoded from the bitstream frame by frame or sequence by sequence or other units. Alternatively or in addition, the decoder may be configured to filter f j The coefficients of are decoded from the bitstream frame by frame or sequence by sequence or other units. Alternatively or in addition, the decoder can be configured to decode from the bitstream frame by frame or sequence by sequence or other units based on some parameters of the function Φ.

[0134] That is, embodiments related to the decoder involve signaling of loop filter coefficients having the basic architecture described in equations (6), (7) and (8), i.e., implemented in one or both decoders. Embodiments involve FIR filtering with a soft classifier and signaling of the filter coefficients. For example, the filter g j and / or f j The coefficients are decoded from the bitstream frame by frame or sequence by sequence or other units.

[0135] Some embodiments of the present invention related to a decoder relate to a decoder configured to decode a flag bit by sequence or by frame or by block, the flag bit determining whether to modify the reconstructed samples on the block according to formula (6).

[0136] Such a decoder can decode a flag bit by sequence or by frame or by block, which determines whether to modify the reconstructed samples on the block according to the following formula:

[0137]

[0138] This can be called a frame-level or block-level flag for turning on or off the first loop filter.

[0139] Some embodiments of the present invention related to a decoder relate to a decoder configured to decode a flag bit by sequence, frame or block, the flag bit determining whether to modify the reconstructed samples of the block according to formula (8).

[0140] This decoder can be configured to decode a flag bit by sequence, frame or block, which determines whether to modify the reconstructed samples of the block according to the following formula:

[0141]

[0142] This can be called a frame-level or block-level flag for turning on or off the first loop filter.

[0143] This decoding of the flag bit can be called a switch related to the loop filter described in this article, and has one of the basic architectures. The decoding of the flag bit can save unnecessary computing power while achieving high-quality decoding results.

[0144] Decoding flags, as described above, indicating whether to modify according to formula (6) on the one hand and whether to modify according to formula (8) on the other hand, in addition to this, alternative or additional implementations may involve decisions on whether and / or which modification to apply. This can be referred to as a frame-level, block-level flag for switching between the first and second filters.

[0145] For example, the decoder is configured to decode a flag bit by frame or by block, which determines whether the reconstructed samples of the block are modified according to the following formula:

[0146] Or modify the reconstructed sample of the block according to the following formula:

[0147]

[0148] This may optionally include that the decoded information does not apply to any of the above.

[0149] The embodiment also provides a decoder having multiple loop filters, which are applied according to the following formula.

[0150]

[0151] According to one embodiment, such a decoder is configured to decode an index by frame or by block, which determines which of these multiple loop filters is applied to a given frame / block, for example, if the functions Φ of multiple applicable loop filters are implemented by a convolutional neural network CNN, they may share some weights on some layers, but differ on other weights / layers. This can be called frame-level, block-level switching, which is used to switch between different configurations of the first loop filter, as shown in formula (6).

[0152] Alternatively or additionally, switching between such configurations may be applied to the second configuration, as shown in equation (8).

[0153] Such a decoder can be configured such that multiple loop filters are applied according to the following formula.

[0154]

[0155]

[0156] The decoder can decode an index by frame or by block, which determines which of these multiple loop filters is applied to a given frame / block; some of the multiple loop filters can be equal in some parameters describing the function Φ, for example, if the functions Φ of multiple applicable loop filters are implemented by a convolutional neural network CNN, they may share some weights on some layers, but differ in other weights / layers. This can be called frame-level or block-level switching, which is used to switch between different configurations of the second loop filter, as shown in formula (8).

[0157] It is worth noting that the decoder according to the embodiment may involve applying the first structure associated with formula (6) or the second structure associated with formula (8). As mentioned above, their combination can also be implemented in the decoder, for example, to benefit from switching between two modes. Therefore, the described advantageous modifications can also be implemented separately, or without different modifications, or can be implemented in any combination.

[0158] In a further embodiment, a loop filter having the basic architecture of formula (6) and / or formula (8) may be performed on luminance in a separate or combined manner, while performing normal ALF / CCALF on chrominance, i.e., luminance and chrominance are processed in different ways.

[0159] The decoder according to said identification is adapted to apply a loop filtering process only to the luma component according to the following formula:

[0160]

[0161] A second loop filtering process that is processed independently of or dependent on the loop filtering process is applied to the chrominance component. In other words, a decoder is proposed in which the loop filtering process according to formula (6) is applied only to the luminance component, while a second loop filtering process that can be processed together with the loop filtering process according to formula (6) or independently of the loop filtering process is applied to the chrominance component, for example, the first loop filtering is applied to the luminance and the normal ALF / CCALF is applied to the chrominance.

[0162] As an alternative implementation or alternative mode of operation, or even in addition, a decoder according to an embodiment may be configured to apply a loop filtering process only to the luma component according to the following formula:

[0163]

[0164]

[0165] A second loop filtering process that is processed independently of or dependent on the loop filtering process is applied to the chrominance component. In other words, a decoder is proposed in which the loop filtering process according to formula (8) is applied only to the luminance component, while a second loop filtering process that can be processed together with the loop filtering process according to formula (8) or independently of the loop filtering process is applied to the chrominance component, for example, the second loop filtering is applied to the luminance and normal ALF / CCALF is applied to the chrominance.

[0166] According to one embodiment, these configurations may also be implemented in an encoder. For example, with respect to the first loop filter in equation (6), the encoder may encode at least some portion of the frame using the loop filter described in equation (6). Another encoder also solves a linear equation to determine the possible differences between the filter f and the frame. j The corresponding filter f k The other encoder also performs a rate-distortion RD search to combine the filter f k / f j and determine optimal flags, switch parameters and other parameters for a decoder operating according to the embodiments described herein.

[0167] Such an encoder encodes at least some parts of a frame using a loop filter according to the following formula.

[0168]

[0169] According to one embodiment, such an encoder may be configured to solve a linear equation to determine the coefficients of the filter fj.

[0170] According to one embodiment, such an encoder may be configured to perform an RD search to combine f jand determining optimal flags, switch parameters and other parameters for a decoder operating according to the aspects described herein.

[0171] The corresponding encoder is defined by the second filtered implementation of equation (8).

[0172] Such an encoder encodes at least some parts of a frame using a loop filter according to the following formula.

[0173]

[0174] According to one embodiment, such an encoder may be configured to solve a linear equation to determine the filter f j or filter g j The coefficient of .

[0175] According to one embodiment, such an encoder may be configured to perform an RD search to combine f j or j and determining optimal flags, switching parameters and other parameters for a decoder operating according to aspects described herein.

[0176] It should be noted that the encoders related to parameter j defined in formulas (6) to (8) are also related to parameter k in the above formulas (1) to (5). That is, index j can correspond to index k. In addition, parameter N described herein can correspond to parameter L.

[0177] Further embodiments relate to methods of operating an encoder or a decoder as described herein. Further embodiments relate to a computer-readable digital storage medium having stored thereon a computer program having a program code for performing the methods as described herein when run on a computer.

[0178] Further embodiments relate to a bitstream generated by an encoder or encoding apparatus as described herein; and / or a bitstream received by a decoder or decoding apparatus as described herein.

[0179] Although some aspects have been described as features in the context of an apparatus, it is clear that this description may also be viewed as a description of corresponding features of a method. Although some aspects have been described as features in the context of a method, it is clear that this description may also be viewed as a description of corresponding features related to the functionality of the apparatus.

[0180] Some or all of the method steps may be performed by (or using) a hardware device (eg, a microprocessor, a programmable computer, or an electronic circuit). In some embodiments, one or more of the most important method steps may be performed by such a device.

[0181] The binary representation of the present invention can be stored on a digital storage medium or can be transmitted on a transmission medium (eg, a wireless transmission medium or a wired transmission medium, such as the Internet).

[0182] According to certain implementation requirements, embodiments of the present invention may be implemented in hardware or software, or at least partially in hardware or at least partially in software. Implementation may be performed using a digital storage medium (e.g., a floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM, or FLASH memory) in which electronically readable control signals are stored that cooperate (or can cooperate) with a programmable computer system to perform the corresponding method. Therefore, the digital storage medium may be computer readable.

[0183] Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

[0184] Generally, the embodiments of the present invention can be implemented as a computer program product with a program code, when the computer program product runs on a computer, the program code is used to perform one of the methods. For example, the program code can be stored on a machine readable carrier.

[0185] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0186] In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0187] Therefore, another embodiment of the inventive method is a data carrier (or a digital storage medium or a computer-readable medium) on which is recorded the computer program for performing one of the methods described herein. The data carrier, the digital storage medium or the recorded medium is typically tangible and / or non-transitory.

[0188] Therefore, a further embodiment of the inventive method is a data stream or a sequence of signals representing the computer program for performing one of the methods described herein.The data stream or the sequence of signals may, for example, be configured for transmission via a data communication connection, for example via the Internet.

[0189] A further embodiment comprises a processing means, for example a computer or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0190] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0191] A further embodiment according to the invention comprises an apparatus or a system configured to transfer (e.g. electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may, for example, be a computer, a mobile device, a storage device, etc. The apparatus or the system may, for example, comprise a file server for transferring the computer program to the receiver.

[0192] In some embodiments, a programmable logic device (e.g., a field programmable gate array) can be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array can collaborate with a microprocessor to perform one of the methods described herein. Typically, these methods are preferably performed by any hardware device.

[0193] The devices described herein may be implemented using hardware devices, computers, or a combination of hardware devices and computers.

[0194] The methods described herein may be performed using a hardware device, a computer, or a combination of a hardware device and a computer.

[0195] In the foregoing detailed description, it can be seen that various features are grouped together in the embodiments in order to simplify the disclosure. This disclosure method should not be interpreted as reflecting an intention that the claimed embodiments require more features than are expressly recited in each claim. On the contrary, as reflected in the following claims, subject matter may be present in all features of a single disclosed embodiment.

[0196] Therefore, the following claims are hereby incorporated into the detailed description, where each claim can stand on its own as a separate embodiment. Although each claim can stand on its own as a separate embodiment, it should be noted that although a dependent claim may mention a specific combination with one or more other claims in a claim, other embodiments may also include a combination of a dependent claim with the subject matter of each other dependent claim or a combination of each feature with other dependent or independent claims. Unless it is stated that a specific combination is not intended, such a combination will be proposed herein. In addition, even if the claim is not directly dependent on the independent claim, it is intended to include the claim features of any other independent claim.

[0197] The above examples are only used to illustrate the principles of the present disclosure. It should be understood that modifications and variations of the arrangements and details described herein will be apparent to other persons skilled in the art. Therefore, it is intended that the scope of the patent claims submitted be limited only, rather than being limited by the specific details presented herein by describing and explaining the embodiments.

[0198] literature

[0199] [1]ITU-T and ISO / IEC“Advanced Video Coding for Generic AudiovisualServices”H.264and ISO / IEC 14496-10,vers.1,2003

[0200] [2]T.Wiegand,G.J.Sullivan,G. and A.Luthra,“Overview of theH.264 / AVC video coding standard,”IEEE Trans.Circuits Syst.Video Technol.,vol.13,pp.560–576,2003.

[0201] [3]ITU-T and ISO / IEC“High Efficiency Video Coding”H.265and ISO / IEC23008-2,vers.1,2013

[0202] [4]G.J.Sullivan,J.-R.Ohm,W.-J.Han and T.Wiegand,“Overview of the highefficiency video coding(HEVC)standard,”IEEE Trans.Circuits Syst.VideoTechnol.,vol.22,pp.1649–1668,2012.

[0203] [5]ITU-T and ISO / IEC“Versatile Video Coding”H.266and ISO / IEC 23090-3,2020

[0204] [6]B.Bross et al.“Overview of the Versatile Video Coding(VVC)Standardand its Applications,”IEEE Trans.Circuits Syst.Video Technol.,vol.31,pp.3736-3764,2021.

[0205] [7]P.List,A.Joch,J.Lainema,G. and M.Karczewicz,“Adaptivedeblocking filter,”IEEE Trans.Circuits Syst.Video Technol.,vol.13,pp.614-619,2003.

[0206] [8]C.-M.Fu et al.,“Sample adaptive offset in the HEVC standard,”IEEETrans.Circuits Syst.Video Technol.,vol.22,pp.1755-1764,2012.

[0207] [9]Tsai,C.-Y.et al.,“Adaptive loop filtering for video coding,”IEEEJournal of Selected Topics in Signal Processing,vol.7,pp.934–945,2013.

[0208]

[10] M.Karczewicz,L.Zhang,W.Chien and X.Li,“Geometry transformation-based adaptive in-loop filter,”in Proc.Picture Coding Symposium(PCS),pp.1-5,2016.

[0209]

[11] M.Karczewicz et al.,“VVC In-Loop Filters,”IEEE Trans.CircuitsSyst.Video Technol.,vol.31,pp.3907-3925,2021.

[0210]

[12] W.-J.Chien et al.,“JVET AHG report:Tool reporting procedureandtesting(AHG13),”17th JVET meeting,no.JVET-Q0013.2020.

[0211]

[13] W.Jia,L.Li,Z.Li,X.Zhang and S.Liu,“Residual GuidedDeblocking WithDeep Learning,”in 2020 IEEE International Conference onImage Processing(ICIP),IEEE,2020,pp.3109-3113.

[0212]

[14] C.Jia et al.,“Content-Aware Convolutional Neural Network forIn-Loop Filtering in High Efficiency Video Coding,”IEEE Transactions onImageProcessing,vol.28,pp.3343-3356,July 2019.

[0213]

[15] M.-Z.Wang,S.Wan,H.Gong and M.-Y.Ma,“Attention-BasedDual-Scale CNNIn-Loop Filter for Versatile Video Coding,”IEEE Access,vol.7,pp.145214-145226,2019.

[0214]

[16] D.Ma,F.Zhang and D.R.Bull,“MFRNet:A New CNNArchitecture for Post-Processing and In-loop Filtering,”IEEE Journal ofSelected Topics in SignalProcessing,vol.15,pp.378-387,2021.

[0215]

[17] S.Ioffe and C.Szegedy,“Batch normalization:Acceleratingdeepnetwork training by reducing internal covariate shift,”inInternationalConference on Machine Learning(ICML),2015,pp.448–456.

[0216]

[18] V.Nair and G.E.Hinton,“Rectified linear units improverestrictedBoltzmann machines,”in Proc.27th Int.Conf.Mach.Learn.(ICML),2010,pp.807–814.

[0217]

[19] A.Howard et.al.,“Mobilenets:Efficient convolutionalneuralnetworks for mobile vision applications,”CVPR,2017.

[0218]

[20] I.Goodfellow,Y.Bengio and A.Courville,“Batch normalization:Accelerating deep network training by reducing internal covariate shift,”inDeep Learning,MIT Press.,2016,pp.180–184.

[0219]

[21] J.Taquet,C.Gisquet,G.Laroche,P.Onno,“Non-Linear AdaptiveLoopFilter,”in 13th JVET meeting,no.JVETM0385,January 2019.

[0220]

[22] D.Ma,F.Zhang and D.R.Bull,“BVI-DVC:a training database fordeepvideo compression,”in arXiv:2003.13552,2020.

[0221]

[23] D.P.Kingma and J.Ba.,“Adam:A method for stochasticoptimization,”nt.Conf.Learn.Represent.(ICLR),pp.1–15,2015.

[0222]

[24] K.Suehring and X.Li,“JVET common test conditions andsoftwarereference configurations,”in 2nd JVET meeting,no.JVET-B1010,February2016.

Claims

1. An apparatus for decoding an image from a binary representation of the image, wherein: The device is configured to: reconstructing samples of the image based on the binary representation; and The reconstructed samples are classified using a soft classification such as a convolutional neural network (CNN) for adaptive loop filtering (ALF).

2. The apparatus according to claim 1, wherein the apparatus is configured to enhance the functionality of an H.266 or VVC decoder. 3 . The apparatus according to claim 1 , wherein the soft classification is configured to classify samples into a plurality of categories and apply an associated filter to each category, wherein each sample is associated with at least one category.

4. The apparatus of claim 3, wherein the associated filter is implemented at least in part as a finite impulse response (FIR) filter.

5. The apparatus according to claim 3, wherein the ALF is implemented according to the following formula: in is the result of adaptive loop filtering; y is the reconstructed frame before loop filtering; L is the number of classification categories; χC k is the characteristic function of class C with index k, and f k is the FIR filter associated with class k; The soft classification is implemented to provide a substitute for the characteristic function.

6. The apparatus of claim 1, wherein the soft classification is at least partially implemented by a CNN comprising a convolutional layer and a first to seventh base layer group.

7. An apparatus according to claim 6, wherein the CNN comprises exactly one convolutional layer and exactly 7 base layer groups.

8. The apparatus of claim 7, wherein the structure of the CNN is based on the following table: in, The sizes (a, b, c, d) refer to the sizes a and b of the convolution kernel, the number of input channels c, and the number of output channels d; wherein the type of the layer indicates that the type of convolution is non-separable NS; or depth-wise separable DS; and wherein #(mul) represents the number of multiplications of the layer.

9. The apparatus of claim 1, wherein the ALF comprises the CNN, the CNN being configured to implement at least a portion of the soft classification, the CNN being based on the following equation: in is the result of adaptive loop filtering; y is the reconstructed frame before loop filtering; L is the number of categories; Φ1,…,Φ L Define the classification output, and f k is the FIR filter associated with class k, and Θ defines the weights of at least one layer, some layers, or all layers in the CNN.

10. The apparatus according to claim 1, wherein the apparatus is configured to implement the soft classification by implementing a rectified linear (ReLU) activation function through convolution and batch normalization.

11. The apparatus of claim 1 , wherein the apparatus is configured to implement the soft classification by using a CNN, the CNN being adapted to classify the reconstructed samples of a frame using at least one of: Quantization parameter QP information, for example, QP parameter plane; a reconstructed version of said frame before the deblocking filter DBF; and • A prediction signal based on inter-frame or intra-frame prediction of the frame.

12. The apparatus according to claim 1, wherein the first basic layer group of the soft-classified CNN is adapted to receive 8 input channels, preferably exactly 8 input channels.

13. The apparatus of claim 12, wherein the eight input channels include: Quantization parameter QP information, for example, QP parameter plane; a reconstructed version of the frame before the deblocking filter DBF; as well as a prediction signal based on inter-frame or intra-frame prediction of the frame; The four output channels of the convolutional layer connected to the first base layer; as well as The reconstructed input frame.

14. The apparatus of claim 1, wherein the soft classification is used to identify dominant features around a sample location.

15. The apparatus of claim 1, wherein the soft classification comprises a subsampler for providing a subsampling operator.

16. The device according to claim 15, wherein: In order to implement the subsampling operator, the soft classification includes a CNN, which includes a maximum pooling operator with a 3x3 window, followed by a 2DN downsampling with a factor of 2 applied to the output channels of the second basic layer group of the CNN; wherein, in the last layer of the CNN, the downsampling step is restored by upsampling using a trained upsampling filter.

17. The apparatus of claim 1, wherein the soft classification is configured for depthwise separable convolution.

18. The apparatus of claim 17, wherein the depthwise separable convolution comprises a two-part filtering process; wherein the first part comprises a 2D convolution with a k1 x k2 convolution kernel, which is performed independently on each input channel of the soft classification; wherein the second part comprises a full convolution with a 1x1 convolution kernel applied to all channels.

19. The apparatus of claim 1, wherein the soft classification is adapted to apply a softmax function to output channels of a last, e.g., seventh, base layer group of the soft classification.

20. The apparatus of claim 1, wherein the softmax function comprises a structure based on: where Φ k (i) is interpreted as the estimated probability that the corresponding sample position i∈I is associated with the class with index k; Φ k is the classification output; and ψ l is the output channel of the last base layer group.

21. The apparatus according to claim 1, wherein the ALF is adapted to combine multiple 2D filters (f k ) is applied to classified samples.

22. The apparatus according to claim 1, wherein the ALF is adapted to filter the classified samples using a clipping function to reduce the impact of adjacent sample values ​​that differ too much from the current sample value being filtered.

23. The apparatus of claim 1, wherein the clipping function is based on the following determination rule: Modify the filtering of the input signal y using a 2D filter f with samples at x, where "Clip" is the clipping function defined by Clip(d;b)=min(b;max(-b;d)) and ρ(i) is the number of pixels used in the filtering process y* * f k and the training clipping parameters of the first convolutional layer of the soft classification CNN.

24. The apparatus of claim 1, wherein filter coefficients of the ALF are received as part of a bitstream.

25. The apparatus of claim 1, wherein the filter of the ALF comprises a shape of a 2D 7x7 diamond.

26. The apparatus of claim 1, wherein the ALF comprises a CNN for implementing at least a portion of the soft classification, the CNN being adapted to perform two-stage filtering based on: in is the result of adaptive loop filtering; y is the reconstructed frame before loop filtering; L is the number of categories; Φ1,…,Φ L Define the classification output, f k is the first-order FIR filter associated with class k, is the second-order FIR filter associated with class k, and Θ defines the weights of at least one layer, some layers, or all layers in the CNN.

27. The apparatus of claim 9, configured to receive parameters of the FIR filter, such as the weights of the filter and / or its size or shape, as part of a bitstream.

28. The apparatus of claim 1, wherein the soft classification is adapted to perform a maximum of 15,000 multiplications per sample, for example 11,302 multiplications, to classify the reconstructed samples.

29. The apparatus of claim 28, wherein the soft classification is adapted to perform a maximum of 2000 multiplications per sample, for example 1550, for implementing the first stage filtering and the second stage filtering.

30. The apparatus of claim 1, wherein the soft classification is adapted to provide a maximum of 35,000 training parameters, for example, 29,873 training parameters.

31. The device according to claim 1, which is a VVC decoder.

32. An apparatus for encoding an image into a binary representation of the image, wherein the apparatus is configured to: receiving a frame associated with a digital representation of an image; deriving samples from the frame; Using a convolutional neural network (CNN) to classify the reconstructed samples for adaptive loop filtering (ALF); and A bitstream is provided which allows reconstruction of the samples.

33. The apparatus of claim 32, wherein the adaptive loop filtering is based on an adaptive adaptive filter in, The encoder is adapted to transmit filter coefficients of the adaptive filter in the bitstream.

34. The apparatus of claim 32, wherein the apparatus is configured to determine coefficients of the adaptive filter to minimize a mean square error between the reconstructed frame and the filtered reconstructed frame.

35. The apparatus of claim 32, wherein the filter of the ALF comprises a 2D7x7 diamond shape.

36. An apparatus for decoding an image from a binary representation of the frame, wherein the The device is configured to: reconstructing samples of the frame based on the binary representation; and classifying the reconstructed samples using soft classification to obtain classified samples; and The classified samples are filtered using a FIR filter.

37. The apparatus of claim 36, wherein the soft classification comprises a mapping that assigns to each sample position (i) in the reconstructed frame a value Φ1(i; y), ..., Φ based on a reconstructed frame (y) providing the sample and optionally providing additional input. N (i; y), such that for sample position i0, there are at least two values ​​Φ k (i0;y),Φ l (i0:y), 1≤k; l≤N satisfies Φ k (i0; y)≠0 and Φ l (i0:y)≠0, and the final reconstructed frame is generated by the decoder, the sample value of the decoder at sample position i Given by: where f j is the FIR filter for class j, * denotes convolution, and Φ1, …, Φ N Defines the classification output.

38. The decoder of claim 37, wherein: The filter f j The coefficients are decoded from the bit stream frame by frame or sequence by sequence or other units.

39. The decoder of claim 37, wherein: Some parameters of the function Φ are decoded from the bit stream by frame or by sequence or by other units.

40. The apparatus of claim 36, wherein the soft classification comprises a mapping that assigns a value Φ to each sample position (i) in the reconstructed frame based on a reconstructed frame (y) providing the sample and optionally providing additional input. i (i; y), ..., Φ N (i; y), FIR filter f j , j = 1, ..., N, so that for j = 1, ..., N, the decoder generates an intermediate reconstructed frame Z j , the decoder obtains the sample value at each sample position i z j (i)=Φ j (i)·(f j *y)(i) And the FIR filter g j , j = 1, ..., N, so that the final reconstructed frame Generated by the decoder, the sample value of the decoder at sample position i Given by the following formula where f j and g j is the FIR filter for class j, * denotes convolution, and Φ1, …, Φ N Defines the classification output.

41. A decoder according to claim 40, wherein the filter g j The coefficients are decoded from the bit stream frame by frame or sequence by sequence or other units.

42. The decoder of claim 40, wherein the filter f j The coefficients are decoded from the bit stream frame by frame or sequence by sequence or other units.

43. The decoder according to claim 40, wherein some parameters of the function are decoded from the bit stream in frames, sequences or other units.

44. A decoder that decodes a flag bit by sequence, frame or block, the flag bit determining whether to modify the reconstructed samples on the block according to the following formula:

45. A decoder that decodes a flag bit by sequence, frame or block, the flag bit determining whether to modify the reconstructed samples on the block according to the following formula:

46. ​​A decoder that decodes a flag bit in sequence, frame or block, the flag bit determining whether to modify the reconstructed samples on the block according to the following formula: Or modify it according to the following formula:

47. A decoder that applies multiple loop filters according to the following formula, And the decoder decodes an index by frame or by block, which determines which of these multiple loop filters is applied to a given frame / block. For example, if the functions Φ of multiple applicable loop filters are implemented by a convolutional neural network CNN, they may share some weights for some layers but differ on other weights / layers.

48. A decoder that applies multiple loop filters according to the following formula, And the decoder decodes an index by frame or by block, the index determining which of these multiple loop filters is applied to a given frame / block; wherein some of these multiple loop filters may be equal in some parameters describing the function Φ, for example, if the function Φ of multiple applicable loop filters is implemented by a convolutional neural network CNN, they may share some weights in some layers, but differ in other weights / layers.

49. A decoder wherein a loop filtering process is applied only to the luminance component according to the following equation, A second loop filtering process, independent of or dependent on the loop filtering process, is applied to the chrominance components.

50. A decoder wherein a loop filtering process is applied only to the luminance component according to the following equation, A second loop filtering process, independent of or dependent on the loop filtering process, is applied to the chrominance components.

51. An encoder for encoding at least some portions of a frame using a loop filter described by:

52. An encoder according to claim 51, configured to solve a linear equation to determine the filter f j , wherein the index j may correspond to the index k in claim 1 or claim 32.

53. The encoder according to claim 51, configured to perform an RD search to combine f j and determining the optimal flags, switch parameters and other parameters of a decoder operating according to claim 37, or claim 44, or claim 47, wherein the index j may correspond to the index k in claim 1 or claim 32.

54. An encoder for encoding at least some portions of a frame using a loop filter described by:

55. An encoder according to claim 54, configured to solve a linear equation to determine the filter f j or filter g j , wherein the index j may correspond to the index k in claim 1 or claim 32.

56. The encoder according to claim 54, configured to perform an RD search to combine f j or j and determining the optimal flags, switch parameters and other parameters of a decoder operating according to claim 41, or claim 45, or claim 46, or claim 48 or claim 50, wherein the index j may correspond to the index k in claim 1 or claim 35.

57. A method for operating an apparatus for decoding an image from a binary representation of the image, the method comprising: Samples of reconstructed images based on binary representation; and The reconstructed samples are classified using a soft classification such as a convolutional neural network (CNN) for adaptive loop filtering (ALF).

58. A method for operating an apparatus for encoding an image into a binary representation of the image, wherein the method comprises: receiving a frame associated with a digital representation of an image; deriving samples from the frame; Use convolutional neural network (CNN) to classify the reconstructed samples for adaptive loop filtering (ALF); and A bitstream is provided which allows reconstruction of the samples.

59. A method for operating an apparatus for decoding an image from a binary representation of a frame, wherein the method comprises: reconstructing a sample of a frame based on the binary representation; as well as Classify the reconstructed samples using soft classification to obtain classified samples; as well as The classified samples are filtered using a FIR filter.

60. A method for operating a decoder, the method comprising: A flag is decoded by sequence, frame or block, and the flag determines whether to modify the reconstructed samples of the block according to the following formula: And act accordingly.

61. A method for operating a decoder, the method comprising: A flag is decoded by sequence, frame or block, and the flag determines whether to modify the reconstructed samples of the block according to the following formula: And act accordingly.

62. A method for operating a decoder, the method comprising: A flag is decoded by sequence, frame or block, and the flag determines whether to modify the reconstructed samples of the block according to the following formula: Or the block is modified to reconstruct the samples according to the following formula: And act accordingly.

63. A method for operating a decoder having a plurality of loop filters, the plurality of loop filters being applied according to the following equation: The method comprises: a per-frame or per-block decode index that determines which of these multiple loop filters is applied to a given frame / block, e.g., if the functions Φ of multiple applicable loop filters are implemented by a convolutional neural network CNN, they may share some weights for some layers but differ on other weights / layers; and Proceed accordingly.

64. A method for operating a decoder having a plurality of loop filters, the plurality of loop filters being applied according to the following equation: The method comprises: An index is decoded by frame or by block, the index determining which of these multiple loop filters is applied to a given frame / block; wherein some of the multiple loop filters may be equal in certain parameters describing the function Φ, for example, if the function Φ of multiple applicable loop filters is implemented by a convolutional neural network CNN, they may share some weights in some layers, but differ in other weights / layers.

65. A method for operating a decoder, the method comprising: The loop filtering process is applied only to the luminance component of the image or frame to be decoded according to the following equation: And, a second loop filtering process, independent of or dependent on the loop filtering process, is applied to the chrominance components of the image or frame.

66. A method for operating a decoder, the method comprising: The loop filtering process is applied only to the luminance component of the image or frame to be decoded according to the following equation: And, a second loop filtering process, independent of or dependent on the loop filtering process, is applied to the chrominance components of the image or frame.

67. A method for operating an encoder, the method comprising: At least some portion of the frame is encoded using a loop filter described by:

68. A method for operating an encoder, the method comprising: At least some portion of the frame is encoded using a loop filter described by:

69. A computer-readable digital storage medium having a computer program stored thereon, the computer program having a program code for executing the method according to claim 57, or claim 58, or claim 59, or claim 60, or claim 61, or claim 62, or claim 63, or claim 64, or claim 65, or claim 66, or claim 67, or claim 68 when the computer is running the method.

70. A bit stream generated by an encoder or encoding device according to claim 32, or claim 51, or claim 54; or received by a decoder or decoding device according to claim 1, or claim 36, or claim 44, or claim 45, or claim 46, or claim 47, or claim 48, or claim 49, or claim 50.