Filtering for Video Encoding and Decoding

Abstract

A method for generating an encoded or decoded video is provided. The method includes obtaining values ​​of reconstructed samples and obtaining input information including any one or a combination of: i) information about filtered samples, ii) information about predicted samples, or iii) information about skipped samples. The method further includes providing the values ​​of the reconstructed samples and the input information to a machine learning (ML) model, thereby generating at least one ML output data, and generating the encoded or decoded video based at least on the at least one ML output data.

Description

[Technical field]

[0001] SUMMARY This disclosure relates to methods and apparatus for performing filtering for video encoding and decoding. [Background technology]

[0002] Video is the dominant form of data traffic in today's networks and is expected to continue to increase its share. One way to reduce data traffic from video is compression. In compression, the source video is encoded into a bitstream that can then be stored and transmitted to an end user. Using a decoder, the end user can extract the video data and display the video data on a screen.

[0003] However, since the encoder does not know what kind of device the encoded bitstream will be sent to, it must compress the video into a standardized format, so that all devices that support the chosen standard can successfully decode the video. The compression can be lossless, i.e., the decoded video will be equivalent to the source video given to the encoder, or lossy, where some degradation of the content is acceptable. Whether the compression is lossless or lossy has a significant impact on the bitrate, i.e., how high the compression ratio is, since factors such as noise can make lossless compression very expensive.

[0004] A video sequence contains a sequence of pictures. The color space typically used in a video sequence is YCbCr, where Y is the luma (brightness) component and Cb and Cr are the chroma components. Sometimes the Cb and Cr components are called U and V. Other color spaces are also used, such as ICtCp (also known as IPT) (where I is the luma component and Ct and Cp are the chroma components), constant-luminance YCbCr (where Y is the luma component and Cb and Cr are the chroma components), RGB (where R, G, and B correspond to the blue, green, and blue components, respectively), and YCoCg (where Y is the luma component and Co and Cg are the chroma components).

[0005] The order in which pictures are arranged in a video sequence is called the "display order." Each picture is assigned a Picture Order Count (POC) value to indicate its display order. In this disclosure, the terms "image," "picture," or "frame" are used interchangeably.

[0006] Video compression is used to compress a video sequence into a sequence of coded pictures. In many existing video codecs, a picture is divided into blocks of different sizes. A block is a two-dimensional array of samples. The blocks serve as the basis for coding. A video decoder then decodes the coded pictures into pictures containing sample values.

[0007] Video standards are usually developed by international organizations because they represent different companies and laboratories with different areas of expertise and interests. Currently, the most applied video compression standard is H.264 / AVC (Advanced Video Coding), which was developed jointly by ITU-T and ISO. The first version of H.264 / AVC was finalized in 2003, with several updates in later years. The successor to H.264 / AVC, also developed by ITU-T (International Telecommunication Union-Telecommunication) and the International Organization for Standardization (ISO), known as H.265 / HEVC (High Efficiency Video Coding), was finalized in 2013. MPEG and ITU-T created the successor to HEVC within the Joint Video Exploratory Team (JVET). The name of this video codec is Versatile Video Coding (VVC), and version 1 of the VVC specification was published as Rec. ITU-T H.266|ISO / IEC (International Electrotechnical Commission) 23090-3, "Versatile Video Coding", 2020.

[0008] The VVC video coding standard is a block-based video codec that utilizes both temporal and spatial prediction. Spatial prediction is achieved using intra (I) prediction from within the current picture. Temporal prediction is achieved using unidirectional (P) or bidirectional inter (B) prediction at the block level from a previously decoded reference picture. At the encoder, the difference between the original and predicted sample data, called the residual, is transformed to the frequency domain, quantized, and then entropy coded before being transmitted along with the necessary prediction parameters, such as the prediction mode and motion vectors (which may also be entropy coded). The decoder performs entropy decoding, inverse quantization, and inverse transform to obtain the residual, which is then added to the intra or inter prediction to reconstruct the picture.

[0009] The VVC video coding standard uses a block structure called the quad-tree plus binary-tree plus ternary-tree block structure (QTBT+TT), in which each picture is first partitioned into square blocks called coding tree units (CTUs). All CTUs are of the same size, and the partitioning of pictures into CTUs is done without any syntax to control the partitioning.

[0010] Each CTU is further partitioned into coding units (CUs) that can have either a square or rectangular shape. The CTUs are first partitioned by a quadtree structure, and then the CTUs can be further partitioned in equal-sized partitions either vertically or horizontally in a binary structure to form coding units (CUs). Thus, blocks can have either a square or rectangular shape. The depth of the quadtree and binary tree can be set by the encoder in the bitstream. The ternary tree (TT) part adds the possibility to split a CU into three partitions instead of two equal-sized partitions. This increases the possibility to use a block structure that better fits the content structure of the picture, such as approximately following an important edge in the picture.

[0011] Intra-coded blocks are I-blocks. Unidirectionally predicted blocks are P-blocks, and bidirectionally predicted blocks are B-blocks. For some blocks, the encoder decides that in some cases the prediction is close enough to the original that it is not necessary to code the residual. The encoder then signals to the decoder that transform coding for that block should be bypassed, i.e., skipped. Such blocks are called skipped blocks.

[0012] At the 20th JVET conference, it was decided to establish an Exploratory Experiment (EE) on Neural Network-Based (NN-Based) Video Coding. The EE continued at the 21st and 22nd JVET conferences with two EE trials, namely NN-Based Filtering and NN-Based Super-Resolution. At the 23rd JVET conference, it was decided that the trials would continue in three categories, namely enhancement filters, super-resolution methods, and intra-prediction. In the enhancement filter category, two settings were considered: (i) the proposed filter used as an in-loop filter and (ii) the proposed filter used as a post-processing filter.

[0013] In-loop filtering in VVC includes deblocking filtering, sample adaptive offset (SAO) operation, and adaptive loop filter (ALF) operation. The deblocking filter is used to remove block artifacts by smoothing discontinuities in the horizontal and vertical directions across block boundaries. The deblocking filter uses a block boundary strength (BS) parameter to determine the filtering strength. The BS parameter can have values ​​of 0, 1, and 2, with larger values ​​indicating stronger filtering. The output of the deblocking filter is further processed by the SAO operation, and then the output of the SAO operation is processed by the ALF operation. The output of the ALF is then put into a display picture buffer (DPB), which is used for prediction of pictures to be coded (or decoded) later. Because the deblocking filter, the SAO filter, and the ALF affect the pictures in the DPB used for prediction, they are classified as in-loop filters, also known as loop filters. It may happen that the decoder further filters the image but does not send the filtered output to the DPB, but only to the display. In contrast to loop filters, such filters do not influence the future prediction and are therefore classified as post-processing filters, also known as post-filters.

[0014] EE1-1.6: Combined Test of EE1-1.2 and EE1-1.4, Y. Li, K. Zhang, L. Zhang, H. Wang, J. Chen, K. Reuze, A.M. Kotra, M. Karczewicz, JVET-X0066, October 2021, and EE1-1.2: Test on Deep In-Loop Filter with Adaptive Parameter Selection and Residual Scaling, Y. Li, K. Zhang, L. Zhang, H. Wang, K. Reuze, A.M. Kotra, M. Karczewicz, JVET-Y0143, January 2022, are two consecutive contributions that describe NN-based in-loop filtering.

[0015] Both contributions use the same NN model for filtering. The NN-based in-loop filter is placed before SAO and ALF, and the deblocking filter is turned off. The purpose of using the NN-based filter is to improve the quality of the reconstructed samples. The NN model can be nonlinear. The deblocking filter, SAO, and ALF all contain nonlinear elements, such as conditions, and are therefore not strictly linear, but all three of them are based on linear filters. In contrast, a sufficiently large NN model can in principle learn any nonlinear mapping and is therefore capable of representing a broader class of functions compared to deblocking, SAO, and ALF.

[0016] In JVET-X0066 and JVET-Y0143, there are four NN models, i.e., four NN-based in-loop filters, i.e., one for luma intra samples, one for chroma intra samples, one for luma inter samples, and one for chroma inter samples. The use of NN filtering can be controlled at the block (CTU) level or the picture level. The encoder can decide whether to use NN filtering for each block or each picture.

[0017] This NN-based in-loop filter substantially increases the compression efficiency of the codec, i.e., it substantially reduces the bit rate without reducing the objective quality measured by MSE (Mean Square Error)-based PSNR (Peak Signal-to-Noise Ratio). The increase in compression efficiency, or simply "gain", is often measured as the Bjontegaard delta rate (BDR) relative to the anchor. As an example, a BDR of -1% means that the same PSNR can be reached with 1% fewer bits. As reported in JVET-Y0143, in the Random Access (RA) setting, the BDR gain for the luma component (Y) is -9.80%, and in the All Intra (AI) setting, the BDR gain for the luma component is -7.39%. The complexity of the NN model used for compression is often measured by the number of per-pixel Multiply-Accumulate (MAC) operations. A high gain of the NN model is directly related to the high complexity of the NN model. The luma intra model described in JVET-Y0143 has a complexity of 430kMAC / sample, i.e. 430000 multiply-accumulate operations per sample. Together with the multiply-accumulate operations required for the chroma model (110kMAC), the overall complexity is 540kMAC / pixel. There are other measures of complexity, such as total model size in terms of stored parameters. Summary of the Invention

[0018] However, the structure of the NN model described in JVET-Y0143 is not optimal. For example, the high complexity of the NN model may be a big challenge for practical hardware implementation. Therefore, it is highly desirable to maintain or improve the performance of the NN model while reducing its complexity.

[0019] Therefore, in one aspect of an embodiment of the present disclosure, a method for generating an encoded or decoded video is provided, the method includes obtaining values ​​of reconstructed samples, obtaining input information including any one or combination of: i) information about filtered samples, ii) information about predicted samples, or iii) information about skipped samples, providing the values ​​of the reconstructed samples and the input information to a machine learning (ML) model, thereby generating at least one ML output data, and generating the encoded or decoded video based at least on the at least one ML output data.

[0020] In another aspect, a method for generating encoded or decoded video is provided. The method includes obtaining machine learning (ML) input data, the ML input data including: i) reconstructed sample values; ii) predicted sample values; iii) block boundary strength (BBS) information indicating strength of filtering applied to sample boundaries; and iv) quantization parameters (QPs). The method further includes providing the ML input data to an ML model, thereby generating ML output data, and generating the encoded or decoded video based at least on the ML output data. The ML input data does not include partitioning information indicating how luma pictures are partitioned into coding tree units (CTUs) and how luma CTUs are partitioned into coding units (CUs).

[0021] In a different aspect, a method for generating an encoded or decoded video is provided. The method includes obtaining machine learning (ML) input data, the ML input data including: i) reconstructed sample values; ii) predicted sample values; and iii) a quantization parameter (QP). The method further includes providing the ML input data to an ML model, thereby generating ML output data, and generating the encoded or decoded video based at least on the ML output data. The ML input data does not include block boundary strength (BBS) information indicating a strength of a filtering applied to a sample boundary.

[0022] In a different aspect, a method for generating encoded or decoded video is provided. The method includes obtaining reconstructed sample values, obtaining a quantization parameter (QP), providing the reconstructed sample values ​​and the quantization parameter to a machine learning (ML) model to generate ML output data, generating first output sample values ​​based at least on the ML output data, and providing the first output sample values ​​to a group of two or more attention residual blocks connected in series. The group of attention residual blocks includes a first attention residual block disposed at one end of the series of attention residual blocks, the first attention residual block configured to receive first input data consisting of first output sample values ​​and generate second output sample values ​​based on the first output sample values.

[0023] In a different aspect, a method for generating encoded or decoded video is provided. The method includes obtaining machine learning (ML) input data. The ML input data includes: i) values ​​of luma components of reconstructed samples; ii) values ​​of chroma components of reconstructed samples; iii) values ​​of luma components of predicted samples; iv) values ​​of chroma components of predicted samples; v) first block boundary strength (BBS) information indicating a strength of filtering applied to a boundary of the luma components of the samples; vi) second BBS information indicating a strength of filtering applied to a boundary of the chroma components of the samples; and vii) a quantization parameter (QP). The method further includes providing the ML input data to an ML model, thereby generating ML output data, and generating the encoded or decoded video based at least on the ML output data.

[0024] In a different aspect, a method for generating an encoded video or a decoded video is provided. The method includes obtaining reconstructed sample values, obtaining a quantization parameter (QP), providing the reconstructed sample values ​​and the quantization parameter to a machine learning (ML) model, thereby generating ML output data, generating a first output sample value based at least on the ML output data, providing the first output sample value to a group of two or more attention residual blocks connected in series, thereby generating a second output sample value, and generating an encoded video or a decoded video based on the second output sample value. The group of attention residual blocks includes a first attention residual block disposed at one end of the series of attention residual blocks, and the first attention residual block is configured to receive input data consisting of the first output sample value and the QP.

[0025] In a different aspect, a computer program is provided comprising instructions (1244) that, when executed by a processing circuit, cause the processing circuit to perform a method according to any one of the embodiments described above.

[0026] In a different aspect, an apparatus for generating encoded or decoded video is provided, the apparatus being configured to: obtain values ​​of reconstructed samples; obtain input information including any one or a combination of: i) information about filtered samples; ii) information about predicted samples; or iii) information about skipped samples; provide the values ​​of the reconstructed samples and the input information to a machine learning (ML) model, thereby generating at least one ML output data; and generate the encoded or decoded video based at least on the at least one ML output data.

[0027] In a different aspect, an apparatus for generating an encoded or decoded video is provided. The apparatus is configured to obtain machine learning (ML) input data, the ML input data including: i) reconstructed sample values; ii) predicted sample values; iii) block boundary strength (BBS) information indicating strength of filtering applied to sample boundaries; and iv) quantization parameters (QPs). The apparatus is configured to provide the ML input data to an ML model, thereby generating ML output data, and generate the encoded or decoded video based at least on the ML output data. The ML input data does not include partitioning information indicating how luma pictures are partitioned into coding tree units (CTUs) and how luma CTUs are partitioned into coding units (CUs).

[0028] In a different aspect, an apparatus for generating an encoded or decoded video is provided. The apparatus is configured to obtain machine learning (ML) input data, the ML input data including: i) reconstructed sample values; ii) predicted sample values; and iii) a quantization parameter (QP). The apparatus is configured to provide the ML input data to an ML model, thereby generating ML output data, and to generate the encoded or decoded video based at least on the ML output data. The ML input data does not include block boundary strength (BBS) information indicating a strength of a filtering applied to a sample boundary.

[0029] In a different aspect, an apparatus for generating encoded or decoded video is provided. The apparatus is configured to obtain reconstructed sample values, obtain a quantization parameter (QP), provide the reconstructed sample values ​​and the quantization parameter to a machine learning (ML) model to generate ML output data, generate first output sample values ​​based at least on the ML output data, and provide the first output sample values ​​to a group of two or more attention residual blocks connected in series. The group of attention residual blocks includes a first attention residual block disposed at one end of the series of attention residual blocks, the first attention residual block configured to receive first input data consisting of first output sample values ​​and generate second output sample values ​​based on the first output sample values.

[0030] In a different aspect, an apparatus for generating encoded or decoded video is provided. The apparatus is configured to obtain machine learning (ML) input data. The ML input data includes: i) values ​​of luma components of reconstructed samples; ii) values ​​of chroma components of reconstructed samples; iii) values ​​of luma components of predicted samples; iv) values ​​of chroma components of predicted samples; v) first block boundary strength (BBS) information indicating a strength of filtering applied to a boundary of the luma components of the samples; vi) second BBS information indicating a strength of filtering applied to a boundary of the chroma components of the samples; and vii) a quantization parameter (QP). The apparatus is configured to provide the ML input data to an ML model, thereby generating ML output data, and generating the encoded or decoded video based at least on the ML output data.

[0031] In a different aspect, an apparatus for generating an encoded or decoded video is provided. The apparatus is configured to: obtain reconstructed sample values; obtain a quantization parameter (QP); provide the reconstructed sample values ​​and the quantization parameter to a machine learning (ML) model, thereby generating ML output data; generate a first output sample value based at least on the ML output data; provide the first output sample value to a group of two or more attention residual blocks connected in series, thereby generating a second output sample value; and generate the encoded or decoded video based on the second output sample value. The group of attention residual blocks includes a first attention residual block disposed at one end of the series of attention residual blocks, and the first attention residual block is configured to receive input data consisting of the first output sample value and the QP.

[0032] Embodiments of the present disclosure provide a way to substantially maintain or improve the performance of a NN model while reducing the complexity of the NN model.

[0033] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate various embodiments. [Brief description of the drawings]

[0034] [Figure 1A] FIG. 1 illustrates a system, according to some embodiments. [Figure 1B] FIG. 1 illustrates a system, according to some embodiments. [Figure 1C] FIG. 1 illustrates a system, according to some embodiments. [Diagram 2] 2 is a schematic block diagram of an encoder according to some embodiments. [Diagram 3] FIG. 2 is a schematic block diagram of a decoder according to some embodiments. [Figure 4] 1 is a schematic block diagram of a portion of a NN filter in accordance with some embodiments. [Diagram 5] 1 is a schematic block diagram of a portion of a NN filter in accordance with some embodiments. [Figure 6A] FIG. 1 is a schematic block diagram of a spatial attention block, according to some embodiments. [Figure 6B] FIG. 1 illustrates an example of a spatial attention block, according to some embodiments. [Figure 6C] FIG. 2 is a schematic block diagram of a residual block according to some embodiments. [Figure 6D] FIG. 1 is a schematic block diagram of a spatial attention block, according to some embodiments. [Figure 7] FIG. 2 illustrates a process according to some embodiments. [Figure 8] FIG. 2 illustrates a process according to some embodiments. [Figure 9] FIG. 2 illustrates a process according to some embodiments. [Figure 10] FIG. 2 illustrates a process according to some embodiments. [Figure 11] FIG. 2 illustrates a process according to some embodiments. [Figure 12]FIG. 1 illustrates an apparatus, according to some embodiments. [Figure 13] FIG. 2 illustrates a process according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0035] The following terminology is used in the description of the embodiments below.

[0036] Neural Network: a generic term for an entity with one or more layers of simple processing units called neurons or nodes that have activation functions and interact with each other via weighted connections and biases, collectively creating a tool in the context of nonlinear transformations.

[0037] Neural network architecture, network architecture, or architecture for short: The layout of a neural network, describing the arrangement of nodes and their connections, usually in the form of several interconnected layers. It may also specify the dimensionality of the input(s) and output(s), as well as the activation functions for the nodes.

[0038] Neural network weights, or weights for short: weight values ​​assigned to connections between nodes in a neural network.

[0039] Neural network model, or model for short: a transformation in the form of a trained neural network. A neural network model may be specified as a neural network architecture, activation functions, biases, and weights.

[0040] Filter: A transform. A neural network model is one implementation of a filter. The term NN filter may be used as a contraction of neural network based filter or neural network filter.

[0041] Neural network training, or training for short: the process of finding values ​​for the weights and biases for a neural network. Typically, a training data set is used to train a neural network, and the goal of training is to minimize a prescribed error. The amount of training data needs to be large enough to avoid overtraining. Training a neural network is usually a time-consuming task and generally involves several iterations over the training data, each iteration being called an epoch.

[0042] 1A illustrates a system 100 according to some embodiments. The system 100 comprises a first entity 102, a second entity 104, and a network 110. The first entity 102 is configured to transmit a video stream (also known as a "video bitstream," "bitstream," "encoded video") 106 towards the second entity 104.

[0043] The first entity 102 may be any computing device (e.g., a network node such as a server) capable of encoding video using an encoder 112 and transmitting the encoded video over a network 110 towards a second entity 104. The second entity 104 may be any computing device (e.g., a network node) capable of receiving the encoded video and decoding the encoded video using a decoder 114. Each of the first entity 102 and the second entity 104 may be a single physical entity or a combination of multiple physical entities. The multiple physical entities may be located in the same location or distributed in a cloud.

[0044] In some embodiments, as shown in FIG. 1B, the first entity 102 is a video streaming server 132 and the second entity 104 is a user equipment (UE) 134. The UE 134 may be either a desktop, a laptop computer, a tablet, a mobile phone, or any other computing device. The video streaming server 132 may transmit a video bitstream 136 (e.g., YouTube™ video streaming) to the video streaming client 134. Upon receiving the video bitstream 136, the UE 134 may decode the received video bitstream 136, thereby generating and displaying a video for video streaming.

[0045] In another embodiment, as shown in FIG. 1C, the first entity 102 and the second entity 104 are a first UE 152 and a second UE 154. For example, the first UE 152 may be a video conference session provider or a video chat initiator, and the second UE 154 may be a video conference session responder or a video chat responder. In the embodiment shown in FIG. 1C, the first UE 152 may transmit a video bitstream 156 for a video conference (e.g., Zoom™, Skype™, MS Teams™, etc.) or a video chat (e.g., Facetime™) to the second UE 154. Upon receiving the video bitstream 156, the UE 154 may decode the received video bitstream 156, thereby generating and displaying a video for the video conference session or the video chat.

[0046] 2 shows a schematic block diagram of the encoder 112 according to some embodiments. The encoder 112 is configured to encode blocks of sample values ​​(hereinafter "blocks") in a video frame of the source video 202. In the encoder 112, a current block (e.g., a block included in a video frame of the source video 202) is predicted from an already provided block in the same frame or in a previous frame by performing motion estimation by the motion estimator 250. The result of the motion estimation is a motion vector or a displacement vector associated with a reference block in case of inter prediction. The motion vector is utilized by the motion compensator 250 to output an inter prediction of the block.

[0047] An intra predictor 249 calculates an intra prediction of the current block. The output from the motion estimator / compensator 250 and the output from the intra predictor 249 are input to a selector 251 which selects either intra or inter prediction for the current block. The output from the selector 251 is input to an error calculator in the form of an adder 241, which also receives the sample values ​​of the current block. The adder 241 calculates and outputs a residual error as the difference in the sample values ​​between the block and the prediction of that block. The error is transformed in a transformer 242, such as by a discrete cosine transform, and quantized by a quantizer 243, followed by coding in an encoder 244, such as by an entropy encoder. In inter coding, the estimated motion vectors are provided to the encoder 244 to generate a coded representation of the current block.

[0048] The transformed and quantized residual error for the current block is also provided to an inverse quantizer 245 and an inverse transformer 246 to retrieve the original residual error. This error is added by an adder 247 to the block prediction output from the motion compensator 250 or the intra predictor 249 to create a reconstructed sample block 280 that can be used in predicting and coding the next block. The reconstructed sample block 280 is processed by an embodiment of the NN filter 230 to perform filtering to eliminate blocking artifacts. The output from the NN filter 230, i.e., output data 290, is then temporarily stored in a frame buffer 248, where the output data 290 is available to the intra predictor 249 and the motion estimator / compensator 250.

[0049] In some embodiments, the encoder 112 may include an SAO unit 270 and / or an ALF 272. The SAO unit 270 and the ALF 272 may be configured to receive output data 290 from the NN filter 230, perform additional filtering on the output data 290, and provide the filtered output data to the buffer 248.

[0050] 2, the NN filter 230 is disposed between the SAO unit 270 and the adder 247, although in other embodiments, the NN filter 230 may replace the SAO unit 270 and / or the ALF 272. Alternatively, in other embodiments, the NN filter 230 may be disposed between the buffer 248 and the motion compensator 250. Furthermore, in some embodiments, a deblocking filter (not shown) may be disposed between the NN filter 230 and the adder 247 such that the reconstructed sample block 280 undergoes a deblocking process and is then provided to the NN filter 230.

[0051] 3 is a schematic block diagram of the decoder 114 according to some embodiments. The decoder 114 comprises a decoder 361, such as an entropy decoder, for decoding the coded representation of the block to obtain a set of quantized and transformed residual errors. These residual errors are dequantized in an inverse quantizer 362 and inverse transformed by an inverse transformer 363 to obtain a set of residual errors. These residual errors are added to the sample values ​​of a reference block in an adder 364. The reference block is determined by a motion estimator / compensator 367 or an intra predictor 366 depending on whether inter or intra prediction is performed.

[0052] Thereby, a selector 368 is interconnected to the adder 364, the motion estimator / compensator 367 and the intra predictor 366. The resulting decoded block 380 output from the adder 364 is input to an embodiment of a NN filter unit 330 for filtering any blocking artifacts. A filtered block 390 is output from the NN filter 330 and is further preferably temporarily provided to the frame buffer 365 where it can be used as a reference block for subsequent blocks to be decoded.

[0053] A frame buffer (e.g., a decoded picture buffer (DPB)) 365 is thereby connected to the motion estimator / compensator 367 to make stored blocks of samples available to the motion estimator / compensator 367. The output from the adder 364 is also preferably input to an intra predictor 366 for use as an unfiltered reference block.

[0054] In some embodiments, the decoder 114 may include an SAO unit 380 and / or an ALF 372. The SAO unit 380 and the ALF 382 may be configured to receive output data 390 from the NN filter 330, perform additional filtering on the output data 390, and provide the filtered output data to a buffer 365.

[0055] 3, the NN filter 330 is disposed between the SAO unit 380 and the adder 364, although in other embodiments, the NN filter 330 may replace the SAO unit 380 and / or the ALF 382. Alternatively, in other embodiments, the NN filter 330 may be disposed between the buffer 365 and the motion compensator 367. Furthermore, in some embodiments, a deblocking filter (not shown) may be disposed between the NN filter 330 and the adder 364 such that the reconstructed sample block 380 undergoes a deblocking process and is then provided to the NN filter 330.

[0056] 4 is a schematic block diagram of a portion of the NN filter 230 / 330 for filtering intra-luma samples according to some embodiments. In this disclosure, a luma (or chroma) intra sample is a luma (or chroma) component of an intra-predicted sample. Similarly, a luma (or chroma) inter sample is a luma (or chroma) component of an inter-predicted sample. In this disclosure, any one or combination of the elements shown in FIG. 4 is referred to as a computation module. For example, a pair of a convolution layer and a PReLU is referred to as a computation module (CM).

[0057] 4, the NN filter 230 / 330 may have six inputs: (1) values ​​of the luma components of the reconstructed samples 280 / 380 ("rec"), (2) values ​​of the luma components of the predicted samples 295 / 395 ("pred"), (3) partitioning information ("part") indicating how the luma components of the samples are partitioned (more specifically, how luma pictures are partitioned into coding tree units (CTUs) and how luma CTUs are partitioned into coding units), (4) block boundary strength (BBS) information ("bs") indicating the strength of filtering applied to the boundaries of the luma components of the samples, (5) a quantization parameter ("qp"), and (6) additional input information. In some embodiments, the additional input information includes values ​​of the luma components of the deblocked samples.

[0058] Each of the six inputs may go through a convolutional layer (labeled as "conv3x3" in FIG. 4) and a parametric rectified linear unit (PReLU) layer (labeled as "PReLU") separately. The six outputs from the six PReLU layers may then be concatenated via a concatenation unit (labeled as "concat" in FIG. 4) and fused together to generate the data (a.k.a. "signal") "y". The convolutional layer "conv3x3" is a convolutional layer with kernel size 3×3, and the convolutional layer "conv1x1" is a convolutional layer with kernel size 1×1. The PReLU may create an activation layer.

[0059] In some embodiments, qp may be a scalar value. In such embodiments, the NN filter 230 / 330 may also include a dimensional operation unit (labeled "unsqueezed widening" in FIG. 4), which may be configured to widen qp such that the widened qp has the same size as the other inputs (i.e., rec, pred, part, bs, and dblk). However, in other embodiments, qp may be a matrix whose size may be the same as the size of the other inputs (e.g., rec, pred, part, and / or bs). For example, different samples inside the CTU may be associated with different qp values. In such embodiments, a dimensional operation unit is not required.

[0060] In some embodiments, the NN filter 230 / 330 may also include a downsampler (labeled "2↓" in FIG. 4), which is configured to perform downsampling by a factor of two.

[0061] 4, data "y" may be provided to a group of N successive attention residual (hereinafter "AR") blocks 402. In some embodiments, the N successive AR blocks 402 may have the same structure, while in other embodiments, the N successive AR blocks 402 may have different structures. N may be any integer greater than or equal to 2. For example, N may be equal to 8.

[0062] 4, a first AR block 402 in a group may be configured to receive data “y” and generate first output data “z0”. A second AR block 402 disposed immediately after the first AR block 402 may be configured to receive the first output data “z0” and generate second output data “z1”.

[0063] If the group includes only two AR blocks 402 (i.e., the first AR block and the second AR block described above), the second output data “z1” may be provided to the final processing unit 550 (shown in FIG. 5) of the NN filter 230 / 330.

[0064] On the other hand, if a group includes three or more AR blocks, each AR block 402 included in the group, except for the first AR block and the last AR block, may be configured to receive output data from the previous AR block 402 and provide the output data to the next AR block. The last AR block 402 may be configured to receive output data from the previous AR block and provide the output data to the final processing unit 550 of the NN filter 230 / 330.

[0065] Referring again to FIG. 4, some or all of the AR blocks 402 may include a spatial attention block 412 configured to generate an attention mask f. The attention mask f may have one channel and its size may be the same as the data "y". Taking the first AR block 402 as an example, the spatial attention block 412 included in the first AR block 402 may be configured to multiply the attention mask f by the residual data "r" to obtain data "rf". The data "rf" may be combined with the residual data "r" and then combined with the data "y", thereby generating a first output data "z0".

[0066] As shown in FIG. 5, the output Z N may be processed by a convolutional layer 502, a PReLU 504, another convolutional layer 506, pixel shuffling (or actually, sample shuffling) 508, and a final scaling 510 to generate the filtered output data 290 / 390. In this disclosure, any one or combination of the elements shown in FIG. 5 is referred to as a computation module.

[0067] Compared to the Luma intra-network architecture used in JVET-X0066, the NN filter 230 / 330 shown in Figures 4 and 5 improves the gain to -7.63% while maintaining the NN filter complexity at 430kMAC / sample (e.g., by removing "part" from the input while adding "dblk" to the input).

[0068] In some embodiments, the NN filters 230 / 330 shown in FIGS. 4 and 5 may be used to filter inter-luma samples, intra-chroma samples, and / or inter-chroma samples according to some embodiments.

[0069] When the NN filter 230 / 330 shown in FIGS. 4 and 5 is used to filter the inter-luma samples, the partition information (“part”) may be excluded from the input of the NN filter 230 / 330 and from the input of the spatial attention block 412.

[0070] When the NN filter 230 / 330 shown in FIGS. 4 and 5 is used to filter intra-chroma samples, the NN filter 230 / 330 receives the following seven inputs (instead of the six inputs shown in FIG. 4): (1) the value of the luma component of the reconstructed sample 280 / 380 (“rec”), (2) the value of the chroma components of the reconstructed sample 280 / 380 (“recUV”), and (3) the value of the chroma components (e.g., Cb and Cr) of the predicted sample 295 / 395 (“p The spatial attention block 412 may have a plurality of inputs, (4) partitioning information ("partUV") indicating how the chroma components of the sample are partitioned (more specifically, how the luma picture is partitioned into coding tree units (CTUs) and how the luma CTUs are partitioned into coding units); (5) block boundary strength (BBS) information ("bsUV") indicating the strength of filtering applied to the boundaries of the chroma components of the sample; (6) a quantization parameter ("qp"); and (7) additional input information. In some embodiments, the additional input information includes values ​​of the chroma components of the deblocked samples. Similarly, the above seven inputs may be used as inputs of the spatial attention block 412.

[0071] When the NN filter 230 / 330 shown in Figures 4 and 5 is used to filter interchroma samples, the partition information ("partUV") may be excluded from the above seven inputs of the NN filter 230 / 330 and from the seven inputs of the spatial attention block 412.

[0072] As described above, in some embodiments, the additional input information includes values ​​of the luma or chroma components of the deblocked samples, however, in other embodiments, the additional input information may include information about the predicted samples (a.k.a. "prediction mode information" or "I / P / B prediction mode information").

[0073] For example, the prediction mode information may indicate whether the sample block to be filtered is an intra-predicted block, a uni-predicted inter-predicted block, or a bi-predicted inter-predicted block. More specifically, the prediction mode information may be set to have a value of 0 if the sample belongs to an intra-predicted block, a value of 0.5 if the sample belongs to a uni-predicted inter-predicted block, or a value of 1 if the sample belongs to a bi-predicted inter-predicted block.

[0074] Because I-frames contain only I-blocks, the prediction mode information may be a constant (e.g., 0) when this architecture is used for a luma intra-network. On the other hand, when this architecture is used for a luma inter-network, the prediction mode information may be set to different values ​​for different samples, which can provide a Bjontegaard delta rate (BDR) gain over architectures that do not utilize this prediction mode information.

[0075] Instead of using the luma or chroma component values ​​of the deblocked samples or prediction mode information as additional input information, in some embodiments, motion vector (MV) information may be used as additional input information. The MV information may indicate the number of MVs (e.g., 0, 1, or 2) used in prediction. For example, 0 MVs may mean that the current block is an I-block, 1 MV may mean a P-block, and 2 MVs may mean a B-block.

[0076] In some embodiments, in addition to prediction mode information or MV information, prediction direction information indicating the direction of prediction for the samples undergoing filtering may be included in the additional input information.

[0077] Instead of using i) the luma or chroma component values ​​of the deblocked samples, ii) prediction mode information, or iii) MV information as additional input information, in some embodiments, coefficient information may be used as additional input information.

[0078] An example of coefficient information is skipped block information that indicates whether a block of samples undergoing NN filtering is a skipped block (i.e., a block that did not undergo the process performed by the transform unit 242, the quantization unit 243, the inverse quantization unit 245, and the inverse transform unit 246, or the process performed by the entropy decoder 361, the inverse quantization unit 362, and the inverse transform unit 363). In one example, the skipped block information may be set to have a value of 0 if the block of samples undergoing NN filtering is a non-skipped block, and set to have a value of 1 if the block is a skipped block.

[0079] For the encoder 112 shown in Figure 2, the skipped blocks may correspond to reconstructed samples 280 obtained based only on the predicted samples 295 (instead of the sum of the predicted samples 295 and the output from the inverse transform unit 246). Similarly, for the decoder 114 shown in Figure 3, the skipped blocks may correspond to reconstructed samples 380 obtained based only on the predicted samples 395 (instead of the sum of the predicted samples 395 and the output from the inverse transform unit 363).

[0080] Because I-frames contain only I-blocks and these blocks cannot be skipped, the skipped block information will be a constant (e.g., 0) when this architecture is used for a luma intra-network. On the other hand, when this architecture is used for a luma inter-network, the skipped block information may have different values ​​for different samples, which can provide a BDR gain over other alternative architectures that do not utilize skipped block information.

[0081] Referring again to Figure 4, as shown in Figure 4, in some embodiments, the NN filter 230 / 330 for filtering intra-luma samples has six inputs. However, in other embodiments, the partition information may be removed from the inputs, which brings the total number of inputs of the NN filter 230 / 330 to five, namely: (1) the value of the luma component of the reconstructed sample 280 / 380 ("rec"), (2) the value of the luma component of the predicted sample 295 / 395 ("pred"), (3) block boundary strength (BBS) information ("bs") indicating the strength of the filtering applied to the boundaries of the luma components of the samples, (4) the quantization parameter ("qp"), and (5) additional input information. In some embodiments, the additional input information includes the value of the luma component of the deblocked sample. 4 is used to filter inter-luma samples, the inputs of the NN filter 230 / 330 will have five inputs (excluding the partitioning information) instead of six inputs. Similarly, the inputs of the spatial attention block 412 will be five inputs instead of six inputs. Thus, in these embodiments, the inputs of the NN filter 230 / 330 used to filter intra-luma samples and the inputs of the NN filter 230 / 330 used to filter inter-luma samples will be the same.

[0082] Instead of removing the partition information from their inputs, in some embodiments, the BBS information may be removed from the inputs. Thus, in these embodiments, the inputs of the NN filter 230 / 330 for filtering intra-luma samples are (1) the value of the luma component of the reconstructed sample 280 / 380 ("rec"), (2) the value of the luma component of the predicted sample 295 / 395 ("pred"), (3) partition information ("part") indicating how the luma component of the sample is partitioned, (4) the quantization parameter ("qp"), and (5) additional input information. If the NN filter 230 / 330 is used to filter inter-luma samples, inter-chroma samples, or intra-chroma samples, the BBS information may be removed from the inputs of the NN filter 230 / 330 and the input of the special attention block 412.

[0083] As described above, in some embodiments, different inputs are provided to the NN filter 230 / 330 for its filtering operations. For example, in some embodiments, "rec", "pred", "part", "bs", "qp", and "dblk" are provided as inputs of the NN filter 230 / 330 for the luma component of intra predicted samples, and "rec", "pred", "bs", "qp", and "dblk" are provided as inputs of the NN filter 230 / 330 for the luma component of inter predicted samples. Similarly, in some embodiments, "rec", "recUV", "predUV", "partUV", "bsUV", "qp", and "dblk" are provided as inputs of the NN filter 230 / 330 for the chroma components of intra predicted samples, and "rec", "recUV", "predUV", "bsUV", "qp", and "dblk" are provided as inputs of the NN filter 230 / 330 for the luma component of inter predicted samples. In other words, four different NN filters 230 / 330 may be used for four different types of samples: inter-luma samples, intra-luma samples, inter-chroma samples, and intra-chroma samples.

[0084] However, in other embodiments, the same NN filter 230 / 330 may be used for the luma components of samples (regardless of whether the samples are inter- or intra-predicted), and the same NN filter 230 / 330 may be used for the chroma components of samples (regardless of whether the samples are inter- or intra-predicted). In such embodiments, instead of using two different filters, "IPB information" may be used to distinguish inter and intra blocks from each other. Thus, in one example, "rec", "pred", "part", "bs", "qp", and "IPB information" are provided as inputs of the NN filter 230 / 330 for the luma components of samples (regardless of whether the samples are inter- or intra-predicted), and "rec", "recUV", "predUV", "partUV", "bsUV", "qp", and "IPB information" are provided as inputs of the NN filter 230 / 330 for the chroma components of samples (regardless of whether the samples are inter- or intra-predicted).

[0085] Alternatively, in some embodiments, the same NN filter 230 / 330 may be used for any component of an intra predicted sample, and the same NN filter 230 / 330 may be used for any component of an inter predicted sample. In these embodiments, the same inputs are used for the luma component of a sample and the chroma component of a sample. Thus, in one example, "rec", "pred", "part", "bs", "recUV", "predUV", "partUV", "bsUV", "qp" are provided as inputs to the NN filter 230 / 330 for intra predicted samples, and "rec", "pred", "bs", "recUV", "predUV", "bsUV", "qp" are provided as inputs to the NN filter 230 / 330 for inter predicted samples. In these embodiments, the outputs of the NN filter 230 / 330 are NN filtered luma samples and NN filtered chroma samples.

[0086] Instead of using two different NN filters 230 / 330, in some embodiments, the same NN filter 230 / 330 may be used for four different types of samples, i.e., inter-luma samples, intra-luma samples, inter-chroma samples, and intra-chroma samples. In these embodiments, the inter or intra information may be provided by "IPB information" and cross component benefits may be provided by incorporating both luma-related and chroma-related inputs. Thus, in one example, "rec", "pred", "part", "bs", "recUV", "predUV", "partUV", "bsUV", "qp", and "IPB information" are provided as inputs of the NN filter 230 / 330 for the four different types of samples.

[0087] In the embodiments described above, the performance and / or efficiency of the NN filter 230 / 330 is improved by adjusting the input provided to the NN filter 230 / 330. However, in some embodiments, the performance and / or efficiency of the NN filter 230 / 330 is improved by modifying the structure of the AR block 402. More specifically, in some embodiments, the spatial attention block 412 may be removed from the first M AR blocks 402 as shown in FIG. 6C (compare the spatial attention block 412 shown in FIG. 4). For example, if eight (or sixteen) AR blocks 402 are included in the NN filter 230 / 330, the first seven (or fifteen) AR blocks 402 may not include the spatial attention block 412, and only the last AR block 402 may include the spatial attention block 412. In another example, none of the AR blocks 402 included in the NN filter 230 / 330 include the spatial attention block 412.

[0088] In some embodiments, instead of or in addition to adjusting the input of the NN filter 230 / 330 and / or removing the spatial attention block 412 from the AR block 402, the performance and / or efficiency of the NN filter 230 / 330 may be improved by adjusting the capacity of the spatial attention block 412. For example, in some embodiments, the number of layers in the spatial attention block 412 may be increased (relative to the number of layers in JVET-X0066) and the layers may be configured to perform downsampling and upsampling to improve the performance of capturing latent correlations. An example of the spatial attention block 412 according to these embodiments is shown in FIG. 6A.

[0089] In some embodiments, instead of or in addition to increasing the capacity of the spatial attention block 412, the output of the spatial attention block 412 may be increased from one channel to multiple channels to provide per-spatial and per-channel attention. For example, in general, in the CNN layer(s) included in the spatial attention block 412, a single kernel (e.g., having a size of 3×3) is used to perform the convolution operation. However, in these embodiments, multiple kernels (e.g., 96) may be used to perform the convolution operation. As a result of using multiple kernels (each of which is associated with a particular channel), multiple channel outputs may be generated.

[0090] In some embodiments, only "qp" may be used in creating or generating per-channel attention. For example, as shown in FIG. 6B, a multi-layer perceptron (MLP) may be used to generate multiple channel outputs using "QP" as an input. MLPs are a class of feed-forward artificial neural networks (ANNs, usually simply called neural networks). The term MLP is sometimes loosely used to mean any feed-forward ANN, and sometimes strictly and ambiguously to refer to a network composed of multiple layers of perceptrons (with threshold activation). Multi-layer perceptrons are sometimes colloquially referred to as "vanilla" neural networks, especially when they have a single hidden layer. MLPs consist of at least three layers of nodes: an input layer, a hidden layer, and an output layer. An exemplary implementation of these embodiments is shown in FIG. 6D. As shown in FIG. 6D, the spatial attention block 412 may include a PReLU layer, a dense layer, and a SoftPlus layer (or layers), which may be used together to generate multiple channel outputs using "QP" as an input. In some embodiments, the Softplus layer (also known as the Softplus activation function) is x) A dense layer is a layer that is deeply connected to its preceding layer, meaning that the neurons of the layer are connected to every neuron in the preceding layer. In some embodiments, instead of performing an attention operation, e.g., multiplication, from the output of the MLP with the residual data "r", the operation is performed by multiplying the output data "z i It should be noted that in this disclosure, any one or combination of the elements shown in Figures 6A-6D is referred to as a computation module.

[0091] Embodiments of the present disclosure provide at least one of the following advantages.

[0092] By retraining the luma intra model from JVET-Y0143, the model gives a luma gain of -7.57% for all intra settings. The difference between the previous gain of 7.39% reported in JVET-X0066 and the retrained network gain of 7.57% is due to the different training procedure. As an example, the training time for the retrained network was longer. By removing the part input "part", the gain is still 7.57% and the complexity is reduced from 430kMAC / pixel to 419kMAC / pixel. By removing the additional bs input "bs", the gain is 7.42% and the complexity is reduced to 408kMAC / pixel.

[0093] By removing the first 7 spatial attention masks, the gain is 7.60% and the complexity is reduced from 430kMAC / pixel to 427kMAC / pixel.

[0094] By using the deblocked information as input and removing the piecewise input, the gain improves to 7.63% while the complexity remains the same at 430kMAC / pixel.

[0095] By removing all spatial attention masks, the gain is 7.72% for class D sequences.

[0096] By increasing the capacity of the attention branch, the gain is improved from 7.70% to 7.85% for class D sequences. The complexity is increased by about 120%.

[0097] By using per-channel attention with qp as input, a gain of 7.78% is obtained for class D sequences and the complexity is reduced to 428.7kMAC / pixel.

[0098] FIG. 7 illustrates a process 700 for generating an encoded or decoded video according to some embodiments. The process 700 may begin with step s702. Step s702 includes obtaining values ​​of reconstructed samples. Step s704 includes obtaining input information including any one or combination of i) information about filtered samples, ii) information about predicted samples, or iii) information about skipped samples. Step s706 includes providing the values ​​of the reconstructed samples and the input information to a machine learning (ML) model, thereby generating at least one ML output data. Step s708 includes generating an encoded or decoded video based at least on the at least one ML output data.

[0099] In some embodiments, the information regarding prediction indicates a prediction mode, the prediction modes including intra-prediction, unidirectional inter-prediction, and bidirectional inter-prediction.

[0100] In some embodiments, the information regarding the prediction indicates the number of motion vectors used for the prediction.

[0101] In some embodiments, the ML model comprises a first computation module (CM) and a second CM, the first CM comprises a first convolutional layer (CL) and a first parametric rectified linear unit (PReLU) coupled to the first CL, the second CM comprises a second CL and a second PReLU coupled to the second CL, and the reconstructed sample values ​​are provided to the first CM and the input information is provided to the second CM.

[0102] In some embodiments, the process 700 further includes obtaining predicted sample values, obtaining block boundary strength information (BBS) indicating the strength of filtering applied to sample boundaries, obtaining a quantization parameter (QP), providing the predicted sample values ​​to a CM, thereby generating first CM output data, providing the BBS information to the CM, thereby generating second CM output data, providing the QP to the CM, thereby generating third CM output data, and combining at least the first CM output data, the second CM output data, and the third CM output data, thereby generating combined CM output data, and the encoded video or decoded video is generated based at least on the combined CM output data.

[0103] In some embodiments, the information about the filtered samples includes values ​​of the deblocked samples.

[0104] In some embodiments, the information about a skipped sample indicates whether the sample belongs to a block that did not undergo a process of processing residual samples, which process includes inverse quantization and inverse transformation.

[0105] In some embodiments, the process 700 includes concatenating values ​​of the reconstructed samples and the input information, thereby generating concatenated CM input data, which is provided to the CM.

[0106] In some embodiments, the ML model comprises a first CM and a second CM, the first CM comprises a first convolutional layer (CL) and a first parametric regularized linear unit (PReLU) coupled to the first CL, the second CM comprises a second CL and a second PReLU coupled to the second CL, and the first CM is configured to perform downsampling and the second CM is configured to perform upsampling.

[0107] In some embodiments, the ML model comprises a first convolutional layer (CL), the first CL configured to convert the concatenated CM input data into N CM output data, where N is the number of kernel filters included in the first CL.

[0108] In some embodiments, the input information includes information regarding the predicted samples, and the method further includes obtaining partition information indicating how the samples are partitioned and providing the partition information to a CM, thereby generating fourth CM output data, and the combined CM output data is generated based on combining at least the first CM output data, the second CM output data, the third CM output data, and the fourth CM output data.

[0109] In some embodiments, the reconstructed sample values ​​include values ​​of the luma component of the reconstructed sample and values ​​of the chroma components of the reconstructed sample, the predicted sample values ​​include values ​​of the luma component of the predicted sample and values ​​of the chroma components of the predicted sample, and the BBS information indicates the strength of filtering applied to the boundaries of the luma components of the sample and the strength of filtering applied to the boundaries of the chroma components of the sample.

[0110] In some embodiments, process 700 includes obtaining first partition information indicating how a luma component of a sample is partitioned, obtaining second partition information indicating how a chroma component of the sample is partitioned, providing the first partition information to a CM, thereby generating fourth CM output data, and providing the second partition information to a CM, thereby generating fifth CM output data, where the input information includes information about the predicted sample, and the combined CM output data is generated based on combining at least the first CM output data, the second CM output data, the third CM output data, the fourth CM output data, and the fifth CM output data.

[0111] FIG. 8 shows a process 800 for generating encoded or decoded video. The process 800 may begin at step s802. Step s802 includes obtaining machine learning (ML) input data, the ML input data including: i) reconstructed sample values; ii) predicted sample values; iii) block boundary strength (BBS) information indicating strength of filtering applied to sample boundaries; and iv) quantization parameters (QPs). Step s804 includes providing the ML input data to an ML model, thereby generating ML output data. Step s806 includes generating the encoded or decoded video based at least on the ML output data (s806). The ML input data does not include partitioning information indicating how luma pictures are partitioned into coding tree units (CTUs) and how luma CTUs are partitioned into coding units (CUs).

[0112] In some embodiments, the ML model comprises a first CM, a second CM, a third CM, and a fourth CM, where the first CM comprises a first convolutional layer (CL) and a first parametric regularized linear unit (PReLU) coupled to the first CL, the second CM comprises a second CL and a second PReLU coupled to the second CL, the third CM comprises a third CL and a third PReLU coupled to the third CL, and the fourth CM comprises a fourth CL and a fourth PReLU coupled to the fourth CL, where the reconstructed sample values ​​are provided to the first CM, the predicted sample values ​​are provided to the second CM, the BBS information is provided to the third CM, and the QP is provided to the fourth CM.

[0113] In some embodiments, the values ​​of the reconstructed samples include values ​​of luma components of the reconstructed samples and values ​​of chroma components of the reconstructed samples, and the ML model comprises a first CM, a second CM, a third CM, a fourth CM, and a fifth CM, the first CM comprises a first convolutional layer (CL) and a first parametric regularized linear unit (PReLU) coupled to the first CL, the second CM comprises a second CL and a second PReLU coupled to the second CL, and the third CM comprises the fourth CM comprises a fourth CL and a fourth PReLU combined with the fourth CL; the fifth CM comprises a fifth CL and a fifth PReLU combined with the fifth CL; values ​​of the luma components of the reconstructed samples are provided to the first CM, values ​​of the chroma components of the reconstructed samples are provided to the second CM, values ​​of the predicted samples are provided to the third CM, BBS information is provided to the fourth CM, and a QP is provided to the fifth CM.

[0114] 9 shows a process 900 for generating encoded or decoded video. The process 900 may begin at step s902. Step s902 includes obtaining machine learning (ML) input data, the ML input data including i) reconstructed sample values, ii) predicted sample values, and iii) a quantization parameter (QP). Step s904 includes providing the ML input data to an ML model, thereby generating ML output data. Step s906 includes generating the encoded or decoded video based at least on the ML output data (s906), the ML input data not including block boundary strength (BBS) information indicating a strength of filtering applied to sample boundaries.

[0115] In some embodiments, the ML model comprises a first CM, a second CM, and a third CM, the first CM comprises a first convolutional layer (CL) and a first parametric regularized linear unit (PReLU) coupled to the first CL, the second CM comprises a second CL and a second PReLU coupled to the second CL, and the third CM comprises a third CL and a third PReLU coupled to the third CL, where the reconstructed sample values ​​are provided to the first CM, the predicted sample values ​​are provided to the second CM, and the QP is provided to the third CM.

[0116] In some embodiments, the values ​​of the reconstructed samples include values ​​of luma components of the reconstructed samples and values ​​of chroma components of the reconstructed samples, the ML model comprises a first CM, a second CM, a third CM, and a fourth CM, the first CM comprises a first convolutional layer (CL) and a first parametric regularized linear unit (PReLU) coupled to the first CL, the second CM comprises a second CL and a second PReLU coupled to the second CL, the third CM comprises a third CL and a third PReLU coupled to the third CL, the fourth CM comprises a fourth CL and a fourth PReLU coupled to the fourth CL, the values ​​of the luma components of the reconstructed samples are provided to the first CM, the values ​​of the chroma components of the reconstructed samples are provided to the second CM, the values ​​of the predicted samples are provided to the third CM, and the QP is provided to the fourth CM.

[0117] In some embodiments, the values ​​of the reconstructed samples include values ​​of luma components of the reconstructed samples and values ​​of chroma components of the reconstructed samples, the ML input data further includes partition information indicating how the samples are partitioned, and the ML model comprises a first CM, a second CM, a third CM, a fourth CM, and a fifth CM, the first CM comprising a first convolutional layer (CL) and a first parametric regularized linear unit (PReLU) coupled to the first CL, and the second CM comprises a second CM, a third CM, a fourth CM, and a fifth CM, the first CM comprising a first convolutional layer (CL) and a first parametric regularized linear unit (PReLU) coupled to the first CL, , the first CM has a second CL and a second PReLU combined with the second CL, the third CM has a third CL and a third PReLU combined with the third CL, the fourth CM has a fourth CL and a fourth PReLU combined with the fourth CL, the values ​​of the luma components of the reconstructed samples are provided to the first CM, the values ​​of the chroma components of the reconstructed samples are provided to the second CM, the values ​​of the predicted samples are provided to the third CM, the QP is provided to the fourth CM, and the partition information is provided to the fifth CM.

[0118] FIG. 10 shows a process 1000 for generating encoded or decoded video. The process 1000 may begin with step s1002. Step s1002 includes obtaining reconstructed sample values. Step s1004 includes obtaining a quantization parameter (QP). Step s1006 includes providing the reconstructed sample values ​​and the quantization parameter to a machine learning (ML) model, thereby generating ML output data. Step s1008 includes generating first output sample values ​​based at least on the ML output data. Step s1010 includes providing the first output sample values ​​to a group of two or more attention residual blocks connected in series. The group of attention residual blocks includes a first attention residual block disposed at one end of the series of attention residual blocks, the first attention residual block configured to receive first input data consisting of first output sample values ​​and generate second output sample values ​​based on the first output sample values.

[0119] In some embodiments, the group of attention residual blocks includes a second attention residual block disposed at an opposite end of the series of attention residual blocks, the second attention residual block configured to receive second input data including reconstructed sample values ​​and / or QPs, and the second attention residual block configured to generate third output sample values ​​based on the reconstructed sample values ​​and / or QPs.

[0120] In some embodiments, the process 1000 includes obtaining predicted sample values, obtaining block boundary strength (BBS) information indicating the strength of filtering applied to the sample boundaries, and providing the predicted sample values ​​and the BBS information to a CM, thereby generating spatial attention mask data, and a third output sample value is generated based on the spatial attention mask data.

[0121] 11 shows a process 1100 for generating encoded or decoded video. The process 1100 includes obtaining machine learning (ML) input data, the ML input data including: i) values ​​of luma components of reconstructed samples; ii) values ​​of chroma components of reconstructed samples; iii) values ​​of luma components of predicted samples; iv) values ​​of chroma components of predicted samples; v) first block boundary strength (BBS) information indicating a strength of filtering applied to a boundary of the luma components of the samples; vi) second BBS information indicating a strength of filtering applied to a boundary of the chroma components of the samples; and vii) a quantization parameter (QP). The process 1100 includes providing the ML input data to an ML model, thereby generating ML output data; and generating the encoded or decoded video based at least on the ML output data.

[0122] FIG. 13 shows a process 1300 for generating encoded or decoded video. The process 1300 may begin at step s1302. Step s1302 includes obtaining reconstructed sample values. Step s1304 includes obtaining a quantization parameter (QP). Step s1306 includes providing the reconstructed sample values ​​and the quantization parameter to a machine learning (ML) model, thereby generating ML output data. Step s1308 includes generating a first output sample value based at least on the ML output data. Step s1310 includes providing the first output sample value to a group of two or more attention residual blocks connected in series, thereby generating a second output sample value. Step s1312 includes generating encoded video or decoded video based on the second output sample value, where the group of attention residual blocks includes a first attention residual block arranged at one end of the series of attention residual blocks, and the first attention residual block is configured to receive input data consisting of the first output sample value and a QP.

[0123] 12 is a block diagram of an apparatus 1200 for implementing the encoder 112, the decoder 114, or a component included in the encoder 112 or the decoder 114 (e.g., the NN filter 280 or 330) according to some embodiments. When the apparatus 1200 implements a decoder, the apparatus 1200 may be referred to as a "decoding apparatus 1200," and when the apparatus 1200 implements an encoder, the apparatus 1200 may be referred to as an "encoding apparatus 1200." As shown in FIG. 12 , the device 1200 includes a processing circuit (PC) 1202 that may include one or more processors (P) 1255 (e.g., a general-purpose microprocessor and / or one or more other processors, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc.), which may be co-located in a single housing or in a single data center or may be geographically distributed (i.e., the device 1200 may be a distributed computing device), and at least one network interface 1248, such that the device 1200 is The PC 1202 may comprise at least one network interface 1248, which may include one or more non-volatile storage devices and / or one or more volatile storage devices, and a storage unit (aka "data storage system") 1208, which may include a transmitter (Tx) 1245 and a receiver (Rx) 1247 for enabling the PC 1202 to transmit data to and receive data from other nodes connected to the network 110 (e.g., an Internet Protocol (IP) network) to which it is connected (directly or indirectly) (e.g., the network interface 1248 may be wirelessly connected to the network 110, in which case the network interface 1248 is connected in an antenna configuration). In an embodiment in which the PC 1202 includes a programmable processor, a computer program product (CPP) 1241 may be provided. The CPP 1241 includes a computer readable medium (CRM) 1242, which stores a computer program (CP) 1243 comprising computer readable instructions (CRI) 1244.The CRM 1242 may be a non-transitory computer-readable medium, such as a magnetic medium (e.g., a hard disk), an optical medium, a memory device (e.g., a random access memory, a flash memory), etc. In some embodiments, the CRI 1244 of the computer program 1243, when executed by the PC 1202, configures the CRI to cause the device 1200 to perform steps described herein (e.g., steps described herein with reference to a flowchart). In other embodiments, the device 1200 may be configured to perform steps described herein without the need for code. That is, for example, the PC 1202 may simply consist of one or more ASICs. Thus, features of the embodiments described herein may be implemented in hardware and / or software.

[0124] Overview of the embodiment A1. A method (700) for generating encoded or decoded video, the method comprising: Obtaining values ​​of the reconstructed samples (s702); and obtaining input information including any one or combination of: i) information about filtered samples; ii) information about predicted samples; or iii) information about skipped samples (s704); providing the reconstructed sample values ​​and the input information to a machine learning (ML) model, thereby generating at least one ML output data (s706); generating (s708) an encoded video or a decoded video based at least on the at least one ML output data; and A method (700). A2. the ML model comprises a first pair of models and a second pair of models; a first pair of models comprising a first convolutional neural network (CNN) and a first parametric rectified linear unit (PReLU) coupled to the first CNN; a second pair of models comprising a second CNN and a second PReLU coupled to the second CNN; The reconstructed sample values ​​are provided to the first CNN, The input information is provided to a second CNN. The method according to embodiment A1. A3. The method is Obtaining predicted sample values; obtaining block boundary strength information (BBS) indicative of strength of filtering applied to sample boundaries; Obtaining a quantization parameter (QP); providing the predicted sample values ​​to the ML model, thereby generating at least a first ML output data; providing the BBS information to an ML model, thereby generating at least second ML output data; providing the QP to the ML model, thereby generating at least a third ML output data; combining the at least one ML output data, the at least one ML output data, the at least a second ML output data, and the at least a third ML output data, thereby generating combined ML output data; Further comprising: The method of embodiment A1 or A2, in which the encoded video or the decoded video is generated based at least on the combined ML output data. A4. The method of any one of embodiments A1 to A3, wherein the information about the filtered samples includes values ​​of deblocked samples. A5. The information about the prediction indicates a prediction mode; Prediction modes include intra prediction, unidirectional inter prediction, and bidirectional inter prediction. The method according to any one of embodiments A1 to A4. A6. The method of any one of embodiments A1 to A5, wherein the information regarding the prediction indicates the number of motion vectors used for the prediction. A7. The information about the skipped sample indicates whether the sample belongs to a block that did not go through the process of processing the residual samples; The process includes inverse quantization and inverse transformation. The method of any one of embodiments A1 to A6. A8. Concatenating the reconstructed sample values ​​and the input information, thereby generating concatenated ML input data; The concatenated ML input data is provided to the ML model; The method according to embodiment A1. A9. the ML model comprises a first pair of models and a second pair of models; a first pair of models comprising a first convolutional neural network (CNN) and a first parametric rectified linear unit (PReLU) coupled to the first CNN; a second pair of models comprising a second CNN and a second PReLU coupled to the second CNN; The first CNN is configured to perform downsampling; The second CNN was configured to perform upsampling. The method according to embodiment A1 or A8. A10. The ML model is equipped with a convolutional neural network (CNN), The CNN is configured to convert the concatenated ML input data into N ML output data; N is the number of kernel filters in the CNN. The method according to embodiment A8 or A9. A11. the input information includes information regarding the predicted samples; The method is: obtaining partition information indicative of how the sample is partitioned; providing the partition information to the ML model, thereby generating a fourth ML output data; Further comprising: a combined ML output data is generated based on combining the at least one ML output data, the first ML output data, the second ML output data, the third ML output data, and the fourth ML output data. The method of embodiment A3. A12. the reconstructed sample values ​​include values ​​of luma components of the reconstructed sample and values ​​of chroma components of the reconstructed sample; the predicted sample values ​​include predicted sample luma component values ​​and predicted sample chroma component values; the BBS information indicates the strength of filtering applied to the boundaries of the luma components of the samples and the strength of filtering applied to the boundaries of the chroma components of the samples; The method of embodiment A3. A13. The method is: Obtaining first partition information indicative of how luma components of the samples are partitioned; obtaining second partitioning information indicative of how chroma components of the sample are partitioned; providing the first partition information to an ML model, thereby generating fourth ML output data; providing the second partition information to the ML model, thereby generating fifth ML output data; Further comprising: the input information includes information regarding the predicted samples; The method of embodiment A12, wherein the combined ML output data is generated based on combining the at least one ML output data, the first ML output data, the second ML output data, the third ML output data, the fourth ML output data, and the fifth ML output data. B1. A method (800) for generating encoded or decoded video, the method comprising: Obtaining machine learning (ML) input data (s802), the ML input data comprising: i) the reconstructed sample values; and ii) the predicted sample values; and iii) Block Boundary Strength (BBS) information indicating the strength of the filtering applied to the sample boundaries; iv) Quantization parameter (QP) obtaining machine learning (ML) input data (s802); providing ML input data to the ML model, thereby generating ML output data (s804); generating (s806) encoded video or decoded video based at least on the ML output data; and Including, A method (800), in which the ML input data does not include partitioning information that indicates how luma pictures are partitioned into coding tree units (CTUs) and how luma CTUs are partitioned into coding units (CUs). B2. the ML models comprise a first pair of models, a second pair of models, a third pair of models, and a fourth pair of models; a first pair of models comprising a first convolutional neural network (CNN) and a first parametric rectified linear unit (PReLU) coupled to the first CNN; a second pair of models comprising a second CNN and a second PReLU coupled to the second CNN; a third pair of models comprising a third CNN and a third PReLU coupled to the third CNN; a fourth pair of models comprising a fourth CNN and a fourth PReLU coupled to the fourth CNN; The reconstructed sample values ​​are provided to the first CNN, The predicted sample values ​​are provided to a second CNN, BBS information was provided to the third CNN, The QP is provided to the fourth CNN. The method of embodiment B1. B3. the reconstructed sample values ​​include values ​​of luma components of the reconstructed sample and values ​​of chroma components of the reconstructed sample; the ML models comprise a first pair of models, a second pair of models, a third pair of models, a fourth pair of models, and a fifth pair of models; a first pair of models comprising a first convolutional neural network (CNN) and a first parametric rectified linear unit (PReLU) coupled to the first CNN; a second pair of models comprising a second CNN and a second PReLU coupled to the second CNN; a third pair of models comprising a third CNN and a third PReLU coupled to the third CNN; a fourth pair of models comprising a fourth CNN and a fourth PReLU coupled to the fourth CNN; a fifth pair of models comprising a fifth CNN and a fifth PReLU coupled to the fifth CNN; The luma component values ​​of the reconstructed samples are provided to a first CNN; The chroma component values ​​of the reconstructed samples are fed to a second CNN, The predicted sample values ​​are provided to a third CNN, BBS information was provided to the fourth CNN, The QP is provided to the fifth CNN. The method of embodiment B1. C1. A method (900) for generating encoded or decoded video, the method comprising: and obtaining (s902) machine learning (ML) input data, the ML input data comprising: i) the reconstructed sample values; and ii) the predicted sample values; and iii) Quantization parameter (QP) obtaining machine learning (ML) input data (s902); providing ML input data to the ML model, thereby generating ML output data (s904); generating (s906) an encoded video or a decoded video based at least on the ML output data; and Including, A method (900) in which the ML input data does not include block boundary strength (BBS) information that indicates the strength of filtering applied to sample boundaries. C2. the ML models comprise a first pair of models, a second pair of models, and a third pair of models; a first pair of models comprising a first convolutional neural network (CNN) and a first parametric rectified linear unit (PReLU) coupled to the first CNN; a second pair of models comprising a second CNN and a second PReLU coupled to the second CNN; a third pair of models comprising a third CNN and a third PReLU coupled to the third CNN; The reconstructed sample values ​​are provided to the first CNN, The predicted sample values ​​are provided to a second CNN, The QP is then fed to the third CNN. The method of embodiment C1. C3. the reconstructed sample values ​​include values ​​of luma components of the reconstructed sample and values ​​of chroma components of the reconstructed sample; the ML models comprise a first pair of models, a second pair of models, a third pair of models, and a fourth pair of models; a first pair of models comprising a first convolutional neural network (CNN) and a first parametric rectified linear unit (PReLU) coupled to the first CNN; a second pair of models comprising a second CNN and a second PReLU coupled to the second CNN; a third pair of models comprising a third CNN and a third PReLU coupled to the third CNN; a fourth pair of models comprising a fourth CNN and a fourth PReLU coupled to the fourth CNN; The luma component values ​​of the reconstructed samples are provided to a first CNN; The chroma component values ​​of the reconstructed samples are fed to a second CNN, The predicted sample values ​​are provided to a third CNN, The QP is provided to the fourth CNN. The method of embodiment C1. C4. the reconstructed sample values ​​include values ​​of luma components of the reconstructed sample and values ​​of chroma components of the reconstructed sample; The ML input data further includes partitioning information indicating how the samples are partitioned; the ML models comprise a first pair of models, a second pair of models, a third pair of models, a fourth pair of models, and a fifth pair of models; a first pair of models comprising a first convolutional neural network (CNN) and a first parametric rectified linear unit (PReLU) coupled to the first CNN; a second pair of models comprising a second CNN and a second PReLU coupled to the second CNN; a third pair of models comprising a third CNN and a third PReLU coupled to the third CNN; a fourth pair of models comprising a fourth CNN and a fourth PReLU coupled to the fourth CNN; The luma component values ​​of the reconstructed samples are provided to a first CNN; The chroma component values ​​of the reconstructed samples are fed to a second CNN, The predicted sample values ​​are provided to a third CNN, The QP is fed to the fourth CNN, The segmentation information is provided to a fifth CNN, The method of embodiment C1. D1. A method (1000) for generating encoded or decoded video, the method comprising: Obtaining values ​​of the reconstructed samples (s1002); and Obtaining a quantization parameter (QP) (s1004); providing the reconstructed sample values ​​and the quantization parameters to a machine learning (ML) model, thereby generating ML output data (s1006); generating first output sample values ​​based at least on the ML output data (s1008); providing the first output sample value to a group of two or more attention residual blocks connected in series (s1010); Including, The group of attention residual blocks includes a first attention residual block disposed at one end of the series of attention residual blocks; A method (1000), wherein a first attention residual block is configured to receive first input data consisting of first output sample values, and to generate second output sample values ​​based on the first output sample values. D2. the group of attention residual blocks includes a second attention residual block disposed at an opposite end of the series of attention residual blocks; a second attention residual block configured to receive second input data including values ​​of the reconstructed samples and / or a QP; the second attention residual block is configured to generate a third output sample value based on the reconstructed sample value and / or the QP; The method of embodiment D1. D3. The method is Obtaining predicted sample values; obtaining block boundary strength (BBS) information indicative of a strength of filtering applied to sample boundaries; providing the predicted sample values ​​and the BBS information to an ML model, thereby generating spatial attention mask data; Further comprising: The method of embodiment D1 or D2, wherein the third output sample values ​​are generated based on the spatial attention mask data. E1. A method (1100) for generating encoded or decoded video, the method comprising: Obtaining machine learning (ML) input data (s1102), the ML input data comprising: i) the value of the luma component of the reconstructed sample; and ii) the values ​​of the chroma components of the reconstructed samples; and iii) the value of the luma component of the predicted sample; and iv) the value of the chroma component of the predicted sample; and v) first block boundary strength (BBS) information indicating a strength of filtering applied to a boundary of the luma component of the sample; vi) second BBS information indicating the strength of the filtering applied to the boundaries of the chroma components of the sample; vii) Quantization parameter (QP) obtaining machine learning (ML) input data (s1102); providing ML input data to the ML model, thereby generating ML output data (s1104); generating encoded or decoded video based at least on the ML output data (s1106); A method (1100). F1. A method (1300) for generating encoded or decoded video, the method comprising: Obtaining values ​​of the reconstructed samples (s1302); and Obtaining a quantization parameter (QP) (s1304); providing the reconstructed sample values ​​and the quantization parameters to a machine learning (ML) model, thereby generating ML output data (s1306); generating first output sample values ​​based at least on the ML output data (s1308); providing the first output sample value to a group of two or more attention residual blocks connected in series to thereby generate a second output sample value (s1310); generating an encoded video or a decoded video based on the second output sample values ​​(s1312); Including, The group of attention residual blocks includes a first attention residual block disposed at one end of the series of attention residual blocks; The method (1300), wherein a first attention residual block is configured to receive input data consisting of a first output sample value and a QP. G1. A computer program (1243) comprising instructions (1244) that, when executed by a processing circuit (1202), cause the processing circuit to perform a method according to any one of embodiments A1 to F1. G2. A carrier containing the computer program of embodiment G1, the carrier being one of an electronic signal, an optical signal, a radio signal, and a computer-readable storage medium. H1. An apparatus (1200) for generating encoded or decoded video, the apparatus comprising: Obtaining values ​​of the reconstructed samples (s702); and obtaining input information including any one or combination of: i) information about filtered samples; ii) information about predicted samples; or iii) information about skipped samples (s704); providing the reconstructed sample values ​​and the input information to a machine learning (ML) model, thereby generating at least one ML output data (s706); generating (s708) an encoded video or a decoded video based at least on the at least one ML output data; and The apparatus (1200) is configured to perform the above. H2. The apparatus of embodiment H1, wherein the apparatus is further configured to perform a method according to any one of embodiments A2 to A13. I1. An apparatus (1200) for generating encoded or decoded video, the apparatus comprising: Obtaining machine learning (ML) input data (s802), the ML input data comprising: i) the reconstructed sample values; and ii) the predicted sample values; and iii) Block Boundary Strength (BBS) information indicating the strength of the filtering applied to the sample boundaries; iv) Quantization parameter (QP) obtaining machine learning (ML) input data (s802); providing ML input data to the ML model, thereby generating ML output data (s804); generating (s806) encoded video or decoded video based at least on the ML output data; and It is set to do An apparatus (1200), in which the ML input data does not include partitioning information indicating how luma pictures are partitioned into coding tree units (CTUs) and how luma CTUs are partitioned into coding units (CUs). I2. The apparatus of embodiment I1, wherein the apparatus is further configured to perform the method of embodiment B2 or B3. J1. An apparatus (1200) for generating encoded or decoded video, the apparatus comprising: and obtaining (s902) machine learning (ML) input data, the ML input data comprising: i) the reconstructed sample values; and ii) the predicted sample values; and iii) Quantization parameter (QP) obtaining machine learning (ML) input data (s902); providing ML input data to the ML model, thereby generating ML output data (s904); generating (s906) an encoded video or a decoded video based at least on the ML output data; and It is set to do An apparatus (1200) in which the ML input data does not include block boundary strength (BBS) information indicating the strength of filtering applied to sample boundaries. J2. The apparatus of embodiment J1, wherein the apparatus is further configured to perform a method according to any one of embodiments C2 to C4. K1. An apparatus (1200) for generating encoded or decoded video, the apparatus comprising: Obtaining values ​​of the reconstructed samples (s1002); and Obtaining a quantization parameter (QP) (s1004); providing the reconstructed sample values ​​and the quantization parameters to a machine learning (ML) model, thereby generating ML output data (s1006); generating first output sample values ​​based at least on the ML output data (s1008); providing the first output sample value to a group of two or more attention residual blocks connected in series (s1010); It is set to do The group of attention residual blocks includes a first attention residual block disposed at one end of the series of attention residual blocks; An apparatus (1200), wherein a first attention residual block is configured to receive first input data consisting of first output sample values, and to generate second output sample values ​​based on the first output sample values. K2. The apparatus of embodiment K1, wherein the apparatus is further configured to perform the method of embodiment D2 or D3. L1. An apparatus (1200) for generating encoded or decoded video, the apparatus comprising: Obtaining machine learning (ML) input data (s1102), the ML input data comprising: i) the value of the luma component of the reconstructed sample; and ii) the values ​​of the chroma components of the reconstructed samples; and iii) the value of the luma component of the predicted sample; and iv) the value of the chroma component of the predicted sample; and v) first block boundary strength (BBS) information indicating a strength of filtering applied to a boundary of the luma component of the sample; vi) second BBS information indicating the strength of the filtering applied to the boundaries of the chroma components of the sample; vii) Quantization parameter (QP) obtaining machine learning (ML) input data (s1102); providing ML input data to the ML model, thereby generating ML output data (s1104); generating encoded or decoded video based at least on the ML output data (s1106); The apparatus (1200) is configured to perform the above. M1. An apparatus (1200) for generating encoded or decoded video, the apparatus comprising: Obtaining values ​​of the reconstructed samples (s1302); and Obtaining a quantization parameter (QP) (s1304); providing the reconstructed sample values ​​and the quantization parameters to a machine learning (ML) model, thereby generating ML output data (s1306); generating first output sample values ​​based at least on the ML output data (s1308); providing the first output sample value to a group of two or more attention residual blocks connected in series to thereby generate a second output sample value (s1310); generating an encoded video or a decoded video based on the second output sample values ​​(s1312); It is set to do The group of attention residual blocks includes a first attention residual block disposed at one end of the series of attention residual blocks; An apparatus (1200), wherein a first attention residual block is configured to receive input data consisting of a first output sample value and a QP. N1. An apparatus (1200), comprising: Memory (1241) and A processing circuit (1202) coupled to a memory; An apparatus (1200) comprising:

[0125] conclusion

[0126] Although various embodiments have been described herein, it should be understood that the embodiments have been presented by way of example only, and not by way of limitation. Thus, the breadth and scope of the present disclosure should not be limited by any of the exemplary embodiments described above. Moreover, unless otherwise indicated herein or clearly contradicted by context, any combination of the above-described elements in all possible variations thereof is encompassed by the present disclosure.

[0127] Additionally, while the processes described above and illustrated in the Figures have been shown as a sequence of steps, this has been done for purposes of illustration only, and it is therefore contemplated that some steps may be added, some steps may be omitted, the order of the steps may be rearranged, and some steps may be performed in parallel.

Claims

1. A method (700) for generating encoded or decoded video, the method comprising: Obtaining (s702) values ​​of the reconstructed samples; Obtaining input information (s704) including any one or combination of: i) information about filtered samples; ii) information about predicted samples; or iii) information about skipped samples; providing (s706) the values ​​of the reconstructed samples and the input information to a machine learning (ML) model, thereby generating at least one ML output; generating (s708) the encoded video or the decoded video based at least on the at least one ML output data; and The method (700).

2. The information about the prediction indicates a prediction mode; the prediction modes include intra prediction, unidirectional inter prediction, and bidirectional inter prediction; The method of claim 1.

3. The method of claim 1 or 2, wherein the information on the prediction indicates a number of motion vectors used for the prediction.

4. the ML model comprises a first calculation module (CM) and a second CM; the first CM comprises a first convolutional layer (CL) and a first parametric rectified linear unit (PReLU) coupled to the first CL; the second CM comprises a second CL and a second PReLU coupled to the second CL; the values ​​of the reconstructed samples are provided to the first CM; The input information is provided to the second CM.

4. The method according to any one of claims 1 to 3.

5. The method further comprising: Obtaining predicted sample values; Obtaining block boundary strength information (BBS) indicative of the strength of filtering applied to sample boundaries; Obtaining a quantization parameter (QP); providing the values ​​of the predicted samples to a CM, thereby generating first CM output data; providing the BBS information to a CM, thereby generating second CM output data; providing the QP to a CM, thereby generating third CM output data; combining at least the first CM output data, the second CM output data, and the third CM output data, thereby generating combined CM output data. Further comprising: The method of claim 1 , wherein the encoded video or the decoded video is generated based at least on the combined CM output data.

6. The method of claim 1 , wherein the information about filtered samples comprises values ​​of deblocked samples.

7. the information on skipped samples indicates whether the samples belong to a block that did not undergo a process of processing residual samples; the process includes inverse quantization and inverse transformation; 7. The method according to any one of claims 1 to 6.

8. and concatenating the values ​​of the reconstructed samples with the input information, thereby generating concatenated CM input data; the concatenated CM input data is provided to a CM; The method of claim 1.

9. the ML model comprises a first CM and a second CM; the first CM comprises a first convolutional layer (CL) and a first parametric rectified linear unit (PReLU) coupled to the first CL; the second CM comprises a second CL and a second PReLU coupled to the second CL; the first CM is configured to perform downsampling; The second CM is configured to perform upsampling. The method according to claim 1 or 8.

10. the ML model comprises a first convolutional layer (CL); the first CL is configured to convert the concatenated CM input data into N CM output data; N is the number of kernel filters included in the first CL.

10. The method according to claim 8 or 9.

11. the input information comprises the information regarding the predicted samples, The method comprises: obtaining partition information indicative of how the sample is partitioned; providing the partition information to a CM, thereby generating fourth CM output data; Further comprising: the combined CM output data is generated based on combining at least the first CM output data, the second CM output data, the third CM output data, and the fourth CM output data. The method according to claim 5.

12. the values ​​of the reconstructed samples include values ​​of luma components of the reconstructed samples and values ​​of chroma components of the reconstructed samples; the values ​​of the predicted samples include values ​​of a luma component of the predicted samples and values ​​of a chroma component of the predicted samples; the BBS information indicates strengths of filtering applied to boundaries of luma components of samples and strengths of filtering applied to boundaries of chroma components of samples; The method according to claim 5.

13. The method comprises: Obtaining first partition information indicative of how a luma component of a sample is partitioned; obtaining second partitioning information indicative of how chroma components of the sample are partitioned; providing the first partition information to a CM thereby generating fourth CM output data; providing the second partition information to a CM, thereby generating fifth CM output data; Further comprising: the input information comprises the information regarding the predicted samples, 13. The method of claim 12, wherein the combined CM output data is generated based on combining at least the first CM output data, the second CM output data, the third CM output data, the fourth CM output data, and the fifth CM output data.

14. A method (800) for generating encoded or decoded video, the method comprising: Obtaining machine learning (ML) input data (s802), the ML input data comprising: i) the values ​​of the reconstructed samples; and ii) the predicted sample values; and iii) Block Boundary Strength (BBS) information indicating the strength of the filtering applied to the sample boundaries; and iv) the quantization parameter (QP); obtaining machine learning (ML) input data (s802); providing (s804) the ML input data to an ML model, thereby generating ML output data; generating (s806) the encoded video or the decoded video based at least on the ML output data; and Including, A method (800), wherein the ML input data does not include partitioning information that indicates how luma pictures are partitioned into coding tree units (CTUs) and how luma CTUs are partitioned into coding units (CUs).

15. the ML model comprises a first CM, a second CM, a third CM, and a fourth CM; the first CM comprises a first convolutional layer (CL) and a first parametric rectified linear unit (PReLU) coupled to the first CL; the second CM comprises a second CL and a second PReLU coupled to the second CL; the third CM comprises a third CL and a third PReLU coupled to the third CL; the fourth CM comprises a fourth CL and a fourth PReLU coupled to the fourth CL; the values ​​of the reconstructed samples are provided to the first CM; The predicted sample values ​​are provided to the second CM; The BBS information is provided to the third CM; The QP is provided to the fourth CM. The method of claim 14.

16. the values ​​of the reconstructed samples include values ​​of luma components of the reconstructed samples and values ​​of chroma components of the reconstructed samples; the ML model comprises a first CM, a second CM, a third CM, a fourth CM, and a fifth CM; the first CM comprises a first convolutional layer (CL) and a first parametric rectified linear unit (PReLU) coupled to the first CL; the second CM comprises a second CL and a second PReLU coupled to the second CL; the third CM comprises a third CL and a third PReLU coupled to the third CL; the fourth CM comprises a fourth CL and a fourth PReLU coupled to the fourth CL; the fifth CM comprises a fifth CL and a fifth PReLU coupled to the fifth CL; the values ​​of the luma components of the reconstructed samples are provided to the first CM; the values ​​of the chroma components of the reconstructed samples are provided to the second CM; the value of the predicted sample is provided to the third CM; The BBS information is provided to the fourth CM; The QP is provided to the fifth CM. The method of claim 14.

17. A method (900) for generating encoded or decoded video, the method comprising: Obtaining machine learning (ML) input data (s902), the ML input data comprising: i) the values ​​of the reconstructed samples; and ii) the predicted sample values; and iii) the quantization parameter (QP); Obtaining machine learning (ML) input data (s902), providing the ML input data to an ML model, thereby generating ML output data (s904); generating the encoded video or the decoded video based at least on the ML output data (s906); and Including, A method (900) in which the ML input data does not include block boundary strength (BBS) information that indicates the strength of filtering applied to sample boundaries.

18. the ML model comprises a first CM, a second CM, and a third CM; the first CM comprises a first convolutional layer (CL) and a first parametric rectified linear unit (PReLU) coupled to the first CL; the second CM comprises a second CL and a second PReLU coupled to the second CL; the third CM comprises a third CL and a third PReLU coupled to the third CL; the values ​​of the reconstructed samples are provided to the first CM; The predicted sample values ​​are provided to the second CM; The QP is provided to the third CM.

20. The method of claim 17.

19. the values ​​of the reconstructed samples include values ​​of luma components of the reconstructed samples and values ​​of chroma components of the reconstructed samples; the ML model comprises a first CM, a second CM, a third CM, and a fourth CM; the first CM comprises a first convolutional layer (CL) and a first parametric rectified linear unit (PReLU) coupled to the first CL; the second CM comprises a second CL and a second PReLU coupled to the second CL; the third CM comprises a third CL and a third PReLU coupled to the third CL; the fourth CM comprises a fourth CL and a fourth PReLU coupled to the fourth CL; the values ​​of the luma components of the reconstructed samples are provided to the first CM; the values ​​of the chroma components of the reconstructed samples are provided to the second CM; the value of the predicted sample is provided to the third CM; The QP is provided to the fourth CM.

20. The method of claim 17.

20. the values ​​of the reconstructed samples include values ​​of luma components of the reconstructed samples and values ​​of chroma components of the reconstructed samples; the ML input data further includes partition information indicating how the samples are partitioned; the ML model comprises a first CM, a second CM, a third CM, a fourth CM, and a fifth CM; the first CM comprises a first convolutional layer (CL) and a first parametric rectified linear unit (PReLU) coupled to the first CL; the second CM comprises a second CL and a second PReLU coupled to the second CL; the third CM comprises a third CL and a third PReLU coupled to the third CL; the fourth CM comprises a fourth CL and a fourth PReLU coupled to the fourth CL; the values ​​of the luma components of the reconstructed samples are provided to the first CM; the values ​​of the chroma components of the reconstructed samples are provided to the second CM; the value of the predicted sample is provided to the third CM; the QP is provided to the fourth CM; The segment information is provided to the fifth CM.

20. The method of claim 17.

21. A method (1000) for generating encoded or decoded video, the method comprising: Obtaining (s1002) values ​​of the reconstructed samples; Obtaining a quantization parameter (QP) (s1004); providing the reconstructed sample values ​​and the quantization parameters to a machine learning (ML) model, thereby generating ML output data (s1006); generating first output sample values ​​based at least on the ML output data (s1008); providing the first output sample values ​​to a group of two or more serially connected attention residual blocks (s1010); Including, the group of attention residual blocks includes a first attention residual block disposed at one end of the series of attention residual blocks; A method (1000), wherein the first attention residual block is configured to receive first input data consisting of the first output sample values, and to generate second output sample values ​​based on the first output sample values.

22. the group of attention residual blocks includes a second attention residual block disposed at an opposite end of the series of attention residual blocks; the second attention residual block is configured to receive second input data including the values ​​of the reconstructed samples and / or the QPs; the second attention residual block is configured to generate third output sample values ​​based on the values ​​of the reconstructed samples and / or the QP.

22. The method of claim 21.

23. The method further comprising: Obtaining predicted sample values; Obtaining block boundary strength (BBS) information indicative of the strength of filtering applied to sample boundaries; providing the values ​​of the predicted samples and the BBS information to a CM, thereby generating spatial attention mask data; Further comprising: The method of claim 21 or 22, wherein the third output sample values ​​are generated based on the spatial attention mask data.

24. A method (1100) for generating encoded or decoded video, the method comprising: Obtaining machine learning (ML) input data (s1102), the ML input data comprising: i) the value of the luma component of the reconstructed sample; and ii) the values ​​of the chroma components of the reconstructed samples; and iii) the value of the luma component of the predicted sample; and iv) the values ​​of the chroma components of the predicted samples; and v) first block boundary strength (BBS) information indicating the strength of filtering applied to the boundaries of the luma components of the samples; vi) second BBS information indicating the strength of the filtering applied to the boundaries of the chroma components of the sample; vii) the quantization parameter (QP); Obtaining machine learning (ML) input data (s1102), providing the ML input data to an ML model, thereby generating ML output data (s1104); generating the encoded video or the decoded video based at least on the ML output data (s1106); The method (1100).

25. A method (1300) for generating encoded or decoded video, the method comprising: Obtaining (s1302) values ​​of the reconstructed samples; Obtaining a quantization parameter (QP) (s1304); providing (s1306) the reconstructed sample values ​​and the quantization parameters to a machine learning (ML) model, thereby generating ML output data; generating first output sample values ​​based at least on the ML output data (s1308); providing (s1310) the first output sample values ​​to a group of two or more attention residual blocks connected in series, thereby generating second output sample values; generating the encoded video or the decoded video based on the second output sample values ​​(s1312); and Including, the group of attention residual blocks includes a first attention residual block disposed at one end of the series of attention residual blocks; The method (1300), wherein the first attention residual block is configured to receive input data consisting of the first output sample value and the QP.

26. A computer program (1243) comprising instructions (1244) which, when executed by a processing circuit (1202), cause the processing circuit to perform a method according to any one of claims 1 to 25.

27. 27. A carrier containing the computer program of claim 26, the carrier being one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium.

28. An apparatus (1200) for generating encoded or decoded video, the apparatus comprising: Obtaining (s702) values ​​of the reconstructed samples; Obtaining input information (s704) including any one or combination of: i) information about filtered samples; ii) information about predicted samples; or iii) information about skipped samples; providing (s706) the values ​​of the reconstructed samples and the input information to a machine learning (ML) model, thereby generating at least one ML output; generating (s708) the encoded video or the decoded video based at least on the at least one ML output data; and The apparatus (1200) is configured to perform the above.

29. 29. Apparatus according to claim 28, wherein the apparatus is further configured to perform a method according to any one of claims 2 to 13.

30. An apparatus (1200) for generating encoded or decoded video, the apparatus comprising: Obtaining machine learning (ML) input data (s802), the ML input data comprising: i) the values ​​of the reconstructed samples; and ii) the predicted sample values; and iii) Block Boundary Strength (BBS) information indicating the strength of the filtering applied to the sample boundaries; and iv) the quantization parameter (QP); obtaining machine learning (ML) input data (s802); providing (s804) the ML input data to an ML model, thereby generating ML output data; generating (s806) the encoded video or the decoded video based at least on the ML output data; and It is set to do The apparatus (1200), wherein the ML input data does not include partitioning information indicating how luma pictures are partitioned into coding tree units (CTUs) and how luma CTUs are partitioned into coding units (CUs).

31. 31. The apparatus of claim 30, wherein the apparatus is further configured to perform the method of claim 15 or 16.

32. An apparatus (1200) for generating encoded or decoded video, the apparatus comprising: Obtaining machine learning (ML) input data (s902), the ML input data comprising: i) the values ​​of the reconstructed samples; and ii) the predicted sample values; and iii) the quantization parameter (QP); Obtaining machine learning (ML) input data (s902), providing the ML input data to an ML model, thereby generating ML output data (s904); generating the encoded video or the decoded video based at least on the ML output data (s906); and It is set to do An apparatus (1200), wherein the ML input data does not include block boundary strength (BBS) information that indicates the strength of filtering applied to sample boundaries.

33. 33. Apparatus according to claim 32, wherein the apparatus is further configured to carry out a method according to any one of claims 18 to 20.

34. An apparatus (1200) for generating encoded or decoded video, the apparatus comprising: Obtaining (s1002) values ​​of the reconstructed samples; Obtaining a quantization parameter (QP) (s1004); providing the reconstructed sample values ​​and the quantization parameters to a machine learning (ML) model, thereby generating ML output data (s1006); generating first output sample values ​​based at least on the ML output data (s1008); providing the first output sample values ​​to a group of two or more serially connected attention residual blocks (s1010); It is set to do the group of attention residual blocks includes a first attention residual block disposed at one end of the series of attention residual blocks; An apparatus (1200), wherein the first attention residual block is configured to receive first input data consisting of the first output sample values, and to generate second output sample values ​​based on the first output sample values.

35. 35. Apparatus according to claim 34, wherein the apparatus is further configured to perform the method of claim 22 or 23.

36. An apparatus (1200) for generating encoded or decoded video, the apparatus comprising: Obtaining machine learning (ML) input data (s1102), the ML input data comprising: i) the value of the luma component of the reconstructed sample; and ii) the values ​​of the chroma components of the reconstructed samples; and iii) the value of the luma component of the predicted sample; and iv) the values ​​of the chroma components of the predicted samples; and v) first block boundary strength (BBS) information indicating the strength of filtering applied to the boundaries of the luma components of the samples; vi) second BBS information indicating the strength of the filtering applied to the boundaries of the chroma components of the sample; vii) the quantization parameter (QP); Obtaining machine learning (ML) input data (s1102), providing the ML input data to an ML model, thereby generating ML output data (s1104); generating the encoded video or the decoded video based at least on the ML output data (s1106); The apparatus (1200) is configured to perform the above.

37. An apparatus (1200) for generating encoded or decoded video, the apparatus comprising: Obtaining (s1302) values ​​of the reconstructed samples; Obtaining a quantization parameter (QP) (s1304); providing (s1306) the reconstructed sample values ​​and the quantization parameters to a machine learning (ML) model, thereby generating ML output data; generating first output sample values ​​based at least on the ML output data (s1308); providing (s1310) the first output sample values ​​to a group of two or more attention residual blocks connected in series, thereby generating second output sample values; generating the encoded video or the decoded video based on the second output sample values ​​(s1312); and It is set to do the group of attention residual blocks includes a first attention residual block disposed at one end of the series of attention residual blocks; An apparatus (1200), wherein the first attention residual block is configured to receive input data consisting of the first output sample value and the QP.

38. An apparatus (1200), comprising: A memory (1241); A processing circuit (1202) coupled to the memory; 26. An apparatus (1200) comprising:

Images