Filtering for video encoding and decoding

Abstract

A method for generating encoded video data or decoded video data is provided. The method includes providing input data to a first convolutional layer (CL), thereby generating first convolutional data, generating residual data based on the first convolutional data, and generating encoded video data or decoded video data based on a combination of the input data and the residual data.

Description

Technical Field

[0001] The present disclosure relates to methods and apparatus for performing filtering for video encoding and decoding. Background Art

[0002] Video is the main form of data traffic in today's networks and is expected to increase its share continuously. One way to reduce data traffic from video is compression. In compression, the source video is encoded into a bitstream, which can then be stored and sent to the end user. Using a decoder, the end user can extract the video data and display it on the screen.

[0003] However, since the encoder does not know what kind of device it is sending the encoded bitstream to, the encoder must compress the video into a standardized format. All devices that support the chosen standard can then successfully decode the video. The compression can be lossless, i.e. the decoded video will be identical to the source video provided to the encoder, or it can be lossy, where some degree of content degradation is accepted. Whether the compression is lossless or lossy has a significant impact on the bitrate (i.e. how high the compression rate is), as factors such as noise can make lossless compression very expensive.

[0004] A video sequence contains a sequence of pictures. A commonly used color space in video sequences is YCbCr, where Y is the luminance component and Cb and Cr are the chrominance components. Sometimes, the Cb and Cr components are referred to as U and V. Other color spaces are also used, such as ICtCp (also known as IPT) (where I is the luminance component and Ct and Cp are the chrominance components), constant brightness YCbCr (where Y is the luminance component and Cb and Cr are the chrominance components), RGB (where R, G, and B correspond to the blue component, the green component, and the blue component, respectively), YCoCg (where Y is the luminance component and Co and Cg are the chrominance components), etc.

[0005] The order in which pictures are placed in a video sequence is called the "display order". Each picture is assigned a picture order count (POC) value to indicate its display order. In this disclosure, the terms "image", "picture" or "frame" may be used interchangeably.

[0006] Video compression is used to compress a video sequence into a sequence of coded pictures. In many existing video codecs, the image is divided into blocks of different sizes. A block is a two-dimensional array of samples. Blocks are used as the basis for encoding. The video decoder then decodes the coded picture into a picture containing the sample values.

[0007] Video standards are usually developed by international organizations, as these organizations represent different companies and research institutions with different areas of expertise and interests. The most widely used video compression standard at the moment is H.264 / AVC (Advanced Video Coding), which was jointly developed by ITU-T and ISO. The first version of H.264 / AVC was completed in 2003, and it was updated several times in the following years. The successor to H.264 / AVC is called H.265 / HEVC (High Efficiency Video Coding), which was also developed by ITU-T (International Telecommunication Union-Telecommunication) and the International Organization for Standardization (ISO) and completed in 2013. MPEG and ITU-T created a successor to HEVC within the Joint Video Exploration Team (JVET). The name of this video codec is Versatile Video Coding (VVC), and version 1 of the VVC specification has been published as Rec.ITU-T H.266|ISO / IEC (International Electrotechnical Commission) 23090-3, “Versatile Video Coding”, 2020.

[0008] The VVC video coding standard is a block-based video codec and utilizes both temporal prediction and spatial prediction. Spatial prediction is implemented using intra (I) prediction within the current picture. Temporal prediction is implemented using unidirectional (P) or bidirectional inter (B) prediction at the block level from a previously decoded reference picture. In the encoder, the difference between the original sample data and the predicted sample data (called the residual) is transformed into the frequency domain, quantized, and then entropy encoded before being transmitted together with the necessary prediction parameters (such as prediction mode and motion vector) (which can also be entropy encoded). The decoder performs entropy decoding, inverse quantization, and inverse transformation to obtain the residual, which is then added to the intra or inter prediction to reconstruct the picture.

[0009] The VVC video coding standard uses a block structure called a quadtree plus binary tree plus ternary tree block structure (QTBT+TT), in which each picture is first divided into square blocks called coding tree units (CTUs). All CTUs have the same size, and the division of pictures into CTUs is done without any syntax controlling it.

[0010] Each CTU is further divided into coding units (CUs) that can have a square or rectangular shape. The CTU is first divided by a quadtree structure, and then it can be further divided vertically or horizontally in a binary structure with equal-sized partitions to form coding units (CUs). Therefore, the block can have a square or rectangular shape. The depth of the quadtree and binary tree can be set by the encoder in the bitstream. The ternary tree (TT) part increases the possibility of dividing the CU into three partitions instead of two equal-sized partitions. This increases the possibility of using a block structure that is more suitable for the content structure of the picture, such as roughly following important edges in the picture.

[0011] Intra-coded blocks are I-blocks. Unidirectionally predicted blocks are P-blocks and bidirectionally predicted blocks are B-blocks. For some blocks, the encoder decides that the residual does not need to be encoded, perhaps because the prediction is close enough to the original block. The encoder then signals to the decoder that transform coding for this block should be bypassed (i.e. skipped). Such blocks are called skipped blocks.

[0012] At the 20th JVET meeting, it was decided to set up Exploratory Experiments (EE) on Neural Network-based (NN-based) video coding. The Exploratory Experiments continued at the 21st and 22nd JVET meetings with two EE tests: NN-based filtering and NN-based super-resolution. At the 23rd JVET meeting, it was decided to continue the tests in three categories: enhancement filters, super-resolution methods, and intra-frame prediction. In the category of enhancement filters, two configurations were considered: (i) the proposed filter was used as an in-loop filter; (ii) the proposed filter was used as a post-processing filter.

[0013] In-loop filtering in VVC includes deblocking filtering, sample adaptive offset (SAO) operation, and adaptive loop filter (ALF) operation. The deblocking filter is used to remove block effects by smoothing discontinuities in the horizontal and vertical directions across block boundaries. The deblocking filter uses a block boundary strength (BS) parameter to determine the filtering strength. The BS parameter can have values ​​of 0, 1, and 2, where a larger value indicates stronger filtering. The output of the deblocking filter is further processed by the SAO operation, and the output of the SAO operation is then processed by the ALF operation. The output of the ALF can then be placed in the display picture buffer (DPB), which is used for prediction of subsequently encoded (or decoded) pictures. Since the deblocking filter, SAO filter, and ALF affect the pictures in the DPB used for prediction, they are classified as in-loop filters, also called loop filters. It is possible that the decoder can further filter the image, but it does not send the filtered output to the DPB, but only to the display. Compared with the loop filter, this filter does not affect future predictions and is therefore classified as a post-processing filter, also called a post-filter.

[0014] The contribution described in EE1-1.6 JVET-X0066: Combined Test of EE1-1.2 and EE1-1.4, Y.Li, K.Zhang, L.Zhang, H.Wang, J.Chen, K.Reuze, A.M.Kotra, M.Karczewicz, JVET-X0066, October 2021 and the contribution described in EE1-1.2 JVET-Y0143: Testing of Deep In-Loop Filters with Adaptive Parameter Selection and Residual Scaling, Y.Li, K.Zhang, L.Zhang, H.Wang, K.Reuze, A.M.Kotra, M.Karczewicz, JVET-Y0143, January 2022 are two consecutive contributions describing NN-based loop filtering.

[0015] Both contributions use the same NN model for filtering. NN-based in-loop filters are placed before SAO and ALF, while the deblocking filter is turned off. The purpose of using NN-based filters is to improve the quality of the reconstructed samples. NN models can be nonlinear. While all deblocking filters, SAO and ALF contain nonlinear elements such as conditions and are therefore not strictly linear, all three are based on linear filters. In contrast, a sufficiently large NN model can in principle learn any nonlinear mapping and is therefore able to represent a wider class of functions than deblocking, SAO and ALF.

[0016] In JVET-X0066 and JVET-Y0143, there are four NN models, i.e., four NN-based in-loop filters - one for luma intra samples, one for chroma intra samples, one for luma inter samples, and one for chroma inter samples. The use of NN filtering can be controlled at the block (CTU) level or the picture level. The encoder can determine whether to use NN filtering for each block or each picture.

[0017] This NN-based in-loop filter significantly improves the compression efficiency of the codec, i.e., it significantly reduces the bitrate without reducing the objective quality as measured by the PSNR (peak signal-to-noise ratio) based on the MSE (mean square error). The increase in compression efficiency, or simply "gain", is usually measured as the Bjontegaard-delta rate (BDR) for the anchor point. For example, a BDR of -1% means that the same PSNR can be achieved with 1% fewer bits. As reported in JVET-Y0143, the BDR gain for the luma component (Y) is -9.80% for the random access (RA) configuration and -7.39% for the full intra (AI) configuration. The complexity of the NN model used for compression is usually measured by the number of multiply-accumulate (MAC) operations per pixel. A high gain of the NN model is directly related to a high complexity of the NN model. The luma intra model described in JVET-Y0143 has a complexity of 430kMAC / sample, i.e., 430,000 multiply-accumulate operations per sample. Together with the multiply-accumulate operations required for the chrominance model (110 kMACs), the total complexity becomes 540 kMACs / pixel.There are other measures of complexity, such as the total model size in terms of stored parameters. Summary of the invention

[0018] However, the structure of the NN model described in JVET-Y0143 is not optimal. For example, the high complexity of the NN model may be a significant challenge for practical hardware implementation. Therefore, it is highly desirable to reduce the complexity of the NN model while maintaining or improving the performance of the NN model.

[0019] Therefore, in one aspect of an embodiment of the present disclosure, a method for generating an encoded video or a decoded video is provided. The method includes obtaining a value of a reconstructed sample; obtaining input information, the input information including any one of the following or a combination thereof: i) information about a filtered sample, ii) information about a predicted sample, or iii) information about a skipped sample; providing the value of the reconstructed sample and the input information to a machine learning ML model, thereby generating at least one ML output data; and generating an encoded video or a decoded video based at least on the at least one ML output data.

[0020] In another aspect, a method for generating an encoded video or a decoded video is provided. The method includes obtaining machine learning ML input data, wherein the ML input data includes: i) values ​​of reconstructed samples; ii) values ​​of predicted samples; iii) block boundary strength BBS information indicating the filter strength applied to boundary samples; and iv) quantization parameter QP. The method also includes providing the ML input data to an ML model to generate ML output data; and generating an encoded video or a decoded video based at least on the ML output data. The ML input data does not include partitioning information indicating how to partition a luma picture into coding tree units CTUs and how to partition a luma CTU into coding units CUs.

[0021] In a different aspect, a method for generating an encoded video or a decoded video is provided. The method includes obtaining machine learning ML input data, wherein the ML input data includes: i) values ​​of reconstructed samples; ii) values ​​of predicted samples; iii) quantization parameters QP. The method also includes providing the ML input data to an ML model to generate ML output data; and generating the encoded video or the decoded video based on at least the ML output data. The ML input data does not include block boundary strength BBS information indicating the filter strength applied to the boundary samples.

[0022] In different aspects, a method for generating an encoded video or a decoded video is provided. The method includes obtaining a value of a reconstructed sample; obtaining a quantization parameter QP; providing the value of the reconstructed sample and the quantization parameter to a machine learning ML model to generate ML output data; generating a first output sample value based at least on the ML output data; and providing the first output sample value to a group of two or more attention residual blocks connected in series. The group of attention residual blocks includes a first attention residual block arranged at one end of the series of attention residual blocks, and the first attention residual block is configured to receive first input data consisting of a first output sample value, and based on the first output sample value, generate a second output sample value.

[0023] In different aspects, a method for generating an encoded video or a decoded video is provided. The method includes obtaining machine learning ML input data. The ML input data includes: i) the value of the luminance component of the reconstructed sample; ii) the value of the chrominance component of the reconstructed sample; iii) the value of the luminance component of the predicted sample; iv) the value of the chrominance component of the predicted sample; v) a first block boundary strength BBS information indicating the filter strength applied to the luminance component of the boundary sample; vi) a second BBS information indicating the filter strength applied to the chrominance component of the boundary sample; and iv) a quantization parameter QP. The method also includes providing the ML input data to an ML model to generate ML output data; and generating an encoded video or a decoded video based at least on the ML output data.

[0024] In different aspects, a method for generating an encoded video or a decoded video is provided. The method includes obtaining a value of a reconstructed sample; obtaining a quantization parameter QP; providing the value of the reconstructed sample and the quantization parameter to a machine learning ML model, thereby generating ML output data; generating a first output sample value based at least on the ML output data; providing the first output sample value to a group of two or more attention residual blocks connected in series, thereby generating a second output sample value; and generating an encoded video or a decoded video based on the second output sample value. The group of attention residual blocks includes a first attention residual block arranged at one end of the series of attention residual blocks, and the first attention residual block is configured to receive input data consisting of a first output sample value and QP.

[0025] In a different aspect, a computer program is provided, comprising instructions (1244), which, when executed by a processing circuit, cause the processing circuit to perform the method of any of the above embodiments.

[0026] In different aspects, a device for generating an encoded video or a decoded video is provided. The device is configured to obtain a value of a reconstructed sample; obtain input information, the input information including any one or a combination of the following: i) information about a filtered sample, ii) information about a predicted sample, or iii) information about a skipped sample; provide the value of the reconstructed sample and the input information to a machine learning ML model, thereby generating at least one ML output data; and generate an encoded video or a decoded video based at least on the at least one ML output data.

[0027] In different aspects, a device for generating a coded video or a decoded video is provided. The device is configured to obtain machine learning ML input data, wherein the ML input data includes: i) values ​​of reconstructed samples; ii) values ​​of predicted samples; iii) block boundary strength BBS information indicating the filter strength applied to boundary samples; and iv) quantization parameter QP. The device is configured to provide the ML input data to an ML model to generate ML output data; and to generate a coded video or a decoded video based at least on the ML output data. The ML input data does not include partitioning information indicating how a luma picture is partitioned into coding tree units CTUs and how a luma CTU is partitioned into coding units CUs.

[0028] In a different aspect, a device for generating an encoded video or a decoded video is provided. The device is configured to obtain machine learning ML input data, wherein the ML input data includes: i) values ​​of reconstructed samples; ii) values ​​of predicted samples; iii) quantization parameters QP. The device is configured to provide the ML input data to an ML model to generate ML output data; and generate the encoded video or the decoded video based on at least the ML output data. The ML input data does not include block boundary strength BBS information indicating the filter strength applied to the boundary samples.

[0029] In different aspects, a device for generating an encoded video or a decoded video is provided. The device is configured to obtain a value of a reconstructed sample; obtain a quantization parameter QP; provide the value of the reconstructed sample and the quantization parameter to a machine learning ML model to generate ML output data; generate a first output sample value based at least on the ML output data; and provide the first output sample value to a group of two or more attention residual blocks connected in series. The group of attention residual blocks includes a first attention residual block arranged at one end of the series of attention residual blocks, and the first attention residual block is configured to receive first input data consisting of a first output sample value, and based on the first output sample value, generate a second output sample value.

[0030] In different aspects, a device for generating an encoded video or a decoded video is provided. The device is configured to obtain machine learning ML input data. The ML input data includes: i) the value of the luminance component of the reconstructed sample; ii) the value of the chrominance component of the reconstructed sample; iii) the value of the luminance component of the predicted sample; iv) the value of the chrominance component of the predicted sample; v) the first block boundary strength BBS information indicating the filter strength applied to the luminance component of the boundary sample; vi) the second BBS information indicating the filter strength applied to the chrominance component of the boundary sample; and iv) the quantization parameter QP. The device is configured to provide the ML input data to the ML model, thereby generating ML output data; and based at least on the ML output data, generate an encoded video or a decoded video.

[0031] In different aspects, a device for generating an encoded video or a decoded video is provided. The device is configured to obtain a value of a reconstructed sample; obtain a quantization parameter QP; provide the value of the reconstructed sample and the quantization parameter to a machine learning ML model, thereby generating ML output data; generate a first output sample value based at least on the ML output data; provide the first output sample value to a group of two or more attention residual blocks connected in series, thereby generating a second output sample value; and generate an encoded video or a decoded video based on the second output sample value. The group of attention residual blocks includes a first attention residual block arranged at one end of the series of attention residual blocks, and the first attention residual block is configured to receive input data consisting of a first output sample value and QP.

[0032] In different aspects, a method for generating encoded video data or decoded video data is provided. The method includes obtaining first input information, wherein the first input information includes any one or more of the following parameters: i) the value of the reconstructed sample; ii) the value of the predicted sample; iii) the partition information indicating how to partition the sample; iv) the block boundary strength BS information indicating the filtering strength applied to the sample at the boundary; v) one or more quantization parameters QP; vi) the value of the deblocked sample or vii) the information about the prediction indicating the prediction mode. The method also includes providing the first input information to a first processing model, thereby generating first output data. The method also includes obtaining second input information, wherein the second input information includes any one or more of the following parameters: i) the value of the reconstructed sample; ii) the value of the predicted sample; iii) the partition information; iv) the BS information; v) the one or more QPs; or vi) the value of the deblocked sample. The method also includes providing the second input information to a neural network, thereby generating one or more weight values. The method also includes generating the encoded video data or decoded video data based on the first output data and the one or more weight values.

[0033] In different aspects, a method for generating encoded video data or decoded video data is provided. The method includes providing input data to a first convolutional layer CL, thereby generating first convolutional data; generating residual data based on the first convolutional data; and generating encoded video data or decoded video data based on a combination of the input data and the residual data.

[0034] In different aspects, a computer program is provided. The computer program includes instructions. When a processing circuit executes the instructions, the instructions cause the processing circuit to perform the method of any one of the above embodiments.

[0035] In different aspects, a device for generating encoded video data or decoded video data is provided. The device is configured to obtain first input information, wherein the first input information includes any one or more of the following parameters: i) the value of the reconstructed sample; ii) the value of the predicted sample; iii) the partition information indicating how to partition the sample; iv) the block boundary strength BS information indicating the filter strength applied to the sample at the boundary; v) one or more quantization parameters QP; vi) the value of the deblocked sample or vii) the information about the prediction indicating the prediction mode. The device is also configured to provide the first input information to a first processing model to generate first output data. The device is also configured to obtain second input information, wherein the second input information includes any one or more of the following parameters: i) the value of the reconstructed sample; ii) the value of the predicted sample; iii) the partition information; iv) the BS information; v) the one or more QPs; or vi) the value of the deblocked sample. The device is also configured to provide the second input information to a neural network to generate one or more weight values. The device is also configured to generate encoded video data or decoded video data based on the first output data and the one or more weight values.

[0036] In different aspects, a device for generating encoded video data or decoded video data is provided. The device is configured to provide input data to a first convolutional layer CL, thereby generating first convolutional data; based on the first convolutional data, generating residual data; and based on a combination of the input data and the residual data, generating encoded video data or decoded video data.

[0037] In different aspects, a device is provided. The device includes a processing circuit and a memory storing instructions executable by the processing circuit, so that the device is operable to perform the method of any of the above embodiments.

[0038] Embodiments of the present disclosure provide a method for reducing the complexity of a NN model while substantially maintaining or improving the performance of the NN model. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings, which are incorporated herein and constitute a part of the specification, illustrate various embodiments.

[0040] Figure 1A A system according to some embodiments is shown.

[0041] Figure 1B A system according to some embodiments is shown.

[0042] Figure 1C A system according to some embodiments is shown.

[0043] Figure 2A schematic block diagram of an encoder according to some embodiments is shown.

[0044] Figure 3 A schematic block diagram of a decoder according to some embodiments is shown.

[0045] Figure 4 A schematic block diagram of a portion of a NN filter according to some embodiments is shown.

[0046] Figure 5 A schematic block diagram of a portion of a NN filter according to some embodiments is shown.

[0047] Fig. 6A A schematic block diagram of an attention block according to some embodiments is shown.

[0048] Figure 6B An example of an attention block in accordance with some embodiments is shown.

[0049] Figure 6C A schematic block diagram of a residual block according to some embodiments is shown.

[0050] Fig.6D A schematic block diagram of an attention block according to some embodiments is shown.

[0051] Figure 7 A process according to some embodiments is shown.

[0052] Figure 8 A process according to some embodiments is shown.

[0053] Fig. 9 A process according to some embodiments is shown.

[0054] Fig.10 A process according to some embodiments is shown.

[0055] Fig.11 A process according to some embodiments is shown.

[0056] Fig.12 An apparatus according to some embodiments is shown.

[0057] Fig.13 A process according to some embodiments is shown.

[0058] Fig.14 A schematic block diagram of an attention residual (AR) block according to some embodiments is shown.

[0059] Fig.15A A schematic block diagram of an attention residual (AR) block according to some embodiments is shown.

[0060] Fig. 15BA schematic block diagram of an attention residual (AR) block according to some embodiments is shown.

[0061] Fig.16A A schematic block diagram of an attention residual (AR) block according to some embodiments is shown.

[0062] Fig. 16B A schematic block diagram of an attention residual (AR) block according to some embodiments is shown.

[0063] Fig.17A A schematic block diagram of an attention residual (AR) block according to some embodiments is shown.

[0064] Fig. 17B A schematic block diagram of an attention residual (AR) block according to some embodiments is shown.

[0065] Fig.18 A process according to some embodiments is shown.

[0066] Fig.19 A process according to some embodiments is shown. DETAILED DESCRIPTION

[0067] The following terms are used in the description of the following embodiments:

[0068] Neural Network: A general term for an entity with one or more layers of simple processing units, called neurons or nodes, that have activation functions and interact via weighted connections and biases, which together create tools in the context of nonlinear transformations.

[0069] Neural network architecture, network architecture or simply architecture: The layout of a neural network, describing the placement of nodes and their connections, usually in the form of several interconnected layers, and may also specify the dimensionality of the inputs and outputs and the activation functions used for the nodes.

[0070] Neural network weights, or simply weights: The weight values ​​assigned to the connections between nodes in a neural network.

[0071] Neural network model, or simply model: A transformation in the form of a trained neural network. A neural network model can be specified as a neural network architecture, activation functions, biases, and weights.

[0072] Filter: Transform. A neural network model is one implementation of a filter. The term NN filter can be used as a shorthand for a neural network based filter or a neural network filter.

[0073] Neural network training, or simply training: The process of finding values ​​for the weights and biases of a neural network. Typically, a training dataset is used to train the neural network, with the goal of training being to minimize a defined error. The amount of training data needs to be large enough to avoid overtraining. Training a neural network is typically a time-consuming task and usually involves multiple iterations of the training data, where each iteration is called an epoch.

[0074] Figure 1A A system 100 according to some embodiments is shown. The system 100 comprises a first entity 102, a second entity 104 and a network 110. The first entity 102 is configured to send a video stream (aka "video bitstream", "bitstream", "encoded video") 106 to the second entity 104.

[0075] The first entity 102 may be any computing device (e.g., a network node, such as a server) capable of encoding a video using an encoder 112 and sending the encoded video to the second entity 104 via the network 110. The second entity 104 may be any computing device (e.g., a network node) capable of receiving the encoded video and decoding the encoded video using a decoder 114. Each of the first entity 102 and the second entity 104 may be a single physical entity or a combination of multiple physical entities. The multiple physical entities may be located at the same location or may be distributed in a cloud.

[0076] In some embodiments, Figure 1B As shown in FIG. 1 , the first entity 102 is a video streaming server 132 and the second entity 104 is a user equipment (UE) 134. The UE 134 may be any one of a desktop, a laptop, a tablet, a mobile phone, or any other computing device. The video streaming server 132 is capable of sending a video bitstream 136 (e.g., a YouTube video client) to the video streaming client 134. TM After receiving the video bitstream 136, the UE 134 may decode the received video bitstream 136 to generate and display the video for the video stream.

[0077] In other embodiments, Figure 1C As shown in FIG. 1 , the first entity 102 and the second entity 104 are respectively a first UE 152 and a second UE 154. For example, the first UE 152 may be a provider of a video conference session or a caller of a video chat, and the second UE 154 may be a respondent of the video conference session or a respondent of a video chat. Figure 1C In the illustrated embodiment, the first UE 152 can send a video conference (e.g., Zoom) to the second UE 154. TM , Skype TM , MS TeamsTM etc.) or video chat (e.g., Facetime TM ) video bitstream 156. After receiving the video bitstream 156, the UE 154 can decode the received video bitstream 156 to generate and display a video for a video conferencing session or a video chat.

[0078] Figure 2 A schematic block diagram of an encoder 112 according to some embodiments is shown. The encoder 112 is configured to encode a block of sample values ​​(hereinafter referred to as a "block") in a video frame of a source video 202. In the encoder 112, a current block (e.g., a block contained in a video frame of the source video 202) is predicted by performing motion estimation by a motion estimator 250 based on blocks already provided in the same frame or a previous frame. In the case of inter-frame prediction, the result of the motion estimation is a motion or displacement vector associated with a reference block. The motion compensator 250 uses the motion vector to output an inter-frame prediction of the block.

[0079] The intra-frame predictor 249 calculates the intra-frame prediction of the current block. The output from the motion estimator / compensator 250 and the intra-frame predictor 249 is input to the selector 251, which selects the intra-frame prediction or inter-frame prediction for the current block. The output from the selector 251 is input to the error calculator in the form of an adder 241, which also receives the sample values ​​of the current block. The adder 241 calculates and outputs the residual error as the sample value difference between the block and its prediction. The error is transformed in the transformer 242, such as by a discrete cosine transform, and quantized by the quantizer 243, and then encoded in the encoder 244, such as by an entropy encoder. In inter-frame coding, the estimated motion vector is taken to the encoder 244 for generating an encoded representation of the current block.

[0080] The residual error for the transformation and quantization of the current block is also provided to the inverse quantizer 245 and the inverse transformer 246 to retrieve the original residual error. The error is added by the adder 247 to the block prediction output from the motion compensator 250 or the intra-frame predictor 249 to create a reconstructed sample block 280, which can be used for prediction and encoding of the next block. According to an embodiment, the reconstructed sample block 280 is processed by the NN filter 230 to perform filtering to combat any blocking effects. The output of the NN filter 230, i.e., the output data 290, is then temporarily stored in the frame buffer 248, where it can be used by the intra-frame predictor 249 and the motion estimator / compensator 250.

[0081] In some embodiments, the encoder 112 may include an SAO unit 270 and / or an ALF 272. The SAO unit 270 and the ALF 272 may be configured to receive the output data 290 from the NN filter 230, perform additional filtering on the output data 290, and provide the filtered output data to the buffer 248.

[0082] Even in Figure 2 In the embodiment shown in , the NN filter 230 is arranged between the SAO unit 270 and the adder 247, and in other embodiments, the NN filter 230 may replace the SAO unit 270 and / or the ALF 272. Alternatively, in other embodiments, the NN filter 230 may be arranged between the buffer 248 and the motion compensator 250. In addition, in some embodiments, a deblocking filter (not shown) may be arranged between the NN filter 230 and the adder 247, so that the reconstructed sample block 280 is deblocked and then provided to the NN filter 230.

[0083] Figure 3 36 is a schematic block diagram of a decoder 114 according to some embodiments. The decoder 114 comprises a decoder 361, such as an entropy decoder, for decoding an encoded representation of a block to obtain a set of quantized and transformed residual errors. These residual errors are inversely quantized in an inverse quantizer 362 and inversely transformed by an inverse transformer 363 to obtain a set of residual errors. These residual errors are added to the sample values ​​of a reference block in an adder 364. The reference block is determined by a motion estimator / compensator 367 or an intra predictor 366, depending on whether inter prediction or intra prediction is performed.

[0084] Thus, the selector 368 is interconnected with the adder 364 and the motion estimator / compensator 367 and the intra predictor 366. According to an embodiment, the resulting decoded block 380 output from the adder 364 is input to the NN filter unit 330 to filter any blocking effects. The filtered block 390 is the output from the NN filter 330 and is further preferably temporarily provided to the frame buffer 365 and can be used as a reference block for subsequent blocks to be decoded.

[0085] Thus, a frame buffer (e.g., decoded picture buffer (DPB)) 365 is connected to a motion estimator / compensator 367 to make the stored sample blocks available to the motion estimator / compensator 367. The output from the adder 364 is preferably also input to an intra predictor 366 for use as an unfiltered reference block.

[0086] In some embodiments, the decoder 114 may include an SAO unit 380 and / or an ALF 372. The SAO unit 380 and the ALF 382 may be configured to receive the output data 390 from the NN filter 330, perform additional filtering on the output data 390, and provide the filtered output data to the buffer 365.

[0087] Even in Figure 3 In the embodiment shown in , the NN filter 330 is arranged between the SAO unit 380 and the adder 364, and in other embodiments, the NN filter 330 may replace the SAO unit 380 and / or the ALF 382. Alternatively, in other embodiments, the NN filter 330 may be arranged between the buffer 365 and the motion compensator 367. In addition, in some embodiments, a deblocking filter (not shown) may be arranged between the NN filter 330 and the adder 364, so that the reconstructed sample block 380 is deblocked and then provided to the NN filter 330.

[0088] Figure 4 is a schematic block diagram of a portion of a NN filter 230 / 330 for filtering intra-frame luma samples according to some embodiments. In the present disclosure, a luma (or chroma) intra-frame sample is a luma (or chroma) component of an intra-frame predicted sample. Similarly, a luma (or chroma) inter-frame sample is a luma (or chroma) component of an inter-frame predicted sample.

[0089] like Figure 4 As shown in , the NN filter 230 / 330 may have six inputs: (1) the value of the luma component of the reconstructed sample ("rec") 280 / 380; (2) the value of the luma component of the predicted sample ("pred") 295 / 395; (3) partition information ("part") indicating how the luma component of the sample is partitioned (more specifically, indicating how the luma picture is partitioned into coding tree units CTUs, and how the luma CTUs are partitioned into coding units); (4) block boundary strength BBS information ("bs") indicating the filtering strength applied to the luma component of the boundary sample; (5) a quantization parameter ("qp"); and (6) additional input information. In some embodiments, the additional input information includes the value of the luma component of the deblocked sample.

[0090] Each of the six inputs can be passed through a convolutional layer (in Figure 4 Then, the six outputs from the six PReLU layers can be passed through the concatenation unit (in Figure 4The convolutional layers "conv3x3" are convolutional layers with a 3x3 kernel size, while the convolutional layers "conv1x1" are convolutional layers with a 1x1 kernel size. PReLU can be used to form activation layers.

[0091] In some embodiments, qp may be a scalar value. In such embodiments, the NN filter 230 / 330 may also include a dimensionality manipulation unit (in Figure 4 In the example of FIG. 10 , the dimensionality manipulation unit may be configured to expand the qp so that the expanded qp has the same size as the other inputs (i.e., rec, pred, part, bs, and dblk). However, in other embodiments, the qp may be a matrix, which may have the same size as the other inputs (e.g., rec, pred, part, and / or bs). For example, different samples within a CTU may be associated with different qp values. In such embodiments, a dimension manipulation unit is not required.

[0092] In some embodiments, the NN filter 230 / 330 may also include a downsampler (in Figure 4 , which is marked as “2↓” in the figure, and is configured to perform downsampling by a factor of 2.

[0093] like Figure 4 As shown, data "y" may be provided to a set of N consecutive attention residual (hereinafter referred to as "AR") blocks 402. In some embodiments, the N consecutive AR blocks 402 may have the same structure, while in other embodiments, they may have different structures. N may be any integer greater than or equal to 2. For example, N may be equal to 8.

[0094] like Figure 4 As shown, the first AR block 402 included in the group may be configured to receive data "y" and generate first output data "z0". The second AR block 402 arranged next to the first AR block 402 may be configured to receive the first output data "z0" and generate second output data "z1".

[0095] In the case where the group includes only two AR blocks 402 (ie, the aforementioned first AR block and the second AR block), the second output data “z1” may be provided to the final processing unit 550 (eg, Figure 5 shown).

[0096] On the other hand, in the case where the group includes more than two AR blocks, each AR block 402 included in the group, except for the first AR block and the last AR block, may be configured to receive output data from the previous AR block 402 and provide the output data thereof to the next AR block. The last AR block 402 may be configured to receive output data from the previous AR block and provide the output data thereof to the final processing unit 550 of the NN filter 230 / 330.

[0097] Reference back Figure 4 , some or all of the AR blocks 402 may include a spatial attention block 412, which is configured to generate an attention mask f. The attention mask f may have one channel and its size may be the same as the data "y". Taking the first AR block 402 as an example, the spatial attention block 412 included in the first AR block 402 may be configured to multiply the attention mask f with the residual data "r" to obtain the data "rf". The data "rf" may be combined with the residual data "r" and then combined with the data "y" to generate the first output data "z0". In the present disclosure, "residual data" may mean any data generated by processing input data (e.g., y or z) and to be combined with the input data.

[0098] like Figure 5 As shown in FIG. , the output Z of the group of AR blocks 402 N It may be processed by a convolutional layer 502, a PReLU 504, another convolutional layer 506, a pixel shuffling (or really a sample shuffling) 508 and a final scaling 510 to generate the filtered output data 290 / 390.

[0099] Compared with the luma intra network architecture used in JVET-X0066, Figure 4 and Figure 5 The NN filter 230 / 330 shown improves the gain to -7 / 63% while keeping the complexity of the NN filter at 430kMAC / sample (e.g., by removing "part" from the input while adding "dblk" to the input).

[0100] In some embodiments, according to some embodiments, Figure 4 and Figure 5 The illustrated NN filter 230 / 330 may be used to filter inter-frame luma samples, intra-frame chroma samples, and / or inter-frame chroma samples.

[0101] exist Figure 4 and 5In the case where the NN filter 230 / 330 shown in is used to filter inter-frame luminance samples, the partition information ("part") can be excluded from the input of the NN filter 230 / 330 and from the input of the spatial attention block 412.

[0102] exist Figure 4 and 5 In the case where the NN filter 230 / 330 shown in FIG. 1 is used to filter intra-frame chroma samples, the NN filter 230 / 330 may have the following seven inputs (instead of Figure 4 ) as shown in the six inputs: (1) the value of the luma component of the reconstructed sample ("rec") 280 / 380; (2) the value of the chroma component of the reconstructed sample ("recUV") 280 / 380; (3) the value of the chroma component (e.g., Cb and Cr) of the predicted sample ("predUV") 295 / 395; (4) partition information ("partUV") indicating how the chroma component of the sample is partitioned (more specifically, indicating how the luma picture is partitioned into coding tree units CTUs, and how the luma CTU is partitioned into coding units); (5) block boundary strength BBS information ("bsUV") indicating the filter strength applied to the chroma component of the boundary sample; (6) a quantization parameter ("qp"); and (7) additional input information. In some embodiments, the additional input information includes the value of the chroma component of the deblocked sample. Similarly, the above seven inputs can be used as inputs to the spatial attention block 412.

[0103] exist Figure 4 and 5 In the case where the NN filter 230 / 330 shown in is used to filter inter-frame chroma samples, the partitioning information ("partUV") can be excluded from the above-mentioned seven inputs of the NN filter 230 / 330 and from the seven inputs of the spatial attention block 412.

[0104] As described above, in some embodiments, the additional input information includes the value of the luminance or chrominance component of the deblocked sample. However, in other embodiments, the additional input information may include information about the prediction sample (also known as "prediction mode information" or "I / P / B prediction mode information").

[0105] For example, the prediction mode information may indicate whether the sample block subject to filtering is an intra-frame prediction block, a unidirectionally predicted inter-frame prediction block, or a bidirectionally predicted inter-frame prediction block. More specifically, if the sample belongs to an intra-frame prediction block, the prediction mode information may be set to have a value of 0; if the sample belongs to a unidirectionally predicted inter-frame prediction block, the prediction mode information may be set to have a value of 0.5; if the sample belongs to a bidirectionally predicted inter-frame prediction block, the prediction mode information may be set to have a value of 1.

[0106] Since I-frames only contain I-blocks, if this architecture is used for luma intra networks, the prediction mode information can be a constant (e.g., 0). On the other hand, if this architecture is used for luma inter networks, the prediction mode information can be set to different values ​​for different samples and can provide a higher Bjontegaard-delta rate (BDR) gain than an architecture that does not use this prediction mode information.

[0107] In some embodiments, motion vector (MV) information may be used as additional input information instead of using the value of the luma or chroma component of the deblocked sample or prediction mode information as additional input information. The MV information may indicate the number of MVs used in the prediction (e.g., 0, 1, or 2). For example, 0 MV may mean that the current block is an I block, 1 MV may mean a P block, and 2 MV may mean a B block.

[0108] In some embodiments, prediction direction information indicating a prediction direction for samples subject to filtering may be included in the additional input information in addition to the prediction mode information or the MV information.

[0109] In some embodiments, coefficient information may be used as additional input information instead of using i) values ​​of luma or chroma components of deblocked samples, ii) prediction mode information or iii) MV information as additional input information.

[0110] One example of the coefficient information is skipped block information, which indicates whether the sample block subject to the NN filtering is a skipped block (i.e., a block that has not passed through the processes performed by the transform unit 242, the quantization unit 243, the inverse quantization unit 245, and the inverse transform unit 246, or a block that has not passed through the processes performed by the entropy decoder 361, the inverse quantization unit 362, and the inverse transform unit 363). In one example, if the sample block subject to the NN filtering is a non-skipped block, the skipped block information may be set to have a value of 0, and if the block is a skipped block, the skipped block information may be set to have a value of 1.

[0111] for Figure 2 , the skipped blocks may correspond to reconstructed samples 280 obtained based only on the prediction samples 295 (rather than the sum of the prediction samples 295 and the output of the inverse transform unit 246). Figure 3 In the decoder 114 shown in FIG. 1 , the skipped blocks may correspond to reconstructed samples 380 obtained based only on the prediction samples 395 (rather than the sum of the prediction samples 395 and the output of the inverse transform unit 363 ).

[0112] Since I-frames contain only I-blocks and these blocks cannot be skipped, if this architecture is used for a luma intra network, the skipped block information will be a constant (e.g., 0). On the other hand, if this architecture is used for a luma inter network, the skipped block information can have different values ​​for different samples and can provide BDR gains over other alternative architectures that do not utilize skipped block information.

[0113] Reference back Figure 4 ,like Figure 4 As shown in , in some embodiments, the NN filter 230 / 330 for filtering intra-frame luma samples has six inputs. However, in other embodiments, the partitioning information can be removed from the input so that the total number of inputs to the NN filter 230 / 330 is five: namely, (1) the value of the luma component of the reconstructed sample ("rec") 280 / 380; (2) the value of the luma component of the predicted sample ("pred") 295 / 395; (3) block boundary strength BBS information ("bs") indicating the filter strength applied to the luma component of the boundary sample; (4) a quantization parameter ("qp"); and (5) additional input information. In some embodiments, the additional input information includes the value of the luma component of the deblocked sample. As described above, in Figure 4 In the case where the NN filter 230 / 330 shown in is used to filter inter-frame luma samples, the input of the NN filter 230 / 330 has five inputs (excluding the partitioning information) instead of six inputs. Similarly, the input of the spatial attention block 412 will be five inputs instead of six inputs. Therefore, in these embodiments, the input of the NN filter 230 / 330 for filtering intra-frame luma samples and the input of the NN filter 230 / 330 for filtering inter-frame luma samples will be the same.

[0114] In some embodiments, BBS information may be removed from the input instead of removing the partition information from the input. Thus, in these embodiments, the inputs to the NN filter 230 / 330 for filtering intra-frame luma samples are: (1) the value of the luma component of the reconstructed sample ("rec") 280 / 380; (2) the value of the luma component of the predicted sample ("pred") 295 / 395; (3) partition information ("part") indicating how the luma component of the sample is partitioned; (4) a quantization parameter ("qp"); and (5) additional input information. In the case where the NN filter 230 / 330 is used to filter inter-frame luma samples, inter-frame chroma samples, or intra-frame chroma samples, the BBS information may be removed from the inputs to the NN filter 230 / 330 and the inputs to the special attention block 412.

[0115] As described above, in some embodiments, different inputs are provided to the NN filter 230 / 330 for its filtering operation. For example, in some embodiments, "rec", "pred", "part", "bs", "qp", and "dblk" are provided as inputs to the NN filter 230 / 330 for the luma component of intra-predicted samples, while "rec", "pred", "bs", "qp", and "dblk" are provided as inputs to the NN filter 230 / 330 for the luma component of inter-predicted samples. Similarly, in some embodiments, "rec", "recUV", "predUV", "partUV", "bsUV", "qp", and "dblk" are provided as inputs to the NN filter 230 / 330 for the chroma component of intra-predicted samples, while "rec", "recUV", "predUV", "bsUV", "qp", and "dblk" are provided as inputs to the NN filter 230 / 330 for the luma component of inter-predicted samples. In other words, four different NN filters 230 / 330 may be used for four different types of samples—inter-frame luma samples, intra-frame luma samples, inter-frame chroma samples, and intra-frame chroma samples.

[0116] However, in other embodiments, the same NN filter 230 / 330 can be used for the luma components of the samples (regardless of whether they are inter-predicted or intra-predicted), and the same NN filter 230 / 330 can be used for the chroma components of the samples (regardless of whether they are inter-predicted or intra-predicted). In such an embodiment, "IPB-info" can be used to distinguish between inter-blocks and intra-blocks instead of using two different filters. Therefore, in one example, "rec", "pred", "part", "bs", "qp" and "IPB-info" are provided as inputs to the NN filter 230 / 330 for the luma components of the samples (regardless of whether they are inter-predicted or intra-predicted), while "rec", "recUV", "predUV", "partUV", "bsUV", "qp" and "IPB-info" are provided as inputs to the NN filter 230 / 330 for the chroma components of the samples (regardless of whether they are inter-predicted or intra-predicted).

[0117] Alternatively, in some embodiments, the same NN filter 230 / 330 can be used for any component of an intra-predicted sample, and the same NN filter 230 / 330 can be used for any component of an inter-predicted sample. In these embodiments, the same input is used for the luma component of the sample and the chroma component of the sample. Thus, in one example, "rec", "pred", "part", "bs", "recUV", "predUV", "partUV", "bsUV", "qp" are provided as inputs to the NN filter 230 / 330 for intra-predicted samples, while "rec", "pred", "bs", "recUV", "predUV", "bsUV", "qp" are provided as inputs to the NN filter 230 / 330 for inter-predicted samples. In these embodiments, the outputs of the NN filter 230 / 330 are NN-filtered luma samples and NN-filtered chroma samples.

[0118] In some embodiments, instead of using two different NN filters 230 / 330, the same NN filter 230 / 330 can be used for four different types of samples - inter-frame luma samples, intra-frame luma samples, inter-frame chroma samples, and intra-frame chroma samples. In these embodiments, the inter-frame information or intra-frame information can be given by "IPB-info", and cross-component benefits can be given by adopting both luma and chroma related inputs. Therefore, in one example, "rec", "pred", "part", "bs", "recUV", "predUV", "partUV", "bsUV", "qp" and "IPB-info" are provided as inputs to the NN filters 230 / 330 for four different types of samples.

[0119] In the embodiments discussed above, the performance and / or efficiency of the NN filter 230 / 330 is improved by adjusting the input provided to the NN filter 230 / 330. However, in some embodiments, the performance and / or efficiency of the NN filter 230 / 330 is improved by changing the structure of the AR block 402.

[0120] More specifically, in some embodiments, Figure 6C As shown (with Figure 4), the spatial attention block 412 may be removed from the first M AR blocks 402. For example, in the case where 8 (or 16) AR blocks 402 are included in the NN filter 230 / 330, the first 7 (or 15) AR blocks 402 may not include the spatial attention block 412, and only the last AR block 402 may include the spatial attention block 412. In another example, none of the AR blocks 402 included in the NN filter 230 / 330 includes the spatial attention block 412.

[0121] In having Figure 6C In the case where the AR block 402 of the structure shown is the first AR block included in the NN filter 230 / 330, the input data y may be provided to the first convolutional layer ( Figure 6C On the other hand, in the case of Figure 6C In the case where the AR block 402 of the structure shown is not the first AR block included in the NN filter 230 / 330, the input data z i-1 can be fed into the first convolutional layer ( Figure 6C The "conv3x3" on the left side of the figure generates the first convolution data. Here z i-1 is the output of the previous AR block.

[0122] like Figure 6C As shown in , the first convolution data can be provided to a parameter rectified linear unit (PReLU) to generate rectified data, which can be provided to a second convolution layer ( Figure 6C Here, the second convolution data is the residual data r. Based on the residual data r and the input data (y or z i-1 ) can generate the output of AR block 402. For example, the output of AR block 402 can be equal to y+r or z i-1 +r.

[0123] In some embodiments, the performance and / or efficiency of the NN filter 230 / 330 may be improved by adjusting the capacity of the spatial attention block 412, instead of or in addition to adjusting the input of the NN filter 230 / 330 and / or removing the spatial attention block 412 from the AR block 402. For example, in some embodiments, the number of layers in the spatial attention block 412 may be increased (relative to the number of layers in JVET-X0066), and the layers may be configured to perform downsampling and upsampling to improve performance in capturing potential correlations. Fig. 6A An example of a spatial attention block 412 according to these embodiments is shown in FIG.

[0124] In some embodiments, instead of or in addition to increasing the capacity of the spatial attention block 412, the output of the spatial attention block 412 can be increased from one channel to multiple channels to provide spatial and channel-by-channel attention. For example, typically, in a CNN layer included in the spatial attention block 412, a single kernel (e.g., having a size of 3x3) is used to perform a convolution operation. However, in these embodiments, multiple kernels (e.g., 96) can be used to perform the convolution operation. Due to the use of multiple kernels (each kernel is associated with a specific channel), multiple channel outputs can be generated.

[0125] In some embodiments, only “qp” can be used in generating channel-by-channel attention. Figure 6B As shown in , a multilayer perceptron (MLP) can be used to generate multiple channel outputs using "QP" as input. MLP is a class of feedforward artificial neural networks (ANN, often simply called neural networks). The term MLP is used ambiguously, sometimes loosely meaning any feedforward ANN, and sometimes strictly referring to a network composed of multilayer perceptrons (with threshold activation). Multilayer perceptrons are sometimes colloquially referred to as "vanilla" neural networks, especially when they have a single hidden layer. MLPs consist of at least three layers of nodes: an input layer, a hidden layer, and an output layer. Fig.6D An exemplary implementation of these embodiments is shown in FIG. Fig.6D As shown in , the spatial attention block 412 may include a PReLU layer, a dense layer, and a SoftPlus layer, and these layers may be used together to generate multiple channel outputs using "QP" as input. In some embodiments, the Softplus layer (also known as the Softplus activation function) may be defined as follows: softplus(x)=log(1+e x ). A dense layer is a layer that is deeply connected to the previous layer, meaning that the neurons of this layer are connected to every neuron in the previous layer. In some embodiments, instead of using the residual data “r” to perform an attention operation (e.g., multiplication) from the output of the MLP, Figure 6C The output data "z i to perform the operation.

[0126] Embodiments of the present disclosure provide at least one of the following advantages.

[0127] By retraining the luma intra model from JVET-Y0143, the model provides a luma gain of -7.57% for the full intra configuration. The difference between the previous gain of 7.39% reported in JVET-X0066 and the gain of 7.57% for the retrained network is due to the different training procedures. For example, the training time for the retrained network can be longer. By removing the partition input "part", the gain is still 7.57% and the complexity is reduced from 430kMAC / pixel to 419kMAC / pixel. By removing the additional bs input "bs", the gain is 7.42% and the complexity is reduced to 408kMAC / pixel.

[0128] By removing the first seven spatial attention masks, the gain is 7.60%, and the complexity is reduced from 430kMAC / pixel to 427kMAC / pixel.

[0129] By using the deblocking information as input and removing the partitioning input, the gain is improved to 7.63%, while the complexity remains the same 430 kMAC / pixel.

[0130] By removing all spatial attention masks, the gain is 7.72% for class D sequences.

[0131] By increasing the capacity of the attention branch, the gain is improved from 7.70% to 7.85% for the D-class sequence. The complexity increases to around 120%.

[0132] By using channel-wise attention with qp as input, a gain of 7.78% is obtained for class D sequences and the complexity is reduced to 428.7 kMAC / pixel.

[0133] like Fig.14 As shown in , in some embodiments, the spatial attention block 412 included in the AR block 402 can be a spatial channel-by-channel attention block (meaning that the block is capable of performing a channel-by-channel attention operation). However, in other embodiments, the spatial attention block 412 can be only a spatial attention block, and a channel-by-channel attention operation can be performed on the output of the spatial attention block 412.

[0134] More specifically, according to some embodiments of the present disclosure, a separate neural network (NN) model (1402, 1502, 1602, or 1702) is provided. The separate NN model is configured to generate one or more weight values ​​(1404, 1504, 1604, or 1704) based on input information, and the input information includes any one or more of the following parameters: i) the value of the reconstructed sample; ii) the value of the predicted sample; iii) the partition information indicating how to partition the sample; iv) the block boundary strength BS information indicating the filter strength applied to the sample at the boundary; v) one or more quantization parameters QP; vi) the value of the deblocked sample or the information about the prediction indicating the prediction mode (e.g., intra-frame prediction, unidirectional inter-frame prediction, or bidirectional inter-frame prediction); vii) skip mode information indicating whether the block is a skip block. In one example, the input information may consist of one or more qp. Each of the above parameters may be a single value or a matrix (i.e., multiple values). In another example, the average value of each matrix may be used instead of the matrix itself. More specifically, in some embodiments, instead of providing a NN model (e.g., including an MLP) with multiple values ​​of a matrix for each channel, a vector of multiple scalar values ​​is provided, each scalar value corresponding to the average of the multiple values ​​of the matrix for each channel.

[0135] The weight values ​​generated by the separate NN model can be used to perform a channel-by-channel attention operation on the output of the spatial attention block 412.

[0136] For example, as mentioned above about Figure 4 , the spatial attention block 412 included in the first AR block 402 may be configured to multiply the attention mask f with the residual data “r” to obtain data “rf”. However, instead of combining the data “rf” with the residual data “r” and the data “y”, as Figure 4 As shown (i.e., in Figure 4 In Fig.14 In the embodiment shown in FIG. 1 , the data “rf” is first multiplied by one or more weight values ​​1404 and then added to the residual data “r” and the data “y” (i.e., in Fig.14 In the equation ( ), z0 = rf × w + r + y).

[0137] Alternatively, in Fig.15A In the embodiment shown in , the data "rf" is first combined with the residual data "r", then multiplied by one or more weight values, and then added to the data "y" (i.e., in Fig.15A , z0=(rf+r)×w+y). Alternatively, Fig. 15B As shown in FIG. 4 , the AR block 402 has Figure 6CIn the case of the structure shown in (i.e., in the case where the AR block 402 does not have the spatial attention block 412), the output of the AR block 402 will be (r×w)+y (or, in the case where the AR block 402 is not the first AR block, it will be z i-1 ).

[0138] Alternatively, in Fig.16A In the illustrated embodiment, the data "rf" is combined with the residual data "r" and the data "y" and then multiplied by one or more weight values ​​(i.e., Fig.16A , z0=(rf+r+y)×w). Alternatively, Fig. 16B As shown in FIG. 4 , the AR block 402 has Figure 6C In the case of the structure shown in (i.e., in the case where the AR block 402 does not have the spatial attention block 412), the output of the AR block 402 will be (r+y)×w or (r+z i-1 )×w (in the case where the AR block 402 is not the first AR block).

[0139] As described above, in some embodiments, the channel-by-channel attention may be performed by applying one or more weight values ​​after generating the data "rf". However, in other embodiments, the channel-by-channel attention may be performed by applying one or more weight values ​​during the process of generating the residual data "r".

[0140] For example, Fig.17A As shown in , during the process of generating residual data "r" using data "y", one or more weight values ​​may be applied. More specifically, Fig.17A In the embodiment shown in , after the data "y" is provided to the CNN 1712 to generate the first convolution data 1722, the first convolution data 1722 can be multiplied by one or more weight values ​​to generate weighted first convolution data 1724. The weighted first convolution data 1724 can be provided to the PReLU 1714 and the CNN 1716 to generate the second convolution data 1726. The second convolution data 1726 can be multiplied by one or more weight values ​​to generate residual data r. In this case, the output of the AR block 402 is rf+r+y, where r corresponds to the residual data r. As shown in FIG. Fig. 17B As shown in FIG. 4 , the AR block 402 has Figure 6C In the case of the structure shown (i.e., in the case where the AR block 402 does not have the spatial attention block 412), the output of the AR block 402 will be r+y or r+z i-1 (In the case where the AR block 402 is not the first AR block).

[0141] In some embodiments, the NN model (1402, 1502, 1602, or 1702) may be a multi-layer perceptron (MLP) NN (similar to Figure 6B The channel-wise attention block shown). Fig.6D An example of an MLP NN is shown. Alternatively, the NN model can have Figure 4 The same structure as shown in the spatial attention block 412. The output can be downsampled and reshaped to provide channel-wise weights.

[0142] As described above, the spatial attention block 412 is configured to perform a spatial attention operation based on input information, wherein the input information includes any one or more of the following parameters: i) the value of the reconstructed sample; ii) the value of the predicted sample; iii) partition information indicating how the sample is partitioned; iv) block boundary strength BS information indicating the filtering strength applied to the sample at the boundary; v) one or more quantization parameters QP; vi) the value of the deblocked sample or information about the prediction indicating a prediction mode (e.g., intra-frame prediction, unidirectional inter-frame prediction, or bidirectional inter-frame prediction).

[0143] Similar to the spatial attention block 412, as described above, the NN model (1402, 1502, 1602 or 1702) is configured to generate one or more weight values ​​(1404, 1504, 1604 or 1704) based on input information, wherein the input information includes any one or more of the following parameters: i) the value of the reconstructed sample; ii) the value of the predicted sample; iii) partition information indicating how the sample is partitioned; iv) block boundary strength (BS) information indicating the filtering strength applied to the sample at the boundary; v) one or more quantization parameters (QP); vi) the value of the deblocked sample or information about the prediction indicating a prediction mode (e.g., intra-frame prediction, unidirectional inter-frame prediction, or bidirectional inter-frame prediction).

[0144] Thus, in some embodiments, the parameters included in the input information for the spatial attention block 412 and the parameters included in the input information for the NN model may at least partially overlap (in one example, they may be the same). However, in other embodiments, the parameters included in the input information for the spatial attention block 412 and the parameters included in the input information for the NN model do not overlap.

[0145] like Figure 3As shown in , in some embodiments, the NN filter 330 may include multiple AR blocks 402. In such an embodiment, the spatial attention block 412 and the NN model (1402, 1502, 1602, or 1702) may be included in each AR block 402 or only in some of the AR blocks 402. For example, the spatial attention block 412 and the NN model may be included in all intermediate AR blocks 402 (i.e., AR blocks that are not arranged at the ends). In another example, the spatial attention block 412 is included in a first group of AR blocks, and the NN model is included in a second group of AR blocks, wherein the first group and the second group do not overlap or partially overlap.

[0146] Figure 7 A process 700 for generating an encoded video or a decoded video is shown. The process 700 may start at step s702. Step s702 includes obtaining values ​​of reconstructed samples. Step s704 includes obtaining input information, the input information including any one or a combination of: i) information about filtered samples, ii) information about predicted samples, or iii) information about skipped samples. Step s706 includes providing the values ​​of the reconstructed samples and the input information to a machine learning (ML) model, thereby generating at least one ML output data. Step s708 includes generating an encoded video or a decoded video based at least on the at least one ML output data.

[0147] In some embodiments, the ML model includes a first pair of models and a second pair of models, the first pair of models includes a first convolutional neural network CNN and a first parameter rectified linear unit PReLU coupled to the first CNN, the second pair of models includes a second CNN and a second PReLU coupled to the second CNN, the value of the reconstructed sample is provided to the first CNN, and the input information is provided to the second CNN.

[0148] In some embodiments, the method further includes: obtaining a value of a prediction sample; obtaining block boundary strength BBS information indicating a filtering strength applied to a boundary sample; obtaining a quantization parameter QP; providing the value of the prediction sample to an ML model, thereby generating at least a first ML output data; providing the BBS information to the ML model, thereby generating at least a second ML output data; providing the QP to the ML model, thereby generating at least a third ML output data; and combining the at least one ML output data, the at least ML output data, the at least second ML output data, and the at least third ML output data, thereby generating a combined ML output data, and the encoded video or the decoded video is generated based at least on the combined ML output data.

[0149] In some embodiments, the information about the filtered samples comprises values ​​of the deblocked samples.

[0150] In some embodiments, the information about prediction indicates a prediction mode, and the prediction mode includes intra prediction, unidirectional inter prediction, and bidirectional inter prediction.

[0151] In some embodiments, the information about the prediction indicates the number of motion vectors used for the prediction.

[0152] In some embodiments, the information about the skipped sample indicates whether the sample belongs to a block that has not been subjected to a process of processing residual samples, and the process includes inverse quantization and inverse transformation.

[0153] In some embodiments, the method further comprises concatenating the value of the reconstructed sample and the input information to generate concatenated ML input data, wherein the concatenated ML input data is provided to the ML model.

[0154] In some embodiments, the ML model includes a first pair of models and a second pair of models, the first pair of models including a first convolutional neural network CNN and a first parameter rectified linear unit PReLU coupled to the first CNN, the second pair of models including a second CNN and a second PReLU coupled to the second CNN, the first CNN is configured to perform downsampling, and the second CNN is configured to perform upsampling.

[0155] In some embodiments, the ML model includes a convolutional neural network (CNN), the CNN is configured to convert the concatenated ML input data into N ML output data, and N is the number of kernel filters included in the CNN.

[0156] In some embodiments, the input information includes information about the prediction sample. The method further includes: obtaining partition information indicating how to partition the sample; and providing the partition information to the ML model to generate fourth ML output data. The combined ML output data is generated based on combining the at least one ML output data, the first ML output data, the second ML output data, the third ML output data, and the fourth ML output data.

[0157] In some embodiments, the value of the reconstructed sample includes the value of the luma component of the reconstructed sample and the value of the chroma component of the reconstructed sample, the value of the predicted sample includes the value of the luma component of the predicted sample and the value of the chroma component of the predicted sample, and the BBS information indicates the filtering strength applied to the luma component of the boundary sample and the filtering strength applied to the chroma component of the boundary sample.

[0158] In some embodiments, the method also includes obtaining first partition information indicating how to partition a luminance component of the sample; obtaining second partition information indicating how to partition a chrominance component of the sample; providing the first partition information to the ML model to generate fourth ML output data; and providing the second partition information to the ML model to generate fifth ML output data, wherein the input information includes information about the predicted sample, and the combined ML output data is generated based on combining the at least one ML output data, the first ML output data, the second ML output data, the third ML output data, the fourth ML output data, and the fifth ML output data.

[0159] Figure 8 A process 800 for generating an encoded video or a decoded video is shown. The process 800 may start at step s802. Step s802 includes obtaining machine learning ML input data, wherein the ML input data includes: i) values ​​of reconstructed samples; ii) values ​​of predicted samples; iii) block boundary strength BBS information indicating the filter strength applied to boundary samples; and iv) a quantization parameter QP. Step s804 includes providing the ML input data to an ML model, thereby generating ML output data. Step s806 includes generating an encoded video or a decoded video based at least on the ML output data. The ML input data does not include partitioning information indicating how a luma image is partitioned into coding tree units CTUs and how a luma CTU is partitioned into coding units CUs.

[0160] In some embodiments, the ML model consists of a first pair of models, a second pair of models, a third pair of models, and a fourth pair of models, the first pair of models including a first convolutional neural network CNN and a first parameter rectified linear unit PReLU coupled to the first CNN, the second pair of models including a second CNN and a second PReLU coupled to the second CNN, the third pair of models including a third CNN and a third PReLU coupled to the third CNN, the fourth pair of models including a fourth CNN and a fourth PReLU coupled to the fourth CNN, the values ​​of the reconstructed samples are provided to the first CNN, the values ​​of the predicted samples are provided to the second CNN, the BBS information is provided to the third CNN, and the QP is provided to the fourth CNN.

[0161] In some embodiments, the value of the reconstructed sample includes the value of the luminance component of the reconstructed sample and the value of the chrominance component of the reconstructed sample, the ML model consists of a first pair of models, a second pair of models, a third pair of models, a fourth pair of models and a fifth pair of models, the first pair of models includes a first convolutional neural network CNN and a first parameter rectified linear unit PReLU coupled to the first CNN, the second pair of models includes a second CNN and a second PReLU coupled to the second CNN, the third pair of models includes a third CNN and a third PReLU coupled to the third CNN, the fourth pair of models includes a fourth CNN and a fourth PReLU coupled to the fourth CNN, the fifth pair of models includes a fifth CNN and a fifth PReLU coupled to the fifth CNN, the value of the luminance component of the reconstructed sample is provided to the first CNN, the value of the chrominance component of the reconstructed sample is provided to the second CNN, the value of the predicted sample is provided to the third CNN, the BBS information is provided to the fourth CNN, and the QP is provided to the fifth CNN.

[0162] Fig. 9 A process 900 for generating an encoded video or a decoded video is shown. The process 900 may start at step s902. Step s902 includes obtaining machine learning (ML) input data, wherein the ML input data includes: i) values ​​of reconstructed samples; ii) values ​​of predicted samples; iii) quantization parameters (QP). Step s904 includes providing the ML input data to an ML model, thereby generating ML output data. Step s906 includes generating an encoded video or a decoded video based at least on the ML output data. The ML input data does not include block boundary strength (BBS) information indicating the strength of filtering applied to boundary samples.

[0163] In some embodiments, the ML model consists of a first pair of models, a second pair of models, and a third pair of models, the first pair of models including a first convolutional neural network CNN and a first parameter rectified linear unit PReLU coupled to the first CNN, the second pair of models including a second CNN and a second PReLU coupled to the second CNN, the third pair of models including a third CNN and a third PReLU coupled to the third CNN, the values ​​of the reconstructed samples are provided to the first CNN, the values ​​of the predicted samples are provided to the second CNN, and the QP is provided to the third CNN.

[0164] In some embodiments, the value of the reconstructed sample includes the value of the luminance component of the reconstructed sample and the value of the chrominance component of the reconstructed sample, the ML model consists of a first pair of models, a second pair of models, a third pair of models and a fourth pair of models, the first pair of models includes a first convolutional neural network CNN and a first parameter rectified linear unit PReLU coupled to the first CNN, the second pair of models includes a second CNN and a second PReLU coupled to the second CNN, the third pair of models includes a third CNN and a third PReLU coupled to the third CNN, the fourth pair of models includes a fourth CNN and a fourth PReLU coupled to the fourth CNN, the value of the luminance component of the reconstructed sample is provided to the first CNN, the value of the chrominance component of the reconstructed sample is provided to the second CNN, the value of the predicted sample is provided to the third CNN, and the QP is provided to the fourth CNN.

[0165] In some embodiments, the value of the reconstructed sample includes the value of the luminance component of the reconstructed sample and the value of the chrominance component of the reconstructed sample, the ML input data also includes division information indicating how to divide the sample, the ML model consists of a first pair of models, a second pair of models, a third pair of models, a fourth pair of models and a fifth pair of models, the first pair of models includes a first convolutional neural network CNN and a first parameter rectified linear unit PReLU coupled to the first CNN, the second pair of models includes a second CNN and a second PReLU coupled to the second CNN, the third pair of models includes a third CNN and a third PReLU coupled to the third CNN, the fourth pair of models includes a fourth CNN and a fourth PReLU coupled to the fourth CNN, the value of the luminance component of the reconstructed sample is provided to the first CNN, the value of the chrominance component of the reconstructed sample is provided to the second CNN, the value of the predicted sample is provided to the third CNN, the QP is provided to the fourth CNN, and the division information is provided to the fifth CNN.

[0166] Fig.10 A process 1000 for generating an encoded video or a decoded video is shown. The process 1000 may start from step s1002. Step s1002 includes obtaining a value of a reconstructed sample. Step s1004 includes obtaining a quantization parameter QP. Step s1006 includes providing the value of the reconstructed sample and the quantization parameter to a machine learning (ML) model, thereby generating ML output data. Step s1008 includes generating (s1008) a first output sample value based on at least the ML output data. Step s1010 includes providing the first output sample value to a group of two or more attention residual blocks connected in series. The group of attention residual blocks includes a first attention residual block arranged at one end of the series of attention residual blocks, the first attention residual block being configured to receive first input data consisting of a first output sample value, and to generate a second output sample value based on the first output sample value.

[0167] In some embodiments, the group of attention residual blocks includes a second attention residual block arranged at the other end of the series of attention residual blocks, the second attention residual block is configured to receive second input data including the value and / or QP of the reconstructed sample, and the second attention residual block is configured to generate a third output sample value based on the value and / or QP of the reconstructed sample.

[0168] In some embodiments, the method also includes obtaining a value of a predicted sample; obtaining block boundary strength (BBS) information indicating the strength of filtering applied to the boundary sample; and providing the value of the predicted sample and the BBS information to an ML model to generate spatial attention mask data, wherein the third output sample value is generated based on the spatial attention mask data.

[0169] Fig.11 A process 1100 for generating an encoded video or a decoded video is shown. The process 1100 may start at step s1102. Step s1102 includes obtaining machine learning (ML) input data, wherein the ML input data includes: i) a value of a luma component of a reconstructed sample; ii) a value of a chroma component of a reconstructed sample; iii) a value of a luma component of a predicted sample; iv) a value of a chroma component of a predicted sample; v) first block boundary strength (BBS) information indicating a filter strength applied to a luma component of a boundary sample; vi) second BBS information indicating a filter strength applied to a chroma component of a boundary sample; and iv) a quantization parameter (QP). Step s1104 includes providing the ML input data to an ML model, thereby generating ML output data. Step s1106 includes generating an encoded video or a decoded video based at least on the ML output data.

[0170] Fig.13 A process 1300 for generating an encoded video or a decoded video is shown. The process 1300 may start from step s1302. Step s1302 includes obtaining a value of a reconstructed sample. Step s1304 includes obtaining a quantization parameter QP. Step s1306 includes providing the value of the reconstructed sample and the quantization parameter to a machine learning ML model, thereby generating ML output data. Step s1308 includes generating (s1308) a first output sample value based on at least the ML output data. Step s1310 includes providing the first output sample value to a group of two or more attention residual blocks connected in series, thereby generating a second output sample value. Step s1312 includes generating an encoded video or a decoded video based on the second output sample value. The group of attention residual blocks includes a first attention residual block arranged at one end of the series of attention residual blocks, the first attention residual block being configured to receive input data consisting of a first output sample value and QP.

[0171] Fig.121 is a block diagram of an apparatus 1200 for implementing an encoder 112, a decoder 114, or a component included in the encoder 112 or the decoder 114 (e.g., the NN filter 280 or 330) according to some embodiments. When the apparatus 1200 implements a decoder, the apparatus 1200 may be referred to as a "decoding apparatus 1200", and when the apparatus 1200 implements an encoder, the apparatus 1200 may be referred to as an "encoding apparatus 1200". Fig.12 As shown in , the device 1200 may include: a processing circuit (PC) 1202, which may include one or more processors (P) 1255 (e.g., a general-purpose microprocessor and / or one or more other processors, such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), etc.), which may be co-located in a single housing or in a single data center, or may be geographically distributed (i.e., the device 1200 may be a distributed computing device); at least one network interface 1248, including a transmitter (Tx) 1245 and a receiver (Rx) 1247, for enabling the device 1200 to send data to and receive data from other nodes connected to a network 110 (e.g., an Internet Protocol (IP) network), to which the network interface 1248 is (directly or indirectly) connected (e.g., the network interface 1248 may be wirelessly connected to the network 110, in which case the network interface 1248 is connected to an antenna device); and a storage unit (also known as a "data storage system") 1208, which may include one or more non-volatile storage devices and / or one or more volatile storage devices. In embodiments where the PC 1202 includes a programmable processor, a computer program product (CPP) 1241 may be provided. The CPP 1241 includes a computer readable medium (CRM) 1242 storing a computer program (CP) 1243 including computer readable instructions (CRI) 1244. The CRM 1242 may be a non-transitory computer readable medium, such as a magnetic medium (e.g., a hard disk), an optical medium, a storage device (e.g., a random access memory, a flash memory), etc. In some embodiments, the CRI 1244 of the computer program 1243 is configured such that when executed by the PC 1202, the CRI causes the apparatus 1200 to perform the steps described herein (e.g., the steps described herein with reference to the flowcharts). In other embodiments, the apparatus 1200 may be configured to perform the steps described herein without code. That is, for example, the PC 1202 may consist solely of one or more ASICs. Thus, the features of the embodiments described herein may be implemented in hardware and / or software.

[0172] Fig.18A process 1800 for generating encoded video data or decoded video data according to some embodiments is shown. The process 1800 may start at step s1802. Step s1802 includes obtaining first input information, wherein the first input information includes any one or more of the following parameters: i) values ​​of reconstructed samples; ii) values ​​of predicted samples; iii) partition information indicating how to partition the samples; iv) block boundary strength BS information indicating the strength of filtering applied to samples at the boundary; v) one or more quantization parameters QP; vi) values ​​of deblocked samples or vii) information about prediction indicating a prediction mode. Step s1804 includes providing the first input information to a first processing model, thereby generating first output data. Step s1806 includes obtaining second input information, wherein the second input information includes any one or more of the following parameters: i) values ​​of reconstructed samples; ii) values ​​of predicted samples; iii) partition information; iv) BS information; v) the one or more QPs; or vi) values ​​of deblocked samples. Step s1808 includes providing the second input information to a neural network, thereby generating one or more weight values. Step s1810 comprises generating encoded video data or decoded video data based on the first output data and the one or more weight values.

[0173] In some embodiments, the one or more weight values ​​are used to perform a channel-by-channel attention operation.

[0174] In some embodiments, the parameters included in the first input information and the parameters included in the second input information do not overlap.

[0175] In some embodiments, the parameters included in the first input information and the parameters included in the second input information at least partially overlap.

[0176] In some embodiments, the second input information consists of the one or more QPs.

[0177] In some embodiments, the method includes generating residual data based on input data; multiplying the residual data with first output data to generate multiplication data; and generating second output data based on the multiplication data and the one or more weight values, wherein the encoded video data or the decoded video data is generated based on the second output data.

[0178] In some embodiments, the second output data is generated based on multiplying the multiplication data with the one or more weight values.

[0179] In some embodiments, z0=(r×f×w)+r+y, wherein z0 is the second output data, f is the first output data, r is the residual data, w is the one or more weight values, and y is the input data.

[0180] In some embodiments, the second output data is generated based on adding the multiplication data to the residual data and multiplying the addition result of the multiplication data and the residual data by the one or more weight values.

[0181] In some embodiments, z0=((r×f)+r)×w+y, wherein z0 is the second output data, f is the first output data, r is the residual data, w is the one or more weight values, and y is the input data.

[0182] In some embodiments, the second output data is generated based on adding the residual data and the input data to the multiplication data.

[0183] In some embodiments, z0=((r×f)+r+y)×w, wherein z0 is the second output data, f is the first output data, r is the residual data, w is the one or more weight values, and y is the input data.

[0184] In some embodiments, the method includes generating residual data based on the input data and the one or more weight values; multiplying the residual data by the first output data to generate multiplication data, wherein the encoded video data or the decoded video data is generated based on the multiplication data.

[0185] In some embodiments, the method includes providing input data to a first convolutional neural network (CNN) to generate first convolution data; and multiplying the first convolution data by the one or more weight values, wherein the residual data is generated based on the multiplication of the first convolution data and the one or more weight values.

[0186] In some embodiments, the method includes providing first convolution data to a first parametric rectified linear unit PReLU and a second CNN coupled to the first PReLU to generate second convolution data; multiplying the second convolution data by the one or more weight values, wherein the residual data is generated based on the multiplication of the second convolution data and the one or more weight values.

[0187] In some embodiments, the first processing model includes: a first concatenation layer for concatenating first input information; a first model CNN coupled to the first concatenation layer; a first model PReLU coupled to the first model CNN; and a second model CNN coupled to the first model PReLU.

[0188] In some embodiments, the NN model includes: a second concatenation layer for concatenating second input information; a third model CNN coupled to the second concatenation layer; a second model PReLU coupled to the second model CNN; and a fourth model CNN coupled to the second model PReLU.

[0189] In some embodiments, the NN model is a multi-layer perceptron NN.

[0190] Fig.19 A process 1900 for generating encoded video data or decoded video data according to some embodiments is shown. The process 1900 may start at step s1902. Step s1902 includes providing input data to a first convolutional layer CL, thereby generating first convolutional data. Step s1904 includes generating residual data based on the first convolutional data. Step s1906 includes generating encoded video data or decoded video data based on a combination of the input data and the residual data.

[0191] In some embodiments, the combination of the input data and the residual data is the sum of the input data and the residual data.

[0192] In some embodiments, the sum of the input data and the residual data comprises r+y, where r is the residual value contained in the residual data and y is the input value contained in the input data.

[0193] In some embodiments, the sum of the input data and the residual data is a weighted sum of the input data and the residual data.

[0194] In some embodiments, process 1900 includes obtaining input information, which includes one or more of the following parameters: i) the value of the reconstructed sample; ii) the value of the predicted sample; iii) partition information indicating how the sample is partitioned; iv) block boundary strength BS information indicating the filtering strength applied to the sample at the boundary; v) one or more quantization parameters QP; vi) the value of the deblocked sample; and providing the input information to a neural network NN model to generate one or more weight values, wherein the encoded video data or the decoded video data is generated based on the one or more weight values.

[0195] In some embodiments, the input information consists of the one or more QPs.

[0196] In some embodiments, the one or more weight values ​​are used to perform a channel-by-channel attention operation.

[0197] In some embodiments, the NN model includes: a concatenation layer for concatenating input information, a CL coupled to the concatenation layer, a parameter rectified linear unit PReLU coupled to the CL coupled to the concatenation layer, and the CL coupled to the PReLU.

[0198] In some embodiments, the NN model is a multi-layer perceptron NN.

[0199] In some embodiments, the encoded video data or the decoded video data is generated based on a weighted sum of input data and residual data, the weighted sum of the input data and the residual data comprising w×r+y, wherein w is one of the one or more weight values, r is a residual value contained in the residual data, and y is an input value contained in the input data.

[0200] In some embodiments, the encoded video data or the decoded video data is generated based on a weighted sum of input data and residual data, the weighted sum of the input data and the residual data comprising w×(r+y), wherein w is one of the one or more weight values, r is the residual value contained in the residual data, and y is the input value contained in the input data.

[0201] In some embodiments, process 1900 includes providing the first convolution data to a parameter rectified linear unit PReLU coupled to a first CL to generate rectified data; providing the rectified data to a second CL to generate second convolution data, wherein the second convolution data is residual data.

[0202] In some embodiments, process 1900 includes multiplying the first convolution data by the one or more weight values ​​to generate weighted first convolution data, wherein the residual data is generated based on the multiplication of the weighted first convolution data.

[0203] In some embodiments, process 1900 includes providing the weighted first convolution data to a PReLU to generate modified data; providing the modified data to a second CL to generate second convolution data; and multiplying the second convolution data by the one or more weight values, wherein the residual data is generated based on multiplying the second convolution data by the one or more weight values.

[0204] Summary of Some Embodiments A1. A method (1800) for generating encoded video data or decoded video data, the method comprising: Obtaining (s1802) first input information (e.g., rec, pred, part, bs, qp, dblk), wherein the first input information includes any one or more of the following parameters: i) a value of a reconstructed sample; ii) a value of a predicted sample; iii) partition information indicating how to partition the sample; iv) block boundary strength BS information indicating a filter strength applied to samples at a boundary; v) one or more quantization parameters QP; vi) a value of a deblocked sample; vii) information on prediction indicating a prediction mode; or viii) skip mode information indicating whether a block is a skip block; Providing (s1804) first input information to a first processing model (e.g., a spatial attention block) to generate first output data (e.g., f); obtaining (s1806) second input information (e.g., rec, pred, part, bs, qp, dblk), wherein the second input information includes any one or more of the following parameters: i) a value of a reconstructed sample; ii) a value of a predicted sample; iii) the partition information; iv) the BS information; v) the one or more QPs; or vi) a value of a deblocked sample; Providing (s1808) the second input information to a neural network (e.g., NN) to generate one or more weight values ​​(e.g., one or more channel-by-channel attention weight values); and Based on the first output data (eg, f) and the one or more weight values ​​(eg, output "w" of the NN), encoded video data or decoded video data is generated (s1810). A1a. The method of embodiment A1, wherein the one or more weight values ​​are used to perform a channel-by-channel attention operation. A2. The method of embodiment A1, wherein the parameters included in the first input information and the parameters included in the second input information do not overlap. A3. The method of embodiment A1, wherein the parameters contained in the first input information and the parameters contained in the second input information at least partially overlap. A4. The method of any one of embodiments A1-A3, wherein the second input information consists of the one or more QPs. A5. The method of any one of embodiments A1-A4, comprising: Based on the input data (e.g. y), generate residual data (e.g. r); multiplying the residual data (eg, r) by the first output data (eg, f) to generate multiplication data (eg, r×f); and generating second output data (e.g., z0) based on the multiplication data (e.g., r×f) and the one or more weight values, in, The encoded video data or the decoded video data is generated based on the second output data. A6. The method of embodiment A5, wherein the second output data (e.g., z0) is generated based on multiplying the multiplication data (e.g., r×f) by the one or more weight values ​​(e.g., w). A7. The method of embodiment A6, wherein z0=(r×f×w)+r+y, wherein z0 is the second output data, f is the first output data, r is the residual data, w is the one or more weight values, and y is the input data. A8. The method of embodiment A5, wherein the second output data (e.g., z0) is generated based on adding the multiplication data (e.g., r×f) to the residual data (e.g., r) and multiplying the addition result of the multiplication data and the residual data by the one or more weight values ​​(e.g., w). A9. The method of embodiment A8, wherein z0=((r×f)+r)×w+y, wherein z0 is the second output data, f is the first output data, r is the residual data, w is the one or more weight values, and y is the input data. A10. The method of embodiment A5, wherein the second output data (e.g., z0) is generated based on adding the residual data (e.g., r) and the input data (e.g., y) to the multiplication data (e.g., r×f). A11. The method of embodiment A10, wherein z0=((r×f)+r+y)×w, wherein z0 is the second output data, f is the first output data, r is the residual data, w is the one or more weight values, and y is the input data. A12. The method of any one of embodiments A1-A4, comprising: generating residual data (e.g., r) based on the input data (e.g., y) and the one or more weight values ​​(e.g., w); The residual data (eg, r) is multiplied by the first output data (eg, f) to generate multiplication data (eg, r×f), where The encoded video data or the decoded video data is generated based on the multiplication data (eg, r×f). A13. The method of embodiment A12, comprising: Providing input data (e.g., y) to a first convolutional neural network CNN (e.g., conv3x3) to generate first convolution data; and The first convolution data is multiplied by the one or more weight values, wherein, The residual data is generated based on the multiplication of the first convolution data and the one or more weight values. A14. The method of embodiment A13, comprising: Providing the first convolution data to a first parametric rectified linear unit (PReLU) and a second CNN coupled to the first PReLU, thereby generating second convolution data; and The second convolution data is multiplied by the one or more weight values, wherein, The residual data is generated based on the multiplication of the second convolution data and the one or more weight values. A15. The method of any one of embodiments A1-A14, wherein the first processing model comprises: A first splicing layer for splicing first input information; A first model CNN coupled to a first concatenation layer; A first model PReLU coupled to the first model CNN; and The second model CNN coupled to the first model PReLU. A16. The method of embodiment A15, wherein the NN model comprises: A second splicing layer for splicing second input information; A third model CNN coupled to the second concatenation layer; A second model PReLU coupled to the second model CNN; and The fourth model CNN coupled to the second model PReLU. A17. The method of any one of embodiments A1-A14, wherein the NN model is a multilayer perceptron NN. B1. A computer program (1243) comprising instructions (1244) which, when executed by a processing circuit (1202), cause the processing circuit to perform the method of any one of embodiments A1-A17. B2. A carrier containing the computer program of embodiment B1, wherein the carrier is one of an electronic signal, an optical signal, a radio signal and a computer-readable storage medium. C1. A device (1200) for generating encoded video data or decoded video data, the device being configured to: Obtaining (s1802) first input information (e.g., rec, pred, part, bs, qp, dblk), wherein the first input information includes any one or more of the following parameters: i) a value of a reconstructed sample; ii) a value of a predicted sample; iii) partition information indicating how to partition the sample; iv) block boundary strength BS information indicating a filter strength applied to samples at a boundary; v) one or more quantization parameters QP; vi) a value of a deblocked sample or vii) information about prediction indicating a prediction mode; Providing (s1804) first input information to a first processing model (e.g., a spatial attention block) to generate first output data (e.g., f); obtaining (s1806) second input information (e.g., rec, pred, part, bs, qp, dblk), wherein the second input information includes any one or more of the following parameters: i) a value of a reconstructed sample; ii) a value of a predicted sample; iii) partition information; iv) BS information; v) the one or more QPs; or vi) a value of a deblocked sample; providing (s1808) the second input information to a neural network (e.g., NN) to generate one or more weight values ​​(e.g., one or more channel-by-channel attention weight values); and Based on the first output data (eg, f) and the one or more weight values ​​(eg, output "w" of the NN), encoded video data or decoded video data is generated (s1810). C2. The apparatus of embodiment C1, wherein the apparatus is configured to perform the method described in at least one of embodiments A2-A17. D1. An apparatus (1200), comprising: processing circuit (1202); and A memory (1241) storing instructions executable by the processing circuit, so that the device is operable to perform the method described in at least one of embodiments A2-A17.

[0205] Although various embodiments are described herein, it should be understood that they are presented only in an exemplary and non-limiting manner. Therefore, the breadth and scope of the present disclosure should not be limited by any of the above exemplary embodiments. In addition, unless otherwise specified herein or the context is clearly contradictory, otherwise, the present disclosure encompasses any combination of the above elements in all possible variations thereof.

[0206] As used herein, "sending" a message "to" or "towards" an intended recipient includes sending the message directly to the intended recipient or indirectly to the intended recipient (i.e., using one or more other nodes to relay the message from a source node to the intended recipient). Similarly, as used herein, "receiving" a message from a sender includes receiving the message directly from the sender or indirectly from the sender (i.e., using one or more nodes to relay the message from the sender to a receiving node). In addition, as used herein, "a" means "at least one" or "one or more".

[0207] In addition, although the processes described above and shown in the drawings are shown as a series of steps, this is for illustrative purposes only. Therefore, it is contemplated that some steps may be added, some steps may be omitted, the order of the steps may be rearranged, and some steps may be performed in parallel.

Claims

1. A method (1900) for generating encoded video data or decoded video data, the method comprising: providing (s1902) input data to a first convolutional layer CL, thereby generating first convolutional data; generating (s1904) residual data based on the first convolution data; as well as Based on a combination of the input data and the residual data, the encoded video data or the decoded video data is generated (s1906).

2. The method according to claim 1, wherein: The combination of the input data and the residual data is the sum of the input data and the residual data.

3. The method according to claim 2, wherein: The sum of the input data and the residual data includes r+y, wherein r is a residual value contained in the residual data and y is an input value contained in the input data.

4. The method according to claim 2, wherein: The sum of the input data and the residual data is a weighted sum of the input data and the residual data.

5. The method according to any one of claims 1 to 4, comprising: Obtaining input information, the input information comprising any one or more of the following parameters: i) a value of a reconstructed sample; ii) predict the value of the sample; iii) division information indicating how to divide the samples; iv) block boundary strength BS information indicating the filtering strength applied to the samples at the boundary; v) one or more quantization parameters QP; vi) the value of the deblocked sample; as well as The input information is provided to a neural network NN model to generate one or more weight values, wherein: The encoded video data or the decoded video data is generated based on the one or more weight values.

6. The method according to claim 5, wherein: The input information consists of the one or more QPs.

7. The method according to any one of claims 5 and 6, wherein: The one or more weight values ​​are used to perform a channel-by-channel attention operation.

8. The method according to any one of claims 5 to 7, wherein: The NN model includes: A splicing layer for splicing the input information, coupled to the stitching layer CL, a parameterized rectified linear unit PReLU coupled to the CL coupled to the concatenated layer, and CL coupled to the PReLU.

9. The method according to any one of claims 5 to 8, wherein: The NN model is a multi-layer perceptron NN.

10. A method according to any one of claims 5 to 9 (when claim 5 is dependent on claims 1 to 2 and 4), wherein: The encoded video data or the decoded video data is generated based on the weighted sum of the input data and the residual data, and The weighted sum of the input data and the residual data includes w×r+y, wherein w is one weight value among the one or more weight values, r is a residual value contained in the residual data, and y is an input value contained in the input data.

11. A method according to any one of claims 5 to 9 (when claim 5 is dependent on claims 1 to 2 and 4), The encoded video data or the decoded video data is generated based on the weighted sum of the input data and the residual data, and The weighted sum of the input data and the residual data comprises w×(r+y), where w is a weight value among the one or more weight values, r is a residual value included in the residual data, and y is an input value included in the input data.

12. The method according to any one of claims 1 to 11, comprising: providing the first convolution data to a parameter rectified linear unit (PReLU) coupled to the first CL, thereby generating rectified data; The modified data is provided to a second CL, thereby generating second convolution data, wherein, The second convolution data is the residual data.

13. A method according to any one of claims 5 to 9 (when claim 5 is dependent on claims 1 to 3), comprising: The first convolution data is multiplied by the one or more weight values ​​to generate weighted first convolution data, wherein: The residual data is generated based on the multiplication of the weighted first convolution data.

14. The method according to claim 13, comprising: Providing the weighted first convolution data to a PReLU to generate modified data; providing the modified data to a second CL, thereby generating second convolution data; as well as The second convolution data is multiplied by the one or more weight values, wherein the residual data is generated based on the second convolution data multiplied by the one or more weight values.

15. A computer program (1200) comprising instructions (1244) which, when executed by a processing circuit (1202), cause the processing circuit to perform the method of any one of claims 1-14.

16. A carrier containing the computer program according to claim 15, wherein: The carrier is one of an electronic signal, an optical signal, a radio signal and a computer-readable storage medium.

17. An apparatus (1200) for generating encoded video data or decoded video data, the apparatus being configured to: providing (s1902) input data to a first convolutional layer CL, thereby generating first convolutional data; generating (s1904) residual data based on the first convolution data; and Based on a combination of the input data and the residual data, the encoded video data or the decoded video data is generated (s1906).

18. The device according to claim 17, wherein: The apparatus is configured to perform the method of any one of claims 2-14.

19. An apparatus (1200), comprising: Processing circuit (1202); and A memory (1241) storing instructions executable by the processing circuit, so that the apparatus is operable to perform the method of any one of claims 1-14.

Images