Neural network (NN)-based loop filter

Abstract

The invention provides a video processing method and system. In some embodiments, a method includes: (i) receiving a video sequence through a neural network (NN)-based loop filter, where the NN-based loop filter includes a residual transformer (NN) (RTNN) filter having a feature extraction module, a backbone module, and a reconstruction module; (ii) extracting, by a feature extraction module, features from input information comprising a quantization parameter (QP) map, a reconstructed image, a predicted image, and a partitioned image; (iii) generating, by a backbone module, a feature map based on the output from the feature extraction module, where the backbone module comprises a transformer block (TB) and a plurality of residual blocks; and (iv) generating, by a reconstruction module, a reduced dimension feature map based on the feature map via a convolution process.

Description

Technical Field

[0001] This disclosure relates to imaging and display technologies. More specifically, this document discloses a video compression scheme including a loop filter based on a neural network (NN). Background Technology

[0002] Existing video compression methods, such as High Efficiency Video Coding (HEVC) and Versatile Video Coding (VVC), perform block processing and quantization during encoding. These processes cause irreversible information loss and various compression artifacts, such as blocking, blurring, and striping artifacts. These problems are particularly pronounced at high compression ratios. Although some methods have attempted to reduce these compression artifacts, these methods are inefficient and require significant computational resources. Therefore, it is beneficial to develop an improved system and method to address these needs. Summary of the Invention

[0003] This disclosure relates to systems and methods for improving video image quality through video compression using neural networks. More specifically, this disclosure provides a neural network (NN)-based loop filter to improve image quality. Although the following systems and methods for video processing have been described, in some embodiments, these systems and methods can be used in other image processing systems and methods. This disclosure also provides a framework that can be trained using deep learning and / or artificial intelligence schemes.

[0004] The NN-based loop filter disclosed herein is based on residual blocks (ResBlocks) and transformer blocks (TBs). The NN-based loop filter in this disclosure can be referred to as an RTNN (Residual Transformer Neural Network) filter. RTNN filters achieve good performance with acceptable computational complexity, thus achieving a good balance / trade-off between complexity and performance.

[0005] This method and system also incorporate auxiliary information (such as partitioning information and quantization parameter (QP) maps) into the attention module of the RTNN filter to achieve effective feature refinement. Furthermore, multi-stage progressive training and iterative training can be used to enhance / maximize the learning ability of the proposed network.

[0006] The RTNN filter includes: (1) a feature extraction part, (2) a backbone part, and (3) a reconstruction part. In some embodiments, one or more of these parts can be designed based on various components (e.g., luminance components and / or chrominance components).

[0007] In some embodiments, for a brightness model, the input information for the feature extraction part may include: (1) reconstructed frames / images, (2) predicted frames / images, (3) partitioned frames / images, and (4) QP graphs.

[0008] In some embodiments, for a chroma model, the input information for the feature extraction part may include: (1) a reconstructed frame / image, (2) a predicted frame / image, (3) a partitioned frame / image, (4) a QP map, and (5) a reconstructed frame / image of a luminance component (e.g., a luminance component corresponding to the chroma component in the processing, e.g., in the same frame / image).

[0009] In some embodiments, the methods discussed herein for an “image” or “frame” can be applied to a portion or region of an “image” or “frame”. For example, the methods disclosed herein can be applied to a sub-image, a region of an image (e.g., a region showing an object of interest), etc.

[0010] In some embodiments, the method may be implemented via a tangible, non-transitory computer-readable medium storing processor instructions that, when executed by one or more processors, cause the processors to perform one or more aspects / features of the method described herein. In other embodiments, the method may be implemented via a system comprising a computer processor and a non-transitory computer-readable storage medium storing instructions that, when executed by the computer processor, cause the computer processor to perform one or more actions of the method described herein. Attached Figure Description

[0011] To more clearly describe the technical solutions in the implementation of this disclosure, the accompanying drawings are briefly described below. The drawings only illustrate some aspects or implementations of this disclosure, and those skilled in the art can still derive other drawings based on these drawings without inventive effort.

[0012] Figure 1 This is a schematic diagram illustrating a system (using a VVC structure) with an RTNN filter (or an NN-based loop filter) according to one or more implementations of this disclosure.

[0013] Figure 2 This is a schematic diagram illustrating an RTNN filter according to one or more implementations of the present disclosure.

[0014] Figure 3 This is a schematic diagram illustrating the processing flow of an RTNN filter according to one or more implementations of this disclosure.

[0015] Figure 4A and Figure 4B This is a schematic diagram illustrating the network architecture of a reconstruction portion according to one or more implementations of this disclosure. Figure 4A The network architecture of the luminance model is shown, while Figure 4B The network architecture of the chromaticity model is shown.

[0016] Figure 5A and Figure 5B This is a schematic diagram illustrating the network architecture of the feature extraction portion according to one or more implementations of this disclosure. Figure 5A The network architecture of the lumen model (I-slice or I-frame; intra-frame coding) is shown, while Figure 5B The network architecture of the chroma model (I-slice or I-frame; intra-frame coding) is shown.

[0017] Figures 6A to 6C This is a schematic diagram illustrating the network architecture for the backbone of the luminance component according to one or more implementations of this disclosure. Figure 6A The residual block (ResBlock) is shown. Figure 6B The residual block group (RBG) is shown, and Figure 6C The backbone of the brightness model is shown.

[0018] Figures 7A to 7C This is a schematic diagram illustrating the network architecture for the backbone of the chroma component according to one or more implementations of this disclosure. Figure 7A The residual block (ResBlock) is shown. Figure 7B The residual attention block (RAB) is shown, and... Figure 7C The backbone of the chromaticity model is shown.

[0019] Figures 8A to 8C This is a schematic diagram illustrating a network architecture for attention blocks targeting chroma components according to one or more implementations of this disclosure. Figure 8A The spatial attention (SA) block is shown. Figure 8B The channel attention (CA) block is shown, and Figure 8C The final attention block in RAB is shown.

[0020] Figure 9A This shows the channel attention block ( Figure 8B A schematic diagram of the intensity channel attention module.

[0021] Figure 9B This shows the channel attention block ( Figure 8B A schematic diagram of the contrast channel attention module.

[0022] Figure 10 This is a schematic diagram illustrating the process of obtaining a dataset.

[0023] Figure 11A and Figure 11B This is a schematic diagram illustrating the training strategy according to this disclosure. Figure 11A A single-stage training strategy for the loop filter is shown, and Figure 11B A multi-stage training strategy for loop filters is shown.

[0024] Figure 12 This is a schematic diagram illustrating an iterative training strategy according to one or more implementations of this disclosure.

[0025] Figure 13 This is a schematic diagram of a wireless communication system according to one or more implementations of this disclosure.

[0026] Figure 14 It is a schematic block diagram of a terminal device according to one or more implementations of this disclosure.

[0027] Figure 15 It is a schematic block diagram of an electronic device according to one or more implementations of this disclosure.

[0028] Figure 16 It is a flowchart of a method according to one or more implementations of this disclosure. Detailed Implementation

[0029] To more clearly describe the technical solutions in the implementation of this disclosure, the accompanying drawings are briefly described below. The drawings only illustrate some aspects or implementations of this disclosure, and those skilled in the art can derive other drawings from these drawings without inventive effort.

[0030] Figure 1 This is a schematic diagram illustrating a system 100 (with a VVC structure) having an RTNN filter 101 (in a loop filter 103) according to one or more implementations of this disclosure. The system 100 is configured according to a VVC structure.

[0031] System 100 includes a video sequence 10, which serves as input to an intra-frame prediction module 11 and / or an inter-frame prediction module 12. The outputs of the intra-frame prediction module 11 and the inter-frame prediction module 12 are directed to a transform module 13. The output of the transform module 13 is then quantized by a quantization module 14. The output of the quantization module 14 is then directed to an inverse quantization module 15 and an inverse transform module 16. Generally, as the quantization parameter (QP) increases, compression artifacts become increasingly significant / severe (i.e., image quality deteriorates).

[0032] like Figure 1 As shown, at adder 17, the outputs of intra-frame prediction module 11 and inter-frame prediction module 12 can be added to the output of inverse transform module 16. The result of the addition can then be fed to loop filter 103. The output of loop filter 103 can then be fed to decoded image buffer 18 for further processing by inter-frame prediction module 12.

[0033] System 100 uses loop filters to suppress compression artifacts and reduce distortion. These loop filters include a deblocking filter (DBF) 105, a sample adaptive offset (SAO) filter 107, and an adaptive loop filter (ALF) 109. Figure 1 As shown, the RTNN filter 101 can be located between the SAO filter 107 and the ALF 109.

[0034] In some embodiments, DBF 105 and SAO filter 107 are two filters designed to reduce artifacts caused by the encoding process. DBF 105 focuses on visual artifacts at block boundaries. SAO filter 107 complementaryly reduces artifacts that may arise from the quantization of intra-block transform coefficients. ALF 109 can enhance the adaptive filtering of the reconstructed signal by reducing the mean square error (MSE) between the original and reconstructed samples using a Wiener-based adaptive filter. As shown, loop filter 103 may also include LMCS (luma mapping with chroma scaling) filter 111. LMCS filter 111 is configured to: (1) map the input luma code values ​​to a new set of code values ​​for use within the encoding loop; and (2) scale the chroma residual values ​​according to the luma code values.

[0035] Figure 2This is a schematic diagram illustrating an RTNN filter 200 according to one or more implementations of the present disclosure. As shown, the RTNN filter 200 includes a feature extraction module 201, a backbone module 203, and a reconstruction module 205. In some embodiments, one or more of these modules can be designed based on various components (e.g., luminance components and / or chrominance components) and image frames (e.g., I-slices, B-slices, etc.). For example, the RTNN filter 200 can be applied to at least four types of models (for different types of input): (1) a luminance model for I-slices, (2) a chrominance model for I-slices, (3) a luminance model for B-slices, and (4) a chrominance model for B-slices.

[0036] Feature extraction module 201 is configured to extract features from the video sequence. More specifically, feature extraction module 201 performs convolution and parametric rectified linear unit (PReLU) processing on each set of input information from the video sequence. Then, feature extraction module 201 concatenates the processed data and performs further convolution and PReLU processing, allowing the data to be further processed by backbone module 203.

[0037] In some embodiments, for a brightness model, the input information of the feature extraction module 201 may include: (1) reconstructed frames / images, (2) predicted frames / images, (3) partitioned frames / images, and (4) quantization parameter (QP) maps. (Refer to...) Figure 5A The relevant embodiments are discussed in detail.

[0038] In some embodiments, for a chroma model, the input information of the feature extraction module 201 may include: (1) a reconstructed frame / image, (2) a predicted frame / image, (3) a partitioned frame / image, (4) a QP map, and (5) a reconstructed frame / image of the luminance component (e.g., the luminance component corresponding to the chroma component in processing, e.g., in the same frame / image). See reference. Figure 5B The relevant embodiments are discussed in detail.

[0039] The backbone module 203 is configured to receive the output of the feature extraction module 201 and feed the feature map to the reconstruction module 205. In the backbone module 203, residual blocks (ResBlocks) and transformer blocks (TBs) are combined to extract and process intermediate features. For example, residual blocks are used to extract shallow features of the input and capture correlations between local features. Transformer blocks can be used to capture long-range correlations between features. (See reference for brightness model) Figures 6A to 6C The relevant implementation examples are discussed in detail, as well as references (for the chromaticity model). Figures 7A to 7C The relevant embodiments are discussed in detail.

[0040] The reconstruction module 205 is configured to take the feature map from the future autonomous module 203 as input and uses a 1×1 convolutional layer to reduce the channel dimension of the feature map to obtain dimensionality-reduced features. For the brightness model, a pixel shuffle (PS) operation can be used to upsample the dimensionality-reduced features to obtain a three-channel residual map. The obtained residual map can be added to the reconstructed frame / image of the input information, thereby improving the image quality of the reconstructed frame / image.

[0041] In some embodiments, for a brightness model, the reconstruction module 205 may be further configured to upsample the dimensionality-reduced features to obtain a three-channel residual map. The obtained residual map can be added to obtain a reconstructed frame / image. (See reference...) Figure 4A (Brightness) and Figure 4B (Chromaticity) The relevant embodiments are discussed in detail.

[0042] Figure 3 This is a schematic diagram illustrating the processing flow 300 of an RTNN filter 200 according to one or more implementations of this disclosure. Figure 3 As shown, the feature extraction module 201 can receive input information 301, which includes prediction frames / images, partitioned frames / images, and QP maps. The feature extraction module 201 can also receive reconstructed frames / images 303 as input.

[0043] In some embodiments, the prediction frame / image may include prediction information for the current frame / image and may be generated from neighboring frames / images. In some implementations, the partitioned frame / image may include block information for the current frame / image. The QP graph may represent the quantization information used for the current frame / image.

[0044] In the main module 203, residual blocks and transformer blocks are used to perform different processing on the luminance and chrominance models. For the luminance model, the residual blocks can be referred to as a residual block group (RBG). Figure 6B and Figure 6C The implementation examples of RBG are discussed in detail.

[0045] For chroma models, residual blocks can have "attention" blocks (e.g., paying special attention to certain features), and are therefore called residual attention blocks (RABs). Figure 7B and Figure 7C The implementation examples of RAB are discussed in detail.

[0046] In some embodiments, the attention block may include a spatial attention (SA) module (e.g., focusing on spatial relationships) and a channel attention (CA) module (e.g., focusing on channels). For example, the input to the spatial attention module may include partitioned frames / images, QP maps, max-pooling maps, and average-pooling maps. In the spatial attention module, multiple inputs may first be combined into a single input through a concatenation operation. Then, some features may be extracted through two convolution operations to obtain the spatial attention map through a sigmoid activation function. Figure 8A The implementation examples of the SA module are discussed in detail.

[0047] The channel attention (CA) block may include an intensity channel attention module and a contrast channel attention module. In some embodiments, the intensity channel attention module may be configured to extract weights for each channel through a global average pooling process, a channel compression process, and a expansion process. The extracted weights are then multiplied by the feature map to obtain the channel attention map. Figure 8B The implementation of the CA module is discussed in detail. The results of the CA module and the SA module can then be combined (see, for example, [link to relevant documentation]). Figure 8C And output it to the reconstruction module 205.

[0048] Then, reconstruction module 205 can combine the output of backbone module 203 with the (input) reconstructed frame / image 303 to generate output 307 for further processing (e.g., output to...). Figure 1 (ALF 109 shown).

[0049] Figure 4A and Figure 4B This is a schematic diagram illustrating the network architecture of a reconstruction portion according to one or more implementations of this disclosure. Figure 4A The network architecture of the luminance model 400A is shown. Figure 4B The network architecture of the chromaticity model 400B is shown.

[0050] exist Figure 4A In the process of reconstructing the brightness model 400A, a convolutional layer 401 (with 64 input channels and 4 output channels) and a pixel rearrangement layer 403 are used to form a reconstructed frame / image 405. Figure 4B In the reconstruction process of the chroma model 400B, a convolutional layer 402 (with 64 input channels and 2 output channels) is used to form a reconstructed frame / image 404.

[0051] Figure 5A and Figure 5B This is a schematic diagram illustrating the network architecture of the feature extraction portion according to one or more implementations of this disclosure. Figure 5A The network architecture of Luminance Model 500A (I-slice or I-frame; intra-frame coding) is shown.

[0052] like Figure 5A As shown, for the luminance model 500A, the input information may include (luminance) reconstruction frame 501, prediction frame 503, partition frame 505, and QP map 507. First, the input information is convolved through convolutional layers with different numbers of input and output channels to extract shallow features. More specifically, convolutional layer 508A is used for reconstruction frame 501 and has 1 input channel and 64 output channels. Similarly, convolutional layer 508B is used for prediction frame 503 and has 1 input channel and 32 output channels. Convolutional layer 508C is used for partition frame 505 and has 1 input channel and 16 output channels. Convolutional layer 508D is used for QP map 507 and also has 1 input channel and 16 output channels. Then, the outputs of convolutional layers 508A to 508D are further processed using PReLU layers 509A to 509D, respectively. Then, the concatenation layer 510 fuses the outputs of PReLU layers 509A to 509D. Further convolutional processing is performed by the convolutional layer 511 (with 64 input / output channels). The PReLU layer 512 further rectifies the data before sending the rectified data to the convolutional layer 513. The convolutional layer 513 downsamples the fused features using convolutions with a stride of 2 (i.e., stride = 2) to conserve computational resources. The result will be used as input to the backbone / module.

[0053] Figure 5B The network architecture of the chroma model 500B (I-slice or I-frame; intra-frame coding) is shown. Figure 5B As shown, in addition to the aforementioned input information (chroma reconstruction frame 515, prediction frame 503, partition frame 505, and QP map 507), the luminance component's luminance reconstruction frame 514 is also used as an additional input. Since the luminance component contains more information, it can be used to provide more accurate structural and textural information for the chroma model, thereby improving the quality of the chroma component.

[0054] Because the input information types are different, a progressive fusion method is adopted to obtain more accurate fusion features. Specifically, firstly, the chroma reconstruction frame 515, the prediction frame 503, and the partition frame 505 are fused to obtain chroma-related fusion features (i.e., through the shown convolutional layers 516A to 516E, PReLU layers 517A to 517E, stitching layer 518, convolutional layer 519, and PReLU layer 520). Then, the luminance reconstruction frame 514, the chroma fusion features, and the QP map 507 are fused through the stitching layer 521, convolutional layer 522, and PReLU layer 523 to obtain the final fusion features.

[0055] Figures 6A to 6C This is a schematic diagram illustrating the network architecture for the backbone of the luminance component according to one or more implementations of this disclosure. Figure 6C The backbone of the luminance model 600 is shown. The luminance model 600 includes three residual block groups (RBGs) 601 and six transformer blocks (TBs) 603. TBs 603 are used to capture long-range correlations between features, thereby helping the network to obtain more effective residual features.

[0056] like Figure 6B As shown, each RBG 601 includes four residual blocks 605 (in Figure 6B (As shown in the diagram, 6051, 6052, 6053, and 6054). Figure 6A As shown, each residual block in residual block 605 includes a “3×3” convolutional layer 607, a PReLU layer 609, and a “3×3” convolutional layer 611.

[0057] The brightness (Y) component can contain rich structural information and is extremely detailed. For neural networks, as depth increases, structural information in the feature maps gradually becomes dominant. Directly using learned deep features makes detail recovery difficult. Since shallow features contain more detailed information, these features are also crucial for the recovery of the brightness (Y) component. Therefore, residual connections 613 ( Figure 6C A feature has been added to the luminance backbone for interaction between shallow and deep features in order to recover higher quality luminance frames.

[0058] Figures 7A to 7C This is a schematic diagram illustrating the network architecture for the backbone of the chroma component according to one or more implementations of this disclosure. Figure 7C The backbone of the chroma model 700 is shown. The luminance model 700 includes three residual attention blocks (RABs) 701 and six transformer blocks (TBs) 703. TBs 703 are used to capture long-range correlations between features, thus helping the network to obtain more efficient residual features.

[0059] like Figure 7B As shown, each RAB in RAB 701 includes four residual blocks 705 (in Figure 7B (Shown as 7051, 7052, 7053, and 7054) and attention block 706. (See also:) Figure 7A As shown, each residual block in residual block 705 includes a “3×3” convolutional layer 707, a PReLU layer 709, and a “3×3” convolutional layer 711.

[0060] Figures 8A to 8C This is a schematic diagram illustrating a network architecture for attention blocks targeting chroma components according to one or more implementations of this disclosure. Attention blocks may include two types: spatial attention (SA) blocks and channel attention (CA) blocks.

[0061] Figure 8C A final attention block 801 is provided, which combines the results of SA block 803 and CA block 805. SA block 803 and CA block 805 are used to extract specific features from the current feature map 807.

[0062] Figure 8A An example of SA block 803 is shown. As shown in the figure, in SA block 803, the input information mainly includes partition frame 8031, QP map 8032, max pooling map 8033, and average pooling map 8034. Partition frame 8031 ​​and QP map 8032 are introduced to better locate regions with block artifacts and distortions in space. Max pooling map 8033 and average pooling map 8034 are used to merge and obtain important spatial features in the current feature map 801.

[0063] The aforementioned inputs (e.g., partitioned frame 8031, QP image 8032, max-pooling image 8033, and average-pooling image 8034) are first combined into a set of inputs through a concatenation operation (through concatenation layer 8035). Then, features are extracted through two convolutional operations (through convolutional layers 8036, 8038, and PReLu layer 8037). A spatial attention map is obtained through a sigmoid activation function 8039. Finally, the attention map is used to emphasize important spatial features through pointwise multiplication 8040.

[0064] Figure 8B An example of CA block 805 is shown. CA block 805 includes two key components: intensity channel attention module 8051 and contrast channel attention module 8052. Details of intensity channel attention module 8051 are shown below. Figure 9A The details of the contrast channel attention module 8052 are shown in [the image]. Figure 9B As shown in the image.

[0065] The intensity channel attention module 8051 uses global average pooling ( Figure 8A The weights for each channel are extracted through a channel compression and expansion process (8034 in the original text). Then, the extracted weights can be multiplied by the input feature map 807 (e.g., via...). Figure 8B The “Mask 1” and “Mask 2” in the image are used to obtain the channel attention map 809.

[0066] The main difference between the contrast channel attention module 8052 and the intensity channel attention module 8051 is that the input is the sum of the mean and variance of the feature map 807, rather than the result of global average pooling. The calculation process of the mean and variance can be shown in the following equation (A).

[0067] Equation (A) In equation (A), "H" represents the height of the block, "W" represents the width of the block, and "F" represents the characteristic function used for calculation.

[0068] In some embodiments, while average pooling can indeed improve the peak signal-to-noise ratio (PSNR) value, it lacks information about structure, texture, and edges that contribute to enhancing image details (as opposed to the structural similarity index (SSIM)). Therefore, the contrast channel attention module 8052 can replace the global average pooling summation of standard deviation and mean (i.e., evaluating the contrast of the feature map) to complement the intensity channel attention module 8052.

[0069] Furthermore, a QP graph 8032 can be introduced to fuse the results of the intensity attention module 8051 and the contrast attention module 8052. This is primarily due to the observation that for larger QP inputs, the network tends to focus on structural features, while for smaller QP inputs, the network tends to focus on texture features. The QP graph 8032 can be processed by two linear layers 8054 and 8055 (i.e., performing a compression and activation process). For example, linear layer 8054 can have 64 input channels and 16 output channels. Linear layer 8055 can have 16 input channels and 64 output channels. Then, the parameters “α” and “β” can be used as weighting parameters for the intensity channel attention module 8051 and the contrast channel attention module 8052.

[0070] In some embodiments, the squeeze and excitation (SE) process can be described as follows.

[0071] compression: First, global average pooling is performed on the input feature map to obtain the compressed features. Each learned filter operates within a local receptive field; therefore, each unit of the transformed output cannot utilize contextual information outside this region. To mitigate this problem, the SE attention mechanism first "compresses" global spatial information into channel descriptors. This is achieved by generating channel-level statistics through global average pooling.

[0072] excitation: The motivation for this step is to better understand the dependencies between each channel. Two conditions need to be met: first, the non-linear relationships between each channel can be learned; second, each channel has an output (e.g., the value cannot be 0). In the illustrated embodiment, the activation function can be "sigmoid" instead of the commonly used ReLU. The activation process is... Two fully connected layers are used to compress and restore the channels. In image processing, 1×1 convolutional layers are used instead of fully connected layers to avoid the conversion between matrices and vectors.

[0073] Figure 9A This shows the intensity channel attention module 8051 ( Figure 8B A schematic diagram of an example is shown. As shown, the intensity channel attention module 8051 includes an average pooling layer 901, a convolutional layer 902 (with 64 input channels and 16 output channels), a ReLU layer 903, a convolutional layer 904 (with 16 input channels and 16 output channels), a ReLU layer 905, a convolutional layer 906 (with 16 input channels and 64 output channels), a sigmoid activation function layer 907, and a multiplication layer 908.

[0074] Figure 9B This shows the contrast channel attention module 8052 ( Figure 8B A schematic diagram of an example is shown. As shown, the contrast channel attention module 8052 includes a contrast layer 909, a convolutional layer 902 (with 64 input channels and 16 output channels), a ReLU layer 903, a convolutional layer 904 (with 16 input channels and 16 output channels), a ReLU layer 905, a convolutional layer 906 (with 16 input channels and 64 output channels), a sigmoid activation function layer 907, and a multiplication layer 908.

[0075] Figure 10 This is a schematic diagram illustrating the process of acquiring a dataset in a loop filter. For example... Figure 10As shown, the current compressed data can be obtained between the LMCS module 111 and the DBF module 105, and the current label set (e.g., to be used as a baseline, reference, ground truth (GT) etc.) can be obtained after the ALF module. With this arrangement, the obtained current compressed data and current label set can be used to train the RTNN filter discussed in this paper.

[0076] Figure 11A and Figure 11B This is a schematic diagram illustrating the training strategy according to this disclosure. Figure 11A A single-stage training strategy for the loop filter is shown, while Figure 11B A multi-stage training strategy for loop filters is illustrated. Figure 11A In this approach, training efficiency is relatively low because training images under different QPs to the same ground truth (GT) requires more time and computational resources. Figure 11B In this study, training images under different QPs to different labels (labeled as "QP-qp_dis") based on the QP "distance" (i.e., the difference between two QP numbers) is more efficient.

[0077] In some embodiments, loss functions can be used to train the RTNN filters discussed herein. For example, L1 loss and L2 loss can be used to train RTNN filters. The loss functions for the luminance model and the chrominance model can be expressed as follows: Equation (B) "Loss" refers to either the "L1 loss" function or the "L2 loss" function. In some embodiments, L1 loss can be used in the early and middle stages of training, while "L2 loss" can be used in the later stages of training.

[0078] The parameter "qp_dis" represents the QP difference between the network input and the label. Since a smaller QP indicates higher quality, the label's QP value is lower than the input's QP value. Initially, the network can be trained with a small "qp_dis" until convergence. Then, "qp_dis" can be gradually increased to train the network. Because the loss function can be a "multi-stage" loss, it can be combined with the training policy.

[0079] In some embodiments, for example, the trainer can set "qp_dis=5" and continuously train the network using L1 and L2 loss functions. Then, "qp_dis" is increased by 10, and the network is trained again using L1 and L2 loss functions. Afterward, "qp_dis" can be increased further, and the training process can be repeated.

[0080] Figure 12This is a schematic diagram illustrating an iterative training strategy according to one or more implementations of this disclosure. The iterative training strategy uses a VTM (VVC test model) as an anchor point to generate training data (step 1201), and then trains a model for processing B slices.

[0081] The proposed four filters (I-luma; I-chroma; B-luma; B-chroma) are initially trained using a multi-stage progressive training strategy (step 1202). Then, the parameters of these filters are fixed and embedded into the VTM (step 1203) to generate new training data (step 1204). Finally, the obtained training data is used to fine-tune B-luma and B-chroma to further improve performance (step 1205). It is important to note that the multi-stage progressive training strategy can still be used during the fine-tuning phase.

[0082] Figure 13 This is a schematic diagram of a wireless communication system 800 according to one or more implementations of this disclosure. The wireless communication system 1300 can implement the framework discussed herein. Figure 13As shown, the wireless communication system 1300 may include a network device (or base station) 1301. Examples of network devices 1301 include a base transceiver station (BTS), a Node B (NB), an evolved Node B (eNB or eNodeB), a next-generation Node B (gNB or gNodeB), a Wi-Fi access point (AP), etc. In some embodiments, network device 1301 may include a relay station, an access point, a vehicle-mounted device, a wearable device, etc. Network equipment 1301 may include wireless connectivity devices for communication networks, such as: Global System for Mobile Communications (GSM) networks, Code Division Multiple Access (CDMA) networks, Wideband CDMA (WCDMA) networks, LTE networks, Cloud Radio Access Networks (CRAN), networks based on Institute of Electrical and Electronics Engineers (IEEE) 802.11 (e.g., Wi-Fi networks), Internet of Things (IoT) networks, Device-to-Device (D2D) networks, next-generation networks (e.g., 5G networks), and future evolution of public land mobile networks (PLMNs). 5G systems or networks may be referred to as new radio (NR) systems or networks.

[0083] exist Figure 13In this context, the wireless communication system 1300 also includes a terminal device 1303. The terminal device 1303 can be a terminal user equipment configured to facilitate wireless communication. The terminal device 1303 can be configured to wirelessly connect to the network device 1301 according to one or more corresponding communication protocols / standards (e.g., via wireless channel 1305). The terminal device 1303 can be mobile or fixed. The terminal device 1303 can be user equipment (UE), access terminal, user unit, user station, mobile station, mobile base station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent, or user device. Examples of terminal devices 1303 include modems, cellular phones, smartphones, cordless phones, session initiation protocol (SIP) phones, wireless local loop (WLL) stations, personal digital assistants (PDAs), handheld devices with wireless communication capabilities, computing devices or other processing devices connected to wireless modems, in-vehicle devices, wearable devices, Internet of Things (IoT) devices, devices used in 5G networks, and devices used in public terrestrial mobile networks. For illustrative purposes, Figure 13 Only one network device 1301 and one terminal device 1303 are shown in the wireless communication system 1300. However, in some cases, the wireless communication system 1300 may include additional network devices 801 and / or terminal devices 1303.

[0084] Figure 14This is a schematic block diagram of a terminal device 1403 (e.g., capable of implementing the methods discussed herein) according to one or more implementations of this disclosure. As shown, the terminal device 1403 includes a processing unit 1410 (e.g., a DSP, CPU, GPU, etc.) and a memory 1420. The processing unit 1410 may be configured to implement instructions corresponding to the methods discussed herein and / or other aspects of the above implementations. It should be understood that the processor 1410 in the implementation of this technology may be an integrated circuit chip and have signal processing capabilities. During implementation, the steps in the foregoing methods may be implemented using integrated logic circuitry in the hardware of the processor 1410 or instructions in software form. The processor 1410 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components. The methods, steps, and logic block diagrams disclosed in the implementations of this technology may be implemented or executed. The general-purpose processor 1410 may be a microprocessor, or alternatively, the processor 1410 may be any conventional processor, etc. The steps in the methods disclosed with reference to the embodiments of this technology may be directly executed or completed by a decoding processor implemented as hardware, or by using a combination of hardware and software modules in the decoding processor. The software modules may be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located at memory 1420, and the processor 1410 reads information from memory 1420 and, in conjunction with its hardware, completes the steps in the aforementioned methods.

[0085] It is understood that the memory 1420 in this implementation can be volatile memory or non-volatile memory, or it can include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random-access memory (RAM) and is used as an external cache. By way of example and not limitation, many forms of RAM can be used, such as static random-access memory (SRAM), dynamic random-access memory (DRAM), synchronous dynamic random-access memory (SDRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), enhanced synchronous dynamic random-access memory (ESDRAM), synchronous link dynamic random-access memory (SLDRAM), and direct Rambus random-access memory (DRRAM). It should be noted that the memory in the systems and methods described herein is intended to include, but is not limited to, these memories and any other suitable types of memory. In some embodiments, the memory may be a non-transitory computer-readable storage medium storing instructions executable by a processor.

[0086] Figure 15This is a schematic block diagram of an electronic device 1500 according to one or more implementations of the present disclosure. The electronic device 1500 may include one or more of the following components: a processing component 1502, a memory 1504, a power supply component 1506, a multimedia component 1508, an audio component 1510, an input / output (I / O) interface 1512, a sensor component 1514, and a communication component 1516.

[0087] Processing component 1502 typically controls the overall operation of an electronic device, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 1502 may include one or more processors 1520 to execute instructions to perform all or part of the steps in the methods described above. Furthermore, processing component 1502 may include one or more modules that facilitate interaction between processing component 1502 and other components. For example, processing component 1502 may include a multimedia module to facilitate interaction between multimedia component 1508 and processing component 1502.

[0088] Memory 1504 is configured to store various types of data to support the operation of the electronic device. Examples of such data include instructions for any application or method running on the electronic device, contact data, phonebook data, messages, images, videos, etc. Memory 1504 can be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, and magnetic or optical disks.

[0089] Power supply component 1506 provides power to various components of an electronic device. Power supply component 1506 may include a power management system, one or more power supplies, and other components associated with the generation, management, and distribution of power for the electronic device.

[0090] Multimedia component 1508 may include a screen that provides an output interface between the electronic device and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen may include a TP, the screen may be implemented as a touchscreen to receive input signals from the user. The TP may include one or more touch sensors to sense touches, swipes, and gestures on the TP. The touch sensors may not only sense the boundaries of touch or swipe actions but also detect the duration and pressure associated with the touch or swipe actions. In some embodiments, multimedia component 1508 may include a front-facing camera and / or a rear-facing camera. When the electronic device is in an operating mode, such as a photo mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each of the front-facing camera and the rear-facing camera may be a fixed optical lens system or have focusing and optical zoom capabilities.

[0091] Audio component 1510 is configured to output and / or input audio signals. For example, audio component 1510 may include a microphone (MIC). When the electronic device is in an operating mode, such as a call mode, recording mode, or voice recognition mode, the MIC is configured to receive external audio signals. The received audio signals may also be stored in memory 1504 or transmitted via communication component 1516. In some embodiments, audio component 1510 may also include a speaker configured to output audio signals.

[0092] I / O interface 1512 provides an interface between processing component 1502 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. Buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0093] Sensor assembly 1514 may include one or more sensors configured to provide state assessments of various aspects of the electronic device. For example, sensor assembly 1514 may detect the on / off state of the electronic device and the relative positioning of components, such as the display and keypad of the electronic device. Sensor assembly 1514 may also detect changes in the position of the electronic device or its components, the presence of contact between a user and the electronic device, the orientation or acceleration / deceleration of the electronic device, and temperature changes of the electronic device. Sensor assembly 1514 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 1014 may also include an optical sensor configured for imaging applications, such as a complementary metal oxide semiconductor (CMOS) or charge coupled device (CCD) image sensor. In some embodiments, sensor assembly 1514 may also include an accelerometer, a gyroscope, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0094] Communication component 1516 is configured to facilitate wired or wireless communication between electronic devices and other devices. The electronic device can access wireless networks based on communication standards, such as Wi-Fi networks, 2nd-generation (2G) or 3G networks, or combinations thereof. In an exemplary embodiment, communication component 1516 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, communication component 1516 may also include a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, BT technology, and other technologies.

[0095] In an exemplary embodiment, the electronic device 1510 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components, and the electronic device 1510 is configured to perform the methods described above.

[0096] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 502 including instructions that can be executed by a processor 502 of an electronic device 1510 to implement the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, optical data storage device, etc.

[0097] Figure 16 This is a flowchart of a method according to one or more implementations of this disclosure. Method 1600 can be implemented by a system or apparatus (e.g., a system or apparatus with an RTNN filter discussed herein). Method 1600 is used to improve image quality. Method 1600 includes: at block 1601, receiving a video sequence through a neural network (NN) based loop filter, wherein the NN-based loop filter includes a residual transform NN (RTNN) filter, the RTNN filter having a feature extraction module, a backbone module, and a reconstruction module.

[0098] At box 1603, method 1600 continues by extracting features from the input information through the feature extraction module, wherein the input information includes a quantization parameter (QP) map and a reconstructed frame.

[0099] At box 1605, method 1600 continues by generating a feature map based on the output from the feature extraction module via the backbone module, wherein the backbone module includes a transformer block (TB) and multiple residual blocks.

[0100] At box 1607, method 1600 continues by generating a dimensionality-reduced feature map based on the feature map via a convolution process through the reconstruction module.

[0101] In some embodiments, the reconstructed frame is a luminance reconstructed frame, and the input information also includes a chrominance reconstructed frame. In some embodiments, the input information also includes partitioned frames and prediction frames.

[0102] In some embodiments, the feature extraction module includes multiple convolutional layers, a concatenation layer, and multiple parametric rectified linear unit (PReLU) layers. In some embodiments, the backbone module includes a transformer block (TB) and multiple residual blocks.

[0103] In some embodiments, method 1600 further includes extracting shallow features from the input information through multiple residual blocks. In some embodiments, method 1600 further includes capturing long-range correlations between the extracted features through transform blocks.

[0104] In some embodiments, the NN-based loop filter further includes a deblocking filter (DBF), a sample adaptive compensation (SAO) filter, and an adaptive loop filter (ALF). The RTNN filter can be located between the SAO filter and the ALF.

[0105] In some examples, the backbone module is used for the brightness model, and wherein the backbone module includes three residual block groups (RBGs) and six transformer blocks (TBs). Each RBG may include four residual blocks. In other embodiments, the backbone module may have different numbers of RBGs, TBs, and residual blocks.

[0106] In some implementations, the backbone module can be used for the chroma model, and the backbone module includes three residual attention blocks (RABs) and six transformer blocks (TBs). Each RAB may include four residual blocks and one attention block. In other embodiments, the backbone module may have different numbers of RABs, TBs, and residual blocks.

[0107] In some embodiments, the attention block includes a special attention block. In some embodiments, method 1600 further includes receiving a max-pooling graph and an average-pooling graph through the special attention block.

[0108] In some embodiments, the attention block includes a channel attention block. The channel attention block may include an intensity channel attention module and a contrast channel attention module. (See reference...) Figure 9A An embodiment of the intensity channel attention module is discussed in detail. (Refer to...) Figure 9B An embodiment of the contrast channel attention module is discussed in detail.

[0109] Additional notes The specific embodiments of the disclosed technology described above are not intended to be exhaustive or to limit the disclosed technology to the precise form disclosed above. While specific examples of the disclosed technology have been described above for illustrative purposes, various equivalent modifications can be made within the scope of the described technology, as will be appreciated by those skilled in the art. For example, although processes or blocks are presented in a given order, alternative implementations may execute routines with multiple steps in a different order or employ systems with multiple blocks, and some processes or blocks may be deleted, moved, added, subdivided, combined, and / or modified to provide alternative implementations or sub-combinations. Each of these processes or blocks can be implemented in various different ways. Furthermore, although processes or blocks are sometimes shown as being executed in series, these processes or blocks may alternatively be executed or implemented in parallel, or may be executed at different times. Moreover, any specific figures indicated herein are merely illustrative; alternative implementations may employ different values ​​or ranges.

[0110] In the detailed description, numerous specific details are set forth to provide a thorough understanding of the technology described herein. In other implementations, the technology described herein may be practiced without these specific details. In other instances, well-known features, such as specific functions or routines, have not been described in detail to avoid unnecessarily obscuring this disclosure. References to “implementation / exemplification,” “an implementation / exemplification,” etc., in this specification mean that a particular feature, structure, material, or characteristic described is included in at least one implementation of the described technology. Therefore, the appearance of these phrases in this specification does not necessarily refer to the same implementation / exemplification. Furthermore, these references are not necessarily mutually exclusive. Moreover, in one or more implementations / exemplifications, a particular feature, structure, material, or characteristic may be combined in any suitable manner. It should be understood that the various implementations shown in the figures are merely illustrative representations and are not necessarily drawn to scale.

[0111] For clarity, this document omits certain details describing the structures or processes described below, which are well-known and commonly associated with communication systems and subsystems, but may unnecessarily obscure some important aspects of the disclosed technology. Furthermore, although the following disclosure sets forth several implementations of different aspects of this disclosure, several other implementations may have configurations different from those described in this section, or have components different from those described in this section. Therefore, the disclosed technology may have other implementations with or without the elements described below.

[0112] Many implementations or aspects of the techniques described herein can take the form of computer-executable instructions or processor-executable instructions, including routines executed by a programmable computer or processor. Those skilled in the art will understand that the described techniques can be practiced on other computer or processor systems besides those shown and described below. The techniques described herein can be implemented in a dedicated computer or data processor specifically programmed, configured, or constructed to execute one or more of the computer-executable instructions described below. Therefore, the terms “computer” and “processor” as commonly used herein refer to any data processor. Information processed by these computers and processors can be presented on any suitable display medium. Instructions for performing computer-executable tasks or processor-executable tasks can be stored in or on any suitable computer-readable medium, including hardware, firmware, or a combination of hardware and firmware. Instructions can be contained in any suitable memory device, including, for example, a flash drive and / or other suitable media.

[0113] The term "and / or" in this specification is used only to describe the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent the following three situations: A exists alone, both A and B exist, and B exists alone.

[0114] These and other modifications can be made to the disclosed technology based on the specific embodiments described above. While the specific embodiments describe certain examples of the disclosed technology and the intended best mode, the disclosed technology can be practiced in many ways, however detailed the description above may appear in the text. The details of the system can vary considerably in its particular implementation, while still being covered by the technology disclosed herein. As stated above, specific terms used in describing certain features or aspects of the disclosed technology should not be construed as implying that such terms are redefined herein to be limited to any particular characteristic, feature, or aspect of the disclosed technology associated with that term. Therefore, the invention is not limited except as defined in the appended claims. Generally, the terms used in the appended claims should not be construed as limiting the disclosed technology to the specific examples disclosed in the specification, unless such terms are expressly defined in the foregoing specific embodiments section.

[0115] Those skilled in the art will recognize that, in conjunction with the examples described in the implementations disclosed herein, the units and algorithm steps can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether the function is performed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0116] While certain aspects of the invention have been presented above in the form of specific claims, the applicant intends to present various aspects of the invention in any number of claims. Therefore, the applicant reserves the right to file additional claims after the filing of this application, in the form of such additional claims in this application or subsequent applications.

Claims

1. A video processing method, comprising: Video sequences are received through a loop filter based on a neural network (NN), wherein the loop filter based on the NN includes a residual transformer NN (RTNN) filter, and the RTNN filter has a feature extraction module, a backbone module and a reconstruction module; Features are extracted from the input information by the feature extraction module, wherein the input information includes a quantization parameter (QP) map and a reconstructed image; The backbone module generates a feature map based on the output from the feature extraction module, wherein the backbone module includes a transformer block (TB) and multiple residual blocks; and The reconstruction module generates a dimensionality-reduced feature map based on the feature map via a convolution process.

2. The method according to claim 1, wherein, The reconstructed image is a luminance reconstructed image, and the input information also includes a chrominance reconstructed image.

3. The method according to claim 1, wherein, The input information also includes partitioned images.

4. The method according to claim 1, wherein, The input information also includes the predicted image.

5. The method according to claim 1, wherein, The feature extraction module includes multiple convolutional layers, one splicing layer, and multiple parametric rectified linear unit (PReLU) layers.

6. The method according to claim 1, wherein, The backbone module includes a converter block (TB) and multiple residual blocks.

7. The method according to claim 6, further comprising: Shallow features are extracted from the input information using the multiple residual blocks.

8. The method according to claim 6, further comprising: The transformer block captures the long-range correlation between the extracted features.

9. The method according to claim 1, wherein, The NN-based loop filter also includes a deblocking filter (DBF), a sample adaptive compensation (SAO) filter, and an adaptive loop filter (ALF).

10. The method according to claim 9, wherein, The RTNN filter is located between the SAO filter and the ALF.

11. The method according to claim 1, wherein, The backbone module is used for the luminance model, and the backbone module includes three residual block groups (RBGs) and six converter blocks (TBs).

12. The method according to claim 11, wherein, Each RBG comprises four residual blocks.

13. The method according to claim 1, wherein, The backbone module is used for the chromaticity model, and the backbone module includes three residual attention blocks (RAB) and six transformer blocks (TB).

14. The method according to claim 13, wherein, Each of the RABs comprises four residual blocks and one attention block.

15. The method according to claim 14, wherein, The attention block includes a dedicated attention block.

16. The method of claim 15, further comprising: The max pooling graph and average pooling graph are received through the dedicated attention block.

17. The method of claim 14, wherein, The attention block includes a channel attention block.

18. The method according to claim 15, wherein, The channel attention block includes an intensity channel attention module and a contrast channel attention module.

19. A video processing apparatus, comprising: A loop filter based on a neural network (NN) includes a feature extraction module, a backbone module, and a reconstruction module; The feature extraction module is configured to extract features from input information, wherein the input information includes a quantization parameter (QP) map, a partitioned image, a predicted image, and a reconstructed image. The feature extraction module includes multiple convolutional layers, concatenation layers, and multiple parametric rectified linear unit (PReLU) layers. The backbone module is configured to generate a feature map based on the output from the feature extraction module. The backbone module includes a transformer block (TB) and multiple residual blocks. The residual blocks are configured to extract shallow features from the input information. The transformer block is configured to capture long-range correlations between the extracted features. The reconstruction module is configured to generate a dimensionality-reduced feature map based on the feature map via a convolution process.

20. A video processing system, comprising: processor; as well as The memory is configured to store instructions, which, when executed by the processor, are used to: Video sequences are received through a loop filter based on a neural network (NN), wherein the loop filter based on the NN includes a residual transformer NN (RTNN) filter, and the RTNN filter has a feature extraction module, a backbone module and a reconstruction module; The feature extraction module extracts features from the input information, wherein the input information includes a quantization parameter (QP) map, a reconstructed image, a predicted image, and a partitioned image; The backbone module generates a feature map based on the output from the feature extraction module, wherein the backbone module includes a transformer block (TB) and multiple residual blocks; and The reconstruction module generates a dimensionality-reduced feature map based on the feature map via a convolution process.

Images