Loop filtering decision method based on deep neural network of coding information

Abstract

The application relates to a loop filtering decision method based on a deep neural network of coding information and belongs to the technical field of image processing. The decision method is applied to the neural network-based loop filter decision in a video coding standard VVC. A neural network-based loop filter is preset as Filter Net, which is used to replace a deblocking effect filter DBF and a sample adaptive offset SAO in the VVC standard. An FFDN is built to extract features of input information, so that distortion conditions are accurately learned, and then a CTU-level Filter Net flag is obtained. The input of the FFDN is reconstruction information of a coding tree unit (CTU), a division depth matrix, slice type information and a QP value. With the coding information, the FFDN network can reduce the complexity of the neural network-based loop filtering scheme under the condition of equivalent coding gain, and then the overall coding time is reduced.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and relates to the field of video coding technology. Specifically, it relates to a loop filtering decision method based on a deep neural network with encoded information. Background Technology

[0002] With the continuous improvement of hardware performance and the iterative development of network technology, coupled with the ever-increasing demands for video content, video is now evolving towards ultra-high definition, wide color gamut, and panoramic formats. The massive video storage and stable video transmission pose new challenges to video coding technologies. The latest generation video coding standard, VVC, reduces the bit rate by approximately 50% compared to its predecessor, HEVC, while maintaining the same perceptual quality. In particular, the loop filtering technology in the video coding standard not only improves the quality of the currently encoded image but also provides a higher-quality reference image for the encoding of subsequent frames.

[0003] Luminosity Mapping with Chroma Scaling (LMCS) is a new technique added to the loop filtering section of the VVC coding standard. It achieves good reconstruction results in both SDR and HDR videos by adaptively modifying the distribution of coded samples through luminosity mapping and chroma scaling components to improve coding efficiency. The Deblocking Filter (DBF) is mainly used to remove block artifacts that occur during the coding process. Video coding processes operate on a block-by-block basis, resulting in discontinuities in pixel values ​​at the boundaries of reconstructed image blocks. When the images on both sides of a block boundary have strong correlation and the image texture is relatively smooth, this pixel discontinuity creates the "block artifact" as perceived by the human eye. The Deblocking Filter can eliminate or mitigate block artifacts during coding, thereby improving the quality of the reconstructed image. The Sample Adaptive Filter (SAO) is used to suppress ringing artifacts at edges and is placed after the Deblocking Filter. In lossy compression coding, due to the influence of prediction, transformation, and quantization processes, reconstructed samples may lose some high-frequency information, leading to ringing artifacts. To eliminate these artifacts, compensation is applied to specific locations in the reconstructed samples, such as peaks, inflection points, and valleys, to reduce the difference between the reconstructed and original samples, thus achieving the filtering purpose. In VVC, based on the characteristics of the reconstructed image, SAO can be divided into two types: edge-compensated (EO) and band-compensated (BO). Different filtering types are selected through control parameters in CTU. VVC integrates a geometry transformation-based ALF (GALF) to reduce its complexity and remove artifacts and distortions generated during the encoding stage. Traditional ALF and GALF are collectively referred to as ALF. ALF minimizes the MSE between the reconstructed and original images through Wiener filtering; it operates after the sample adaptive filter SAO.

[0004] Loop filtering modules can effectively improve the subjective and objective quality of video. The smaller the compression distortion of the reconstructed frame after filtering, and the closer it is to the original frame, the more beneficial it is for the encoding of subsequent frames. Specifically, the filtered image frame may be used in the encoding process of subsequent image frames. The loop filtering technology in the VVC video codec standard can reduce the compression distortion of the reconstructed frame to a certain extent. However, since its parameters are mostly manually designed, the quality of the reconstructed frame after filtering is not ideal. Therefore, some researchers have proposed a loop filtering scheme based on neural networks. Although existing neural network-based loop filtering can improve the quality of the reconstructed frame to some extent, the overall network complexity is high. Moreover, current neural network-based loop filtering schemes need to perform filtering processing through the neural network first, and then use rate-distortion optimization technology to compare the cost before and after filtering to set the CTU-level filtering flag. For regions with simple textures, the filtering processing in the VVC coding standard can achieve efficient artifact removal without the need for filtering through complex neural networks. However, this scheme of performing neural network filtering processing and then using rate-distortion optimization technology to set the filtering flag introduces a lot of unnecessary computation, which increases the overall encoding complexity. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a loop filtering decision method based on deep neural networks with encoded information. This method fully considers the impact of partition depth on filtering decisions, using it as input to the network to make the decision-making process more efficient and accurate. Furthermore, this method considers the impact of quantization during video compression on filtering, also using slice type and QP as network inputs for key information extraction. Based on the above analysis, this method can fully utilize and efficiently fuse various types of information from the encoding process to achieve the goal of accurate filtering decisions. With comparable encoding gain, this invention, by utilizing richer encoded information and a more efficient fusion method, reduces the overall encoding time compared to other loop filtering schemes based on deep neural networks.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A loop filtering decision method based on deep neural networks for encoded information is proposed and applied to the loop filter decision based on neural networks in the video coding standard VVC. First, the loop filter based on neural networks is preset to FilterNet to replace the deblocking filter DBF and sample adaptive compensation SAO in the VVC standard. Then, FFDN is built to extract features from the input information to accurately learn the distortion situation, thereby obtaining the FilterNet flag bit at the CTU level. Here, FFDN represents Fast Filter Decision Network and CTU represents Coding Tree Unit.

[0008] The method specifically includes the following steps:

[0009] S1: Construct the training dataset and validation dataset;

[0010] Images from the DIV2K dataset were converted to YUV. Meanwhile, 160 video sequences were randomly selected from four resolution sequences in the BVI-DVC video dataset to form a new dataset. The dataset was then compressed using VTM-14.0 at QP = 22, 27, 32, 37, and 42. The data required for the training and validation sets were collected and divided into information blocks of the same size according to CTU. Thus, the samples for the training and validation sets were obtained.

[0011] S2: Construct FFDN;

[0012] In FFDN, the feature extraction module is handled by SE-ResBlock, which is a combination of residual blocks (ResBlock) and attention layers (SELayer). ResBlock first reduces dimensionality using a 1×1 convolution, then extracts information using a 3×3 convolution, and finally adjusts the dimensionality using another 1×1 convolution. SELayer first compresses the spatial dimension using global pooling, then reduces dimensionality using a first fully connected layer, learns the channel weights and adjusts the dimensionality using a second fully connected layer, and finally maps the weights using a sigmoid function and multiplies them with the residuals from ResBlock. The network input can be divided into three parts: the first part is a reconstruction information preprocessing module, which uses convolutional layers... The network consists of three main parts: a first part and a second part. The first part uses two SE-ResBlocks to extract features from the reconstructed information; the second part is a depth matrix processing module, which uses convolutional layers and an SE-ResBlock to extract information from the depth matrix; the third part is a Slice type and QP value co-processing module, where the Slice type extracts features through a convolutional operation, and the QP is encoded using One-Hot encoding, multiplied by the weight matrix, and then concatenated with the Slice type; the output module mainly uses convolutional operations and SE-ResBlocks to further extract features and fully fuse the inputs, and then uses a global pooling operation and a fully connected layer to obtain the final filtered flag output.

[0013] S3: Train the network using the training set obtained in step S1 to generate a filtering decision network model. Then, determine hyperparameters such as the learning rate based on the model's performance on the corresponding validation set, and select the optimal model based on its performance on the validation set to finally obtain the fast filtering decision network model.

[0014] S4: Use the LibTorch library to convert the fast filtering decision network model and embed it into VTM-14.0, and disable the DBF and SAO modules in the loop filter, while using the default configuration for the rest; after the above process, the processed image can be used as the input of the ALF filtering module to complete subsequent processing.

[0015] Furthermore, in step S2, in the FFDN, for the luma component, the network input information is the CTU reconstruction information after passing through the LMCS module, the segmentation depth matrix during the encoding process, the slice type information, and the QP value, and the network output information is the CTU-level Filter Net filter flag bit; for the chroma component, the filter flag bit of the luma CTU at the corresponding position is used directly.

[0016] Furthermore, in step S2, the calculation formula of the reconstructed information preprocessing module is as follows:

[0017] D1 = F SE-Res (F SE-Res (w r *X reco ))

[0018] Where D1 represents the output of the reconstruction information preprocessing module, w r The convolution kernel w represents the kernel at the preprocessing module for reconstructed information. r *X reco This indicates the reconstruction information X reco Perform convolution operation, F SE-Res This represents a combination of residual blocks (ResBlock) and the attention mechanism (SElayer).

[0019] Assume the input of the combination of ResBlock and SELayer is x t The output is y t Then F SE-Res Represented as:

[0020] y t =F SE-Res (x t )=x t +F SE (F Res (x t ))

[0021] Among them, F Res ∈R H×W×C F represents the result of residual block processing. SE ∈R 1×1×C This represents the result after applying the attention mechanism, where H represents the height of the output feature map, W represents the width of the output feature map, and C represents the number of channels in the output feature map; F Res It can be specifically represented as:

[0022] F Res (x)=y Res =w 1×1 *max(0,w 3×3 *max(0,w 1×1 *x t ))

[0023] Among them, w 1×1 This represents a convolution kernel with a size of 1×1, w 3×3 This represents a convolution kernel with a kernel size of 3×3, and max(0,x) represents the ReLU activation function;

[0024] F SE This can be specifically expressed as:

[0025]

[0026] Where σ represents the Sigmoid function operation, f GAP This indicates Global Adaptive Pooling. This indicates that a full connection operation is being performed.

[0027] Furthermore, in step S2, the calculation formula for the depth matrix processing module is as follows:

[0028] D2=f Softmax (F SE-Res (w d *X depth_map ))

[0029] Where D2 represents the output of the depth matrix partitioning module, f Softmax This indicates that a Softmax operation is performed on the channel dimension, X depth_map The depth matrix w represents the partitioning depth. d w represents the convolution kernel that divides the depth matrix processing module. d *X depth_map This indicates that a convolution operation is performed on the partitioned depth matrix. Further, in step S2, the QP branch of the Slice type and QP value co-processing module: first, the QP value is encoded using One-Hot encoding to obtain E... QP ∈R 1×d Where d is the length of the QP value interval, and a weight vector V is used. μ ∈R d×1 After multiplication, φ is obtained by convolution. QP ∈R 1×1×C The calculation process can be specifically represented as follows:

[0030] φ QP =w q *(E QP V μ )

[0031] Among them, w q w represents the convolution kernel for the QP value processing part. q *(E QP V μ ) indicates the convolution operation performed after processing the QP value.

[0032] Furthermore, in step S2, the calculation formula for the output module of FFDN is as follows:

[0033]

[0034] Where, x o ∈R H×W×C For the input of the output module, y o This is the output flag.

[0035] The beneficial effects of this invention are as follows: This invention uses a Fast Filtering Decision Network (FFDN) to replace the Loop Filtering Decision (RDO) method in the VVC video codec standard. By efficiently utilizing information during the encoding process, it effectively extracts feature information from each CTU, avoiding the time redundancy caused by RDO. Correct decisions can be made directly through the FFDN network without first performing actual Filter Net filtering operations. Compared to existing neural network-based loop filtering schemes and rate-distortion optimization decision schemes, the fast filtering decision scheme used in this invention makes fuller use of information and improves the efficiency of filtering decisions. Furthermore, it further reduces the overall encoding complexity based on neural network-based loop filtering.

[0036] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0037] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0038] Figure 1 This is a flowchart of the decision-making process of the loop filter module of the present invention;

[0039] Figure 2 This is a schematic diagram of the FFDN network structure;

[0040] Figure 3 This is a schematic diagram of the combined structure of ResBlock and SELayer. Detailed Implementation

[0041] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0042] Please see Figures 1-3This invention provides a loop filtering decision method based on a deep neural network with encoded information. This method fuses multiple pieces of information from the encoding process as input to the network, builds a network model, and then trains the network using a training set. After the network model converges, it is embedded into the VVC standard for application in loop filtering decisions.

[0043] The loop filtering section of the VVC video codec standard includes, in sequence: an adaptive loop shaper (LMCS), a deblocking filter (DBF), a sample adaptive offset filter (SAO), and an adaptive loop filter (ALF). This invention, based on the VVC loop filtering section, pre-defines the neural network used for loop filtering as a Filter Net to replace the original DBF and SAO filtering modules, and uses a fast filter decision network (FFDN) to obtain the flag bits controlling the Filter Net. The overall process is as follows: Figure 1 As shown. The loop filtering decision method of the present invention specifically includes the following steps:

[0044] S1: Construct the training dataset and validation dataset.

[0045] The training and validation datasets were generated using the DIV2K image dataset and the BVI-DVC video dataset. Nine hundred images from the DIV2K dataset were selected, and 160 video sequences were randomly selected from four resolution sequences in the BVI-DVC video dataset. First, the datasets were compressed using VTM-14.0, the reference software for the VVC video codec standard, at QP values ​​of 22, 27, 32, 37, and 42. Then, the images after passing through the LMCS module were saved according to Code Tree Units (CTUs), and the corresponding partition depth matrix, slice type information, and QP value for each CTU were recorded. Assuming the neural network-based loop filtering scheme is a FilterNet, the CTU-level filtering flags obtained from the RDO decision after FilterNet filtering are saved. After the above operations, the training set and validation set with QP = 22, 27, 32, 37, and 42 are obtained. Each set of data contains the reconstruction information of the coding tree unit (CTU) without FilterNet filtering, the partitioning depth matrix of the CTU, the slice type information and QP value, and the corresponding filtering flag.

[0046] S2: Build the FFDN network and train the FFDN network using the training set obtained in step S1 to generate the corresponding fast filtering decision network. Then, determine the optimal hyperparameters based on the performance of the network model on the corresponding validation set, select the optimal model, and finally obtain the network model applied to the filtering decision.

[0047] FFDN network structure as follows Figure 2 As shown, the main information extraction module of the network is a combination of ResBlock and SELayer, namely SE-ResBlock. The network input can be divided into three parts. The first part is the reconstruction information preprocessing module, which extracts features from the reconstruction information through convolutional layers and two SE-ResBlocks. The second part is the segmentation depth matrix processing module, which extracts information from the segmentation depth through convolutional layers and one SE-ResBlock. The third part is the Slice type and QP value collaborative processing module. The Slice type extracts features through a convolutional operation, and the QP is multiplied by a weight matrix after one-hot encoding, and then concatenated with the Slice type after applying a convolutional operation. In the network output part, the main functions are to further extract features and fully fuse the above inputs through convolutional operations and SE-ResBlock. Then, a global pooling operation (GAP) and a fully connected layer (FC) are used to obtain the final filtered flag output.

[0048] The training process in step S2 is detailed as follows: During network model training, the cross-entropy loss function L = -[ylogy + (1-y)log(1-y)] is used, where y represents the label value of the sample and y represents the probability that the sample is predicted as positive. The initial learning rate is set to 1e-3, and it decays to 0.5 times the original rate every 20 epochs, for a total of 80 epochs.

[0049] The inputs to the FFDN network include the reconstruction information of the coding tree unit (CTU), the partitioning depth matrix, the slice type information, and the QP value. The output of the network is the filtering decision made by the neural network after learning the features associated with filtering from each input, indicating whether the Filter Net is enabled or disabled.

[0050] (1) For the luminance component, the network model uses reconstruction information, partitioning depth matrix, slice type information, and QP value as input information, and the network output information is the Filter Net filter flag. The following is a further explanation of the structural details of the network.

[0051] (2) The first input section is the reconstruction information preprocessing module. The input information for this module, i.e., the reconstruction information, is represented as X. reco This module can be represented as:

[0052] D1 = F SE-Res (F SE-Res (w r *X reco ))

[0053] Where D1 represents the output of the reconstruction information preprocessing module, wr The convolution kernel w represents the kernel at the preprocessing module for reconstructed information. r *X reco This indicates the reconstruction information X reco Perform convolution operation, F SE-Res This represents a combination of residual blocks (ResBlock) and the attention mechanism (SElayer).

[0054] Assume the input of the combination of ResBlock and SELayer is x t The output is y t Then F Res+SE This can be further expressed as:

[0055] y t =F SE-Res (x t )=x t +F SE (F Res (x t ))

[0056] Among them, F Res ∈R H×W×C F represents the result of residual block processing. SE ∈R 1×1×C This indicates the result after applying an attention mechanism.

[0057] F Res It can be specifically represented as:

[0058] F Res (x)=y Res =w 1×1 *max(0,w 3×3 *max(0,w 1×1 *x t ))

[0059] Among them, w 1×1 This represents a convolution kernel with a size of 1×1, w 3×3 This represents a convolution kernel with a kernel size of 3×3, and max(0,x) represents the ReLU activation function.

[0060] F SE This can be specifically expressed as:

[0061]

[0062] Where σ represents the Sigmoid function operation, f GAP This indicates Global Adaptive Pooling. This indicates that a full connection operation is being performed.

[0063] (3) The second input part is the depth matrix partitioning module. The input information for this module, i.e., the depth matrix partitioning, is represented as X. depth_map This module can be represented as:

[0064] D2=f Softmax (F SE-Res (w d *X depth_map Among them, f Softmax This indicates that a Softmax operation is performed on the channel dimension.

[0065] (4) The third input section is the Slice type and QP value co-processing module. The input information for this module, i.e., the Slice type, can be represented as X. ST The QP value can be represented as X. QP The QP branch first obtains E by One-Hot encoding the QP value. QP ∈R 1×d Where d is the length of the QP value interval, and a weight vector V is used. μ ∈R d×1 After multiplication, φ is obtained by convolution. QP ∈R 1×1×C The calculation process can be specifically represented as: φ QP =w q *(E QP V μ ).

[0066] (5) The network ends at the output module. Assume the input to this part is x. o ∈R H×W×C The output is the flag bit y. o The calculation process of this module can then be represented as:

[0067] S3: Use the LibTorch library to convert the model obtained in step S2 into a C++ usable type, and use the C++ API in PyTorch to embed the converted network model into the reference software VTM-14.0 provided by the VVC video codec standard. When encoding the standard video sequence provided by JVET, first disable the DBF filter module and SAO filter module in the loop filter, and use the default configuration for the rest. For the luma component, first use the reconstructed information of the coding tree unit (CTU) after passing through the LMCS module as the main input, and record the partition depth information, slice type information and QP value corresponding to the CTU during the encoding process. Use the network model that has been trained and converged in step S2, and input the pre-prepared information into this network model. The output information is the filter flag bit of the coding tree unit. The derivation of the filter flag bit of the chroma component directly uses the filter flag bit of the luma component at the corresponding position. After the above process, the processed image can be used as the input of the ALF filter module to complete the subsequent processing.

[0068] This invention presents a neural network-based loop filtering decision scheme that reduces video coding complexity by utilizing encoded information for fusion. Addressing the problems existing in the Video Coding Standard (VVC) and existing neural network-based loop filtering methods, this invention fully utilizes and efficiently fuses information from the coding process, reducing coding complexity and time compared to traditional neural network-based loop filtering methods without significantly sacrificing the subjective and objective quality of the video.

[0069] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A loop filtering decision method based on a deep neural network with encoded information, characterized in that, This decision-making method is applied to the neural network-based loop filter decision in the video coding standard VVC. First, the neural network-based loop filter is preset as Filter Net to replace the deblocking filter DBF and sample adaptive compensation SAO in the VVC standard. Then, FFDN is built to extract features from the input information to accurately learn the distortion situation, and then obtain the Filter Net flag bit at the CTU level. Here, FFDN represents Fast Filter Decision Network and CTU represents Coding Tree Unit. In the constructed FFDN, the feature extraction module is processed by SE-ResBlock, which is obtained by combining residual blocks ResBlock and attention layers SELayer; among them, ResBlock first passes through a... Convolution reduces dimensionality, then through Convolution is used to extract information, and then through... The network uses convolution to adjust dimensionality. SELayer first compresses the spatial dimension through global pooling, then reduces dimensionality through a first fully connected layer, and a second fully connected layer learns and adjusts the dimension of the channel weights. Finally, it maps the weights using a sigmoid function and multiplies them with the residuals in the ResBlock. The network input consists of three parts: the first is a reconstruction information preprocessing module, which extracts features from the reconstruction information through convolutional layers and two SE-ResBlocks; the second is a segmentation depth matrix processing module, which extracts information from the segmentation depth through convolutional layers and an SE-ResBlock; the third is a Slice type and QP value collaborative processing module, where the Slice type extracts features through a convolutional operation, and the QP is multiplied by the weight matrix after one-hot encoding, then concatenated with the Slice type after another convolutional operation. In the network output module, convolutional operations and SE-ResBlocks further extract features and fully fuse the inputs, then a global pooling operation and a fully connected layer are used to obtain the final filtered flag output. In FFDN, for the luma component, the network input information consists of the CTU reconstruction information after passing through the LMCS module, the segmentation depth matrix during the encoding process, the slice type information, and the QP value. The network output information is the FilterNet filter flag at the CTU level. For the chroma component, the filter flag of the luma CTU at the corresponding position is used directly. The calculation formula of the reconstruction information preprocessing module is as follows: in, This represents the output of the reconstruction information preprocessing module. This represents the convolution kernel at the reconstruction information preprocessing module. Indicates information about reconstruction Perform convolution operations. This represents a combination of residual blocks (ResBlock) and the attention mechanism (SElayer). Assume the input of the combination of ResBlock and SELayer is The output is ,but Represented as: in, This indicates the result of the residual block processing. This indicates the result after applying the attention mechanism. H Indicates the height of the output feature map. W This indicates the width of the output feature map. C This indicates the number of channels in the output feature map; Specifically, it is expressed as follows: in, Indicates the core size is convolution kernel, Indicates the core size is convolution kernel, Represents the ReLU activation function; Specifically, it is expressed as follows: in, This indicates the operation of the Sigmoid function. This indicates global adaptive pooling. This indicates that a full connection operation is being performed; The calculation formula for the depth matrix partitioning module is as follows: in, This represents the output of the depth matrix partitioning module. This indicates that a Softmax operation is performed on the channel dimension. Represents the partitioning depth matrix. This represents the convolution kernel that divides the depth matrix processing module. This indicates that a convolution operation is performed on the partitioned depth matrix; The QP branch of the Slice type and QP value co-processing module: First, the QP value is obtained through One-Hot encoding. ,in The length of the QP value range, and a weight vector. Multiply and then perform a convolution to obtain The calculation process is specifically expressed as follows: in, This represents the convolution kernel used for processing QP values. This indicates a convolution operation performed after processing the QP value; The calculation formula for the output module of FFDN is: in, For the input of the output module, This is the output flag.

2. The loop filtering decision method according to claim 1, characterized in that, The method specifically includes the following steps: S1: Construct the training dataset and validation dataset; S2: Construct FFDN; S3: Train the network using the training set obtained in step S1 to generate a filtering decision network model. Then, determine the hyperparameters based on the model's performance on the corresponding validation set, and select the optimal model based on its performance on the validation set to finally obtain the fast filtering decision network model.

3. The loop filtering decision method according to claim 2, characterized in that, The fast filtering decision network model is converted using the LibTorch library and embedded into VTM-14.

0. The DBF and SAO modules in the loop filter are turned off, while the rest are configured with default settings. After the above process, the processed image can be used as the input to the ALF filtering module to complete subsequent processing.

Images