Single-frame super-resolution reconstruction method based on VVC video coding characteristics

Abstract

The invention discloses a single-frame super-resolution reconstruction method based on VVC video coding characteristics, and belongs to the field of super-resolution reconstruction. The method comprises the following steps: 1, a feature fusion method based on coding features; and 2, single-frame super-resolution reconstruction based on coding characteristics. The method is based on the current latest general video coding VVC standard, fully utilizes the guiding advantage of video coding characteristics on the super-resolution reconstruction process of the coded video, is combined with the thought based on deep learning in the existing super-resolution reconstruction, and guides a super-resolution reconstruction model to perform high-quality fitting by fusing single-frame video coding characteristics, so that the super-resolution reconstruction efficiency is improved. And on the premise of not increasing the complexity of the model framework, the super-resolution reconstruction quality of the decoded video is effectively improved.

Description

Technical Field

[0001] The present invention belongs to the field of image super-resolution reconstruction, and relates to a single-frame super-resolution reconstruction method based on coding characteristics in the VVC coding standard. Background Art

[0002] With the increasing maturity of the Internet and the progress of communication technologies such as 5G, information interaction has stepped into thousands of households, and people's dependence on and demand for information from various media are also increasing. The development of information technology has made the early text-form information unable to meet people's daily needs, and new types of content such as pictures, short videos, and emoticons have gradually become the mainstream. Compared with text-type data, image-type data has the characteristics of higher content richness, larger amount of information, and stronger visual perception. People spend more and more time surfing the Internet every day, and there are more and more channels for communication. The progress of technology has led to a faster corresponding speed. All major social software, portal websites, and media are filled with image information. Even with the development of "media integration", short videos are gradually mixed in.

[0003] At the same time, in order to provide a better user experience, terminal devices have also been rapidly iterated with the development of information technology. From the screen display range of a smart watch of a hundred square millimeters to the giant spherical outer surface screen of 54,000 square meters of the "MSG Sphere" in the bowl-shaped theater in Las Vegas, USA. Facing the increasingly complex terminal devices, the existing unified encoding and decoding process cannot well meet the current complex scenario requirements, and it is difficult to adapt to the terminal display, resulting in problems such as video compression distortion, blurred and jerky display of video streams that cannot be effectively solved.

[0004] Super-resolution reconstruction technology often pursues the improvement of the key parts and detail quality of images. Its emergence provides technical possibilities for applications such as subsequent recognition, processing, and display of low-quality and small targets in video frames. However, the existing video super-resolution reconstruction technology mainly restores the size and degradation problems of the video source, and does not consider the reasons for the image quality degradation caused by the video coding process under the block-based hybrid coding framework. Therefore, the existing super-resolution reconstruction technology has very limited effect on improving the quality of compressed videos.

[0005] Therefore, when resolving the obvious video compression distortion and the problems of blurred and jerky display of the terminal after decoding, if the coding characteristics can be fully explored and utilized, and the coding characteristics are coupled with the super-resolution reconstruction technology, it will undoubtedly improve the accuracy and efficiency of super-resolution reconstruction for coded videos. To achieve this goal, the present invention is based on the latest general video coding VVC standard, and by fusing the single-frame video coding characteristics, guides the super-resolution reconstruction model to perform high-quality fitting, and effectively improves the super-resolution reconstruction quality of the decoded video without increasing the complexity of the model framework. Summary of the Invention

[0006] The technical problem to be solved by the present invention is that existing video super-resolution reconstruction technologies mainly restore the size and degradation problems of video sources, and the improvement effect on the impact of block partitioning, intra-frame prediction mode, and loop filtering introduced during the VVC encoding process on video reconstruction quality is very limited. Therefore, without increasing the complexity of the model framework, a single-frame super-resolution reconstruction method based on the characteristics of VVC video encoding is proposed. By associating the single-frame video encoding characteristics of the video with the video content characteristics, fusing and extracting the video content features and encoding features, and making full use of the guiding role of the encoding features for the super-resolution reconstruction model, the model can achieve high-quality fitting for the degradation problems generated by the VVC encoder, effectively improving the single-frame super-resolution reconstruction quality of VVC videos.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] A single-frame super-resolution reconstruction method based on the characteristics of VVC video encoding includes the following contents:

[0009] First: A feature fusion method based on encoding characteristics;

[0010] This method mainly realizes channel-level fusion of the low-resolution image with the block partition map (BPM), intra-prediction mode map (IPM), and loop filter map (LFM) in the encoding characteristics generated during the VVC encoding process, and preliminarily extracts the fusion features through convolution operations to provide necessary encoding prior information for the subsequent network.

[0011] First, extract the low-resolution image I LR , with a size of H×W×3, the block partition map (BPM), with a size of H×W×1, where each pixel records the CU type in the corresponding area, the intra-prediction mode map (IPM), with a size of H×W×1, where each pixel records the prediction mode of the PU in the corresponding area, and the loop filter map (LFM), with a size of H×W×1, recording the filter type applied to the corresponding area. For each encoding characteristic map, perform normalization operation to ensure that its numerical distribution is suitable for the processing of the subsequent convolutional network.

[0012] Secondly, perform channel-level fusion on each of the above-extracted features. Concatenate the obtained low-resolution image with the three encoding characteristic maps of BPM, IPM, and LFM along the channel dimension. The size of the resulting fusion input tensor is H×W×6.

[0013] Xinput = concat(I LR , BPM, IPM, LFM) (1)

[0014] Then, perform preliminary convolutional feature extraction. Apply consecutive convolutional operations to the fused input tensor X input using two 3×3 convolutional layers to perform feature extraction in sequence.

[0015] F 1 = ReLU(K 1 * X input + b 1 ) (2)

[0016] F 2 = ReLU(K 2 * F 1 + b 2 ) (3)

[0017] where K 1 , K 2 are the convolutional kernels of the first layer and the second layer respectively, and b 1 , b 2 are the biases of the first layer and the second layer respectively. ReLU is the rectified linear unit, referring to the ramp function in mathematics, which is used to introduce non-linear transformation.

[0018] Through the convolutional feature extraction operation, further fuse each input feature and map it to a higher-dimensional feature space, so as to extract the preliminary fused feature information F 2 .

[0019] Finally, output the fused features obtained in the above steps. Take the obtained feature F 2 as the output of the feature fusion method based on coding characteristics and pass it to the subsequent single-frame super-resolution reconstruction model based on coding characteristics. At this time, the feature fusion method based on coding characteristics not only retains the pixel information of the original low-resolution image, but also carries the structural coding prior information obtained from the VVC coding process, providing a guiding basis for subsequent deep feature extraction and super-resolution reconstruction.

[0020] Second: Single-frame super-resolution reconstruction based on coding characteristics;

[0021] This method constructs a super-resolution reconstruction model based on the output features of the above method. First, take the fused coding feature F 2 as the initial input of the subsequent multiple coding-aware residual blocks. The coding-aware residual block mainly consists of the following three parts: the first layer of convolution and non-linear activation, the attention mechanism, and the second layer of convolution. Specifically, in each residual block, first perform an operation on the input feature F i-1Perform a 3×3 convolution operation to extract local feature information. After ReLU activation, the intermediate feature F′ is obtained.

[0022]

[0023] Among them, W 1 (i) and respectively represent the weights and biases of the first-layer convolution within the i-th residual block.

[0024] To better focus on feature information, the intermediate feature F′ is input into the attention mechanism module SE for weighting. A global average pooling operation is performed on F′ to generate the channel description vector z. Then, z is passed through two fully connected layers and activation functions to generate the attention weights s for each channel. The generated attention weights s are then applied to F′ to generate the weighted feature F″, thereby strengthening the important channel features and suppressing noise and redundant information. After passing through the attention mechanism module, it is input into the second-layer convolution. The weighted feature F″ is further obtained through a 3×3 convolution operation to get the output F i . Then, using the residual connection, the convolution output is added to the input F i-1 of the block to obtain the output F i of the current encoded perception residual block.

[0025]

[0026] s = σ(W 2 ·δ(W 1 ·z)) (6)

[0027] F″c = s c ·F c ′, c = 1,..., C (7)

[0028]

[0029] F i = F i + F i-1 (9)

[0030] Among them, W 1 and W 2 are the weight matrices of the fully connected layers, δ represents the ReLU activation function, and σ represents the Sigmoid activation function.

[0031] Then, by stacking multiple residual blocks, a deep feature representation is formed. The final output feature F output

[0032]

[0033] where N represents the number of stacked residual blocks, and ΔF i represents the incremental feature extracted by the i-th residual block.

[0034] Secondly, the above output feature F output is subjected to feature expansion. A 3×3 convolution operation is performed on F output to expand the feature channels to s 2 ×C channels, where s is the upsampling magnification and C is the base number of channels. The sub-pixel convolution is applied to rearrange the expanded feature into high-resolution spatial information, obtaining the upsampled feature F up .

[0035] F up = PixelShuffle(W up *F output + b up ) (11)

[0036] where W up and b up represent the convolution kernel and the bias. The PixelShuffle operation rearranges the channel information into high-resolution spatial information.

[0037] Finally, a 3×3 convolutional layer is used for the upsampled feature F up to map the number of channels to the number of channels of the final image, obtaining the target super-resolution reconstructed image I SR .

[0038] I SR = W re *F up + b re (12)

[0039] where W re and b re represent the convolution kernel and the bias.

[0040] Facing the current situation of obvious video compression distortion after encoding and decoding, as well as blurred and jerky display on the terminal, compared with the prior art, on the one hand, the present invention reasonably integrates the encoding characteristics and pixel information into the video super-resolution reconstruction method, with low computational complexity and the ability to improve the information basis for reconstruction; on the other hand, it draws on the idea of the residual dense block for image feature extraction in the existing single-frame super-resolution reconstruction, and can well fit the fusion of pixel information and structural encoding prior information. Without increasing the complexity of the model framework, high-quality super-resolution reconstruction of encoded videos within 4 times is achieved. It is of great significance to ensure the full restoration of video frame details and textures and improve the quality of super-resolution reconstruction of encoded videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a flowchart of a single-frame super-resolution reconstruction method based on coding characteristics.

[0042] Figure 2 It is a structural diagram of a single-frame super-resolution reconstruction network based on coding characteristics. Specific implementation manners

[0043] What the present invention mainly realizes is single-frame super-resolution reconstruction based on the coding characteristics of VVC video coding. The specific method adopted by the present invention will be introduced in detail below with reference to the accompanying drawings.

[0044] Specifically, the flowchart of the single-frame super-resolution reconstruction method based on the coding characteristics of VVC video coding is as Figure 1 shown, including the following aspects: First: A feature fusion method based on coding characteristics. Second: Single-frame super-resolution reconstruction based on coding characteristics.

[0045] First: A feature fusion method based on coding characteristics:

[0046] The first step: Extract a low-resolution image from the VVC encoder, with a size of H×W×3, a block partition feature map BPM with a size of H×W×1, an intra-frame prediction mode map IPM with a size of H×W×1, and a loop filter information map LFM with a size of H×W×1. Among them, BPM records the CU type of each region, IPM records the prediction mode of the PU, and LFM records the type of filter applied;

[0047] The second step: Perform normalization operations on the extracted BPM, IPM, and LFM respectively to ensure that their numerical distributions are suitable for subsequent convolutional network processing;

[0048] The third step: Concatenate the low-resolution image with BPM, IPM, and LFM in the channel dimension according to formula 1 to generate a fused input tensor with a size of H×W×6;

[0049] The fourth step: Apply two convolutional layers of formula 2 and formula 3 to the fused input tensor in sequence, and extract preliminary fused features through the ReLU activation function;

[0050] The fifth step: Transmit the extracted preliminary fused features to the subsequent super-resolution reconstruction module as the joint input of coding prior information and pixel information;

[0051] Second: Single-frame super-resolution reconstruction based on coding characteristics:

[0052] The sixth step: Using the fused features as input, perform deep feature extraction through multiple coding-aware residual blocks. Each residual sub-block uses formula 4 to formula 8 for the first layer of convolution and non-linear activation, attention mechanism, and the second layer of convolution. Formula 9 combines the residual connection to generate the final output of the residual block;

[0053] Step 7: Repeat Step 6, stack the coded perception residual blocks using Formula 10, and gradually extract deep features;

[0054] Step 8: Perform a convolution on the deep features to expand the number of channels, and apply sub-pixel convolution to rearrange the spatial information using Formula 11 to generate high-resolution features;

[0055] Step 9: Perform a final convolution on the upsampled features using Formula 12 to map the number of channels to the dimension consistent with the reconstructed image, generate the reconstructed image, and complete the single-frame super-resolution reconstruction of the VVC-encoded video.

[0056] The above specific implementation manners are only used to illustrate the technical solutions of the present invention, rather than limiting it. Those skilled in the art should understand that: the above implementation manners do not limit the present invention in any form, and all similar technical solutions obtained by using equivalent replacements or equivalent transformations and the like belong to the protection scope of the present invention.

Claims

1. A single-frame super-resolution reconstruction method based on the characteristics of VVC video coding, characterized in that: It contains the following two contents: First: feature fusion method based on encoding characteristics; The low-resolution image is fused with the block partition feature map BPM, intra-frame prediction mode map IPM and loop filter information map LFM in the coding characteristics generated during the VVC coding process at the channel level, and the fusion features are initially extracted through convolution operations to provide coding prior information for the subsequent network; Second: single frame super-resolution reconstruction based on coding characteristics; A super-resolution reconstruction model is constructed based on the output feature fusion.

2. The single-frame super-resolution reconstruction method based on VVC video coding characteristics according to claim 1 is characterized in that: First, extract the low-resolution image I from the VVC encoder LR , with a size of H×W×3, a block partition feature map BPM with a size of H×W×1, each pixel of which records the CU type in the corresponding area, an intra-frame prediction mode map IPM with a size of H×W×1, each pixel of which records the prediction mode of the PU in the corresponding area, and a loop filter information map LFM with a size of H×W×1, which records the filter type applied to the corresponding area; for each coding characteristic map, a normalization operation is performed to ensure that its numerical distribution is suitable for the subsequent convolutional network processing; Secondly, the extracted features are fused at the channel level; the acquired low-resolution image is concatenated with the three encoding characteristic maps of BPM, IPM, and LFM according to the channel dimension; the size of the obtained fused input tensor is H×W×6; X input =concat(I LR ,BPM,IPM,LFM) (1) Then, perform preliminary convolution feature extraction; for the fused input tensor X input Apply continuous convolution operations, using two 3×3 convolutional layers to extract features in sequence; F1=ReLU(K1*X input +b1) (2) F2=ReLU(K2*F1+b2) (3) Among them, K1 and K2 are the first and second convolution kernels respectively, b1 and b2 are the first and second biases respectively, and ReLU is a linear rectification function, which refers to the ramp function in mathematics and is used to introduce nonlinear transformation; The convolution feature extraction operation is used to further fuse the input features and map them to a higher-dimensional feature space, thereby extracting preliminary fusion feature information F2; Finally, the fused features obtained in the above steps are output; the obtained feature F2 is used as the output of the feature fusion method based on coding characteristics, and is passed to the subsequent single-frame super-resolution reconstruction model based on coding characteristics; at this time, the feature fusion method based on coding characteristics not only retains the pixel information of the original low-resolution image, but also carries the structural coding prior information obtained from the VVC encoding process, which provides a guiding basis for subsequent deep feature extraction and super-resolution reconstruction.

3. The single-frame super-resolution reconstruction method based on VVC video coding characteristics according to claim 2 is characterized in that: First, the fused coding feature F2 is used as the initial input of the subsequent multiple coding-aware residual blocks; the coding-aware residual block consists of the following three parts: the first layer of convolution and nonlinear activation, the attention mechanism, and the second layer of convolution; in each residual block, the input feature F i-1 Perform a 3×3 convolution operation to extract local feature information, and then activate it with ReLU to obtain the intermediate feature F′; in and Respectively represent the weight and bias of the first convolution layer in the i-th residual block; In order to better focus on the feature information, the intermediate feature F′ is input into the attention mechanism module SE for weighting, and a global average pooling operation is performed on F′ to generate a channel description vector z. Then z is passed through two fully connected layers and an activation function to generate the attention weight s of each channel. The generated attention weight s is then applied to F′ to generate a weighted feature F″, thereby strengthening important channel features and suppressing noise and redundant information. After passing through the attention mechanism module, it is input into the second layer of convolution, and the weighted feature F″ is subjected to a 3×3 convolution operation to obtain the output F i ; Then, the convolution output is connected to the input F of the block using residual connection i-1 Add together to get the output F of the current coded perceptual residual block i ; s=σ(W2·δ(W1·z)) (6) F″c=s c ·F c ′, c=1,...,C (7) F i =F i +F i-1 (9) Among them, W1 and W2 are the weight matrices of the fully connected layer, δ represents the ReLU activation function, and σ represents the Sigmoid activation function; Then, by stacking multiple residual blocks, a deep feature representation is formed; the final output feature F output Where N represents the number of stacked residual blocks, ΔF i represents the incremental features extracted by the i-th residual block; Secondly, the above output feature F output Perform feature expansion and output Perform a 3×3 convolution operation to expand the feature channel to s 2 ×C channels, where s is the upsampling ratio and C is the number of base channels; sub-pixel convolution is applied to rearrange the expanded features into high-resolution spatial information to obtain the upsampled features F up ; F up =PixelShuffle(W up *F output +b up ) (11) Where W up and b up Represents the convolution kernel and bias. The PixelShuffle operation rearranges the channel information into high-resolution spatial information. Finally, the upsampled feature F up Use a 3×3 convolution layer to map the number of channels to the number of channels of the final image, and obtain the target super-resolution reconstructed image I of the reconstructed output SR ; I SR =W re *F up +b re (12) Where W re and b re Represents the convolution kernel and bias.

Images