Single-frame super-resolution reconstruction method based on characteristics of vvc video coding

Abstract

The application discloses a single-frame super-resolution reconstruction method based on VVC video coding characteristics, and belongs to the field of super-resolution reconstruction.The application comprises the following aspects: first, a feature fusion method based on coding characteristics;second, a single-frame super-resolution reconstruction based on coding characteristics.The application is based on the latest general video coding VVC standard, fully utilizes the guidance advantage of video coding characteristics on the super-resolution reconstruction process of coded video, combines the idea of deep learning in the existing super-resolution reconstruction, fuses single-frame video coding characteristics, guides the high-quality fitting of the super-resolution reconstruction model, and effectively improves the super-resolution reconstruction quality of the decoded video without increasing the complexity of the model framework.

Description

Technical Field

[0001] This invention belongs to the field of image super-resolution reconstruction and relates to a single-frame super-resolution reconstruction method based on coding characteristics in the VVC coding standard. Background Technology

[0002] With the increasing maturity of the internet and the advancement of communication technologies such as 5G, information exchange has rapidly spread to every household, and people's reliance on and demand for information from various media are growing. The development of information technology has rendered early text-based information insufficient to meet people's daily needs, leading to the rise of new types of content such as images, short videos, and emojis. Compared to text-based data, image-based data is characterized by its richer content, greater information content, and stronger visual appeal. People are spending increasingly more time online each day, utilizing a wider range of communication channels, and the rapid pace of technological advancements has resulted in a proliferation of image-based information across social media platforms, portals, and media outlets. Furthermore, with the development of "converged media," short videos are increasingly being incorporated into these platforms.

[0003] Meanwhile, to provide a better user experience, terminal devices are rapidly iterating along with the development of information technology, ranging from the small screen display area of ​​a smartwatch (a few hundred square millimeters) to the massive 54,000-square-meter spherical screen of the MSG Sphere, a bowl-shaped theater in Las Vegas, USA. Faced with increasingly complex terminal devices, the existing unified encoding and decoding process cannot adequately handle the demands of today's complex scenarios and is difficult to adapt to terminal displays. This results in unresolved issues such as video compression distortion, blurry video streams, and stuttering.

[0004] Super-resolution reconstruction technology often aims to improve the quality of key parts and details in an image. Its emergence has provided technical possibilities for applications such as the subsequent recognition, processing, and display of low-quality, small targets in video footage. However, existing video super-resolution reconstruction technologies mainly restore the size and quality degradation of the video source, without considering the image quality degradation caused by the video encoding process under a block-based hybrid coding framework. Therefore, the effect of existing super-resolution reconstruction technologies on improving the quality of compressed videos is very limited.

[0005] Therefore, to address the issues of significant compression distortion in decoded videos and blurry, stuttering displays on terminals, fully exploring and utilizing coding characteristics and coupling these characteristics with super-resolution reconstruction technology can undoubtedly improve the accuracy and efficiency of super-resolution reconstruction for coded videos. To achieve this, this invention is based on the latest universal video coding standard, VVC, and by fusing single-frame video coding features, guides the super-resolution reconstruction model to perform high-quality fitting, effectively improving the quality of super-resolution reconstruction of decoded videos without increasing the complexity of the model framework. Summary of the Invention

[0006] The technical problem this invention aims to solve is that existing video super-resolution reconstruction techniques primarily address the size and degradation issues of the video source, offering limited improvement to the reconstruction quality of video from the perspectives of block partitioning, intra-frame prediction modes, and loop filtering introduced during VVC coding. Therefore, without increasing the complexity of the model framework, this invention proposes a single-frame super-resolution reconstruction method based on the characteristics of VVC video coding. This method correlates the single-frame video coding characteristics with the video content characteristics, fusing and extracting video content features and coding features. By fully utilizing the guiding role of coding features in the super-resolution reconstruction model, it achieves high-quality fitting of the model to the degradation problems caused by the VVC encoder, effectively improving the single-frame super-resolution reconstruction quality of VVC videos.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A single-frame super-resolution reconstruction method based on the characteristics of VVC video coding includes the following:

[0009] First: Feature fusion methods based on coding characteristics;

[0010] This method mainly achieves channel-level fusion of low-resolution images with the Block Partition Map (BPM), Intra Prediction Mode Map (IPM), and Loop Filter Map (LFM) generated during the VVC coding process. The fused features are initially extracted through convolution operations to provide necessary coding prior information for subsequent networks.

[0011] First, extract the low-resolution image I from the VVC encoder. LR The system uses a multi-map structure with dimensions H×W×3. A Block Partition Feature Map (BPM) of size H×W×1 records the CU type within each pixel of the corresponding region. An Intra-Prediction Mode Map (IPM) of size H×W×1 records the prediction mode of the PU within each pixel of the corresponding region. A Loop Filter Information Map (LFM) of size H×W×1 records the filter type applied to the corresponding region. For each coding feature map, a normalization operation is performed to ensure that its numerical distribution is suitable for subsequent processing by the convolutional network.

[0012] Secondly, channel-level fusion is performed on the extracted features. The acquired low-resolution image is concatenated with the three coded feature maps (BPM, IPM, and LFM) according to the channel dimension. The resulting fused input tensor has a size of H×W×6.

[0013] Xinput =concat(I LR (1) (BPM, IPM, LFM)

[0014] Then, preliminary convolutional feature extraction is performed. This is applied to the fused input tensor X. input By applying consecutive convolution operations, two 3×3 convolutional layers are used sequentially for feature extraction.

[0015] F1 = ReLU(K1*X) input +b1) (2)

[0016] F2=ReLU(K2*F1+b2) (3)

[0017] Where K1 and K2 are the first and second layer convolution kernels, respectively, b1 and b2 are the first and second layer biases, respectively, and ReLU is a linear rectified function, referring to the ramp function in mathematics, used to introduce nonlinear transformations.

[0018] The input features are further fused through convolutional feature extraction and mapped to a higher-dimensional feature space, thereby extracting preliminary fused feature information F2.

[0019] Finally, the fused features obtained from the above steps are output. The obtained feature F2 is used as the output of the feature fusion method based on coding characteristics and passed to the subsequent single-frame super-resolution reconstruction model based on coding characteristics. At this time, the feature fusion method based on coding characteristics not only retains the pixel information of the original low-resolution image, but also carries the structural coding prior information obtained from the VVC coding process, providing guidance for subsequent deep feature extraction and super-resolution reconstruction.

[0020] Second: Single-frame super-resolution reconstruction based on coding characteristics;

[0021] This method constructs a super-resolution reconstruction model based on the feature fusion output of the aforementioned method. First, the fused encoded feature F2 is used as the initial input to multiple subsequent encoded-aware residual blocks. Each encoded-aware residual block mainly consists of three parts: a first convolutional layer with nonlinear activation, an attention mechanism, and a second convolutional layer. Specifically, in each residual block, the input feature F2 is first processed... i-1 Perform a 3×3 convolution operation to extract local feature information, and then activate it with ReLU to obtain the intermediate feature F′.

[0022]

[0023] Among them W1 (i) and These represent the weights and biases of the first convolutional layer within the i-th residual block, respectively.

[0024] To better focus on feature information, the intermediate feature F′ is input into the attention mechanism module SE for weighting. Global average pooling is then performed on F′ to generate channel description vectors z. z is then passed through two fully connected layers and activation functions to generate attention weights s for each channel. These attention weights s are then applied to F′ to generate the weighted feature F″, thereby strengthening important channel features and suppressing noise and redundant information. After passing through the attention mechanism module, the feature F″ is input into the second convolutional layer, where the weighted feature F″ is further processed through a 3×3 convolutional layer to obtain the output F. i Then, using residual connections, the convolution output is connected to the block input F. i-1 Add them together to obtain the output F of the current coded sensing residual block. i .

[0025]

[0026] s=σ(W2·δ(W1·z)) (6)

[0027] F″c=s c ·F c ′, c=1,...,C (7)

[0028]

[0029] F i =F i +F i-1 (9)

[0030] Where W1 and W2 are the weight matrices of the fully connected layer, δ represents the ReLU activation function, and σ represents the Sigmoid activation function.

[0031] Then, by stacking multiple residual blocks, a deep feature representation is formed. The final output feature F output

[0032]

[0033] Where N represents the number of stacked residual blocks, ΔF i This represents the incremental feature extracted from the i-th residual block.

[0034] Secondly, the above output feature F output Perform feature expansion on F output Perform a 3×3 convolution operation to expand the feature channels to s 2 The feature has ×C channels, where s is the upsampling factor and C is the number of base channels. Subpixel convolution is applied to rearrange the expanded features into high-resolution spatial information, resulting in the upsampled feature F. up .

[0035] Fup =PixelShuffle(W up *F output +b up (11)

[0036] Among them W up and b up Indicates the convolution kernel and bias; the PixelShuffle operation rearranges channel information into high-resolution spatial information.

[0037] Finally, the upsampled feature F is analyzed. up Using a 3×3 convolutional layer, the number of channels is mapped to the number of channels in the final image, resulting in the reconstructed target super-resolution image I. SR .

[0038] I SR =W re *F up +b re (12)

[0039] Among them W re and b re This represents the convolution kernel and bias.

[0040] Addressing the issues of significant video compression distortion and blurry, stuttering displays on terminals after encoding and decoding, this invention, compared to existing technologies, offers several advantages. Firstly, it rationally integrates encoding characteristics and pixel information into the video super-resolution reconstruction method, resulting in lower computational complexity while enhancing the informational foundation of the reconstruction. Secondly, it borrows the concept of residual dense blocks from image feature extraction in existing single-frame super-resolution reconstruction, effectively fitting the fusion of pixel information and structural encoding prior information. Without increasing the complexity of the model framework, it achieves high-quality super-resolution reconstruction of coded videos up to four times the complexity. This is of great significance for ensuring the full recovery of video frame details and textures and improving the quality of coded video super-resolution reconstruction. Attached Figure Description

[0041] Figure 1 This is a flowchart of a single-frame super-resolution reconstruction method based on coding characteristics.

[0042] Figure 2 This is a diagram of the network structure for single-frame super-resolution reconstruction based on coding characteristics. Detailed Implementation

[0043] This invention primarily achieves single-frame super-resolution reconstruction based on the characteristics of VVC video coding. The specific methods employed in this invention will be described in detail below with reference to the accompanying drawings.

[0044] Specifically, the process of single-frame super-resolution reconstruction based on the characteristics of VVC video coding is as follows: Figure 1As shown, it includes the following aspects: First: Feature fusion method based on coding characteristics. Second: Single-frame super-resolution reconstruction based on coding characteristics.

[0045] First: Feature fusion method based on encoding characteristics:

[0046] Step 1: Extract the low-resolution image (H×W×3), block partitioning feature map (BPM) (H×W×1), intra-frame prediction mode map (IPM) (H×W×1), and loop filter information map (LFM) (H×W×1) from the VVC encoder. The BPM records the CU type for each region, the IPM records the prediction mode of the PU, and the LFM records the type of filter applied.

[0047] Step 2: Normalize the extracted BPM, IPM, and LFM to ensure that their numerical distribution is suitable for subsequent convolutional network processing.

[0048] Step 3: Concatenate the low-resolution image with BPM, IPM, and LFM according to Formula 1 to generate a fused input tensor with a size of H×W×6.

[0049] Step 4: Apply the convolutional layers of formulas 2 and 3 sequentially to the fused input tensor, and extract the initial fused features using the ReLU activation function;

[0050] Step 5: Pass the extracted preliminary fusion features to the subsequent super-resolution reconstruction module as a joint input of encoded prior information and pixel information;

[0051] Second: Single-frame super-resolution reconstruction based on coding characteristics:

[0052] Step 6: Using the fused features as input, deep feature extraction is performed through multiple encoded perceptual residual blocks. Each residual sub-block undergoes a first-layer convolution with non-linear activation and attention mechanism, followed by a second-layer convolution, using Equations 4 to 8. Equation 9 combines the residual connections to generate the final residual block output.

[0053] Step 7: Repeat step 6, using formula 10 to stack coded perceptual residual blocks to gradually extract deep features;

[0054] Step 8: Perform a convolution on the deep features to expand the number of channels, and use Formula 11 to apply sub-pixel convolution to rearrange spatial information and generate high-resolution features;

[0055] Step 9: Use Formula 12 to perform a final convolution on the upsampled features, mapping the number of channels to the same dimension as the reconstructed image, generating the reconstructed image, and completing the single-frame super-resolution reconstruction of the VVC encoded video.

[0056] The above specific embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Those skilled in the art should understand that the above embodiments do not limit the present invention in any way, and all similar technical solutions obtained by equivalent substitution or equivalent transformation are within the protection scope of the present invention.

Claims

1. A single-frame super-resolution reconstruction method based on the characteristics of VVC video coding, characterized in that, It includes the following two contents, First: Feature fusion methods based on coding characteristics; The low-resolution image is fused with the block partitioning feature map (BPM), intra-frame prediction mode map (IPM), and loop filter information map (LFM) generated during the VVC coding process at the channel level, and the fused features are initially extracted through convolution operation to provide coding prior information for subsequent networks. Second: Single-frame super-resolution reconstruction based on coding characteristics; A super-resolution reconstruction model is constructed based on the fusion of output features; First, the fused coding features will be... As the initial input for multiple subsequent code-aware residual blocks; a code-aware residual block consists of three parts: a first convolutional layer with non-linear activation, an attention mechanism, and a second convolutional layer; in each residual block, the input features are first processed... Perform once Convolutional operations are used to extract local feature information, followed by ReLU activation to obtain intermediate features. ; (4); in and They represent the first The weights and biases of the first convolution within each residual block; To better focus on feature information, intermediate features The input is weighted and fed into the attention mechanism module SE for processing. Perform global average pooling to generate channel description vectors. Then Attention weights for each channel are generated using two fully connected layers and an activation function. ; Then generate attention weights Acting on Generate weighted features This strengthens important channel features and suppresses noise and redundant information; after passing through the attention mechanism module, the input is fed into the second convolutional layer, where the weighted features are processed. Through a layer The convolution operation produces the output. ; Then, using residual connections, the convolution output is connected to the block input. Add them together to obtain the output of the current coded-aware residual block. ; (5); (6); (7); (8); (9); in, and This is the weight matrix of the fully connected layer. Represents the ReLU activation function. This represents the Sigmoid activation function; Then, by stacking multiple residual blocks, a deep feature representation is formed; the final output feature is... (10); in This represents the number of stacked residual blocks. Indicates the first Incremental features extracted from each residual block; Secondly, the above output features Perform feature expansion, for Perform once Convolution operations expand the feature channels to One channel, among which This is the upsampling factor. The base number of channels is used; sub-pixel convolution is applied to rearrange the expanded features into high-resolution spatial information, resulting in upsampled features. ; (11); in and Indicates the convolution kernel and bias; the PixelShuffle operation rearranges channel information into high-resolution spatial information. Finally, the upsampled features are processed... Use one The convolutional layer maps the number of channels to the number of channels in the final image, resulting in the reconstructed super-resolution image of the target. ; (12); in and This represents the convolution kernel and bias.

2. The single-frame super-resolution reconstruction method based on VVC video coding characteristics according to claim 1, characterized in that, First, extract the low-resolution image from the VVC encoder. The size is Block partitioning feature map (BPM), size is Each pixel records the CU type within the corresponding region. The intra-prediction mode map (IPM) has a size of [size missing]. Each pixel records the prediction mode of the PU within the corresponding region, and the loop filter information map (LFM) has a size of [missing information]. Record the filter type applied to the corresponding region; perform normalization operation on each coding feature map to ensure that its numerical distribution is suitable for subsequent processing by the convolutional network. Secondly, channel-level fusion is performed on the extracted features; the acquired low-resolution image is concatenated with the three encoded feature maps (BPM, IPM, and LFM) according to the channel dimension; the resulting fusion input tensor has a size of [size missing]. ; (1); Then, preliminary convolutional feature extraction is performed; the fused input tensor is then processed. Apply consecutive convolution operations, using two The convolutional layers extract features sequentially; (2); (3); in, , These are the first and second layer convolution kernels, respectively. , These are the first and second layer biases, respectively. ReLU is a linear rectified function, referring to the ramp function in mathematics, used to introduce nonlinear transformations. The various input features are further fused through convolutional feature extraction operations and mapped to a higher-dimensional feature space, thereby extracting preliminary fused feature information. ; Finally, output the fused features obtained from the above steps; the obtained features As the output of the feature fusion method based on coding characteristics, it is passed to the subsequent single-frame super-resolution reconstruction model based on coding characteristics. At this time, the feature fusion method based on coding characteristics not only retains the pixel information of the original low-resolution image, but also carries the structural coding prior information obtained from the VVC coding process, providing guidance for subsequent deep feature extraction and super-resolution reconstruction.

Images