A lightweight super-resolution method based on video data

Abstract

The application provides a lightweight super-resolution method based on video data. A lightweight super-resolution network is designed, including feature extraction, feature alignment, feature reconstruction and other modules, which respectively utilize basic convolution operation and channel separation, channel disorder, self-correcting convolution and other processing, can better utilize the features at all levels of the video frame channel and learn differently while ensuring that the model parameter quantity will not be too large, and obtain a better panoramic video super-resolution result.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically relating to a lightweight super-resolution method based on video data. Background Technology

[0002] Super-resolution (SR) is a fundamental computer vision task. It aims to map a low-resolution (LR) image to a high-resolution (HR) image while enhancing its details for a more visually appealing result. SR primarily processes two types of data: images and videos. Image-based SR techniques enhance details while increasing the resolution of LR images, while video-based SR techniques must achieve the same results as image SR while maintaining the continuity and consistency between video frames. Otherwise, the synthesized video will play disjointedly, negatively impacting the viewing experience.

[0003] Existing super-resolution algorithms can be divided into two main categories: traditional methods and deep learning-based methods. Traditional super-resolution algorithms include bilinear interpolation, bicubic interpolation, sparse representation, and so on. Deep learning-based methods, due to the strong feature learning capabilities of convolutional neural networks (CNNs), far outperform traditional super-resolution methods in performance. Many deep learning-based image and video super-resolution algorithms have been proposed, attempting to improve super-resolution effects from different perspectives, including the original data itself, network structure, and loss functions. The earliest application of deep learning to the field of super-resolution was the SRCNN (Deep Convolution Network for Image Super-Resolution) algorithm, which consists of three convolutional layers and achieves the goal of recovering LR images through feature extraction, nonlinear mapping, and reconstruction. Subsequently, in response to the shortcomings of existing methods, many excellent image SR algorithms have been proposed from the perspectives of network input format, network structure, loss function, and information utilization. These include FSRCNN (Accelerating the Super-Resolution Convolutional Neural Network), VDSR (Very Deep Convolutional Network), ESPCN (Efficient Sub-Pixel Convolutional Neural Network), EDSR (Enhanced Deep Residual Networks for Single Image Super-Resolution), RDN (Residual Dense Network), RCAN (Very Deep residual channel attention network), and DRN (Deep Residual Network).

[0004] Video super-resolution algorithms include early methods such as Deep-DE and VSRnet (Video Segment Retrival Network). VSRnet, an improvement on SRCNN, also uses three convolutional layers, but the input is changed from a single image to multiple consecutive frames. Subsequently, many more excellent algorithms have been proposed, including VESPCN (Video Efficient Sub-Pixel Convolutional Neural Network), DUF (Dynamic Upsampling Filters Video Super-Resolution Network), DRVSR (Dense Memory Network for Video Super-Resolution), TDAN (Temporally Deformable Alignment Network), RBPN (Recurrent Back-Projection Network for Video Super-Resolution), and EDVR (Video Restoration with Enhanced Deformable Convolutional Networks). To more effectively utilize inter-frame information in video and capture as much useful information as possible from neighboring frames for super-resolution of the target frame, such as high-frequency details, methods such as 3D convolution, non-local networks, RCNN (Region-CNN), deformable convolution, and motion estimation and compensation based on optical flow have been proposed. Among these, optical flow-based methods suffer from inaccurate estimated optical flow motion information due to application limitations, thus affecting subsequent super-resolution processes and ultimately limiting performance. Deformable convolution-based methods, on the other hand, can obtain more accurate inter-frame information and achieve superior performance; typical examples include EDVR, VESRnet (Video Enhancement and Super-Resolution net), and TDAN.

[0005] With the development of virtual reality and 5G mobile communication technologies, 360-degree panoramic video has become increasingly important for immersive technologies. Panoramic video is captured by 360-degree cameras and stored in two-dimensional planar formats, such as equal rectangular projection (ERP) and cube projection. During playback, panoramic video can be projected onto ordinary flat-panel displays and spherical displays. In recent years, panoramic video has been widely used in entertainment, education, healthcare, communications, advertising, and many other fields. However, this video format faces significant challenges because it requires very high resolution to cover the entire 360-degree space to ensure visually satisfying captured video. Given a portion of the panoramic video, called a viewpoint, it can be utilized by a VR display, but currently, the resolution of viewpoints is relatively low, significantly reducing the visual effect. Therefore, in VR systems based on spherical video, high-resolution video is crucial for ensuring a high-quality viewing experience. However, currently, both user systems and network bandwidth struggle to process and transmit such large-scale video in real time. An effective way to solve this problem is to first capture low-resolution video and then super-resolution it to high-resolution video.

[0006] Currently, there are relatively few publicly available panoramic video super-resolution algorithms. A typical example is the SMFN (Single Frame and Multi-Frame Joint Network) algorithm, which is a panoramic video super-resolution algorithm based on the joint use of single-frame and multi-frame super-resolution. It consists of a single-frame super-resolution network, a multi-frame super-resolution network, a dual network, and a fusion module to combine the advantages of single-frame and multi-frame image super-resolution. Single-frame image super-resolution can effectively recover spatial information, while multi-frame image super-resolution can effectively capture temporal information between frames. The dual network is responsible for constraining the solution space during training. The network's goal is to output a high-resolution result of the target frame with a super-resolution factor of s, given 2N+1 consecutive low-resolution video frames (N represents the number of frames adjacent to the target frame on the left (right), and the target frame plus the 2N adjacent frames on the left and right equals 2N+1 consecutive frames). The feature extraction module first takes the target frame and its adjacent frames as input to generate corresponding feature maps. Then, the alignment module aligns the feature maps of adjacent frames with the feature map of the target frame. The aligned features are then input into the reconstruction module to obtain the high-resolution image. The single-frame super-resolution network directly performs single-frame super-resolution processing on the target frame. To further improve the quality of the reconstructed video, the SMFN algorithm also designs a fusion module to post-process the results of the reconstruction module and the single-frame super-resolution network. While this design ensures the model's performance, it also leads to a complex model structure, an excessive number of network parameters, and, given the high resolution of the video frames generated for panoramic video super-resolution, excessive demands on GPU server memory and inference performance, making it difficult to implement in practical applications. Summary of the Invention

[0007] To overcome the shortcomings of existing technologies, this invention provides a lightweight super-resolution method based on video data. A lightweight super-resolution network is designed, including modules for feature extraction, feature alignment, and feature reconstruction. It utilizes basic convolution operations and processes such as channel separation, channel shuffling, and self-correcting convolution to effectively utilize the features at various levels of video frame channels and perform differentiated learning while ensuring that the number of model parameters is not excessive, thus obtaining better panoramic video super-resolution results.

[0008] A lightweight super-resolution method based on video data, characterized by the following steps:

[0009] S1. The low-resolution 360° panoramic video is split into continuous video frames. The continuous video frames are input into the network in a sliding window manner. The window size is set to 3 frames and the step size is 1 frame. When processing the first and last frames, the second and second-to-last frames of the original video frame sequence are added to the first and last frames of the sequence, respectively. The video frame corresponding to the center of the window is the target frame, and the other two video frames are the nearest neighbor frames.

[0010] S2. The input video frame is processed by the feature extraction module, which consists of several residual blocks composed of two convolutional layers.

[0011] S3. The extracted features are processed by the feature alignment module to extract inter-frame information between neighboring frames and the target frame. This inter-frame information is then used to align the feature maps of the neighboring frames and the target frame before outputting the results. The specific process includes:

[0012] S301. The nearest neighbor frame features and target frame features extracted in step S2 are concatenated according to the channel dimension, and then feature extraction is performed through a convolutional layer.

[0013] S302. Input the feature information obtained in step S301 into the self-calibrating convolutional network. First, the input feature information channels are divided into two. One branch's features, obtained after passing through an average pooling layer, a convolutional layer, and an upsampling operation, are added to and fused with the input features. The fused features are then passed through a Sigmoid layer and multiplied with the output of the input features after passing through a convolutional layer. The fused features are then passed through a convolutional layer to obtain the output features of this branch. The other branch's features are extracted through a convolutional layer. The output features of the two branches are then concatenated along the channel dimension and output.

[0014] S303. Pass the output of the self-calibrating convolutional network through another convolutional layer to obtain the offset feature.

[0015] S304. Use the offset feature to perform feature transformation on the feature information obtained in step S301, and then generate and output the aligned feature map through a convolutional layer.

[0016] S4. Input the aligned feature map into the residual dense block-based reconstruction module, and output it after channel shuffling, channel separation, and feature fusion. The specific process includes:

[0017] S401. Pass the feature map output in step S3 through a convolutional layer;

[0018] S402. Use a shuffle operation to randomly shuffle the channels of the feature map;

[0019] S403. After the channels of the shuffled feature map are divided into two, one branch is passed through a convolutional layer and then concatenated with the other branch along the channel dimension to obtain a new feature map.

[0020] S404. Using the new feature map obtained in step S403 as input, return to step S402 for iterative processing until the specified number of iterations is reached. Then, add the corresponding elements of the new feature map obtained in step S3 after passing it through a 1×1 convolution to obtain the fused feature map and output it.

[0021] S5. Upsample the original input target frame and the feature map obtained in step S4 respectively, and add the two upsampled results together. Use the summed result as the super-resolution panoramic video frame corresponding to the target frame.

[0022] The beneficial effects of this invention are: by adopting lightweight module design such as channel separation operation, the number of model parameters can be kept small, the computational complexity is low, and the algorithm performance is improved; by adopting feature alignment, self-correcting convolution and other processing, inter-frame information can be better utilized to obtain better super-resolution effect. Attached Figure Description

[0023] Figure 1 This is a flowchart of the lightweight super-resolution method based on video data of the present invention;

[0024] Figure 2 This is a schematic diagram of the self-correcting convolution structure of the present invention;

[0025] Figure 3 This is a schematic diagram of the lightweight residual dense block (LWRDB) structure of the present invention;

[0026] Figure 4 This is a visualization of the relationship between the weighted spherical peak signal-to-noise ratio (WS-PSNR) and the parameters of each model.

[0027] Figure 5This is a visualization of the relationship between the weighted spherical structural similarity (WS-SSIM value) and the parameters of each model.

[0028] Figure 6 This is a visualization of the relationship between peak signal-to-noise ratio (PSNR) and the parameters of each model.

[0029] Figure 7 This is a visualization of the relationship between structural similarity (SSIM) and the parameters of each model.

[0030] Figure 8 These are images showing the results of different algorithms performing 4x super-resolution on video 005.

[0031] Figure 9 These are images showing the results of different algorithms performing 4x super-resolution on video 006. Detailed Implementation

[0032] The present invention will be further described below with reference to the accompanying drawings and embodiments. The present invention includes, but is not limited to, the following embodiments.

[0033] This invention provides a lightweight super-resolution method based on video data, which can effectively super-resolution panoramic videos, and the lightweight model design makes it easier to apply to real-world generation. Figure 1 As shown, continuous video frames are input, and after passing through the feature extraction module, feature alignment module, and reconstruction module, a super-resolution image is obtained. The specific implementation process is as follows:

[0034] S1. The low-resolution 360° panoramic video is split into continuous video frames. The continuous video frames are input into the network in a sliding window manner. The window size is set to 3 frames and the step size is 1 frame. When processing the first and last frames, the second and second-to-last frames of the original video frame sequence are added to the first and last frames of the sequence, respectively. The video frame corresponding to the center of the window is the target frame, and the other two video frames are the neighboring frames.

[0035] S2. The input video frame is processed by the feature extraction module, which consists of several residual blocks composed of two convolutional layers.

[0036] S3. The extracted features are processed by the feature alignment module to extract inter-frame information between neighboring frames and the target frame. This inter-frame information is then used to align the feature maps of the neighboring frames and the target frame before outputting the results. The specific process includes:

[0037] S301. The input neighbor frame features and target frame features are concatenated according to the channel dimension, and then feature information is extracted through a convolutional layer. The neighbor frame features and target frame features are obtained in step S2.

[0038] S302. Input the feature information obtained in step S301 into the self-calibrating convolutional network, such as... Figure 2 As shown, firstly, channel splitting is performed, dividing the input feature information into two channels. One branch, after passing through an average pooling layer (AvgPool), a convolutional layer (Conv), and an upsampling layer, has its feature map fused with the corresponding elements of the input feature map. The fused feature map then passes through a sigmoid layer and is multiplied with the corresponding elements of the input's output after the convolutional layer. The output of the convolutional layer after fusion is used as the output of this branch. The other branch extracts features through a convolutional layer to preserve the original spatial context features of the input. Finally, the output features of the two branches are concatenated along the channel dimension and output.

[0039] S303. Pass the output of the self-calibrating convolutional network through another convolutional layer to generate offset features composed of the offsets of each position in the feature map obtained in S301.

[0040] S304. The offset feature is used to perform feature transformation on the input feature map obtained from S301, and then the aligned feature map is generated and output through a convolutional layer.

[0041] S4, such as Figure 3 As shown, the aligned feature map (F) in The video frame features are then processed by the reconstruction module, which performs channel shuffling, channel splitting, and feature fusion. This reconstruction module is based on residual dense blocks and uses a lighter residual dense block (LWRDB) designed by introducing channel shuffling and channel splitting operations to fuse video frame features. The specific process includes:

[0042] S401. Input the feature map output from step S3, and pass it through a convolutional layer;

[0043] S402. Randomly shuffle the channels of the feature map using a shuffle operation;

[0044] S403. After the channels of the shuffled feature map are split into two, one branch is passed through a convolutional layer and then concatenated with the other branch (without any operation) along the channel dimension to obtain a new feature map.

[0045] S404. Using the new feature map obtained in step S403 as input, return to step S402 for repeated processing until the specified number of times is reached. Then, pass the new feature map obtained in the last step through a 1×1 convolution and add it to the corresponding elements of the feature map output in step S3 to obtain the fused feature map and output it.

[0046] S5. Upsample the original input target frame and the fused feature map respectively, and add the two upsampled results. Use the sum as the super-resolution panoramic video frame (super-resolution image) corresponding to the target frame. At the same time, the output super-resolution image can be downsampled by a downsampled operation consisting of two convolutional layers as the nearest neighbor frame of the input target frame for the next reprocessing.

[0047] To verify the effectiveness of the method of this invention, a self-compiled panoramic video dataset was used as the experimental dataset. Super-resolution experiments were conducted on this dataset using different methods, and the WS-PSNR (Weighted to Spherically uniform Peak Signal-to-Noise Ratio), WS-SSIM (Weighted to Spherically uniform Structural Similarity), PSNR (Peak Signal-to-Noise Ratio), and SSIM (Structural Similarity) indices of different methods were compared. Among them, the WS-PSNR value represents the peak signal-to-noise ratio of a clean 360-degree video and a noisy video; the higher the value, the higher the similarity between the two videos. The WS-SSIM value represents the structural similarity between a clean 360-degree video and a noisy video; the higher the value, the higher the similarity between the two videos. The PSNR value represents the pixel difference between a clean image and a noisy image; the higher the value, the higher the similarity between the two images. The SSIM value represents the similarity in brightness, contrast, and structure between a clean image and a noisy image; the higher the value, the higher the similarity between the two images.

[0048] The experimental comparison methods included SR360, VSRnet (Video segment REtrival Network), VESPCN (Video Efficient Sub-Pixel Convolutional Neural Network), FRVSR (Frame-Recurrent Video Super-Resolution), TDAN (Temporally-Deformable Alignment Network), SOFVSR20 (Video Super Resolution through HR Optical Flow Estimation), EDVR (Video Restoration with Enhanced Deformable Convolutional Networks), and SMFN (Single Frame and Multi-Frame Joint Network). Among these, SR360, VSRnet, VESPCN, FRVSR, TDAN, and SOFVSR20 are relatively lightweight video super-resolution algorithms, with model parameters all below 2M, while EDVR and SMFN are relatively large-scale video super-resolution algorithms. The quantitative calculation results of WS-PSNR and WS-SSIM values ​​for different methods on the panoramic video dataset are shown in Table 1. As can be seen, the method of this invention is only 0.05 dB lower than the best-performing SMFN method in terms of WS-PSNR and WS-SSIM, respectively. However, the number of parameters in this invention is only about one-fifth of that in SMFN, indicating that the method of this invention achieves a good balance between model complexity and performance. Furthermore, Table 1 shows that compared to lightweight video super-resolution algorithms, although the model parameters of this invention are larger, the difference is not significant, while the performance is greatly improved, proving the effectiveness of the method of this invention.

[0049] Table 1

[0050]

[0051]

[0052] Table 2 shows the quantitative calculation results of PSNR and SSIM values ​​of different methods on the panoramic video dataset. It can be seen that the method of this invention is very close to the best-performing SMFN method in terms of metrics, but the number of parameters in this invention is only about one-fifth of that in SMFN. This indicates that the method of this invention achieves a good balance between model complexity and performance, proving the effectiveness of the method.

[0053] Table 2

[0054]

[0055] Figure 4 , 5 Tables 6 and 7 present visualizations of the relationship between WS-PSNR, WS-SSIM, PSNR, SSIM values, and model parameter count for different methods. The horizontal axis represents the number of parameters (in meters), and the vertical axis represents the WS-PSNR, WS-SSIM, PSNR, and SSIM values ​​(all in dB). Ours represents the method of this invention. It can be seen that the method of this invention achieves a good balance between performance and model complexity, making it more suitable for practical applications.

[0056] Figure 8 , 9 The results of 4x super-resolution processing on panoramic video datasets 005 and 006 using different methods are presented. In the figures, the numbers under the method names represent the WS-PSNR / WS-SSIM values ​​of that method. The upper right corner of the screenshot shows a small patch of the result displaying the original ground truth (GT) image, while the remaining small images show the results of various models. It can be seen that, compared to existing algorithms, the method of this invention recovers more realistic and reliable details, closely resembling the original high-resolution image. Therefore, the method of this invention achieves a good balance between performance and algorithmic complexity, exhibiting better super-resolution performance and being suitable for practical applications.

Claims

1. A lightweight super-resolution method based on video data, characterized in that... The steps are as follows: S1. The low-resolution 360° panoramic video is split into continuous video frames. The continuous video frames are input into the network in a sliding window manner. The window size is set to 3 frames and the step size is 1 frame. When processing the first and last frames, the second and second-to-last frames of the original video frame sequence are added to the first and last frames of the sequence, respectively. The video frame corresponding to the center of the window is the target frame, and the other two video frames are the nearest neighbor frames. S2. The input video frame is processed by the feature extraction module, which consists of several residual blocks composed of two convolutional layers. S3. The extracted features are processed by the feature alignment module to extract inter-frame information between neighboring frames and the target frame. This inter-frame information is then used to align the feature maps of the neighboring frames and the target frame before outputting the results. The specific process includes: S301. The nearest neighbor frame features and target frame features extracted in step S2 are concatenated according to the channel dimension, and then feature extraction is performed through a convolutional layer. S302. Input the feature information obtained in step S301 into the self-calibrating convolutional network. First, the input feature information channels are divided into two. One branch's features, obtained after passing through an average pooling layer, a convolutional layer, and an upsampling operation, are added to and fused with the input features. The fused features are then passed through a Sigmoid layer and multiplied with the output of the input features after passing through a convolutional layer. The fused features are then passed through a convolutional layer to obtain the output features of this branch. The other branch's features are extracted through a convolutional layer. The output features of the two branches are then concatenated along the channel dimension and output. S303. Pass the output of the self-calibrating convolutional network through another convolutional layer to obtain the offset feature. S304. Use the offset feature to perform feature transformation on the feature information obtained in step S301, and then generate and output the aligned feature map through a convolutional layer. S4. Input the aligned feature map into the residual dense block-based reconstruction module, and output it after channel shuffling, channel separation, and feature fusion. The specific process includes: S401. Pass the feature map output in step S3 through a convolutional layer; S402. Use a shuffle operation to randomly shuffle the channels of the feature map; S403. After the channels of the shuffled feature map are divided into two, one branch is passed through a convolutional layer and then concatenated with the other branch along the channel dimension to obtain a new feature map. S404. Using the new feature map obtained in step S403 as input, return to step S402 for iterative processing until the specified number of iterations is reached. Then, add the corresponding elements of the new feature map obtained in step S3 after passing it through a 1×1 convolution to obtain the fused feature map and output it. S5. Upsample the original input target frame and the feature map obtained in step S4 respectively, and add the two upsampled results together. Use the summed result as the super-resolution panoramic video frame corresponding to the target frame.

Images