Video space-time super-resolution method and device based on improved deformable convolution correction

Abstract

The invention discloses a video space-time super-resolution method and device based on improved deformable convolution correction. The method comprises the following steps: constructing a video space-time super-resolution network comprising a feature extraction module, an inter-frame correction module and an image reconstruction module; performing network parameter optimization on the video space-time super-resolution for later use; during application, using the feature extraction module for extracting feature maps from input low-fraction adjacent video frames, and using the inter-frame correction module for carrying out correction processing according to the feature maps corresponding to the adjacent video frames to synthesize an intermediate frame feature map; and performing inter-frame and intra-frame feature extraction on the input intermediate frame feature map and the feature map corresponding to the adjacent video frame by using an image reconstruction module, and reconstructing and outputting a high-resolution and high-frame-rate image sequence. By improving a deformable convolution mode and introducing explicit optical flow estimation, an attention network and other skills, an inter-frame correction network is enabled to be better competent for a video space-time super-resolution task, and a restoration effect is greatly improved.

Description

technical field [0001] The present invention relates to the field of computer science image processing, in particular to a method and device for video space-time super-resolution based on improved deformable convolution correction. Background technique [0002] Video spatiotemporal super-resolution is the combination of video and super-resolution and video frame interpolation, two basic problems in the field of video processing. In recent years, the rapid development of deep learning networks provides an efficient solution for video super-resolution and video frame interpolation algorithms, such as a deep learning-based video super-resolution reconstruction method disclosed in a patent application with publication number CN109102462A. Another example is the patent application with publication number CN104463793A, which discloses a video super-resolution reconstruction method and system based on sparse expression and vector continued fraction interpolation under polar coordin...

AI Technical Summary

Problems solved by technology

However, the traditional convolutional network has seldom studied the method of simultaneously completing video spatio-temporal super-resolution in one stage.

[0003] In the space-time super-resolution problem, the motion estimation of the intermediate frame is particularly difficult when the intermediate frame lacks reference and the adjacent input frame is of low resolution.

Although some newer research attempts to introduce methods such as deformable convolution to improve the inter-frame motion compensation effect, the existing deep learning network is often still unable to restore the position and detail information of the intermediate frame at the same time.

The traditional optical flow method models the motion relationship between the intermediate frame and the input frame and two adjacent input frames with a preset ratio, which lacks adaptability

However, the recently emerging motion correction method based on deformable convolution has a poor compensation effect on video with large motion, and it is difficult to refer to the real motion situation, and there is a problem of insufficient generalization performance.

Images