Video image super-resolution reconstruction method based on time domain correlation

A super-resolution reconstruction and super-resolution technology, which is applied in the field of video image super-resolution reconstruction based on temporal correlation, can solve the problems of lack of temporal correlation of aggregated video images, difficulty in distinguishing, and waste of computing resources, etc. Achieve the effect of realizing quality and calculation amount, optimizing distribution, and ensuring reconstruction quality

Active Publication Date: 2020-09-15
NORTHEASTERN UNIV
View PDF5 Cites 4 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, many of the existing video super-resolution reconstruction methods still use a network for super-resolution reconstruction of video images, and do not distinguish the degree of difficulty of reconstruction. In this way, a lot of computing resources will be wasted in areas with relatively simple content.
In addition, the existing technical solutions mainly extract feature information from low-resolution video images and then perform "alignment-fusion-reconstruction" operations, and do not integrate temporal correlation of video images, making full use of the reconstruction of previous frames As a result, this further wastes some computing resources
Therefore, there is still room for performance improvement in the current video super-resolution technology.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Video image super-resolution reconstruction method based on time domain correlation
  • Video image super-resolution reconstruction method based on time domain correlation
  • Video image super-resolution reconstruction method based on time domain correlation

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0026] Embodiment 1 is a flowchart of a video image super-resolution reconstruction method based on temporal correlation proposed according to the present invention, wherein figure 1 for the flowchart, figure 2 (a) is the output classification of the decision maker, and (b) is an example diagram of the output classification of the decision maker. figure 1In , firstly, it is judged whether the input video frame is the starting frame, and if it is the starting frame, the super-resolution network is used for reconstruction. If it is a non-starting frame, the current frame is divided into non-overlapping blocks, and H.254 / AVC macroblocks or H.265 / HEVC coding tree units can be used as the basis for block division. The comparison of the change degree between the block of the current frame and the corresponding position block of the previous frame is carried out sequentially, and this process is executed by a decision device. The decider outputs a probability distribution, which i...

Embodiment 2

[0027] Embodiment 2 is a super-resolution reconstruction network architecture proposed according to the present invention. Such as image 3 (a) The network architecture is a network structure based on residual learning, which can accelerate the convergence of training by learning the difference between the original high-resolution video image and the upsampled image of the low-resolution video image. In this embodiment, only the current frame P is used i with its predecessor frame P i-1 , which is mainly to be compatible with the reconstruction requirements of real-time video services. Meanwhile, this embodiment does not limit the use of only one preceding frame of the current frame. The current frame and its preceding frame are subjected to motion compensation processing to obtain a motion-compensated frame, which is then input together with the current frame into the super-resolution reconstruction network for super-resolution reconstruction. image 3 (b) is a network se...

Embodiment 3

[0028] Embodiment 3 is a nonlinear mapping network architecture proposed according to the present invention. The design of the network architecture is mainly to reuse the reconstruction results of the previous frame of the current frame, in order to reduce unnecessary calculations. The input of the network is the reconstruction result of the previous frame of the current frame, that is, the residual data learned by the residual network, and the difference between the block pair corresponding to the current frame and its previous frame. The network implements a nonlinear warped mapping of difference data to residual data. The output is the residual data corresponding to the block of the current frame. Non-linearly distorted networks use convolutional networks with fewer layers.

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a video image super-resolution reconstruction method based on time domain correlation. The method comprises the steps: determining whether a reconstruction result of a preorderframe is used or not through obtaining the space-time correlation feature information of a video frame, so as to reduce the unnecessary repeated calculation; meanwhile, the reconstruction process ofthe to-be-reconstructed frame is guided and the quality is enhanced in combination with the inter-frame difference and the reconstruction condition of the preorder frame. And performing super-resolution reconstruction on the low-resolution video sequence by using a deep learning technology to obtain a high-resolution video sequence, multiplexing a reconstruction result of the preorder frame sequence, and optimizing calculation resource configuration while ensuring the reconstruction quality.

Description

technical field [0001] The invention belongs to the field of video image processing, in particular to a video image super-resolution reconstruction method based on temporal correlation. Background technique [0002] According to the white paper "Cisco Visual Networking Index: Forecast and Trends, 2017–2022" released by Cisco in 2019, video services have become the mainstream services on the Internet. High-Definition, or 4K) has become the main video resolution format, and videos in these formats usually require high transmission bandwidth (generally 5-18Mbps). Since the transmission resources of the Internet are time-varying, when the network transmission resources change dynamically, the transmission of video data, especially the transmission of high-resolution video services, will be greatly affected. Due to the strong spatial correlation between video image pixels, the representation of video content may not require a higher resolution. Therefore, a simple and feasible b...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06T3/40G06K9/62G06N3/04
CPCG06T3/4053G06T3/4076G06N3/044G06N3/045G06F18/214
Inventor 雷为民曹航刘晓雯李玉婷王一达
Owner NORTHEASTERN UNIV
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap