Apparatus and method for unsupervised video super-resolution using generative adversarial networks

Abstract

A method of video super-resolution (VSR) with temporal consistency using a generative adversarial network (VistGAN) that only requires training high-resolution video sequences to generate high-resolution / low-resolution video frame pairs for training without the need for pre-synthesized high-resolution / low-resolution video frame pairs for training. With this unsupervised learning method, an encoder degrades the input high-resolution video frames of a training high-resolution video sequence to their low-resolution counterparts, and a decoder attempts to recover the original high-resolution video frames from the low-resolution video frames. To improve temporal consistency, the unsupervised learning method provides a sliding window that explores the temporal correlation in the high-resolution domain and the low-resolution domain. It maintains temporal consistency and also leverages high-frequency details from the last generated reconstructed high-resolution video frame.

Description

[0001] Cross-reference with related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 100,272, filed March 5, 2020, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] This invention generally relates to a technique for reconstructing high-resolution (HR) video from its low-resolution (LR) counterpart, a technique known as video super-resolution (VSR). More specifically, this invention relates to VSR using machine learning models. Background Technology

[0004] Video Reconstruction Streaming (VSR) reconstructs an HR video sequence from its LR counterpart. Due to the development of high-definition (HD) displays and the widespread application of VSR in video surveillance, storage, and streaming, VSR has recently attracted considerable attention. VSR aims to reconstruct an HR video sequence from its input LR video sequence. Estimating HR video sequences The HR video sequence should closely resemble the actual HR video sequence. The problem can be formulated as follows:

[0005] ;

[0006] Where Θ represents the network parameter set.

[0007] In the real world, the process of generating an LR video sequence from an HR video sequence can be represented by the following formula:

[0008] ;

[0009] in B Represents a fuzzy matrix. D Let represent the downsampling matrix, and n This represents the noise matrix.

[0010] In most existing methods, the following is given B and D Therefore, training datasets can be artificially synthesized, and the main challenge lies in the ill-posed inverse; however, when it comes to... B and D Performance degrades when dealing with unknown real-world Lightroom videos.

[0011] In general, VSR is a challenging ill-posed problem because the LR frame loses high frequency details, making the recovery based on the spatial dependency within the image only ineffective. Although deep learning methods have recently been used for VSR and shown promising peak signal-to-noise ratio (PSNR) on certain datasets, these current techniques still suffer from the following drawbacks:

[0012] Insufficient training dataset for supervised learning. In the knowledge of supervised learning based on HR / LR degradation methods, previous works are based on supervised learning models trained on synthetic LR / HR pairs, assuming that the training data and test data are processed by the same degradation operation. However, in practice, the original HR version can not exist. Even for the case where the HR version does exist, the degradation operation for the test video is often not known. Because of the lack of corresponding training dataset, they usually show unsatisfactory results in real test videos.

[0013] Temporal inconsistency. Previous works on VSR independently reconstruct the HR frames and thus do not sufficiently explore the correlation or continuity between the HR frames and the high frequency details from the previous HR frames. As a result, temporal discontinuity and flickering artifacts appear in the super-resolved video sequence.

[0014] Ignoring scene changes. Scene changes often occur, especially in movies and TV series. When it happens, the neighboring frames have little correlation. However, previous methods are based on motion compensation to fuse the LR frames and assume that all frames are temporally correlated, which can introduce noisy information from the previous frames. SUMMARY

[0015] To address the above drawbacks, the present invention proposes VistGAN - a VSR method with temporal consistency using a generative adversarial network (GAN). VistGAN is an encoder-decoder architecture based on unsupervised learning. VistGAN only needs a training HR video sequence to generate HR / LR video frame pairs for training, without the need for pre-synthetic HR / LR video frame pairs for training. With this unsupervised learning method, the encoder degrades the input HR video frames of the training HR video sequence to their LR counterparts, and the decoder tries to recover the original HR video frames from the LR video frames. To improve temporal consistency, unlike the sliding window in conventional methods that only consider temporal correlation in the LR domain, the unsupervised learning method provides a sliding window that explores temporal correlation in the HR and LR domains. It maintains temporal consistency and also fully utilizes the high frequency details from the last generated reconstructed HR video frame. In short, the present invention includes the following main advantageous features:

[0016] The unsupervised learning model and temporal consistency based on GAN for video super-resolution do not require any prior assumption or knowledge of the degradation operation. VistGAN trains an adaptive model by matching the features of its degraded LR video frames with the features of the test LR video sequence in an unsupervised manner through a GAN.

[0017] Metric learning in the discriminator. VistGAN employs a metric learning approach in the discriminator to map the degradation operation of the LR video frames to a feature space in which the matching features are minimally separated and vice versa. Furthermore, the degradation feature vectors are used in the generator module in the VistGAN architecture to make the training more stable.

[0018] According to one embodiment of the present invention, the VistGAN comprises a LR frame generation network (LFGNet) configured to synthesize, during training of the device, LR video frames of an intermediate LR video sequence from input HR video frames of a training HR video sequence.

[0019] The VistGAN further comprises an HR frame estimation network (HFENet) configured to generate, from a currently processed LR video frame and a last processed LR video frame of an input LR video sequence and a last generated output HR video frame of a reconstructed HR video sequence, an output HR video frame of the reconstructed HR video sequence, wherein the HFENet is trained using the intermediate LR video sequence as the input LR video sequence and the currently processed LR video frame and the last processed LR video frame are video frames of the intermediate LR video sequence.

[0020] The VistGAN further comprises a kernel estimation network (KENet) configured to, during training, extract degradation features of the currently processed LR video frame and a LR video frame of a test LR video sequence corresponding to the currently processed LR video frame, shrink the extracted degradation features to reduce the Euclidean distance between the same degradation features and expand the Euclidean distance between different degradation features, feed back the shrunk degradation features to the LFGNet, add to the training HR video sequence, and judge whether the degradation of the intermediate LR video sequence is the same as the degradation of the test LR video sequence. BRIEF DESCRIPTION OF DRAWINGS

[0021] In the following, embodiments of the present invention are described in more detail, with reference to the accompanying drawings, in which:

[0022] FIG. 1AA simplified system block and dataflow diagram of a GAN architecture for VSR under unsupervised training is shown according to one embodiment of the present application;

[0023] FIG. 1B A simplified system block and dataflow diagram of a GAN architecture for VSR in testing is shown; and

[0024] FIG. 2 A detailed system block and dataflow diagram of a GAN architecture is shown. DETAILED DESCRIPTION

[0025] In the following description, devices, training methods, and GAN architectures for VSR, etc. are set forth as preferred examples. Modifications, including additions and / or substitutions, will be apparent to one skilled in the art without departing from the scope and spirit of the application. Particular details can be omitted in order not to obscure the application. However, the present disclosure is written to enable one skilled in the art to practice the teachings of the present application without undue experimentation.

[0026] It should be apparent to those skilled in the art that the foregoing examples of digital driving methods are used only to illustrate the working principles of the present application. They are not intended to exhaust or limit the present application to the disclosed exact forms.

[0027] The Euclidean distance used herein, also known as the Euclidean distance, is a distance measure that measures the absolute distance between two points in a multi-dimensional space, such as a two-dimensional or three-dimensional space.

[0028] The optical flow used herein is used to characterize the "instantaneous velocity" of the pixel motion of a spatial moving object on an observation imaging plane. According to the velocity vector characteristics of each pixel point, dynamic analysis of the image can be performed, such as target tracking.

[0029] According to the embodiments of the present application, the VistGAN provided is an encoder-decoder architecture based on unsupervised learning, and can be implemented by a combination of a series of software and / or firmware machine instructions executed by one or more specially configured and interconnected computer processors. These series of software and / or firmware machine instructions can be grouped or represented by logical execution modules.

[0030] In one embodiment, the VistGAN includes at least the following logical execution modules: an LR frame generation network (LFGNet), an HR optical flow estimation network (HFE Net), and a kernel estimation network (KENet). Reference is made to FIG. 1A , FIG. 1AThe data flow of VistGAN in training configuration is exemplified, with LFGNet 101 and HFEnet 102 cascaded together. LFGNet 101 acts as a generator and an encoder, synthesizing intermediate LR video sequence 112 from training HR video sequence 111, where the process is represented by the following equation:

[0031] and

[0032] where B denotes a blur matrix, D denotes a down-sampling matrix, and n denotes a noise matrix, for training HFENet 102. HFENet 102 does not directly generate HR video frames, but rather acts as a decoder and estimates HR video optical flow in a coarse-to-fine manner, which is later used to generate reconstructed HR video sequence 113.

[0033] KENet 103 acts as a discriminator in the VistGAN architecture, extracting features from intermediate LR video sequence 112 to produce intermediate LR video feature map 114, and extracting features from test LR video sequence 115 to produce test LR video feature map 116. KENet 103 then operates to expand the Euclidean distance of features between intermediate LR video feature map 114 and test LR video feature map 116, for judging whether the degradation of intermediate LR video sequence 112 is the same as the degradation of test LR video sequence 115, rather than just judging true or false, and thus producing degradation features.

[0034] LR frame generation network (LFGNet)

[0035] Deep learning-based single image super-resolution (SISR) methods show that a convolutional neural network (CNN) model can learn a mapping from LR images to HR images, which is a one-to-many problem. A CNN model can also mimic a mapping from HR images to LR images, which is a many-to-one problem.

[0036] LFGNet takes training HR video sequence 111 as input and aims to synthesize real LR video frames of intermediate LR video sequence 112, represented by with the same degradation operation as the real LR video frames of test LR video sequence 115. For detailed information about the logical architecture of VistGAN 100, see FIG. 2 , where LFGNet 101 is shown on the left. In one embodiment, LFGNet 101 includes a video sequence synthesizer G 1the video sequence synthesizer G 1 is a generator configured to downsample the HR video frames of the training HR video sequence 111 when generating the LR video frames of the intermediate LR video sequence 112, the training HR video sequence 111 being represented by . The KENet 103 acts as a discriminator, denoted by D, for judging whether the degradation of the intermediate LR video sequence 112 is the same as the degradation of the test LR video sequence 115, the test LR video sequence 115 being represented by . In this way, the generation of the LR video frames of the intermediate LR video sequence 112 can be represented by the following equation:

[0037] ;

[0038] wherein denotes a set of network parameters of the LFGNet 101. Furthermore, the GAN loss :

[0039] ;

[0040] wherein N is the number of samples.

[0041] Since the ground truth is not available, to maintain the content similarity between the pairs of HR video frames of the training HR video sequence 111 and the pairs of LR video frames of the intermediate LR video sequence 112, the computation of a content loss, including a pixel loss and a VGG (Visual Geometry Group algorithm) loss, is introduced to the intermediate LR video sequence synthesis. The pixel loss is used to maintain the down-sampling property of the intermediate LR video sequence synthesis, while the VGG loss makes the output LR video frames produced from G 1 to have the same semantic content as the original input HR video frames to G 1. The pixel loss and the VGG loss are computed by the following equations:

[0042] ; and

[0043] ;

[0044] wherein denotes the video frame down-sampled (i.e. by bicubic down-sampling) from the input HR video frame; and denotes the feature map between the j th convolutional layer and the i th max-pooling layer in the pre-trained VGG-19 network.

[0045] Although and degradation methods are different, the minimization of pixel loss and VGG loss can protect the output LR video frames from bias in the down-sampling operation. Although the real degradation is unknown, the prior information adopted by LFGNet 101 is a kind of down-sampling operation. Although bicubic down-sampling can be used in the calculation of pixel loss , the purpose is not to obtain the bicubic down-sampling result, but to ensure that the synthesis of the intermediate LR video sequence by G 1 is indeed a kind of down-sampling operation. Since the VGG-19 network can extract high-level information from images, although the HR video frames down-sampled by bicubic are different from the results produced from G 1, they are similar to some extent, the low-level information is different, but the high-level information is the same. Training GANs can generate irrelevant content. To mitigate, pixel loss is introduced to make the training more stable.

[0046] HR optical flow estimation network (HFENet)

[0047] After LFGNet 101 produces many pairs of HR video frames of the training HR video sequence 111 and many pairs of LR video frames of the intermediate LR video sequence 112, the LR video frames of the intermediate LR video sequence 112 are used to train HFENet 102 to generate output HR video frames of the reconstructed HR video sequence 113. HFENet 102 adopts an HR frame recurrent architecture to improve the temporal consistency of the output sequence. Instead of generating each HR video frame of the reconstructed HR video sequence 113 independently, the recurrent architecture of HFENet 102 utilizes the high-frequency details of the last generated HR video frame . The generation of the HR video frames of the reconstructed HR video sequence 113 can be represented as:

[0048] ;

[0049] Although it can also be obtained directly by fusing and , in this case the high-frequency details in are not fully utilized. In this way, HFENet 102 is configured to estimate the HR optical flow to warp , preserving its high-frequency details, thereby improving temporal consistency. In addition, sometimes the pixel values of the same feature in different video frames of a video sequence can change, from residual frame that restores high frequency details, and an occlusion mask matrix that includes the residual frame and the deformation with the fusion weights to generate .

[0050] With reference to FIG. 2 , for detailed information about the logical architecture of the VistGAN 100, see FIG. 2 , the HFENet 102 is shown on the right.

[0051] In one embodiment, the HFENet 102 includes a coarse optical flow estimator that includes a FlowNet and an up-scaler. The coarse optical flow estimator is configured to estimate, by the FlowNet, a LR optical flow between a currently processed LR video frame of the intermediate LR video sequence 112 and a last processed LR video frame of the intermediate LR video sequence 112; and then up-scale, by the up-scaler, the LR optical flow to obtain a coarse HR optical flow This operation can be represented by the following equation:

[0052] .

[0053] The HFENet 102 also includes a fine optical flow estimator that includes a first warper, a space-to-depth mapper denoted by StoD, and a generator denoted by Gen G 2 for generating a fine HR optical flow, an occlusion mask matrix, and a residual frame. The generator G 2 is a neural network, which can be a GAN. The fine optical flow estimator is configured to first warp, by the warper, a last generated HR video frame of the reconstructed HR video sequence 113 and the coarse HR optical flow to produce a first warped HR video frame ; then map, by the StoD, the first warped HR video frame to a first warped LR video frame ; and finally, obtain, by the generator G 2, from the first warped LR video frame and the currently processed LR video frame of the intermediate LR video sequence 112, the fine HR optical flow , the occlusion mask matrix having values between 0 and 1 M t , and the residual frame R t The operation of the fine optical flow estimator can be represented by the following equation:

[0054] .

[0055] Finally, HFENet 102 also includes an HR frame synthesizer, which includes a second deformer and a mask fusion unit. The desired result is the reconstructed HR video sequence 113 based on the previously generated HR video frames. Details are preserved, but the intermediate LR video sequence 112 currently being processed is the LR video frame. There may be new details. Furthermore, since scene changes do occur in the video, filtering is necessary in these cases. High-frequency details in [the image]. Thus, by using the occlusion mask matrix... M t Fusion from Details and from New details to synthesize the reconstructed HR video sequence 113 HR video frames The HR frame synthesizer is configured to achieve fine HR optical flow via a second deformer. The deformed HR video frame of the reconstructed HR video sequence 113 was generated in the previous generation. The first deformed HR video frame is deformed to produce the second deformed HR video frame, and then processed by a mask fusion unit according to the occlusion mask matrix. M t Fusion of the second deformed HR video frames and residual frames R t To synthesize and reconstruct HR video frames from HR video sequence 113 The operation of the HR frame synthesizer can be expressed by the following equation:

[0056] .

[0057] refer to FIG. 1B During the testing of VistGAN 100, only HFENet 102 was active. The input to the active HFENet 102 was the LR video frames of the test LR video sequence 115, not the LR video frames of the intermediate LR video sequence 112. During runtime, real LR video sequences were input into HFENet 102 to generate reconstructed HR video sequences.

[0058] Kernel estimation network (KENet)

[0059] The KENet 103 acts as a discriminator in the logical architecture of the VistGAN 100. The KENet 103 includes multiple convolutional layers and fully connected layers, and is configured to extract degradation features of the LR video frames of the intermediate LR video sequence 112 and the corresponding LR video frames of the test LR video sequence 115. After obtaining the degradation features, a metric learning method is employed to shrink (or cluster) the degradation features to reduce the Euclidean distance between the same degradation features and to enlarge the Euclidean distance between the different degradation features. The shrinking of the degradation features can be achieved by minimizing a contrastive loss which is expressed as:

[0060] ; and

[0061]

[0062] wherein margin is the expected Euclidean distance of the different degradation features, a and b are two degradation feature vectors, d is the Euclidean distance between a and b, and n is the comparison number. The same category only includes the LR video frames of the test LR video sequence 115. To avoid the KENet 103 learning the content information of the text video frames, the warped test LR video frames are obtained by warping the last processed LR video frames of the test LR video sequence 115 and the currently processed LR video frames of the test LR video sequence 115. The LR video frames of the test LR video sequence 115, the warped test LR video frames, and the LR video frames of the intermediate LR video sequence 112 are added to the training data to make the KENet 103 learn to distinguish them, and to the KENet 103, these input video frames are used as different categories, which have similar content but different degradation operations. Since the LFGNet 101 is prone to learn other noise information, by using the metric learning method, the shrunk degradation features can be fed back to the LFGNet 101, thereby adding to the training HR video sequence 111 to make the training more stable.

[0063] loss function

[0064] ​​In the LFGNet 101, to synthesize the real LR video frames of the intermediate LR video sequence 112 corresponding to the input HR video frames of the training HR video sequence 111, a GAN loss is introduced to imitate the LR video frames of the test LR video sequence 115 by reducing the Euclidean distance between the degraded features in the LR video frames of the intermediate LR video sequence 112 and the degraded features in the LR video frames of the test LR video sequence 115, and a content loss is introduced to constrain the relationship of the HR / LR video frame pairs of the training HR video sequence 111 and the intermediate LR video sequence 112. The purpose of introducing these two losses is to make the intermediate LR video sequence 112 have the same content as the input training HR video sequence 111, but have the same degradation operation as the test LR video sequence 115. In addition, a cycle loss is introduced to make the adversarial training of the LFGNet 101 more stable and prevent the training process from deviating from the down-sampling and up-scaling operations. The cycle loss is defined as:

[0065] ;

[0066] wherein is the LR video frame of the test LR video sequence 115; and is the HR video frame of the reconstructed HR video sequence 111 output by the HFENet 102 according to the test LR video sequence 115.

[0067] Total loss in the LFGNet 101 can be expressed as:

[0068] .

[0069] During the reconstruction of the HR video frames, mean square error (MSE) is frequently used to obtain high PSNR. This can be achieved by introducing the L2 loss into the HFENet 102, which is given by the following equation:

[0070] .

[0071] During the coarse optical flow estimation, since the ground truth of the optical flow cannot be obtained, a deformation loss is introduced to supervise the coarse optical flow estimation network, and it is given by the following equation:

[0072] .

[0073] During the fine optical flow estimation, it is desired to make the estimated optical flow close to the optical flow between the input HR video frames, which is used as another learning target to enhance the reconstruction quality. The deformation loss introduced in the fine optical flow estimation is given by the following equation:

[0074] .

[0075] MSE loss is beneficial for high PSNR, and deformation loss can help to ensure temporal consistency, which also preserves high frequency details from previous HR frames and helps to improve PSNR. The total loss in HFENet 102 can be expressed as:

[0076] .

[0077] Thus, the total loss of VistGAN 100 is:

[0078] .

[0079] The function of KENet 103 is to extract and cluster the same degradation features together. The purpose is to reduce the Euclidean distance between the same degradation features and expand the Euclidean distance between different degradation features. Thus, the loss in KENet 103 is:

[0080] .

[0081] The embodiments disclosed herein can be implemented using computing devices, computer processors, or electronic circuitry, including but not limited to digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and other programmable logic devices configured or programmed according to the teachings of the present disclosure. Those of skill in the software and electronics arts will readily recognize that the teachings of the present disclosure can be used in a variety of well-known computer systems, and that the computer instructions or software code can be readily prepared by practitioners of ordinary skill in the software and electronics arts based on the teachings of the present disclosure.

[0082] In some embodiments, the present disclosure includes computer storage media having computer instructions or software code stored thereon, which can be used to program a computer or microprocessor to perform any of the processes of the present disclosure. The storage media can include, but is not limited to, ROM, RAM, flash memory devices, or any type of media or device suitable for storing instructions, code, and / or data.

[0083] The foregoing description of the present disclosure has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the present disclosure to the precise form disclosed. Many modifications and variations will be apparent to practitioners skilled in the art.

[0084] The embodiments are chosen and described in order to best explain the principles of the application and its practical application to thereby enable others skilled in the art to best utilize the application in various embodiments and with various modifications as are suited to the particular use contemplated. It is intended that the scope of the application be defined by the following claims and their equivalents.

Claims

1. A device for video super-resolution (VSR) using a generative adversarial network (GAN), comprising: a set of low-resolution (LR) frame generation networks LFGNet configured to synthesize, during training of the device, low-resolution video frames of an intermediate low-resolution video sequence from input high-resolution video frames of a training high-resolution (HR) video sequence; a set of high-resolution frame estimation networks HFENet configured to generate, from a currently processing low-resolution video frame and a last processed low-resolution video frame of an input low-resolution video sequence and a last generated output high-resolution video frame of a reconstructed high-resolution video sequence, an output high-resolution video frame sequence of the reconstructed high-resolution video sequence, wherein the high-resolution frame estimation networks are trained during training of the device using the intermediate low-resolution video sequence as the input low-resolution video sequence and the currently processing low-resolution video frame and the last processed low-resolution video frame are video frames of the intermediate low-resolution video sequence; and a set of kernel estimation networks KENet configured, during training of the device, to: extract degradation features of the currently processing low-resolution video frame and a low-resolution video frame of a test low-resolution video sequence corresponding to the currently processing low-resolution video frame; shrink the extracted degradation features to reduce the Euclidean distance between identical degradation features and to enlarge the Euclidean distance between different degradation features; feed back the shrunk degradation features to the low-resolution frame generation networks, added to the training high-resolution video sequence; and determine whether the degradation of the intermediate low-resolution video sequence is identical to the degradation of the test low-resolution video sequence; wherein the high-resolution frame estimation networks comprise: a coarse optical flow estimator configured to estimate a low-resolution optical flow between the currently processing low-resolution video frame and the last processed low-resolution video frame and to upscale the estimated low-resolution optical flow to obtain a coarse high-resolution optical flow; a fine optical flow estimator configured to: deform the last generated output high-resolution video frame of the reconstructed high-resolution video sequence and the coarse high-resolution optical flow to produce a first deformed high-resolution video frame; perform a spatial-to-depth mapping of the first deformed high-resolution video frame to produce a first deformed low-resolution video frame; and generate, from the first deformed low-resolution video frame and the currently processing low-resolution video frame, a fine high-resolution optical flow, an occlusion mask matrix, and a residual frame; and a high-resolution frame synthesizer configured to: deform the fine high-resolution optical flow and the first deformed high-resolution video frame to produce a second deformed high-resolution video frame; and synthesize the output high-resolution video frame of the reconstructed high-resolution video sequence by fusing the second deformed high-resolution video frame and the residual frame according to the occlusion mask matrix. ​ ​ wherein the occlusion mask matrix includes one or more blending weights for blending the second warped high-resolution video frame and the residual frame; and wherein the residual frame includes high-frequency details from the current low-resolution video frame being processed.

2. The device of claim 1, wherein synthesizing, by the low-resolution frame generation network, the output low-resolution video frame of the intermediate low-resolution video sequence from the input high-resolution video frame of the training high-resolution video sequence during training comprises: down-sampling the high-resolution video frame of the training high-resolution video sequence to generate a down-sampled high-resolution video frame of the training high-resolution video sequence; and minimizing content loss in the synthesis of the output low-resolution video frame based on the down-sampled high-resolution video frame, wherein the content loss includes a pixel loss and a visual geometry group algorithm (VGG) loss.

3. The device of claim 1, wherein the kernel estimation network is further configured to perform a metric learning method in warping the extracted degradation features; wherein the metric learning method includes computing and minimizing a contrastive loss of the extracted degradation features.

4. A method of training a device for video super-resolution (VSR) using a generative adversarial network (GAN), comprising: wherein the device comprises: a set of low-resolution (LR) frame generation networks LFGNet; a set of high-resolution (HR) frame estimation networks HFENet; and a set of kernel estimation networks KENet; wherein the method comprises: synthesizing, by the low-resolution frame generation network, a current low-resolution video frame of an intermediate low-resolution video sequence from an input high-resolution video frame of a training high-resolution video sequence; generating, by the high-resolution frame estimation network, an output high-resolution video frame of a reconstructed high-resolution video sequence from the current low-resolution video frame and a last processed low-resolution video frame of the intermediate low-resolution video sequence and a last generated output high-resolution video frame of the reconstructed high-resolution video sequence; extracting, by the kernel estimation network, degradation features of the current low-resolution video frame and a low-resolution video frame of a test low-resolution video sequence corresponding to the current low-resolution video frame; warping, by the kernel estimation network, the extracted degradation features to reduce the Euclidean distance between the same degradation features and to enlarge the Euclidean distance between different degradation features; feeding the warped degradation features back to the low-resolution frame generation network, added to the training high-resolution video sequence; and judging, by the kernel estimation network, whether the degradation of the intermediate low-resolution video sequence is the same as the degradation of the test low-resolution video sequence, wherein the high-resolution frame estimation network comprises: a coarse optical flow estimator configured to estimate a low resolution optical flow between the current low resolution video frame being processed and the last low resolution video frame processed, and upscaling the estimated low resolution optical flow to obtain a coarse high resolution optical flow; a fine optical flow estimator configured to: deform the last generated output high resolution video frame of the reconstructed high resolution video sequence and the coarse high resolution optical flow to produce a first deformed high resolution video frame; spatial-to-depth map the first deformed high resolution video frame to produce a first deformed low resolution video frame; and generate a fine high resolution optical flow, an occlusion mask matrix, and a residual frame from the first deformed low resolution video frame and the current low resolution video frame being processed; and a high resolution frame synthesizer configured to: deform the fine high resolution optical flow and the first deformed high resolution video frame to produce a second deformed high resolution video frame; and synthesize the output high resolution video frame of the reconstructed high resolution video sequence by fusing the second deformed high resolution video frame and the residual frame according to the occlusion mask matrix; wherein the occlusion mask matrix includes one or more fusion weights for fusing the second deformed high resolution video frame and the residual frame; and wherein the residual frame includes high frequency details from the current low resolution video frame being processed.

5. The method of claim 4, wherein synthesizing, by the low resolution frame generation network, the current low resolution video frame of the intermediate low resolution video sequence from the input high resolution video frame of the training high resolution video sequence includes: downsampling the high resolution video frames of the training high resolution video sequence to generate downsampled high resolution video frames of the training high resolution video sequence; and minimizing a content loss in the synthesizing of the current low resolution video frame based on the downsampled high resolution video frames, wherein the content loss includes a pixel loss and a visual geometry group algorithm (VGG) loss.

6. The method of claim 4, wherein the warping of the extracted degraded features includes performing a metric learning method that includes computing and minimizing a contrastive loss of the extracted degraded features.

Images