Multi-stage multi-reference bootstrapping for video super-resolution

Abstract

An embodiment method includes performing a first convolution filtering on a first tensor constructed using a current frame in a video and a reference frame (or a digital world reference image) of the current frame to generate a first estimated image of the current frame having a resolution higher than a resolution of an image of the current frame. The method also includes performing a second convolution filtering on a second tensor constructed using the first estimated image and an estimated reference image of the reference frame to generate a second estimated image of the current frame having a resolution higher than a resolution of the image of the current frame. The estimated reference image of the reference frame is a reconstructed high-resolution image of the reference image.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Provisional Application No. 62 / 861,884, filed on June 14, 2019, entitled “Multi-Stage Multi-Reference Bootstrapping for Video Super-Resolution,” and U.S. Provisional Application No. 62 / 843,135, filed on May 3, 2019, entitled “Multi-Stage Multi-Reference Bootstrapping for Video Super-Resolution,” which are incorporated herein by reference as if reproduced in their entirety. Technical Field

[0003] The present invention generally relates to video signal processing and, in particular embodiments, to techniques and mechanisms for multi-stage multi-reference bootstrapping for video super-resolution. Background Art

[0004] Improving the spatial resolution of visual data using a low-resolution version of the original visual data through non-optical means has been one of the goals of image enhancement for many years. Video super-resolution (VSR), a technique for generating high-resolution images of visual data, has become increasingly important as the use of video data has increased significantly. It is estimated that in 2016, video data accounted for about 73% of all Internet traffic and is expected to continue to increase in the future. For example, it is predicted that in 2021, video data may account for about 82% of Internet traffic, reaching a staggering 187 exabytes (187 billion gigabytes) of bandwidth in a few years. The demand may come from various types of Internet video, such as on-demand content, webcam viewing, traditional TV options available on the Internet, as well as live video delivered through social media, live sports games, video surveillance, and real-time over-the-top bundles from content providers. VSR can ensure that not only bandwidth is reduced for all these applications, but also latency is reduced. Summary of the invention

[0005] Technical advantages are generally achieved by embodiments of the present invention, which describe multi-stage multi-reference bootstrapping for video super-resolution.

[0006] According to one aspect of the present invention, a computer-implemented method is provided, the method comprising: obtaining a current frame in a video and multiple reference frames of the current frame; performing a first convolution filtering on a first tensor constructed using the current frame and the multiple reference frames to generate a first estimated image of the current frame, the resolution of the first estimated image being higher than the resolution of the image of the current frame; obtaining estimated reference images of the multiple reference frames, the resolution of the estimated reference image being higher than the resolution of the images of the multiple reference frames; performing a second convolution filtering on a second tensor constructed using the first estimated image of the current frame and the estimated reference images of the multiple reference frames to generate a second estimated image of the current frame.

[0007] The above method can enhance the image quality of the current frame and generate a time-consistent result by using multiple reference video frames. The method can also gradually improve the image resolution of the current frame by iteratively performing the steps of acquiring and performing convolution filtering.

[0008] Optionally, in any of the above aspects, the multiple reference frames of the current frame include previous frames of the current frame.

[0009] Optionally, in any of the above aspects, the multiple reference frames of the current frame include frames before the current frame and frames after the current frame.

[0010] Optionally, in any of the above aspects, the method also includes: before performing the first convolution filtering, determining an extended area in a reference frame among the multiple reference frames, the extended area corresponding to an area in the current frame, and the extended area in the reference frame includes an enlarged scene of the area in the current frame; assigning a utility score to each pixel of the reference frame according to whether each pixel of the reference frame belongs to the extended area, thereby generating a utility mask of the reference frame, the utility mask including a set of utility scores of pixels of the reference frame; and constructing the first tensor using the current frame, the multiple reference frames and the utility mask of the reference frame.

[0011] Optionally, in any of the above aspects, the method also includes: generating a scene flow using the current frame and the multiple reference frames, the scene flow including images of the multiple reference frames motion compensated according to the image of the current frame; generating a flow graph for each reference frame in the multiple reference frames; wherein determining the extended area in the reference frame is based on the scene flow and the flow graph.

[0012] Optionally, in any of the above aspects, generating the scene flow includes generating the scene flow using the current frame, the multiple reference frames and a digital world reference image of the current frame.

[0013] Optionally, in any of the above aspects, the digital world reference image is acquired from a digital world image database.

[0014] Optionally, in any of the above aspects, the method further comprises: acquiring visual positioning system (VPS) information of the current frame; and searching the digital world reference image in the digital world image database according to the VPS information.

[0015] Optionally, in any of the above aspects, the method further comprises: adjusting a size of the digital world reference image so that the digital world reference image has the same size as a size of the image of the current frame.

[0016] Optionally, in any of the above aspects, the method further includes: generating a utility mask for each reference frame in the multiple reference frames; and constructing the first tensor using the current frame, the multiple reference frames and the utility masks of the multiple reference frames.

[0017] Optionally, in any of the above aspects, constructing the first tensor includes: sorting the current frame and the multiple reference frames according to a sequence of the current frame and the multiple reference frames; and sorting the utility masks of the multiple reference frames according to the sequence.

[0018] Optionally, in any of the above aspects, constructing the first tensor includes: multiplying the pixel value of each reference frame in the multiple reference frames and the utility mask of the corresponding reference frame.

[0019] Optionally, in any of the above aspects, a size of the first estimated image or the second estimated image is larger than a size of the image of the current frame.

[0020] Optionally, in any of the above aspects, the method also includes: before performing the second convolution filtering, determining an extended area in the estimated reference image in the multiple reference frames, the extended area corresponding to an area in the first estimated image, and the extended area in the estimated reference image includes an enlarged scene of the area in the first estimated image; assigning a utility score to each pixel of the estimated reference image according to whether each pixel of the estimated reference image belongs to the extended area, thereby generating a utility mask of the estimated reference image, the utility mask of the estimated reference image including a set of utility scores of pixels of the estimated reference image; and constructing the second tensor using the first estimated image, the estimated reference images of the multiple reference frames, and the utility mask of the estimated reference image.

[0021] Optionally, in any of the above aspects, the method also includes: generating a scene flow using the first estimated image and the estimated reference images of the multiple reference frames, the scene flow including an image of the estimated reference image motion compensated according to the first estimated image; generating a flow graph for each of the estimated reference images; wherein determining the extended area in the estimated reference image is based on the scene flow and the flow graph.

[0022] Optionally, in any of the above aspects, the method further comprises: generating a utility mask for each of the estimated reference images.

[0023] Optionally, in any of the above aspects, the method further includes: constructing the second tensor using the first estimated image, the estimated reference images of the multiple reference frames, and a utility mask of the estimated reference image.

[0024] Optionally, in any of the above aspects, the method also includes: performing convolution filtering on the estimated first image and the estimated reference image of the multiple reference frames to generate feature maps of the estimated first image and the estimated reference image; wherein, constructing the second tensor includes: constructing the second tensor using the feature map, the utility mask of the estimated reference image and the first tensor.

[0025] According to another aspect of the present invention, a computer-implemented method is provided, the method comprising: acquiring a current frame in a video and multiple reference frames of the current frame; determining an extended area in a reference frame among the multiple reference frames, the extended area corresponding to an area in the current frame, and the extended area in the reference frame includes an enlarged scene of the area in the current frame; assigning a utility score to each pixel of the reference frame according to whether each pixel of the reference frame belongs to the extended area, thereby generating a utility mask of the reference frame, the utility mask including a set of utility scores of pixels of the reference frame; constructing a first tensor using the current frame, the multiple reference frames and the utility mask of the reference frame; and performing convolution filtering on the first tensor to generate a first reconstructed image of the current frame.

[0026] The above method can improve the image resolution of the current frame by using the reference frame of the current frame.

[0027] Optionally, in any of the above aspects, the method also includes: generating a scene flow using the current frame and the multiple reference frames, the scene flow including images of the multiple reference frames motion compensated according to the image of the current frame; generating a flow graph for each reference frame in the multiple reference frames; wherein determining the extended area in the reference frame is based on the scene flow and the flow graph.

[0028] Optionally, in any of the above aspects, generating the scene flow includes generating the scene flow using the current frame, the multiple reference frames and a digital world reference image of the current frame.

[0029] Optionally, in any of the above aspects, the digital world reference image is acquired from a digital world image database.

[0030] Optionally, in any of the above aspects, the method further comprises: acquiring visual positioning system (VPS) information of the current frame; and searching the digital world reference image in the digital world image database according to the VPS information.

[0031] Optionally, in any of the above aspects, the method further comprises: adjusting a size of the digital world reference image so that the digital world reference image has the same size as a size of the image of the current frame.

[0032] Optionally, in any of the above aspects, the method further includes: generating a utility mask for each reference frame in the multiple reference frames; and constructing the first tensor using the current frame, the multiple reference frames and the utility masks of the multiple reference frames.

[0033] Optionally, in any of the above aspects, constructing the first tensor includes: sorting the current frame and the multiple reference frames according to a sequence of the current frame and the multiple reference frames; and sorting the utility masks of the multiple reference frames according to the sequence.

[0034] Optionally, in any of the above aspects, constructing the first tensor includes: multiplying the pixel value of each reference frame in the multiple reference frames and the utility mask of the corresponding reference frame.

[0035] According to another aspect of the present invention, a device is provided, comprising: a non-volatile memory including instructions; and one or more processors communicating with the memory, wherein the one or more processors execute the instructions to perform the method described in any one of the above aspects.

[0036] According to another aspect of the present invention, a non-transitory computer-readable medium is provided, wherein the non-transitory computer-readable medium stores computer instructions, and when the computer instructions are executed by one or more processors, the one or more processors execute the method described in any one of the above aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] For a more complete understanding of the present invention and its advantages, reference is now made to the following description taken in conjunction with the accompanying drawings.

[0038] Figure 1 A diagram illustrating an embodiment architecture of a neural network;

[0039] Figure 2 A diagram showing an embodiment operation of an input subnet in stage 1 and an input subnet in stage 2;

[0040] Figure 3 A diagram showing an embodiment hybrid backbone subnet and an embodiment spatial upsampling subnet;

[0041] Figure 4 A diagram illustrating the operation of an embodiment for generating a utility mask for a video frame;

[0042] Figure 5 A diagram illustrating an embodiment operation for determining an expansion region and calculating a utility mask for a video frame;

[0043] Figure 6 A flow chart illustrating an embodiment method for generating a high resolution image of a low resolution (LR) frame;

[0044] Figure 7 A flow chart showing another embodiment method for generating a high resolution image of an LR frame;

[0045] Figure 8 is a block diagram of an embodiment processing system that may be used to implement the methods disclosed herein;

[0046] Fig. 9 A block diagram of another embodiment processing system that can be used to implement the methods disclosed herein is shown.

[0047] Corresponding numerals and symbols in the different figures generally refer to corresponding parts unless otherwise indicated.The figures are drawn to clearly illustrate the relevant aspects of the embodiments and are not necessarily drawn to scale. DETAILED DESCRIPTION

[0048] The making and using of embodiments of the present invention are discussed in detail below. However, it should be understood that the concepts disclosed herein can be embodied in a variety of specific contexts, and the specific embodiments discussed herein are merely illustrative and are not intended to limit the scope of the claims. Furthermore, it should be understood that various changes, substitutions, and modifications may be made herein without departing from the spirit and scope of the present invention as defined by the appended claims.

[0049] Embodiments of the present invention provide methods for reconstructing or estimating a high resolution (HR) image of a video frame (current frame) from a low resolution (LR) image of a video frame. The resolution of the HR image is higher than that of the LR image. These embodiments utilize a reference frame of the current frame, or a digital world reference image of the current frame having a similar image (or content or scene) of the current frame, in order to obtain information that can be used to reconstruct the HR image. The reference frame is an adjacent frame of the current frame (including directly adjacent and adjacent enough to have a substantially similar image) and has an image resolution similar to that of the LR image of the current frame. However, the HR version of the reference frame (HR image or SR image) can be pre-generated, for example, using an embodiment method, and is available for use. These embodiments iteratively use LR images and reference frames to estimate the HR image of the current frame in multiple stages. In some embodiments, the initial stage (stage 1) uses LR images and reference frames (digital world reference images can also be used) to construct a tensor, which is then processed to estimate the stage 1 HR image. Each stage after stage 1 uses the HR image generated in the previous stage (e.g., stage 1 HR image) and the HR image of the reference frame to generate another HR image in each stage, thereby gradually enhancing the quality of the generated HR images. This is described in detail below.

[0050] For many years, improving the spatial resolution of visual data using the original low-resolution version of the visual data through non-optical means has been one of the goals of image enhancement. Super-resolution (SR) is an imaging technique that transforms a low-resolution (LR) image into a high-resolution image. Video super-resolution (VSR) extends the super-resolution task to video data. The VSR method is an adaptation of the image super-resolution technique. VSR aims to improve the spatial and temporal resolution of a given input video. VSR techniques have been shown to have the advantages of reduced bandwidth and reduced latency.

[0051] However, reconstructing a high-resolution (HR) version from a low-resolution (LR) input (for both images and videos) is often an ill-posed problem due to the loss of the original HR information. To address this problem, additional restrictions are imposed that rely mainly on data redundancy or multiple observations. Redundancy is usually enforced in the form of local correlations by imposing sparsity constraints or assuming constancy of various image statistics, such as multi-scale patch reproduction to recover lost high-frequency details. In cases where different observations of the same scene are available, the shared redundancy between the observations is used to regulate the problem to invert the downscaling process. Video data naturally has a high correlation between consecutive frames within the same shot, which can be exploited to reconstruct its HR version.

[0052] As the demand for video data increases, a new trend in super-resolution based on recent successes in convolutional neural networks has also revolutionized the field of VSR. Deep learning-based techniques have rapidly achieved better high-resolution estimates. Despite great progress, existing VSR schemes still suffer from several issues. For example, patch-based neural networks, especially those with shallower structures, lack global context and focus on local features. On the other hand, deeper networks with more parameters are harder to train because the choice of optimal hyperparameters such as learning rate becomes critical but more difficult to estimate during the initial training phase. Due to GPU memory limitations, VSR networks are usually trained in a patch-based manner. However, the patch size (even large 128×128 patches) may not support covering large object motions (e.g., the classic aperture problem) or learning beneficial aspects from a larger receptive field. Single image super-resolution frameworks that directly adapt to video usually require upscaling together at the end of the pipeline, which can lead to a learning bottleneck when the target upsampling factor is 4× or higher.

[0053] Past single-image resolution methods can be categorized based on the reference examples used. Internal reference based methods try to identify internal redundancy using patch recurrence to obtain essential information when upscaling the patch. Recurrence assumes that patches in a single image encode multi-scale relationships, which may enable inferring missing high-frequency content at a given scale from patches at coarser scales. Dictionary learning methods define the input image as a sparse linear combination of dictionary atoms coupled to a HR dictionary. Both internal and external reference examples can be used, as such dictionaries are learned in a data-driven manner using internal and external training pairs.

[0054] Deep learning based methods mainly build models based on external references while approximating the complex nonlinear functions inherent in the super-resolution task. One of the pioneering convolutional neural networks, the super-resolution convolutional neural network (SRCNN), consists of a simple three-layer model. Since then, neural networks have been achieving superior performance, usually leveraging deep structures and residual learning. These methods take the bicubic upscaled LR image as the input of their first convolutional layer, which leads to a high computational burden. To address this problem, one scheme keeps the spatial resolution of the feature maps in the network as in the LR image and obtains the HR image by simply reordering the multi-channel output feature maps. A similar scheme to improve efficiency has been proposed based on transposed convolutions. The performance is further improved by reusing the saved memory (which can be 40% of the original network) to use a larger model and deferring upsampling to the very end of the network, thereby removing the batch normalization layer. It is argued that batch normalization leads to loss of scale information and hinders the range flexibility of the network.

[0055] VSR methods extend the super-resolution task from images to videos by aggregating spatial information between consecutive frames while compensating for inter-frame motion, exposing and exploiting temporal correlations to access multiple references of the same scene. Regarding deep learning-based schemes, a main idea is to align and warp adjacent frames relative to the current frame before all images are fed to the super-resolution network. Therefore, it is very common for VSR to explicitly compute motion, for example using off-the-shelf optical flow algorithms. On the other hand, computing motion is usually expensive. Alternatively, a motion estimation layer can be integrated into the super-resolution as part of the network. Similarly, another scheme uses a sliding window approach and combines frame alignment and super-resolution steps. It is also possible to use a fairly shallow recurrent bidirectional network for joint motion compensation and super-resolution without explicit motion compensation. However, training an integrated, usually low-capacity motion estimation layer is challenging and may distort accuracy. Instead of compensating for motion, methods are used to operate on a batch of frames simultaneously to generate different high-resolution images and then compress the results into a single image. However, these methods are sensitive to the degree of motion in the video. Another notable VSR approach applies a perceptual loss in an adversarial recurrent network using optical flow in order to exploit temporal cues between frames and a temporal consistency loss term to enforce consistency across frames. Such networks defer upscaling to the very end of the generative network, which can be considered to make learning more difficult than stage-wise learning. Upsampling based on pixel reorganization has been introduced in one approach, where the network also has a dedicated optical flow subnet. However, block-based motion compensated training may not generalize and account for large motions. Subsequently, some small improvements were provided by switching from explicit motion compensation to 3D filters.

[0056] An embodiment of the present invention provides a method for generating a second image of a video frame according to a first image of a video frame, wherein the resolution of the second image is higher than the resolution of the first image. The second image is a reconstructed super-resolution image of the first image. Compared with the first image, the second image includes high-resolution (HR) or super-resolution (SR) details of the video frame. In the present invention, the second image with a higher resolution is generally referred to as an SR image of a video frame, and the first image with a lower resolution size is generally referred to as a low-resolution (LR) image of a video frame. The method may generally be referred to as a VSR process or a VSR method. The size of the second image may be the same as or greater than the size of the first image. The image resolution used herein is represented by the number of pixels per area unit (e.g., per square inch, per square millimeter), or any other unit used to represent the image resolution. The size of an image or frame is represented by the length and width of the image. For example, the size of the image may be 5cm*5cm, and its resolution is 8 pixels per square millimeter. In another example, the size of the image may be 5cm*5cm, and its resolution is 16 pixels per square millimeter. When two images have the same resolution, such as the same predefined resolution, their size can also be represented by the number of pixels of the image. For example, multiple images may have the same size of 64*64 pixels. The images in question may also have a predefined size, such as a unit size, such as 4 inches*4 inches. In this case, the resolution of each image can be represented by the number of pixels of each image. For example, the resolution of one image may be 800*600 (pixels) and the resolution of another image may be 1600*900.

[0057] The ratio of the SR image to the LR image is called the magnification factor R, where R can be a positive integer or a non-integer greater than 1. That is, the resolution ratio of the SR image to the LR image is R. The magnification factor can also be called R×. The resolution of the SR image can be expressed as R×. For example, the resolution can be 2×, 4×, 8×, 1.1×, 2.5×.

[0058] Specifically, these embodiments provide a neural network model (e.g., for 4× or higher video super-resolution) that is capable of generating sharp video frames with high-resolution details by utilizing motion compensated reference frames and reusing high-resolution versions of reference frames estimated in previous stages of a bootstrapped (in other words, recursive) resolution enhancement process. These embodiments do not make any assumptions about objects in the image and are therefore class-agnostic.

[0059] Intuitively, the super-resolution performance of a neural network improves when the number of low-resolution references available to build evidence for high-resolution details increases. Therefore, multiple motion-compensated reference frames of the current frame can be used. Optimizing a neural network to obtain temporally consistent results is usually not straightforward because information from previous super-resolved frames is not directly included in the current step. Temporal consistency refers to the absence of visual artifacts, flickering, texture changes, vibration patterns, and similar dynamic changes between consecutive frames of the reconstructed HR video. Temporally consistent video frames do not contain any artificial and distracting motion of background scenes and objects. In other words, the SR versions of two consecutive video frames have similar motion characteristics. To achieve temporally consistent results, a bootstrapped frame recursion method can be used, where the reconstructed high-resolution frame of the previous step can be scheduled into the network after its pixels are rearranged into multiple images of a size smaller than the size of the reconstructed high-resolution frame.

[0060] These embodiments are described generally below, followed by a detailed description.

[0061] According to one embodiment, a neural network model for generating an SR image of a video frame may include three components, namely, an input subnet for reorganizing and combining multiple motion compensated reference frames, a hybrid backbone subnet for applying convolution blocks to LR feature maps, and a spatial upsampling subnet for reconstructing a high-resolution image. The neural network may also be referred to as a convolutional neural network. The neural network model is used to implement or perform a VSR process for reconstructing an SR image from an LR image.

[0062] The hybrid backbone subnet may include convolutional residual units or blocks. After a long chain of residual units (layers), direct skip connections may be embedded from the first feature layer to the last feature layer to preserve the influence of the original reference frame on the feature map of the last layer. Therefore, the hybrid backbone subnet is conditioned to reconstruct residual information that includes high-resolution patterns that are missing in the visual data. Residual blocks and direct skip connections also enable deepening the hybrid backbone subnet, thereby improving the overall representation capacity of the network and increasing the receptive field area of ​​higher convolutional layers, thereby achieving better contextual feedback. The representation capacity should be proportional to the number of network parameters. A neural network consists of multiple layers of neurons and filters connected to each other. Each neuron applies a combination of linear or nonlinear operations. These operations are defined in terms of weights, bias terms, and function parameters. These weights, bias terms, and function parameters are also called network parameters. The number of network parameters is the count of these parameters of the entire neural network. The hybrid backbone subnet utilizes different combinations of motion compensated reference frames, including reorganized and low-resolution mapped versions of the estimated high-resolution output of the current image from the previous stage. This can transfer the initial model to a progressively more complex network in subsequent stages.

[0063] After the hybrid backbone subnet, the spatial upsampling subnet is applied to reconstruct a high-resolution image from the feature maps generated by the hybrid backbone subnet. The spatial upsampling subnet uses pixel reorganization with learned weights, so deconvolution is not required. It can be considered that the hybrid backbone subnet prepares the best feature maps that are generally possible, which have a large number of channels, and the spatial upsampling subnet layers reorganize the feature maps into HR images using the learned weights of the filters of these layers.

[0064] The feature map in these embodiments is the output of the convolutional neural network layer. This output can be 3D. The convolutional neural network layer can include filters, whose output can be considered to represent the "features" of the input of the convolutional neural network layer. Both the input and the output can be tensors. Tensors can have any high-dimensional data structure. For example, a vector is a 1D tensor, a grayscale image is a 2D tensor, a matrix is ​​a 2D tensor, a color image is a 3D tensor (with 3 color channels), and a color video is a 4D tensor.

[0065] The VSR process can then be bootstrapped by spatially-to-depth rearranging the estimated HR image into multiple LR channels, updating the motion compensation, combining the estimated frame with the original frame, and applying a similar network again as the next stage. This frame-recursive approach can bootstrap the estimated high-resolution results and provide additional performance improvements.

[0066] An embodiment of the bootstrapped VSR scheme may include multiple stages. Each stage uses a neural network model and has an input subnet, a deep hybrid backbone subnet, and a spatial upsampling subnet, such as Figure 1 It should be noted that each stage is an independent network.

[0067] Figure 1A diagram of an embodiment architecture of a neural network 100 is shown. The neural network 100 can be used to generate an SR image of a current video frame from an LR image of the current video frame. The neural network 100 includes N stages for generating an SR image of the current video frame. Each stage uses the above-mentioned neural network model and generates an intermediate image / result of the current video frame. The intermediate image / result generated by the last stage can also be referred to as the final result of the neural network 100 because it is used as a reconstructed SR image of the current frame generated by the neural network 100. In this embodiment and other following embodiments, the input image of the current frame of the neural network 100 (i.e., the input image to the first stage) is referred to as an "LR" image. An SR image is reconstructed based on the LR image. In this embodiment and other following embodiments, the final result generated / output by the neural network 100 (i.e., the result generated by the last stage of the neural network 100) is referred to as an "SR image" (of the LR image / frame). The resolution of the SR image is higher than the resolution of the LR image, and the SR image is a reconstructed image of the input LR image. The intermediate image generated by each stage during the operation of the neural network 100 can be described as an "HR image" (of the LR image / frame). The resolution of the HR image is higher than that of the LR image and may be the same or lower than that of the SR image. Therefore, the image generated by the last stage can be called an HR image or an SR image. Figure 1 In the above example, the intermediate images generated by each stage are represented by SR t,i Denotes, where i denotes the i-th stage. The terms “LR”, “HR”, and “SR” are used herein to denote relative resolutions of images and should not be construed as being limited to any particular resolution.

[0068] The size of the LR image (e.g., size 1) may be smaller than the size of the SR image, or the same as the size of the SR image. The neural network 100 may be considered as a bootstrapped VSR scheme. The neural network 100 receives a plurality of video frames 102 (including the current video frame 101) as input, and generates an SR image of the current video frame based on the video frame 102. A video frame is generally an image in a video. The image source of the video frame is a video, and there may be a time associated with the video frame. The plurality of video frames include a current video frame 101 for which an SR image is to be generated and a plurality of reference frames 103 of the current video frame. The plurality of video frames 102 correspond to images having the same LR resolution and the same image size (e.g., size 1). The plurality of video frames 102 may also be referred to as LR frames / images. In the following description, for simplicity of explanation, "video frame" is also referred to as "frame". The terms "video frame" and "frame" are used interchangeably. The terms "frame" and "image" are used interchangeably in the present invention. The distinction between the terms "frame" and "image", and the distinction between the terms "video frame" and "frame" are provided when necessary.

[0069] As shown in the figure, the neural network 100 includes multiple stages, namely stage 1, stage 2...stage N. N can be an integer greater than 1. N can be a predetermined number, such as 3, 5 or 8. N can also be adjusted dynamically. For example, the neural network 100 can continue to run stage by stage (so N increases) until the target magnification factor is reached (that is, when the output image of the current stage of the operation has reached the target resolution determined for the LR current frame). In one example, each stage provides 2× magnification, and the target magnification factor is 16×, then four stages (N is 4) need to be run to achieve the target magnification. Multiple stages are then executed. Stage 1 can also be called the initial stage. Stage 1 takes multiple video frames 102 as input and generates an HR image of the current video frame 101 in stage 1, that is, the result SR t,1 118. The resolution of the HR image 118 may be higher than the resolution of the LR image 102. Each of the other stages (except the initial stage) takes as input the SR image of the reference frame 103 (e.g., previously estimated using the neural network 100) and the output of the previous stage (i.e., the estimated HR image of the current video frame, e.g., 118), and processes the input in stage 2 to generate another HR image of the current video frame 101, e.g., the resultant SR t,2 128. The input may also include a digital world reference image. The SR image of the reference frame 103 (including the SR image of each reference frame of the reference frame 103) may be generated in a manner similar to the SR image of the current video frame 101, as described below in the present invention. The neural network 100 continues to run to stage 3, stage 4, ..., until stage N. The output of the last stage (i.e., stage N) includes the final result SR t,N 104, which is the SR image of the current video frame 101. The SR image is a reconstructed high-resolution image of the current video frame 101. The final result SR t,N104 is referred to as the final result of the neural network 100 when reconstructing the current video frame. The final result can be stored in the memory 140 together with the SR image of the reference frame 103. When reconstructing the SR image of the next video frame of the current frame, these stored SR images can be used to estimate the SR image of the next video frame. Reconstructing the SR images of multiple video frames can be performed frame by frame. For example, the first frame of multiple video frames is taken as the current frame, and as described above, the neural network 100 reconstructs the SR image of the first frame as the current frame through N stages in the first cycle. Then in the second cycle (or the next cycle), the second frame of the multiple video frames (that is, the next frame of the first frame) is taken as the current frame, and as described above, the neural network 100 reconstructs the SR image of the second frame as the current frame through N stages. The remaining video frames of the multiple video frames are reconstructed similarly. The multiple video frames can be sequential frames of the video. When the current frame is the first frame of the video, that is, there is no frame before the current frame that can be used as a reference frame to reconstruct the current frame, any other available resource (such as a digital world reference image) can be used to provide an image as a reference frame for the current frame. For example, an image containing similar or key content, images, scenes of the current frame may be selected from, for example, digital world reference images and used as a reference frame for the current frame. The memory 140 may store the SR image estimated using the neural network 100. The memory 140 may also store other data, such as any results generated during the execution of the stage.

[0070] Each stage includes an input subnet, a hybrid backbone subnet, and a spatial upsampling subnet. As shown, stage 1 includes an input subnet 112, a hybrid backbone subnet 114, and a spatial upsampling subnet 116. Stage 2 includes an input subnet 122, a hybrid backbone subnet 124, and a spatial upsampling subnet 126. Stage N includes an input subnet 132, a hybrid backbone subnet 134, and a spatial upsampling subnet 136. Each stage can be an independent network.

[0071] The input subnet 112 includes an LR scene stream block 111 and an initial multi-reference block 113. The LR scene stream block 111 uses multiple video frames 102 to calculate the LR stream relative to the current video frame 101. The LR scene stream block 111 can also calculate the LR stream using a digital world reference image. The digital world reference image may include high-resolution information, which can be used to generate an HR image of the current image 101. Detailed information is provided below in the present invention. The initial multi-reference block 113 generates a tensor based on the LR stream. The operation of the input subnets of other stages except stage 1 is similar, and each stage includes a HR scene stream block and an intermediate multi-reference block. As shown in the figure, the input subnet 122 includes an HR scene stream block 121 and an intermediate multi-reference block 123. The input subnet 132 includes an HR scene stream block 131 and an intermediate multi-reference block 133. The HR scene stream block (e.g., HR scene stream block 121) calculates the HR stream relative to the current video frame using multiple reference frames 103 of the current video frame and the HR image of the current frame generated from the previous stage (e.g., stage 1). Specifically, the HR scene stream block calculates the HR stream using the SR image of the reference frame 103 and the HR image of the current frame 101. The intermediate multi-reference block (e.g., intermediate multi-reference block 123) generates a tensor based on the HR stream, the tensor generated from the previous stage (e.g., stage 1), the SR image of the reference frame 103, and the HR image generated from the previous stage (e.g., stage 1). Detailed information about the input subnet, the hybrid backbone subnet, and the spatial upsampling subnet is provided below in the present invention.

[0072] Each reference frame may be an RGB image, which is normalized to [-1, 1], for example, for efficient back propagation. The reference frame is determined based on the current frame to be reconstructed, and all stages for reconstructing the current frame use the same reference frame. The reference frame may include adjacent frames of the current frame. These adjacent frames may be continuous frames. The reference frame may be selected within a time window centered on the current frame. In one example, the time window may be set to have a size of [t–n, t+n], where t represents the current video frame and n is an integer greater than 0. According to the time window, n continuous frames before the current video frame and n continuous frames after the current video frame are selected as reference frames for the current video frame. In one example, n may be set to 2, i.e., the time window size is set to [t–2; t+2], as a trade-off between speed and accuracy in generating an SR image of the current video frame. In this example, four reference frames within the time window are selected, and a total of five frames including the current frame are used as inputs to the neural network 100. In another example, a time window [t–3; t+3] (i.e., n=3) may be used. In this case, the graphics processing unit (GPU) inference time for adding two reference frames (compared to n=2) is short (about 1 millisecond). However, one issue that may be of concern is the cost of the additional optical calculations, which may take about 400 milliseconds. In another example, the time window can be set to [t–k,t], where t represents the current video frame and k is an integer greater than 0. In this case, k consecutive frames before the current video frame are selected as reference frames for the current video frame. Figure 1 As shown, the current time window T1 ([t-k, t]) is used to select multiple frames 102. The previous time window T2 ([t-k-1, t-1]) can be used to select frame I t–1 The previous k frames are used to estimate frame I t–1 SR image. Figure 1 In the embodiment of the present invention, the time window [t-k, t] is used to select the reference frame of the current frame, but other time windows can also be used. For example, the time window [t-n, t+n] or [t-m, t+n] can be used, where n and m are integers greater than 0.

[0073] In some embodiments, a time window [t–m, t+n] (where m may be different from or equal to n) is used to determine a reference frame for the current frame, i.e., the reference frame for the current frame includes frames after the current frame. This may be the case where subsequent frames of the current frame may provide useful information for reconstructing an SR image for the current frame. In this case, the above-described stage-by-stage process for reconstructing an SR image for the current frame may be adjusted so that HR images or even SR images of subsequent frames of the current frame may be acquired for use. For example, the reference frames for reconstructing the current frame i include frames i–2, i–1, i+1, and i+2 (i.e., using a time window [t–2, t+2]). Frames i–2 and i–1 have been reconstructed before the current frame i, so their SR / HR images are already available. The current frame i and subsequent frames i+1 and i+2 will be reconstructed. In one embodiment, the first stage (stage 1) of the neural network 100 may be run for each of the current frame i and subsequent frames i+1 and i+2 to generate a first estimated HR image for each of the current frame i and subsequent frames i+1 and i+2, for example, respectively by SR i,1 , SR i+1,1 and SR i+2,1 However, in the first stage, the SR i,1 , SR i+1,1 and SR i+2,1 The time window is [t–k, t]. In this way, an HR image of each of the subsequent frames i+1 and i+2 is obtained, and the HR image can be used to reconstruct the current frame i using the time window [t–2, t+2] in one or more subsequent stages. For example, it can be compared with the above Figure 1 Similar to the description, the second stage is run to generate the second estimated HR image of the current frame i using the time window [t–2, t+2], i.e., SR i,2 That is, in the second stage, the SR images of frames i–2 and i–1 and the HR images SR of frames i+1 and i+2 are used. i+1,1 and SR i+2,1 To estimate SR i,2 . In another embodiment, stage 1 and stage 2 can be run to generate estimated HR images of the current frame i and subsequent frames i+1 and i+2 using the time window [t–k, t]. In stage 3, the estimated HR images of subsequent frames i+1 and i+2 can then be used to estimate the HR / SR image of the current frame using the time window [t–m, t+n]. Those of ordinary skill in the art will recognize that other variations of using the reference frame of the current frame according to the time window [t–m, t+n] may also be applied to embodiments of the present invention without departing from the principles of the present invention.

[0074] The plurality of video frames 102 may be uniformly sampled. The sampling frequency may be selected based on motion or frame difference. For example, when the motion is small, the sampling frequency may be reduced, and vice versa.

[0075] For example, each reference frame can be warped (i.e., motion compensated) relative to the current video frame using the inter-frame motion from each reference image / frame to the current frame. Motion compensation predicts a motion compensated (i.e., warped) version of a given frame in a video by taking into account the inter-frame motion of the given frame. The warped (motion compensated) reference frame can be calculated by assigning a value to each reference frame pixel that is offset by the inter-frame motion at the current pixel coordinate relative to the current pixel coordinate. The input subnet takes multiple motion compensated (i.e., warped) reference frames as input tensors and applies a convolutional filter layer to the input tensor. The dense optical flow of the input tensor can be estimated using another optical flow network model in the optical flow process. The optical flow network model relies on an arrangement of stacked optical flow networks that capture large displacements in coarse optical flow estimates, which can then be refined by subsequent optical flow networks. In the last step of the optical flow process, these multi-scale estimates are fused by a shallow optical flow fusion network. Dense optical flow can be calculated by other methods such as phase correlation, block matching, differential methods, and discrete optimization.

[0076] In an example where four reference frames 103 are selected for a current video frame 101 (the plurality of video frames 102 includes five frames, namely the current frame 101 and four previous frames 103, referred to as LR frames 102), the input network 112 may arrange the four warped images of the reference frames 103 and the current video frame 101 in a 15-channel tensor, and then apply 256 filters to the 15-channel tensor, each filter having a size of 3 × 3 × 15. The number of filters and the size of each filter may vary.

[0077] In stage 1, the HR image SR is estimated / reconstructed for the current frame 101 t,1 118. For the following stages, other references can be used. Taking stage 2 as an example, in stage 2, HR image SR is used t,1 118 and the SR / HR images of the reference frames 103 (including the SR images of each of the four reference frames 103) are taken as input, and the input includes five (5) HR images. In one example, the pixels of each image input in stage 2 can be rearranged (i.e., spatial to depth) into 16 LR images. Therefore, the five HR images are converted into LR images. Then, all the resulting LR images (16*5=80 LR images) from the 5 HR images can be combined into a 240-channel tensor. The 240-channel tensor can then be concatenated with the five LR frames 102 (i.e., the 15-channel tensor) to obtain a 255-channel tensor with the original LR spatial resolution. The 255 channels can then be used to reconstruct the SR t,1118. For 2× super-resolution (ie, the ratio of the resolution of the current frame 101 to the resolution of the SR image size to be generated for the current frame 101), the number of LR images after spatial-to-depth rearrangement will be four.

[0078] It should be noted that the embodiment neural network 100 calculates and processes images with the same LR resolution as the original frame (i.e., video frame 102) in all its subnets and stages. By using the same LR image resolution and RGB mode for all images processed in the neural network 100 (e.g., the original LR frame and the rearranged image of the estimated HR frame), learning becomes more efficient. Multiple references provide local patterns of spatial vibration for super-resolution tasks.

[0079] The reconstructed SR image of the LR frame can be applied to various purposes, as recognized by those skilled in the art. For example, the reconstructed SR image of the LR frame can be displayed on a screen for an enhanced view of the LR frame. In another example, the SR image of the LR frame can be reconstructed and transmitted to a receiving point. In yet another example, the SR image of the LR frame can be reconstructed to further process the LR frame, such as extracting features from the image of the LR frame, or performing image recognition. In yet another example, the reconstructed SR image of the LR frame can be used to process images of other frames.

[0080] Figure 2 It shows that Figure 1 FIG200 is a diagram of an embodiment of the operation of the input subnet 112 in stage 1 and the input subnet 122 in stage 2. Figure 1 and Figure 2 describe Figure 2 The input subnetwork in each stage after stage 2 has similar operations as stage 2. In this example, k+1 video frames 202, including the current frame I t and k reference frames I t–k ,I t–k+1 ……I t–1 Used as input to the neural network.

[0081] The LR scene stream block 111 obtains the input video frame 202 and uses the input video frame 202 to calculate the LR stream (also called the LR scene stream) relative to the current frame (block 212). The LR stream includes a flow graph for each reference frame and a warped video frame of the video frame 202. The LR stream can be calculated using an optical flow network that uses the LR version of the current video frame and the reference frame 202. The LR stream can also be calculated using other optical flow calculation methods. The LR scene stream block 111 can generate a 6D scene flow graph relative to the camera, 2D optical flow, or motion vector that captures the video frame. Then, the calculated LR flow graph is sent to the initial multi-reference block 113. The LR scene stream block 111 also generates warped video frames, which are aligned versions of the input video frame 202 generated using the LR flow graph. The output of the LR scene stream block 111 includes an LR flow graph and a warped video frame. Using the LR flow graph, each warped video frame is aligned with the current frame relative to the scene (or image) of the current frame. The LR stream includes a digitally warped image of the video frame 202.

[0082] In stage 1, the initial multi-reference block 113 obtains the LR stream and searches (or selects) the backward extension region in one or more reference frames (block 214). Generally, for a region in the current frame, a high backward extension region in the reference frame corresponding to a region in the current frame is more likely to contain more useful HR information (e.g., HR texture information) for the region in the current frame. The backward extension region in the reference frame of the current frame is a collection of image pixels depicting a scene and / or an object (or a portion of a scene or object). The scene or object can also be found in the current frame. However, the resolution of the scene or object in the reference frame is higher than the resolution of the scene or object in the current frame or the size is larger than the size of the scene or object in the current frame. In other words, the backward extension region in the reference frame of the current frame depicts an enlarged version of the scene (or region) in the current frame. That is, the region depicting the scene in the current frame is enlarged in the reference frame. Since the resolution of the backward extension region in the reference frame is higher than the resolution of the corresponding region in the current frame or the size is larger than the size of the corresponding region in the current frame, the backward extension region may include information that can be used to improve the resolution of the current frame, or at least improve the resolution of the corresponding region in the current frame.

[0083] The initial multi-reference block 113 may determine / identify a backward expansion region in the reference frame corresponding to each region of the current frame. The backward expansion region may then be used to reconstruct the HR image of the current frame. In one embodiment, the initial multi-reference block 113 may determine a backward expansion scale (or backward expansion level) of pixels in the reference frame. The backward expansion scale may be determined based on the degree to which the region in the current frame is enlarged in the backward expansion of the reference frame. The initial multi-reference block 113 then assigns a utility score determined based on the expansion scale to pixels in the video frame 202, thereby calculating and constructing a utility mask 217 (i.e., U ) for each video frame of the video frame 202 including the current frame. t–k , U t–k+1 ...U t )(block 216). A utility mask for a frame is constructed based on the utility scores assigned to the pixels of the frame. The utility mask for a frame includes a set of utility scores for the pixels of the frame. The utility mask for a video frame (comprising m*n pixels) may be an m*n matrix comprising m*n utility scores assigned to the corresponding m*n pixels of the frame. The utility mask for a reference frame of a current frame indicates which region(s) in a reference frame (e.g., a frame preceding the current frame in this example) is a larger version of a corresponding region in the current frame and therefore may be important for generating an SR image for the current frame (e.g., including detailed HR information for the SR image). The utility mask for the current frame may include a score indicating that the current frame does not include a backward extended region of the current frame. The utility score may be in the range of [0,1]. A utility score close to 0 indicates that the pixel contributes little to generating an SR image. A utility score close to 1 indicates that the pixel may include features (at different levels) that may be used to generate an SR image and therefore may contribute to generating an SR image. The utility score may also be in binary form, where each pixel value is either zero (0) or 1. Examples of determining the expansion region and constructing the utility mask are described below in the present disclosure.

[0084] Then, the input video frames 202 and the utility masks 217 are arranged to generate a tensor 220 (block 218). The tensor 220 may be referred to as a feature map. Each image or utility mask is a channel in the tensor 220. The input video frames 202 and the utility masks 217 may be arranged in various ways to generate the tensor 220. In one embodiment, the input video frames 202 and the utility masks 217 may be arranged in a pattern to form the tensor 220. For example, the input video frames 202 may be sorted in ascending or descending order according to the sequence of the input video frames 202, such as from I to V. t To I t–k The utility masks of the input video frames 202 may also be sorted in ascending or descending order according to the sequence of the input video frames 202, for example, from U t To U t–kThen, the input video frame 202 and the utility mask 217 are arranged so that they alternate with each other in sequence (as shown in A), i.e., t , U t ,I t–1 , U t–1 In another example, the input video frames 202 and the utility masks 217 may be sorted in ascending or descending order, respectively, according to the sequence, for example, from I t To I t–k and from U t To U t–k , and the input video frame 202 is placed before the utility mask 217 (as shown in B), that is, I t ……I t–k , U t ...U t–k ; Alternatively, the input video frame 202 is placed after the utility mask 217. In another example, the input video frame 202 is multiplied by the utility mask 217 (the pixel values ​​of the input video frame 202 are multiplied by the utility scores of their corresponding utility masks), and the multiplication results are used to form a tensor 220 (as shown in C). Then, the tensor 220 can be sent to the hybrid backbone subnet 114 of stage 1. Based on the tensor 220, stage 1 generates an estimated HR image 232 (e.g., SR image 232) of the current frame by performing convolution filtering on the tensor 220 (through the hybrid backbone subnet 114 and the spatial upsampling subnet 116). t,1 ), the resolution of the estimated HR image 232 is higher than the resolution of the image of the current frame. The tensor 220 can also be sent to the intermediate multi-reference block 123 to participate in the processing of stage 2.

[0085] In stage 2, the HR scene stream block 121 takes as input the HR estimation of the previous stage (i.e., the estimated HR image 232 generated in stage 1) and the SR image of the previously estimated reference frame, and uses this input to calculate the HR stream (also called HR scene stream) relative to the estimated HR image 232 of the current frame (block 234). In other words, the input of the HR scene stream block 121 in stage 2 is a high-resolution image of multiple video frames 202. In stage 2, the input of the HR scene stream block 121 may also include a retrieved and resized digital world reference image (block 262). Detailed information about the digital world reference image is provided below in the present invention. The SR image of the reference frame can be retrieved from the memory 260. The memory 260 is similar to Figure 1Memory 140 shown. Memory 260 may include SR images of video frames (e.g., frames before and / or after the current frame) that have been estimated using similar methods as described in the present invention. The calculation of the HR stream is similar to the calculation of the LR stream described with respect to block 212. The HR stream includes a digitally warped image of the SR image of the reference frame (obtained from memory 260) and the estimated HR image 232 of the current frame (estimated from the previous stage, i.e., stage 1), as well as a flow map for each SR image of the reference frame.

[0086] The intermediate multi-reference block 123 obtains the HR stream, searches for a backward extended region corresponding to a region of the estimated HR image 232 of the current frame in the SR image of the reference frame, and calculates another set of utility masks of the SR image of the reference frame and the estimated HR 232 according to the HR stream (block 236). For example, the intermediate multi-reference block 123 may determine one or more backward extended regions in one or more SR images of the reference frame according to the HR stream, and assign utility scores to pixels of the SR image of the reference frame and the estimated HR image 232 according to the determined one or more backward extended regions, thereby generating a utility mask for each of the SR image of the reference frame and the estimated HR image 232 according to the HR stream.

[0087] Each of the estimated HR image 232 and the SR image of the reference frame is a high-resolution image and can be displayed as an image 238 formed by rearranging the image pixels of its LR image. As shown, the image 238 includes a plurality of differently colored squares. Squares with the same shading represent pixels located at the same position in the LR image. The image 238 can be downsampled and passed to a convolutional layer (e.g., a full convolutional layer) 240, which generates feature maps (tensors) of the estimated HR image 232 and the SR image of the reference frame. The feature map can be referred to as the output of the convolutional neural network layer (i.e., the convolutional layer 240). The output can be 3D, corresponding to the input of the mapping. The input and output are tensors. The full convolutional layer consists of a plurality of convolutional filters applied to the input, which can be an image, a matrix, or a high-dimensional matrix (tensor). These filters can be three-dimensional (3D), 2D, or 1D. These filters can be separable. These full convolutional layers are used to transform the input (e.g., a tensor) into an output tensor by applying convolution filtering. They can transform the data space. The feature maps may be rearranged with respect to the magnification factor (block 244) to generate a plurality of LR images 246. The size of the LR images 246 is smaller than the size of the estimated HR image 232 and the SR image of the reference frame. The magnification factor is the ratio of the estimated HR image 232 to the current frame / image. For example, the magnification factor may be 2×, 4×, or 6×.

[0088] The LR image 246 and tensor 220 from stage 1 are then arranged together with the utility mask calculated in block 236 to generate tensor 250 (block 248). Tensor 250 may also be referred to as a feature map. Tensor 250 may then be sent to the hybrid backbone subnet 124 in stage 2 for further processing. For example, tensor 250 may be convolutionally filtered in stage 2 to generate an estimated HR image for the current frame. Tensor 250 may also be sent to the input subnet of the next stage (i.e., stage 3) to participate in the processing of stage 3.

[0089] For example, in stage 3, the estimated HR image of the current frame and the SR image of the reference frame in stage 2 are used to calculate another HR stream (similar to block 234), and the stage 3 utility mask of the estimated HR image of the current frame and the SR image of the reference frame in stage 2 is calculated based on the HR stream (similar to block 244). The estimated HR image of the current frame and the SR image of the reference frame in stage 2 can also be downsampled and passed to the convolution layer (similar to block 240), and then rearranged with respect to the magnification factor (similar to block 244), thereby generating multiple LR images in stage 3. These LR images, the stage 3 utility mask, and the tensor 250 generated in stage 2 are then used to form a stage 3 tensor. The stage 3 tensor can then be used to generate an estimated image of the current frame in stage 3 by performing convolution filtering on the stage 3 tensor, or passed to the hybrid backbone subnet and the spatial upsampling subnet in stage 3. The stage 3 tensor can also be passed to stage 4 to participate in the processing of stage 4, similar to the tensor 250 in stage 2 participating in the processing of stage 3.

[0090] Taking five LR video frames 202 (including one current frame and four reference frames of the current frame) as an example, in stage 1, the input subnetwork 112 arranges the five LR video frames 202 and their corresponding utility masks to generate a 15-channel tensor 220, wherein each LR video frame has three channels, and the five LR frames correspond to 15 channels, and generates an estimated HR image 232 by performing convolution filtering on the tensor 220, etc. In stage 2, the estimated HR image 232 and the SR images of the four reference frames are used as input frames (five HR frames). Each of the five input HR frames in stage 2 can be represented similarly to an image 238. In stage 2, each of the five input HR frames can be rearranged (i.e., spatial to depth) to generate 16 LR images for a 2× magnification factor, thereby generating 80 LR images 246 for the five input HR frames. Then, the 80 LR images 246 (forming a 240-channel tensor) and the 15-channel tensor 220 and the utility masks of the five input HR frames can be combined (e.g., concatenated) to form a 255-channel tensor 250. In this case, the utility masks of the five input HR frames can be arranged with the five input HR frames in the pattern shown as C with respect to block 218. The utility masks of the five input HR frames can also be arranged with the five input HR frames in other applicable patterns, in which case the number of tensor channels of the tensor 255 can be increased. Then, in stage 2, the hybrid backbone subnet 124 and the spatial upsampling subnet 126 use the tensor 250 to generate an HR image (e.g., SR 400) of the current frame in stage 2. t,2 ). Tensor 250 may also be sent to stage 3 to participate in the processing of tensors generated in stage 3 (in a manner similar to tensor 220 sent and processed in stage 2).

[0091] In addition to the reference frame, the LR scene stream block 111 can also take the resized digital world reference image of the current frame as input. These digital world reference images are processed by the LR scene stream block 111 of stage 1 or the HR scene stream block in the subsequent stage in the same manner as the reference frame. The digital world reference image can be retrieved from the digital world image database 208, for example, based on information obtained from the visual positioning system (visual positioning system, VPS) 204, wherein the information (hereinafter referred to as VPS information) can include geospatial position, attitude, angular velocity, linear velocity and position relative to the global reference coordinates, and inertial measurement unit (IMU) data. The inertial measurement unit is used to measure the force, angular velocity and magnetic field around the processor using a combination of accelerometers, gyroscopes and / or magnetometers. In one embodiment, the processing system (such as the processing system 800 or 900 shown below of the present invention) can be installed in a host device using a camera. In this case, VPS information can be obtained, which includes the position and direction of the camera that captures the video. VPS information can also be obtained or provided manually, or obtained or provided by another process that inputs the video. The digital world image database 208 may include images of streets, landmarks, outdoor and indoor environments, buildings, and all geospatial locations. Such images may be captured by users, for example, using digital cameras as well as other cameras (e.g., cameras used to capture street-level images, aerial and satellite images). These images may have associated geospatial tags, which may be in the form of 6-degrees-of-freedom (DoF) pose information, global positioning system (GPS) information, or map coordinates.

[0092] The retrieve best view block 206 can obtain the current frame and the VPS information of the current frame obtained from the VPS 204, and search the digital world image database 208 for an image similar to the current frame with similar VPS information. It can retrieve one or more matching images of the current image from the database 208. Rules can be defined to determine the matching images. For example, one or more overall best matching images can be retrieved. Then, the size of the matching image can be adjusted to the LR image size (block 210) and sent to the LR scene stream block 111 for calculating the LR stream. The matching image can also be sent to the HR scene stream block 121 for calculating the HR stream. In one example, the current frame shows an image of a tower in the background. The retrieve best view block 206 searches the database 208 for one or more images with similar geospatial tags (the images in the database have geospatial tags) and depicting the tower. It can return one or more images with the most similar pose (e.g., 6DoF pose) and imaging conditions as the current frame. The output of the retrieve best view block 206 can be one or more images of the tower.

[0093] Figure 3 300 is shown, which shows an embodiment hybrid backbone subnet 310 and an embodiment spatial upsampling subnet 340 in a neural network (e.g., neural network 100). Hybrid backbone subnet 310 can be used as Figure 1 The hybrid backbone subnetworks 114, 124 and 134 are shown. The spatial upsampling subnetwork 340 can be used as Figure 1 The spatial upsampling subnets 116, 126, and 136 are shown. In general, the hybrid backbone subnet 310 and the spatial upsampling subnet 340 perform convolution filtering on the tensor generated by the input subnet (e.g., the input subnet 112 or 122) and generate a reconstructed HR image of the current frame.

[0094] The hybrid backbone subnet applies a full convolution block (or unit) to the low-resolution feature map (i.e., tensor) generated by the input subnet. The hybrid backbone subnet 310 obtains the tensor (i.e., feature map) 302 generated by the input subnet and generates a hybrid backbone feature map based on the tensor 302. The tensor 302 and the hybrid backbone feature map each have the image size of the current frame for the spatial dimension. The input tensor 302 to the hybrid backbone subnet 310 includes a rearranged LR image and a utility mask having the image size of the current frame for the spatial dimension (e.g., the first two dimensions of the tensor), and the depth size (third dimension) of the tensor may vary according to the number of reference frames, the upsampling ratio, and / or the number of digital world reference images. The feature map generated by the hybrid backbone subnet 310 has the same spatial dimension as the input tensor 302; however, its third dimension may be different from the input tensor 302. The feature map of the hybrid backbone subnet includes enhanced information for the super-resolution task (for the current frame) after applying multiple residual units. The mixed backbone feature map is then provided to the spatial upsampling subnet 340 for further processing.

[0095] The hybrid backbone subnet 310 includes a convolutional layer 312 and multiple fully convolutional residual units, such as fully convolutional residual units 314 and 316 as shown. The residual unit can increase the number of layers in the hybrid backbone subnet 310, and efficiently train all subnets of the stage together. Multiple fully convolutional residual units are connected in series. The number of multiple fully convolutional residual units can vary, such as 16, 10, or 20. Each residual unit includes multiple layers, with direct skip connections from the input to the output of the residual unit, where the input tensor is added to the output tensor element by element. The residual unit can increase the depth of the hybrid backbone subnet, thereby improving the overall representation capacity of the neural network. The representation capacity is proportional to the number of network parameters. By increasing the depth of the hybrid backbone subnet, the receptive field area of ​​the continuous convolutional layer increases, so that a deeper hybrid backbone subnet can provide better context information for super-resolution tasks. Each convolutional layer in the hybrid backbone subnet 310 performs a convolutional filter (e.g., a 3D, 2D, or 1D convolutional filter) on an input tensor (e.g., tensor 302) and generates an output that is fed to a successive convolutional layer, such as a fully convolutional residual unit 314. The convolutional layer 312 takes the input feature map (i.e., tensor) 302, applies a convolutional filter, and generates an output feature map of the convolutional layer 312, wherein the convolutional filter can incorporate local information within the receptive field of the convolutional filter. The output feature map is enhanced with information and local context. The successive application of convolutional layers (e.g., residual units 314 and 316) can increase the local receptive field of subsequent layers projected back onto the initial feature map, thereby better incorporating local context within the final feature map. This can improve super-resolution results. Each fully convolutional residual unit has a front convolutional layer followed by a rectified linear unit (ReLU) and a second convolutional layer, with a skip connection from the input of the front convolutional layer to the output of the second convolutional layer. For example, the fully convolutional residual unit 314 has a front convolutional layer 318, followed by a ReLU 320 and a second convolutional layer 322, with a skip connection 324 from the input of the front convolutional layer 318 to the output of the second convolutional layer 322. The fully convolutional residual unit 316 has a front convolutional layer 326, followed by a ReLU 328 and a second convolutional layer 330, with a skip connection 332 from the input of the front convolutional layer 326 to the output of the second convolutional layer 330. Each fully convolutional residual unit works as a convolutional filter, performs a convolutional filter on its input and generates an output, which is the filtered result of the input. Specifically, each fully convolutional residual unit performs a nonlinear transformation on its input and feeds its output to the next fully convolutional residual unit. ReLU is a rectified linear unit, which can be an activation function. Mathematically, the activation function can be defined as the maximum value of 0 and the input of ReLU, that is, ReLU=max(0,x).ReLU can be replaced by nonlinear activation functions such as leaky ReLU, exponential linear unit (ELU), Swish (where the output is scaled by its inverse exponential plus one), hyperbolic tangent, parameterized ReLU, capped ReLU (where if the output of the ReLU is greater than the value of the parameter, it is set to a fixed value), and sigmoid.

[0096] The hybrid backbone subnet 310 also has a direct skip connection 334 from the input of the first fully convolutional residual unit (i.e., 314 in this example) to the output of the last fully convolutional residual unit (i.e., 316 in this example). The skip connection 334 supports the hybrid backbone subnet 310 to learn the missing high-resolution details by reconstructing the residual information. Figure 1 The structure of the hybrid backbone subnetwork at each stage in the neural network 100 is shown to be the same as the hybrid backbone subnetwork 310 .

[0097] Fully convolutional residual blocks or units (e.g., 314 and 316) and direct skip connections (e.g., 324, 332, and 334) can further deepen the hybrid backbone subnetwork at each stage. This improves the overall capacity of the neural network by supporting the neural network to be composed of deeper (i.e., multiple) layers (convolutional layers), thereby having more parameters, and increases the receptive field size of the convolutional filter. The receptive field is the effective area of ​​the convolutional filter projected onto the initial input feature map. For example, the 3×3 filter in the current (first) layer 318 of the fully convolutional residual block 314 has a receptive field of 3×3 on its input feature map (i.e., the output feature map of 312). After two layers of 3×3 filters (i.e., current layer 3×3, second layer 3×3), the 3×3 filter of the second layer 322 of the fully convolutional residual block 314 has a receptive field of 5×5 on the input feature map (i.e., the output feature map of 312). Therefore, the hybrid backbone feature map can better access the contextual information in the reference image.

[0098] The spatial upsampling subnet 340 is used to reconstruct a high-resolution image of the current frame from the feature map generated by the hybrid backbone subnet 310. Since the image pixels are reorganized and a set of convolution filters are applied, upsampling does not require a deconvolution operation. The spatial upsampling subnet 340 includes a plurality of convolution layers (e.g., convolution layers 342, 344) and an upsampling layer 346 connected in series. These convolution layers 342 and 344 are convolution layers, each of which includes a plurality of filters (e.g., 3D filters). These filters may also be 2D or 1D filters. These filters perform convolution filtering on the input of the spatial upsampling subnet 340. The last layer of the spatial upsampling subnet 340 (i.e., the upsampling layer 346) may include filters for generating images. For example, the upsampling layer 346 may have three filters for generating a 3-channel color image. The upsampling layer 346 uses filters to generate an enlarged image with a higher resolution, such as image 348. There can be different numbers of layers (including multiple convolutional layers and one upsampling layer) in the spatial upsampling subnets of different stages. For example, there can be 3 layers in the spatial upsampling subnet of the first stage (stage 1), and there can be 5 layers in the spatial upsampling subnet of subsequent stages.

[0099] The feature maps generated by the hybrid backbone subnet 310 may include a considerable number of feature map channels (i.e., tensor channels). For example, the number of tensor channels may be 256. The feature map channels are filtered by the convolutional layers 342, 344 (by performing convolution filtering) and then rearranged into a high-resolution image for each pixel by the last upsampling layer 346 using the weights of the filters of the upsampling layer 346. These weights and the weights of other subnets (e.g., the hybrid backbone subnet) may be learned separately during the training process. The number of convolutional layers 342, 344 of the spatial upsampling subnet 340 of the first stage (stage 1) and the second stage (stage 2) may be set to 3 and 5, respectively, because the second stage hybrid backbone feature map has more channels. A preset upscaling factor may be provided for each stage, such as 2× super-resolution, but when the feature map is deep enough, the upsampling factor may be set to a larger ratio.

[0100] Figure 4 A diagram 400 of an embodiment operation for generating a utility mask for a video frame is shown. Embodiment operations may be used in a neural network to generate a utility mask for an input video frame 102 or 202, such as Figure 1 and Figure 2 As shown. As shown, image 1 of the current frame 402 and image 2 of the reference frame (or digital world reference image) 404 of the current frame 402 are used to calculate the flow of the image (block 420). The flow can be called scene flow. The flow can be LR flow or HR flow. The flow can be calculated using an optical flow method involving solving differential equations. The flow can also be calculated by other methods, such as phase correlation, block matching, differential methods, or discrete optimization.

[0101] Flow indicates the amount of movement of a pixel in image 1 (402) in image 2 (404). In one embodiment, for a pixel at position p1 (defined in the xy coordinate plane) in image 1 (402), the corresponding position p2 of the pixel in the second image can be determined, and the difference between positions p1 and p2 (e.g., in the x and y directions) can be determined. The difference is a vector, which is called the motion or flow of the pixel. Flow can be calculated for all pixels (i.e., dense flow) or for a few representative pixels (e.g., blocks). In this example, the input to the flow calculation is two images, such as images 402 and 404. The output of the flow calculation includes a flow map 430 (also called a flow image) of the reference frame 404. The flow map 430 includes pixels, where each pixel has two numbers, one number corresponding to the movement in the x direction and the other number corresponding to the movement in the y direction. The purpose of the flow calculation (i.e., the flow map) is to warp the reference frame (and / or the digital world reference image) of the current frame onto the current frame. Therefore, the flow map also displays the warped (motion compensated) reference frame based on the current frame. Therefore, around each pixel in the current frame, there may be multiple reference pixels (in the reference frame or digital world reference image) showing the content of that pixel. If one LR image is missing some details, another LR image (e.g., the reference image or digital world reference image) may contain these details. Flow calculation may produce multiple versions of the same image (current image), thereby enabling the generation of a super-resolution image of the current image. That is, the output of the flow calculation includes an image of the reference frame that is motion compensated based on the current frame / image. The flow map is determined based on the motion compensated reference frame.

[0102] As shown, image 402 includes a human head image 406 and a car image 408, and image 404 includes a human head image 410 that is larger than 406, and a car image 412. Flow chart 430 shows the flow of human head image 406 and car image 408. The arrow on the head image position indicates that the head image in the reference frame 404 is larger than the head image in the current frame 402, and the car images in images 402 and 404 are the same size, but the arrow on the car position indicates that car 412 flows to the left relative to car 408.

[0103] In one embodiment, during the process of finding the backward extension area in the reference frame 404 (block 440), the flow map 430 is analyzed. For a given area in the flow map 430, the target area is calculated by translating the pixels according to the calculated flow of the pixels of the given area. If the target area is larger than the given area, the target area is the extension area. For example, the human head image 442 in the current frame 402 is the given area, and the human head image 444 in the reference frame 404 is the target area. Since the target area 444 is larger than the given area 442, the target area 444 is the extension area found. The car image in the current frame 402 is another given area, and the car image in the reference frame 404 is analyzed to determine whether the car image is the target area of ​​the given area. Since the areas 446 and 448 are the same size, even if the car 412 is located slightly to the left relative to the car 408, the area 448 is not the extension area. Other methods for determining the extension area in the current frame can also be used.

[0104] In the case where the backward extension region is determined, different scores are assigned to the pixels of the frame according to whether each pixel belongs to the backward extension region, thereby calculating (or constructing) the utility mask of the frames 402 and 404 (block 450). Figure 4 A utility mask 460 is shown for the reference image 404. The utility mask can be viewed as an image and placed in a tensor as a channel. The utility mask can also be multiplied with its corresponding reference frame or digital world reference image and placed in a tensor.

[0105] Figure 5 Another diagram 500 of an embodiment operation for determining an expansion region and calculating a utility mask for a video frame is shown. These operations may be performed at Figure 4is used in blocks 430 and 450 of . In this example, the flow map 502 is analyzed to determine the backward extension area in the reference frame of the current frame or the digital world reference image. A spatial area Ro(p) is assigned (or determined / selected) for each pixel p at coordinates (x, y) in the current frame (block 512). For example, the spatial area can be a 5×5 block centered on pixel p. The area size and shape may vary for different pixels. Using the flow map, an area Ru(p) (called an update area) corresponding to the area Ro(p) is generated (block 514). This can be done by adding the flow map value to the pixel coordinates in Ro(p) for each pixel in the area Ro(p) to find the update area Ru(p). For example, for pixel p=(100, 100), Ro(p) can be a 5×5 area with angular coordinates at (98, 98), (102, 98), (98, 102) and (102, 102). The flow map vector values ​​at these coordinates may be (-3, -3), (3, -3), (-3, 3), and (3, 3), respectively, where the first value in the flow map vector is the movement of the x-axis and the second value is the movement of the y-axis. The Ru(p) angular coordinates are then (95, 95), (105, 95), (95, 105), and (105, 105), which are obtained by adding the corresponding flow map vector values ​​to the angular coordinates of Ro(p). The Ru(p) angular coordinates define the update region Ru(p). The area of ​​Ro(p) is then calculated (block 516). For example, for a 5×5 region, the area is 25. The area of ​​the update region Ru(p) is also calculated (block 518). In this example, the area of ​​Ru(p) is 121 because it is an 11×11 region. The ratio of the area of ​​Ru(p) / Ro(p) is calculated (block 520), i.e., 121 / 25=4.84. This ratio can be used as an expansion scale for pixel p. The ratio 4.84 can be assigned as the utility score at the pixel (100, 100). In one embodiment, the ratio can be converted to a value in the range between 0 and 1. For example, an activation function 524 can be applied to each utility score of the utility mask (block 522). The activation function 524 helps to map the scores in the range [0, 1]. The activation function can be a logical function, i.e., 1 / (1+exp(–score)), where the score in the function is the ratio (or expansion scale) as described above. Taking the ratio 4.84 calculated above as an example, the result is 1 / (1+exp(–4.84))=0.992, which will be used as the final utility score of the pixel p at (100, 100) and assigned to the pixel p (block 526). The activation function can be a step function, a hyperbolic tangent function, an inverse tangent function, a sigmoid function, a threshold function, and a similarity function.In the case of using a logistic function, pixels with a ratio of Ru(p) / Po(p) greater than 1 (expanded) have a utility mask score close to 1, pixels with a ratio equal to 1 have a score of 0.5 (same size), and pixels with a ratio less than 1 (contracted) have a score close to 0. The utility scores may be thresholded to map them to 0 and 1. The utility scores of other pixels in the reference frame may be similarly calculated and assigned to generate a utility mask 504 for the reference frame.

[0106] The backward extended region may also be calculated by segmenting the current frame and the reference frame using a segmentation function, then finding corresponding segments in the two segmentation results, and analyzing the ratio of the areas of the corresponding segments to assign a utility mask score.

[0107] All subnetworks of a neural network (e.g., neural network 100) have layers consisting of filters, and the filters are represented by weights called network parameters. These parameters can be learned separately using a loss function in the training process. In order to improve the peak signal-to-noise ratio (PSNR), a simple loss function can be based on the mean squared error (MSE), which is the average of the square of the error of the mean squared error metric. For example, the loss function can calculate the average squared difference between the estimated value and the estimated value. However, MSE severely penalizes outliers. In the initial stage of training, using a loss function based on MSE can lead to instability. However, learning is slower using a loss function based on mean absolute error (MAE) in the later stage. Alternatively, the Huber loss function can be used, which is differentiable and combines the benefits of MAE and MSE. The function is defined as the pixel-by-pixel difference between the target and the estimate. The cutoff value is set to 1, the point where the Huber loss function changes from quadratic to linear.

[0108] The neural network described in the present invention (e.g., neural network 100) can be trained stage by stage. For example, the first stage network can be trained first, and then the second stage network can be trained by initializing with the first stage network parameters to accelerate the convergence speed of the loss function and the training process of the second stage network.

[0109] Although the above embodiment uses the convolutional layers in the hybrid backbone subnet and the spatial upsampling subnet to perform convolution filtering on the feature map to generate HR images and SR images, other methods can also be used in the neural network of the embodiment to achieve the same purpose. For example, a layer with connected neuron layers, a decision stump network, a spectral filtering network, or a pulse neural network can be used in one or more stages to generate HR images and SR images.

[0110] The utility mask is used to indicate the importance of information provided by each reference frame when reconstructing the HR image of the current frame. This can also be understood as indicating the level of correlation between each reference frame and the current frame. These embodiments determine the backward expansion area to construct the utility mask. However, other applicable methods may also be used without departing from the spirit of the embodiments.

[0111] In summary, these embodiments provide a general, class-agnostic fully convolutional network for video super-resolution that processes multiple reference frames in their original low-resolution format in its hybrid backbone subnetwork and then reconstructs a high-resolution output from the rearranged feature maps. These embodiments recursively apply the network to progressively bootstrap and enhance image quality with the reconstructed high-resolution output from the previous stages. The embodiment method generates temporally consistent results and handles complex real-world scenes depicting moving objects, fast camera motion, uncontrolled lighting conditions, and shadows without inducing perceptual artifacts.

[0112] The embodiment methods recursively utilize the reconstructed high-resolution output from the previous stage, so that the quality of the generated SR image can be bootstrapped and thus progressively enhanced. It is computationally efficient and memory-efficient because they process multiple reference frames of the original resolution (smaller size) in all its subnetworks. Processing at the original resolution can also increase the capacity of the convolution filtering process in terms of the number of learnable parameters. The embodiment methods generate temporally consistent results by using multiple reference video frames. By using digital world reference images as reference frames, it can achieve excellent performance. These embodiment methods do not require the input video frames to have a fixed size, so videos with video frames of different sizes can be processed. The upscaling factor of the embodiment method is adjustable, so the embodiment method can be used to generate super-resolution results for different upscaling factors.

[0113] Figure 6 A flow chart of an embodiment method 600 for generating a high resolution image of an LR frame is shown. Method 600 may be a computer-implemented method. As shown, method 600 includes obtaining a current frame in a video and multiple reference frames of the current frame (block 602). Method 600 includes performing a first convolution filter on a first tensor constructed using the current frame and multiple reference frames to generate a first estimated image of the current frame (block 604). The resolution of the first estimated image may be higher than the resolution of the image of the current frame. Method 600 also includes obtaining estimated reference images of multiple reference frames, wherein the resolution of the estimated reference image is higher than the resolution of the images of the multiple reference frames (block 606). Method 600 also includes performing a second convolution filter on a second tensor constructed using the first estimated image of the current frame and the estimated reference images of multiple reference frames to generate a second estimated image of the current frame (block 608). The resolution of the second estimated image may be higher than the resolution of the image of the current frame.

[0114] Figure 7 A flow chart of another embodiment method 700 for generating a high resolution image of an LR frame is shown. The method 700 may be a computer-implemented method. As shown, the method 700 includes obtaining a current frame in a video and a plurality of reference frames of the current frame (block 702). The method 700 also includes determining an extended region in one of the plurality of reference frames (block 704). The extended region corresponds to a region in the current frame, and the extended region in the reference frame includes an enlarged scene of the region in the current frame. The method 700 also includes assigning a utility score to each pixel of the reference frame according to whether each pixel of the reference frame belongs to the extended region, thereby generating a utility mask of the reference frame (block 706). The utility mask includes a set of utility scores of pixels of the reference frame. The method 700 also includes constructing a first tensor using the current frame, the plurality of reference frames, and the utility mask of the reference frame (block 708). The method 700 also includes performing convolution filtering on the first tensor to generate a first reconstructed image of the current frame (block 710). The resolution of the first estimated image may be higher than the resolution of the image of the current frame.

[0115] Figure 8 is a block diagram of a processing system that can be used to implement the methods disclosed herein. A particular device may use all of the components shown or only a subset of the components, and the degree of integration between devices may vary. In addition, a device may include multiple instances of a component, such as multiple processing units, processors, memories, transmitters, receivers, etc. A processing system may include a processing unit equipped with one or more input / output devices, such as a speaker, microphone, mouse, touch screen, keyboard, printer, display, etc. A processing unit may include a central processing unit (CPU), a memory, a mass storage device, a video adapter, and an I / O interface connected to a bus.

[0116] The bus may be one or more of any type of several bus architectures, including a memory bus or memory controller, a peripheral bus, a video bus, etc. The CPU may include any type of electronic data processor. The memory may include any type of non-transient system memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), read-only memory (ROM), or a combination thereof. In one embodiment, the memory may include a ROM used at boot time and a DRAM used to store programs and data when executing programs.

[0117] The mass storage device may include any type of non-transient storage device for storing data, programs, and other information and enabling the data, programs, and other information to be accessed via a bus. For example, the mass storage device may include one or more of a solid state drive, a hard disk drive, a magnetic disk drive, an optical disk drive, and the like.

[0118] The video adapter and I / O interface provide interfaces to couple external input and output devices to the processing unit. As shown, examples of input and output devices include a display coupled to the video adapter, and a mouse / keyboard / printer / button / key coupled to the I / O interface. Other devices can be coupled to the processing unit, and additional or less interface cards can be utilized. For example, a serial interface (not shown) such as a universal serial bus (USB) can be used to provide an interface for a printer.

[0119] The processing unit also includes one or more network interfaces, which may include wired links such as Ethernet cables, and / or wireless links for accessing nodes or different networks. The network interface supports the processing unit to communicate with the remote unit through a network. For example, the network interface can provide wireless communication through one or more transmitters / transmitting antennas and one or more receivers / receiving antennas. In one embodiment, the processing unit is coupled to a local area network or a wide area network to perform data processing and communicate with remote devices such as other processing units, the Internet, remote storage facilities, etc.

[0120] Fig. 9 A block diagram of an embodiment processing system 900 that can be installed in a host device for executing the methods described herein is shown. As shown, the processing system 900 includes a processor 904, a memory 906, and interfaces 910-914, which may (or may not) be as described in the example of FIG. Fig. 9 . The processor 904 may be any component or a collection of components for performing calculations and / or other processing-related tasks, and the memory 906 may be any component or a collection of components for storing programs and / or instructions executed by the processor 904. In one embodiment, the memory 906 includes a non-transitory computer-readable medium. The interfaces 910, 912, and 914 may be any component or a collection of components that support the processing system 900 to communicate with other devices / components and / or users. For example, one or more of the interfaces 910, 912, and 914 may be used to transmit data, control, or management messages from the processor 904 to applications installed in a host device and / or a remote device. As another example, one or more of the interfaces 910, 912, and 914 may be used to support a user or a user device (e.g., a personal computer (PC), etc.) to interact / communicate with the processing system 900. The processing system 900 may include Figure 6 Additional components not shown, such as long-term storage (e.g., non-volatile memory, etc.).

[0121] In some embodiments, processing system 900 is included in a network device that accesses a telecommunication network or otherwise becomes a part of a telecommunication network. In one example, processing system 900 is located in a network side device in a wireless or wired telecommunication network, such as a base station, a relay station, a scheduler, a controller, a gateway, a router, an application server, or any other device in a telecommunication network. In other embodiments, processing system 900 is located in a user side device that accesses a wireless or wired telecommunication network, such as a mobile station, a user equipment (UE), a personal computer (PC), a tablet computer, a wearable communication device (e.g., a smart watch, etc.), or any other device for accessing a telecommunication network. In some embodiments, one or more interfaces in interfaces 910, 912, 914 connect processing system 900 to a transceiver for sending and receiving signaling through a telecommunication network.

[0122] It should be understood that one or more steps of the embodiment method provided herein can be performed by corresponding units or modules. For example, a signal can be sent by a sending unit or a sending module. A signal can be received by a receiving unit or a receiving module. A signal can be processed by a processing unit or a processing module. Other steps can be performed by the following: a reorganization unit / module, an acquisition unit / module, a determination unit / module, an allocation unit / module, a construction unit / module, an execution unit / module, a scene flow calculation unit / module, a utility mask calculation unit / module, an extended area determination unit / module, a convolution filtering unit / module, a tensor construction unit / module, a generation unit / module, a reconstruction unit / module, a distortion unit / module, a sampling unit / module, a display unit, a combination unit / module, a cascade unit / module, an up-sampling unit / module, a rearrangement unit / module and / or a training unit / module. The corresponding unit / module can be hardware, software or a combination thereof. For example, one or more units / modules can be integrated circuits, such as field programmable gate arrays (FPGA) or application-specific integrated circuits (ASIC).

[0123] The following documents are included herewith as attachments and are incorporated herein by reference in their entirety:

[0124] Ratheesh Kalarot, Fatih Porikli, “MultiBoot VSR: Multi-Stage Multi-Reference Bootstrapping for Video Super-Resolution”.

[0125] Although described in detail, it should be understood that various changes, substitutions and modifications may be made without departing from the spirit and scope of the present invention as defined by the appended claims. In addition, the scope of the present invention is not intended to be limited to the specific embodiments described herein, and it is readily apparent to those of ordinary skill in the art from the present invention that processes, machines, manufacturing processes, material compositions, components, methods or steps (including those currently existing or to be developed later) can perform substantially the same functions as the corresponding embodiments described herein or achieve substantially the same effects as the corresponding embodiments described herein. Therefore, the appended claims are intended to include within their scope such processes, machines, manufacturing processes, material compositions, components, methods or steps.

Claims

1. A computer-implemented method, characterized in that: include: Acquire a current frame in a video and multiple reference frames of the current frame; Performing a first convolution filtering on a first tensor constructed using the current frame and the plurality of reference frames to generate a first estimated image of the current frame; Acquire an estimated reference image of the multiple reference frames, wherein a resolution of the estimated reference image is higher than a resolution of images of the multiple reference frames; Performing a second convolution filtering on a second tensor constructed using the first estimated image of the current frame and the estimated reference images of the plurality of reference frames to generate a second estimated image of the current frame; Before performing the first convolution filtering, determining an extended region in a reference frame among the plurality of reference frames, the extended region corresponding to a region in the current frame, and the extended region in the reference frame including an enlarged scene of the region in the current frame; assigning a utility score to each pixel of the reference frame according to whether each pixel of the reference frame belongs to the extended region, thereby generating a utility mask of the reference frame, wherein the utility mask includes a set of utility scores of pixels of the reference frame; The first tensor is constructed using the current frame, the plurality of reference frames, and the utility masks of the reference frames.

2. The computer-implemented method of claim 1, wherein: The plurality of reference frames of the current frame include previous frames of the current frame.

3. The computer-implemented method of claim 1 , wherein: The multiple reference frames of the current frame include a frame before the current frame and a frame after the current frame.

4. The computer-implemented method of claim 1, wherein: Also includes: generating a scene stream using the current frame and the plurality of reference frames, the scene stream comprising images of the plurality of reference frames motion compensated according to the image of the current frame; generating a flow graph for each reference frame in the plurality of reference frames; Wherein, determining the extended area in the reference frame is based on the scene flow and the flow graph.

5. The computer-implemented method of claim 4, wherein: Generating the scene stream includes generating the scene stream using the current frame, the plurality of reference frames, and a digital world reference image of the current frame.

6. The computer-implemented method of claim 5, wherein: The digital world reference image is obtained from a digital world image database.

7. The computer-implemented method of claim 6, wherein: Also includes: Obtaining visual positioning system information of the current frame; The digital world reference image is searched in the digital world image database according to the visual positioning system information.

8. The computer-implemented method of claim 5, wherein: Also includes: The digital world reference image is resized so that the digital world reference image has the same size as the image of the current frame.

9. The computer-implemented method of claim 1, wherein: Also includes: generating a utility mask for each reference frame in the plurality of reference frames; The first tensor is constructed using the current frame, the plurality of reference frames, and utility masks of the plurality of reference frames.

10. The computer-implemented method of claim 9, wherein: Constructing the first tensor includes: Sort the current frame and the multiple reference frames according to a sequence of the current frame and the multiple reference frames; The utility masks of the plurality of reference frames are ordered according to the sequence.

11. The computer-implemented method of claim 9, wherein: Constructing the first tensor includes: The pixel value of each reference frame of the plurality of reference frames is multiplied by a utility mask of the corresponding reference frame.

12. The computer-implemented method according to any one of claims 1 to 11, characterized in that: A size of the first estimated image or the second estimated image is larger than a size of the image of the current frame.

13. The computer-implemented method according to any one of claims 1 to 11, characterized in that: Also includes: Before performing the second convolution filtering, determining an extended region in an estimated reference image in the plurality of reference frames, the extended region corresponding to a region in the first estimated image, and the extended region in the estimated reference image including an enlarged scene of the region in the first estimated image; assigning a utility score to each pixel of the estimated reference image according to whether each pixel of the estimated reference image belongs to the extended area, thereby generating a utility mask of the estimated reference image, wherein the utility mask of the estimated reference image includes a set of utility scores of pixels of the estimated reference image; The second tensor is constructed using the first estimated image, the estimated reference images of the plurality of reference frames, and the utility mask of the estimated reference image.

14. The computer-implemented method of claim 13, wherein: Also includes: generating a scene stream using the first estimated image and the estimated reference images of the plurality of reference frames, the scene stream comprising images of the estimated reference images motion compensated according to the first estimated image; generating a flow map for each of the estimated reference images; Wherein, determining the extended area in the estimated reference image is based on the scene flow and the flow map.

15. The computer-implemented method of claim 13, wherein: Also includes: A utility mask is generated for each of the estimated reference images.

16. The computer-implemented method of claim 15, wherein: Also includes: The second tensor is constructed using the first estimated image, the estimated reference images of the plurality of reference frames, and a utility mask of the estimated reference image.

17. The computer-implemented method of claim 16, wherein: Also includes: Performing convolution filtering on the estimated first image and the estimated reference images of the multiple reference frames, thereby generating feature maps of the estimated first image and the estimated reference image; Wherein, constructing the second tensor includes: The second tensor is constructed using the feature map, the utility mask of the estimated reference image, and the first tensor.

18. A computer-implemented device, characterized in that: include: Non-transitory memory, including instructions; One or more processors in communication with the memory, wherein the one or more processors execute the instructions to perform operations including: Acquire a current frame in a video and multiple reference frames of the current frame; Performing a first convolution filtering on a first tensor constructed using the current frame and the plurality of reference frames to generate a first estimated image of the current frame; Acquire an estimated reference image of the multiple reference frames, wherein a resolution of the estimated reference image is higher than a resolution of images of the multiple reference frames; Performing a second convolution filtering on a second tensor constructed using the first estimated image of the current frame and the estimated reference images of the plurality of reference frames to generate a second estimated image of the current frame; Before performing the first convolution filtering, determining an extended region in a reference frame among the plurality of reference frames, the extended region corresponding to a region in the current frame, and the extended region in the reference frame including an enlarged scene of the region in the current frame; assigning a utility score to each pixel of the reference frame according to whether each pixel of the reference frame belongs to the extended region, thereby generating a utility mask of the reference frame, wherein the utility mask includes a set of utility scores of pixels of the reference frame; The first tensor is constructed using the current frame, the plurality of reference frames, and the utility masks of the reference frames.

19. The device according to claim 18, characterized in that The plurality of reference frames of the current frame include previous frames of the current frame.

20. The device according to claim 18, characterized in that The multiple reference frames of the current frame include a frame before the current frame and a frame after the current frame.

21. The device according to claim 18, characterized in that The operations also include: generating a scene stream using the current frame and the plurality of reference frames, the scene stream comprising images of the plurality of reference frames motion compensated according to the image of the current frame, the scene stream further comprising a flow graph of each of the plurality of reference frames; Wherein, determining the extended area in the reference frame is based on the scene flow and the flow graph.

22. The device according to claim 21, characterized in that Generating the scene stream includes generating the scene stream using the current frame, the plurality of reference frames, and a digital world reference image of the current frame.

23. The device according to claim 22, characterized in that The digital world reference image is obtained from a digital world image database.

24. The device according to claim 23, characterized in that The operations also include: Obtaining visual positioning system information of the current frame; The digital world reference image is searched in the digital world image database according to the visual positioning system information.

25. The device according to claim 22, characterized in that The operations also include: The digital world reference image is resized so that the digital world reference image has the same size as the image of the current frame.

26. The device according to claim 18, characterized in that The operations also include: generating a utility mask for each reference frame in the plurality of reference frames; The first tensor is constructed using the current frame, the plurality of reference frames, and utility masks of the plurality of reference frames.

27. The device according to claim 26, characterized in that Constructing the first tensor includes: Sort the current frame and the multiple reference frames according to a sequence of the current frame and the multiple reference frames; The utility masks of the plurality of reference frames are ordered according to the sequence.

28. The device according to claim 26, characterized in that Constructing the first tensor includes: The pixel value of each reference frame of the plurality of reference frames is multiplied by a utility mask of the corresponding reference frame.

29. The device according to any one of claims 18 to 28, characterized in that A size of the first estimated image or the second estimated image is larger than a size of the image of the current frame.

30. The device according to any one of claims 18 to 28, characterized in that The operations also include: Before performing the second convolution filtering, determining an extended region in an estimated reference image in the plurality of reference frames, the extended region corresponding to a region in the first estimated image, and the extended region in the estimated reference image including an enlarged scene of the region in the first estimated image; assigning a utility score to each pixel of the estimated reference image according to whether each pixel of the estimated reference image belongs to the extended area, thereby generating a utility mask of the estimated reference image, wherein the utility mask of the estimated reference image includes a set of utility scores of pixels of the estimated reference image; The second tensor is constructed using the first estimated image, the estimated reference images of the plurality of reference frames, and the utility mask of the estimated reference image.

31. The device according to claim 30, characterized in that The operations also include: generating a scene stream using the first estimated image and the estimated reference images of the plurality of reference frames, the scene stream comprising images of the estimated reference images motion compensated according to the first estimated image; generating a flow map for each of the estimated reference images; Wherein, determining the extended area in the estimated reference image is based on the scene flow and the flow map.

32. The device according to claim 30, characterized in that The operations also include: A utility mask is generated for each of the estimated reference images.

33. The device according to claim 30, characterized in that The operations also include: The second tensor is constructed using the first estimated image, the estimated reference images of the plurality of reference frames, and a utility mask of the estimated reference image.

34. The device according to claim 33, characterized in that Also includes: Performing convolution filtering on the estimated first image and the estimated reference images of the multiple reference frames, thereby generating feature maps of the estimated first image and the estimated reference image; Wherein, constructing the second tensor includes: The second tensor is constructed using the feature map, the utility mask of the estimated reference image, and the first tensor.

35. A non-transitory computer readable medium storing computer instructions, characterized in that: The computer instructions, when executed by one or more processors, cause the one or more processors to: Acquire a current frame in a video and multiple reference frames of the current frame; Performing a first convolution filtering on a first tensor constructed using the current frame and the plurality of reference frames to generate a first estimated image of the current frame; Acquire an estimated reference image of the multiple reference frames, wherein a resolution of the estimated reference image is higher than a resolution of images of the multiple reference frames; Performing a second convolution filtering on a second tensor constructed using the first estimated image of the current frame and the estimated reference images of the plurality of reference frames to generate a second estimated image of the current frame; Before performing the first convolution filtering, determining an extended region in a reference frame among the plurality of reference frames, the extended region corresponding to a region in the current frame, and the extended region in the reference frame including an enlarged scene of the region in the current frame; assigning a utility score to each pixel of the reference frame according to whether each pixel of the reference frame belongs to the extended region, thereby generating a utility mask of the reference frame, wherein the utility mask includes a set of utility scores of pixels of the reference frame; The first tensor is constructed using the current frame, the plurality of reference frames, and the utility masks of the reference frames.

Images