Video super-resolution method and apparatus

By decomposing video frames into blocks and applying tailored super-resolution models based on motion parameters, the method improves video super-resolution, particularly in videos with stationary elements, enhancing image quality.

JP2024545535AActive Publication Date: 2024-12-09DOUYIN VISION CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2024539756
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2022-03-17
Filing Date
2023-03-17
Publication Date
2024-12-09
Estimated Expiration
2043-03-17

AI Technical Summary

Technical Problem

Existing video super-resolution technologies struggle to achieve optimal results when dealing with videos containing stationary objects or backgrounds, as time-domain redundant information from neighboring frames can hinder the super-resolution effect.

Method used

A method and apparatus that decompose each image frame and its neighboring frames into image blocks, calculate motion parameters to determine appropriate super-resolution network models for each block sequence, and adaptively apply these models to enhance the super-resolution process.

Benefits of technology

Enhances the super-resolution effect by tailoring network models to specific motion parameters, improving the quality of video frames by addressing redundant information from stationary elements.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024545535000001_ABST
    Figure 2024545535000001_ABST
Patent Text Reader

Abstract

The embodiment of the present disclosure provides a video super-resolution method and apparatus, which relates to the field of image processing technology. The method includes: respectively decomposing a t-th image frame of a video to be super-resolved and a neighboring image frame of the t-th image frame into N image blocks; generating N image block sequences, each image block in the image block sequences being located at the same position in a different image frame; calculating motion parameters of each image block sequence, the motion parameters of the image block sequences including motion parameters between image blocks of each adjacent image frame in the image block sequences; determining a super-resolution network model corresponding to each image block sequence according to the motion parameters of each image block sequence; adopting the corresponding super-resolution network model to super-resolve an image block of the t-th image frame in each image block sequence to obtain each super-resolved image block of the t-th image frame; and generating a super-resolved image frame of the t-th image frame according to each super-resolved image block of the t-th image frame.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] [CROSS REFERENCE TO RELATED APPLICATIONS] This disclosure is based on and claims priority from Chinese application number 202210265124.7, filed on March 17, 2022, the disclosure of which is hereby incorporated by reference in its entirety into this disclosure.

[0002] [Technical field] The present disclosure relates to the technical field of image processing, and in particular to a method and apparatus for video super-resolution. [Background technology]

[0003] Video super-resolution technology, also known as video super-resolution technology, is a technology that restores high-resolution video from low-resolution video. Video super-resolution has become an important task in enhancing video quality, so video super-resolution technology is currently one of the research hotspots in the field of image processing.

[0004] In recent years, with the development of deep learning technology, video super-resolution network models based on deep learning neural networks have achieved many breakthroughs, including better super-resolution effect and better real-time performance. At present, most mainstream sliding window video super-resolution network models take advantage of the fact that most image frames in a video are in motion, and when super-resolving each image frame in a video, the video super-resolution network model can utilize the fact that the neighboring image frames can provide a large amount of time domain information for the current image frame to be super-resolved. Summary of the Invention [Means for solving the problem]

[0005] According to a first aspect, an embodiment of the present disclosure provides a video super-resolution method, the video super-resolution method comprising: Decomposing a t-th image frame of a video to be super-resolved and a neighboring image frame of the t-th image frame into N image blocks, respectively, where t and N are both positive integers; generating a sequence of N image blocks based on image blocks obtained by decomposing the t-th image frame and the neighboring image frames, where each image block in the sequence of N image blocks is located at a same position in a different image frame; calculating motion parameters for each image block sequence, the motion parameters of any one image block sequence being used to represent optical flow between image blocks of each adjacent image frame in the image block sequence; determining a super-resolution network model corresponding to each image block sequence based on the motion parameters of each image block sequence; adopting a super-resolution network model corresponding to each image block sequence to super-resolve an image block of the t-th image frame in each image block sequence, to obtain each super-resolved image block of the t-th image frame; and generating a super-resolved image frame of the t-th image frame based on each super-resolved image block of the t-th image frame.

[0006] According to a second aspect, an embodiment of the present disclosure provides a video super-resolution device, the video super-resolution device comprising: an image decomposition module for decomposing a t-th image frame of a video to be super-resolved and a neighboring image frame of the t-th image frame into N image blocks, respectively, where t and N are both positive integers; a sequence generation module for generating a sequence of N image blocks based on image blocks obtained by decomposing the t-th image frame and the neighboring image frames, where each image block in the image block sequence is located at the same position in a different image frame; a parameter calculation module for calculating a motion parameter for each image block sequence, the motion parameter for any one image block sequence being used to represent an optical flow between image blocks of adjacent image frames in the image block sequence; a model determination module for determining a super-resolution network model corresponding to each image block sequence based on the motion parameters of each image block sequence; an image super-resolution module for adopting a super-resolution network model corresponding to each image block sequence to super-resolve an image block of the t-th image frame in each image block sequence, to obtain each super-resolved image block of the t-th image frame; and an image generation module for generating a super-resolved image frame of the t-th image frame based on each super-resolved image block of the t-th image frame.

[0007] According to a third aspect, an embodiment of the present disclosure provides an electronic device, the electronic device comprising a memory and a processor, the memory being adapted to store a computer program, and the processor being adapted to cause the electronic device to implement a video super-resolution method as described in the first aspect or any one of the optional embodiments of the first aspect when invoking the computer program.

[0008] According to a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, the computer program, when executed by a computing device, causing the computing device to implement the video super-resolution method described in the first aspect or any one of the optional embodiments of the first aspect.

[0009] According to a fifth aspect, an embodiment of the present disclosure provides a computer program product, which, when running on a computer, causes the computer to implement the video super-resolution method described in the first aspect or any one of the optional embodiments of the first aspect. [Brief description of the drawings]

[0010] The drawings herein are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification serve to explain the principles of the present disclosure.

[0011] In order to more clearly explain the technical solutions in the embodiments or related technologies of the present disclosure, the following briefly introduces drawings that need to be used in the embodiments or related technology description, and obviously, those skilled in the art can also derive other drawings based on these drawings without exerting creative efforts. [Figure 1] 1 is a step flowchart of a video super-resolution method according to an embodiment of the present disclosure. [Diagram 2] FIG. 2 is a schematic diagram of image blocks obtained by decomposing an image frame according to an embodiment of the present disclosure. [Diagram 3] FIG. 2 is a schematic diagram of an image block sequence according to an embodiment of the present disclosure. [Figure 4] FIG. 2 is a schematic diagram of an image block scheme according to an embodiment of the present disclosure. [Diagram 5] FIG. 2 is a schematic diagram of a model for implementing a super-resolution method according to an embodiment of the present disclosure. [Figure 6] FIG. 2 is a schematic diagram of an adaptive super-resolution module according to an embodiment of the present disclosure. [Figure 7] FIG. 2 is a schematic diagram of a first super-resolution network model according to an embodiment of the present disclosure. [Figure 8] FIG. 2 is a schematic diagram of a second super-resolution network model according to an embodiment of the present disclosure. [Figure 9] FIG. 13 is a schematic diagram of a third super-resolution network model according to an embodiment of the present disclosure. [Figure 10]1 is a schematic diagram of a video super-resolution device according to an embodiment of the present disclosure; [Figure 11] FIG. 2 is a schematic diagram of a hardware structure of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] In order to make the above objects, features and advantages of the present disclosure more clearly understandable, the present disclosure will be further described below. It should be noted that, unless contradictory, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0013] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure; however, the present disclosure may be embodied in other ways not described herein, and it is apparent that the embodiments in the specification are merely some embodiments of the present disclosure, but not all embodiments.

[0014] It should be explained that, in order to easily and clearly describe the technical solutions of the embodiments of the present disclosure, the embodiments of the present disclosure employ the words "first", "second" and so on to distinguish identical or similar items having essentially the same functions or actions, and as can be understood by those skilled in the art, the words "first", "second" and so on do not limit the number and execution steps, for example, the first feature image set and the second feature image set are merely for distinguishing different feature image sets, and do not limit the order of the feature image sets, etc.

[0015] In the embodiments of the present disclosure, terms such as "exemplary" or "for example" are used to denote an example, illustration, or explanation. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present disclosure should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of terms such as "exemplary" or "for example" is intended to present the related concept in a concrete manner. In the description of the embodiments of the present disclosure, unless otherwise specified, "plurality" means two or more.

[0016] In the related art, the sliding window type video super-resolution network model utilizes that most of the image frames in a video are in motion, and when each image frame in a video is super-resolved, the neighboring image frames can provide a large amount of time domain information for the video super-resolution network model to super-resolve the current image frame. However, some areas in some videos are always stationary objects or backgrounds, and when super-resolving such videos, due to the time domain redundant information caused by stationary objects or backgrounds, the ideal video super-resolution effect cannot be obtained when neighboring image frames are used as input, and the super-resolution effect is often inferior to the super-resolution effect of super-resolving based on a single image frame. In summary, when there is time domain redundant information in a video, how to improve the video super-resolution effect is an urgent problem to be solved.

[0017] Therefore, the present disclosure provides a video super-resolution method and apparatus for enhancing the video super-resolution effect.

[0018] An embodiment of the present disclosure provides a video super-resolution method, and referring to the step flowchart shown in FIG. 1, the video super-resolution method according to the embodiment of the present disclosure includes steps S11 to S16 as follows.

[0019] S11: Decompose the t-th image frame of the video to be super-resolved and the neighboring image frames of the t-th image frame into N image blocks respectively.

[0020] Here, t and N are both positive integers.

[0021] In some embodiments, an implementation of decomposing any one image frame into N image blocks includes sampling each position of the image frame using a sampling window that is the size of one image block, starting from the first pixel of the image frame and sliding with a preset step size, so that each sampling area of ​​the sampling window is an image block, thereby decomposing the image frame into N image blocks.

[0022] For example, as shown in FIG. 2, when the t-th image frame of the video to be super-resolved includes 1024*512 pixels, the size of the sampling window is 72*72, and the step size is 64, the t-th image frame of the video to be super-resolved can be decomposed into 16*8 image blocks, each image block includes 72*72 pixels, and there is an overlapping area between adjacent image blocks, and the width of the overlapping area is 8 pixels.

[0023] S12: Generate a sequence of N image blocks based on image blocks obtained by decomposing the t-th image frame and the neighboring image frames.

[0024] Here, each image block in the sequence of image blocks is located at the same position in a different image frame.

[0025] 3, the neighboring image frames of the t-th image frame 33 include the t-2th image frame 31, the t-1th image frame 32, the t+1th image frame 34, and the t+2th image frame 35. Each image library sequence includes five image blocks, which are the t-2th image frame 31, the t-1th image frame 32, the t-th image frame 33, the t+1th image frame 34, and the t+2th image frame 35, respectively, and the image blocks located in the same image block sequence are located at the same position in the image frame to which they belong.

[0026] S13: Calculate the motion parameters of each image block sequence.

[0027] Here, the motion parameters of any one image block sequence are used to represent the optical flow between image blocks of each adjacent image frame in the image block sequence.

[0028] Exemplarily, a sequence of image blocks

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

[0029] As one alternative embodiment of the present disclosure, the above-mentioned calculating the motion parameters of each image block sequence includes performing steps a to c for each image block sequence as follows.

[0030] Step a: Calculate optical flow between image blocks of each adjacent image frame in the image block sequence.

[0031] As mentioned above, the image block sequence

number

number

number

number

number

number

number

number

number

number

number

number

number

number

[0032] Exemplarily, the optical flow between image blocks of each adjacent image frame in the image block sequence can be calculated based on an optical flow algorithm called Dense Inverse Search (DIS).

[0033] Step b: for the optical flows between the image blocks of each pair of adjacent image frames, calculate the average value of the absolute values ​​of the optical flows corresponding to each pixel, and obtain the motion parameters between the image blocks of the adjacent image frames.

[0034] Image Block Sequence

number

number

number

number

number

number

number

number

number

number

number

[0035] As one optional embodiment of the embodiment of the present disclosure, the neighboring image frames of the t-th image frame include the t-2-th image frame, the t-1-th image frame, the t+1-th image frame and the t+2-th image frame, and the above step S14 (determining a super-resolution network model corresponding to each image block sequence based on the motion parameters of each image block sequence) includes performing steps 1 to 5 as follows for each image block sequence.

[0036] Step 1: determining whether a first motion parameter and a second motion parameter of the image block sequence are both smaller than a preset threshold.

[0037] Here, the first motion parameter is a motion parameter between an image block of the t-th image frame and an image block of the t-1-th image frame, and the second motion parameter is a motion parameter between an image block of the t-th image frame and an image block of the t+1-th image frame.

[0038] Image block of the t-th image frame

number

number

number

number

number

number

number

[0039] In the above step 1, if both the first motion parameter and the second motion parameter are smaller than the preset threshold value, the following step 2 is executed.

[0040] Step 2: Determine a super-resolution network model corresponding to the image block sequence as a first super-resolution network model.

[0041] In one alternative embodiment of the present disclosure, the first super-resolution network model is a single-frame super-resolution network model.

[0042] In the above step 1, if the first motion parameter and / or the second motion parameter is equal to or greater than the preset threshold, the following step 3 is executed.

[0043] Step 3: determining whether the third motion parameter and the fourth motion parameter of the image block sequence are both smaller than a preset threshold;

[0044] Here, the third motion parameter is a motion parameter between the image block of the t-2th image frame and the image block of the t-1th image frame, and the fourth motion parameter is a motion parameter between the image block of the t+1th image frame and the image block of the t+2th image frame.

[0045] Image block of the t-2th image frame

number

number

number

number

number

number

number

[0046] In the above step 3, when both the third motion parameter and the fourth motion parameter are smaller than the preset threshold value, the following step 4 is executed.

[0047] Step 4: Determine the super-resolution network model corresponding to the image block sequence as a second super-resolution network model.

[0048] In one optional embodiment of the embodiment of the present disclosure, the second super-resolution network model is used to super-resolve an image block of the t-1th image frame based on an image block of the t-th image frame, an image block of the t-th image frame, and an image block of the t+1th image frame.

[0049] In the above step 3, when the third motion parameter and / or the fourth motion parameter is equal to or greater than the preset threshold, the following step 5 is executed.

[0050] Step 5: Determine the super-resolution network model corresponding to the image block sequence as a third super-resolution network model.

[0051] In one alternative embodiment of the present disclosure, the third super-resolution network model is used to super-resolve an image block of the t-th image frame based on all image blocks in the image block sequence.

[0052] Image block of the t-th image frame

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

[0053]

number

[0054] Specifically, one super-resolved image block of the t-th image frame can be obtained based on each image block sequence, and since a total of N image block sequences are included, N super-resolved image blocks of the t-th image frame can be obtained.

[0055] S16: Generate a super-resolved image frame of the t-th image frame based on each super-resolved image block of the t-th image frame.

[0056] As an optional embodiment of the embodiment of the present disclosure, when adjacent image blocks in the image blocks obtained by decomposing the t-th image frame and the neighboring image frame do not have an overlapping area, the above step S16 (generating a super-resolved image frame of the t-th image frame based on each super-resolved image block of the t-th image frame) is stitching each super-resolved image block of the t-th image frame as a super-resolved image frame of the t-th image frame.

[0057] As one optional embodiment of the embodiment of the present disclosure, when adjacent image blocks among the image blocks obtained by decomposing the t-th image frame and the neighboring image frame have an overlapping area, the above step S16 (generating a super-resolved image frame of the t-th image frame based on each super-resolved image block of the t-th image frame) is stitching each super-resolved image block of the t-th image frame to generate a stitched image; and setting the pixel value of each pixel in the overlapping area of ​​each super-resolved image block in the stitched image as an average value of the pixel values ​​of corresponding pixels in each super-resolved image block to generate a super-resolved image frame of the t-th image frame.

[0058] 4, the starting pixel column of the super-resolved image block 41 in FIG. 4 is column P, the ending pixel column of the super-resolved image block 41 is column P+m, the starting pixel column of the super-resolved image block 42 is column P+n, and the ending pixel column of the super-resolved image block 42 is column P+m+n. When the super-resolved image block 41 and the super-resolved image block 42 are stitched, a region 411 of the super-resolved image block 41 overlaps with a region 421 of the super-resolved image block 42, so that the pixel value of any one pixel in the overlapping region 400 is the average value of the pixel values ​​corresponding to this pixel in the regions 411 and 421. For example, the pixel value of the pixel (x1, y1) in the overlapping region 400 is the average value of the pixel value of the pixel (x1, y1) in the region 411 and the pixel value of the pixel (x1, y1) in the region 412.

[0059] As shown in Fig. 5, Fig. 5 is a structural schematic diagram of a video super-resolution network for realizing the above-mentioned video super-resolution method. The video super-resolution network for realizing the above-mentioned video super-resolution method includes an image decomposition module 51, a sequence generation module 52, a redundant information detection module 53, an adaptive super-resolution module 54 and an image stitching module 55.

[0060] Here, the image decomposition module 51 decomposes the t-2th image frame

number

number

number

number

number

number

number

number

number

number

[0061] The sequence generation module 52 includes:

number

number

[0062] The redundancy information monitoring module 53 monitors each image block sequence

number

number

number

[0063] The adaptive super-resolution module 54 includes a super-resolution network model corresponding to each image block sequence, and employs the super-resolution network model corresponding to each image block sequence to super-resolve the image block of the t-th image frame in each image block sequence, and each super-resolved image block of the t-th image frame is

number

[0064] The image stitching module 55 stitches each super-resolved image block of the t-th image frame

number

number

[0065] In a video super-resolution method according to an embodiment of the present disclosure, when super-resolving a t-th image frame image, firstly, decompose the t-th image frame of the video to be super-resolved and the neighboring image frames of the t-th image frame into N image blocks, generate N image block sequences based on the image blocks obtained by decomposing the t-th image frame and the neighboring image frames, calculate motion parameters of each image block sequence, determine a super-resolution network model corresponding to each image block sequence based on the motion parameters of each image block sequence, determine a super-resolution network model corresponding to each image block sequence based on the motion parameters of each image block sequence, adopt the super-resolution network model corresponding to each image block sequence to super-resolve the image blocks of the t-th image frame in each image block sequence, obtain each super-resolved image block of the t-th image frame, and generate a super-resolved image frame of the t-th image frame based on each super-resolved image block of the t-th image frame. In the video super-resolution method according to an embodiment of the present disclosure, when super-resolving the tth image frame image, a super-resolution network model corresponding to each image block sequence is determined based on the motion parameters of each image block sequence, and different super-resolution network models are adaptively adopted for super-resolution in different cases, so that the video super-resolution method according to an embodiment of the present disclosure can improve the video super-resolution effect.

[0066] 5 includes a first super-resolution network model 541, a second super-resolution network model 542 and a third super-resolution network model 543. The first super-resolution network model 541 selects an image block of the t-th image frame when super-resolving the image block of the t-th image frame.

number

number

number

[0067] As an optional embodiment of the embodiment of the present disclosure, as shown in the model structure schematic diagram of the first super-resolution network model in Figure 7, an implementation method of super-resolving the image block of the t-th image frame using the first super-resolution network model includes steps I to IV as follows.

[0068] Step I: Pyramid Cascading and Deformable Convolutions (PCD) alignment module 71 aligns the image block of the t-th image frame.

number

[0069] As shown in FIG. 7, the input of the PCD alignment module 71 is two image blocks, but in the above step I, the input of the PCD alignment module contains only one image block (the image block of the t-th image frame), so the image block of the t-th image frame can be copied once and used as the input of the PCD alignment module together with the original image block.

[0070] Step II: Process the first features through feature fusion module 72 to obtain second features T2.

[0071] Here, the second feature is a feature obtained by stitching five of the first features in the channel dimension. As can be understood by those skilled in the art, the second feature may be a feature obtained by stitching a plurality of the first features in the channel dimension, and there is no limit to the number of first features to be stitched here.

[0072] That is, if the extension of the first feature is C*H*W, then the extension of the second feature is 5*C*H*W, where C is the number of channels of the first feature, H is the length of the first feature, and W is the width of the first feature.

[0073] Illustratively, as shown in Fig. 7, the feature fusion module 72 may include a time attention unit 721, a feature copy unit 722, a feature fusion unit 723 and a spatial attention unit 724. Here, the feature copy unit 722 is used to copy a first feature four times and stitch the original first feature.

[0074] Step III: Reconstruct the second feature T2 by a reconstruction module 73 to obtain a first image block B1.

[0075] Step IV: The first image block B1 is upsampled by the upsampling module 74 to obtain a super-resolved image block B1 corresponding to the image block of the t-th image frame.

number

[0076] As an optional embodiment of the embodiment of the present disclosure, as shown in the model structure schematic diagram of the second super-resolution network model in Figure 8, the implementation method of super-resolving the image block of the t-th image frame using the second super-resolution network model includes steps i to iv as follows.

[0077] Step i: The image block of the t-1-th image frame is aligned by the PCD alignment module 81.

number

number

number

[0078] Here, the third feature T3 is a feature obtained by stitching the fourth feature T4, the fifth feature T5, and the sixth feature T6 in the channel dimension, and the fourth feature T4 is aligned to the image block of the t-1th image frame by the PCD alignment module.

number

number

number

number

number

[0079] 8, the PCD alignment module 81 includes a first PCD alignment unit 811, a second PCD alignment unit 812, a third PCD alignment unit 813 and a stitching unit 814. Here, the first PCD alignment unit 811 aligns an image block of the t-1th image frame.

number

number

number

number

number

[0080] Step ii: The third feature T3 is processed by a feature fusion module 82 to obtain a seventh feature T7.

[0081] Here, the seventh feature T7 is a feature obtained by stitching the fourth feature T4, the third feature T3, and the fifth feature T5 in the channel dimension.

[0082] Illustratively, as shown in Fig. 8, the feature fusion module 82 may include a time attention unit 821, a feature copy unit 822, a feature fusion unit 823 and a space attention unit 824. Here, the feature copy unit 822 is used to copy the fourth feature T4 and the fifth feature T5 in the third feature T3 once, and stitch the third feature T3.

[0083] Step iii: Reconstruct the seventh feature T7 by the reconstruction module 83 to obtain a second image block B2.

[0084] Step iv: upsampling the second image block B2 by the upsampling module 84 to obtain a super-resolved image block B2 corresponding to the image block of the t-th image frame.

number

[0085] As one optional embodiment of the embodiment of the present disclosure, as shown in the model structure schematic diagram of the third super-resolution network model in Figure 9, super-resolving the image block of the t-th image frame in the image block sequence by the third super-resolution network model includes steps 1 to 4 as follows.

[0086] Step 1: Image block sequence by PCD alignment module 91

number

[0087] Here, the eighth feature T8 is a ninth feature T9, a tenth feature T 10 The eleventh feature 11 The twelfth characteristic 12 And the thirteenth feature T 13 The ninth feature T9 is obtained by stitching the image block of the t-2-th image frame by the PCD alignment module.

number

number

number

number

number

number

number

number

number

[0088] 9, the PCD alignment module 91 includes a first PCD alignment unit 911, a second PCD alignment unit 912, a third PCD alignment unit 913, a fourth PCD alignment unit 914, a fifth PCD alignment unit 915 and a stitching unit 916. The first PCD alignment unit 911 aligns an image block of the t-2th image frame.

number

number

number

number

number

number

number

number

number

[0089] Step 2: The feature fusion module 92 processes the eighth feature T8 and derives a fourteenth feature T 14 Get the.

[0090] Illustratively, as shown in FIG. 9, the feature fusion module 92 may include a temporal attention unit 921, a feature fusion unit 922 and a spatial attention unit 923.

[0091] Step 3: The reconstruction module 93 uses the fourteenth feature T 14 to obtain a third image block B3.

[0092] Step 4: The third image block B3 is upsampled by the upsampling module 94 to obtain a super-resolved image block B3 corresponding to the image block of the t-th image frame in the image block sequence.

number

[0093] Based on the same inventive idea, as an implementation of the above method, the embodiments of the present disclosure further provide a video super-resolution device, and the embodiments of the device correspond to the embodiments of the above method. For ease of reading, the embodiments of the device will not describe the detailed contents of the embodiments of the above method, but it should be clear that the video super-resolution device in this embodiment can correspondingly realize all the contents of the embodiments of the above method.

[0094] An embodiment of the present disclosure provides a video super-resolution device, and FIG. 10 is a structural schematic diagram of the video super-resolution device. As shown in FIG. 10, the video super-resolution device 100 includes: an image decomposition module 101 for decomposing a t-th image frame of a video to be super-resolved and a neighboring image frame of the t-th image frame into N image blocks, respectively, where t and N are both positive integers; a sequence generation module 102 for generating a sequence of N image blocks based on image blocks obtained by decomposing the t-th image frame and the neighboring image frames, where each image block in the image block sequence is located at the same position in a different image frame; a parameter calculation module 103 for calculating motion parameters for each image block sequence, the motion parameters of any one image block sequence being used to represent the optical flow between image blocks of adjacent image frames in this image block sequence; a model determination module 104 for determining a super-resolution network model corresponding to each image block sequence based on the motion parameters of each image block sequence; an image super-resolution module 105 for adopting a super-resolution network model corresponding to each image block sequence to super-resolve an image block of the t-th image frame in each image block sequence, to obtain each super-resolved image block of the t-th image frame; and an image generation module 106 for generating a super-resolved image frame of the t-th image frame based on each super-resolved image block of the t-th image frame.

[0095] As one optional embodiment of the present disclosure, the parameter calculation module 103 is specifically used to calculate, for each image block sequence, an optical flow between image blocks of each adjacent image frame in the image block sequence, calculate an average value of the absolute value of the optical flow corresponding to each pixel for the optical flow between the image blocks of each pair of adjacent image frames, obtain motion parameters between the image blocks of the adjacent image frames, and obtain motion parameters of the image block sequence based on the motion parameters between the image blocks of each adjacent image frame in the image block sequence.

[0096] In one alternative embodiment of the present disclosure, the neighboring image frames of the t-th image frame include: The to-be-super-resolved video includes the t-2th image frame, the t-1th image frame, the t+1th image frame and the t+2th image frame.

[0097] In one alternative embodiment of the present disclosure, the model determination module 104 specifically determines, for each image block sequence, whether a first motion parameter and a second motion parameter of the image block sequence are both smaller than a preset threshold, where the first motion parameter is a motion parameter between the image block of the t-th image frame and the image block of the t-1-th image frame, and the second motion parameter is a motion parameter between the image block of the t-th image frame and the image block of the t+1-th image frame; if the first motion parameter and the second motion parameter are both smaller than the preset threshold, determine a super-resolution network model corresponding to the image block sequence as a first super-resolution network model; if the first motion parameter and / or the second motion parameter are equal to or greater than the preset threshold, determine a super-resolution network model corresponding to the image block sequence as a first super-resolution network model. The method is used to determine whether a third motion parameter and a fourth motion parameter of an image block sequence are both smaller than a predetermined threshold, wherein the third motion parameter is a motion parameter between an image block of the t-2th image frame and an image block of the t-1th image frame, and the fourth motion parameter is a motion parameter between an image block of the t+1th image frame and an image block of the t+2th image frame; if the third motion parameter and the fourth motion parameter are both smaller than the predetermined threshold, determine a super-resolution network model corresponding to the image block sequence as a second super-resolution network model; and if the third motion parameter and / or the fourth motion parameter are equal to or greater than the predetermined threshold, determine a super-resolution network model corresponding to the image block sequence as a third super-resolution network model.

[0098] As an alternative embodiment of the present disclosure, the first super-resolution network model is a single-frame super-resolution network model; the second super-resolution network model is used to super-resolve the image block of the t-th image frame based on the image block of the t-1-th image frame, the image block of the t-th image frame, and the image block of the t+1-th image frame; The third super-resolution network model is used to super-resolve an image block of the t-th image frame based on all image blocks in a sequence of image blocks.

[0099] As one optional embodiment of the embodiment of the present disclosure, the image super-resolution module 105 is specifically used to process the image block of the t-th image frame by a cascade deformable convolution PCD alignment module to obtain a first feature, process the first feature by a feature fusion module to obtain a second feature, the second feature being a feature obtained by stitching five of the first features in the channel dimension, reconstruct the second feature by a reconstruction module to obtain a first image block, and upsample the first image block by an upsampling module to obtain a super-resolved image block corresponding to the image block of the t-th image frame.

[0100] In one alternative embodiment of the present disclosure, the image super-resolution module 105 is specifically used to process the image block of the t-1th image frame, the image block of the tth image frame, and the image block of the t+1th image frame through a PCD alignment module to obtain a third feature, the third feature being a feature obtained by stitching a fourth feature, a fifth feature, and a sixth feature in a channel dimension, the fourth feature being a feature obtained by processing the image block of the t-1th image frame and the image block of the tth image frame through the PCD alignment module, and the fifth feature being a feature obtained by stitching the image block of the t-1th image frame and the image block of the tth image frame through the PCD alignment module. the sixth feature is a feature obtained by processing the image block of the tth image frame and the image block of the t+1th image frame by the PCD alignment module; the third feature is processed by a feature fusion module to obtain a seventh feature; the seventh feature is a feature obtained by stitching the fourth feature, the third feature and the fifth feature in a channel dimension; the seventh feature is reconstructed by a reconstruction module to obtain a second image block; and an upsampling module upsamples the second image block to obtain a super-resolved image block corresponding to the image block of the tth image frame.

[0101] In one alternative embodiment of the present disclosure, the image super-resolution module 105 is specifically used to process all image blocks in the image block sequence through a PCD alignment module to obtain an eighth feature, the eighth feature being obtained by stitching the ninth feature, the tenth feature, the eleventh feature, the twelfth feature, and the thirteenth feature in a channel dimension, the ninth feature being obtained by processing the image block of the t-2th image frame and the image block of the t-1th image frame through the PCD alignment module, the tenth feature being obtained by processing the image block of the t-1th image frame and the image block of the tth image frame through the PCD alignment module, the eleventh feature being obtained by processing the image block of the t-1th image frame and the image block of the tth image frame through the PCD alignment module, and the eleventh feature being obtained by stitching the image block of the t-2th image frame and the image block of the t-1th image frame through the PCD alignment module. the tth feature is a feature obtained by processing the image block of the tth image frame and the image block of the t+1th image frame by the PCD alignment module; the thirteenth feature is a feature obtained by processing the image block of the t+1th image frame and the image block of the t+2th image frame by the PCD alignment module; the eighth feature is processed by a feature fusion module to obtain a fourteenth feature; the fourteenth feature is reconstructed by a reconstruction module to obtain a third image block; and an upsampling module upsamples the third image block to obtain a super-resolved image block corresponding to the image block of the tth image frame in the image block sequence.

[0102] In one alternative embodiment of the present disclosure, adjacent image blocks among the image blocks obtained by decomposing the t-th image frame and the neighboring image frame have an overlapping region; The image generation module 106 is specifically used to stitch each super-resolved image block of the t-th image frame to generate a stitched image, set the pixel value of each pixel in the overlapping area of ​​each super-resolved image block in the stitched image as the average value of the pixel values ​​of corresponding pixels in each super-resolved image block, and generate a super-resolved image frame of the t-th image frame.

[0103] The modules may be implemented as software assemblies running on one or more general-purpose processors, or may be implemented as hardware, such as programmable logic devices and / or application specific integrated circuits, performing certain functions or a combination thereof. In some embodiments, the modules may be embodied in the form of a software product, which may be stored in a non-volatile storage medium, and the non-volatile storage medium may include a program that causes a computer device (such as a personal computer, a server, a network device, a mobile terminal, etc.) to implement the method described in the embodiments of the present disclosure. In other embodiments, the modules may be implemented on a single device or distributed across multiple devices. The functions of the modules may be integrated with each other or divided into multiple sub-modules.

[0104] The video super-resolution device according to this embodiment can implement the video super-resolution method according to the above-mentioned method embodiment, and the realization principle and technical effect thereof are similar, so they will not be further described here.

[0105] Based on the same inventive idea, an embodiment of the present disclosure further provides an electronic device. Figure 11 is a structural schematic diagram of an electronic device according to an embodiment of the present disclosure, as shown in Figure 11, the electronic device according to this embodiment includes a memory 111 and a processor 112, the memory 111 is used to store a computer program, and the processor 112 is used to execute the video super-resolution method according to the above embodiment when calling the computer program.

[0106] Based on the same inventive idea, an embodiment of the present disclosure further provides a computer-readable storage medium having a computer program stored therein, which, when executed by a processor, causes the computing device to realize the video super-resolution method according to the above embodiment.

[0107] Based on the same inventive idea, an embodiment of the present disclosure further provides a computer program product, which, when running on a computer, enables the computing device to realize the video super-resolution method according to the above embodiment.

[0108] Based on the same inventive idea, an embodiment of the present disclosure further provides a computer program, which includes instructions, when executed by a processor, causing the processor to perform the video super-resolution method according to the above embodiment.

[0109] It should be understood by those skilled in the art that the embodiments of the present disclosure can be provided as a method, a system, or a computer program product. Therefore, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment that combines software and hardware aspects. And, the present disclosure may take the form of a computer program product embodied on one or more computer usable storage mediums that contain computer usable program code.

[0110] The processor may be a Central Processing Unit (CPU) or other general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware assembly, etc. The general purpose processor may be a microprocessor, or the processor may be any common processor, etc.

[0111] The memory may include non-persistent memory, random access memory (RAM) and / or non-volatile memory in the form of a computer readable medium, such as read only memory (ROM) or flash memory (flash RAM). The memory is an example of a computer readable medium.

[0112] Computer-readable media includes persistent and non-persistent storage media, removable and non-removable storage media. Storage media may implement information storage by any method or technology, and information may be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only optical disk read-only memory (CD-ROM), digital versatile optical disk (DVD) or other optical storage, magnetic cassette tape, magnetic disk storage or other magnetic storage device, or any other non-transmission medium that can be used to store information accessed by a computing device. As defined herein, computer-readable media does not include transitory media, such as modulated data signals and carriers.

[0113] Finally, it should be explained that the above embodiments are only used to describe the technical solutions of the present disclosure, and are not intended to limit the same. Although the present disclosure has been described in detail with reference to the above embodiments, it should be understood by those skilled in the art that the technical solutions described in the above embodiments can still be modified or some or all of the technical features can be replaced with equivalents, and such modifications or replacements do not cause the essence of the corresponding technical solutions to depart from the scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. 1. A method for video super-resolution comprising the steps of: Decomposing a t-th image frame of a video to be super-resolved and a neighboring image frame of the t-th image frame into N image blocks respectively, where t and N are both positive integers; generating a sequence of N image blocks based on image blocks obtained by decomposing the t-th image frame and the neighboring image frames, where each image block in the sequence of N image blocks is located at a same position in a different image frame; calculating motion parameters for each image block sequence, the motion parameters of any one image block sequence being used to represent optical flow between image blocks of each adjacent image frame in the image block sequence; determining a super-resolution network model corresponding to each image block sequence based on the motion parameters of each image block sequence; adopting a super-resolution network model corresponding to each image block sequence to super-resolve an image block of the t-th image frame in each image block sequence, to obtain each super-resolved image block of the t-th image frame; generating a super-resolved image frame of the t-th image frame based on each super-resolved image block of the t-th image frame.

2. Calculating the motion parameters for each image block sequence includes: calculating, for each sequence of image blocks, optical flow between image blocks of each adjacent image frame in the sequence of image blocks; For the optical flows between the image blocks of each pair of adjacent image frames, calculating an average value of the absolute values ​​of the optical flows corresponding to each pixel to obtain motion parameters between the image blocks of the adjacent image frames; and obtaining motion parameters for the sequence of image blocks based on motion parameters between image blocks of each adjacent image frame in the sequence of image blocks.

3. The neighboring image frames of the t-th image frame are The video super-resolution method of claim 1 , comprising the t−2 th image frame, the t−1 th image frame, the t+1 th image frame and the t+2 th image frame.

4. Determining a super-resolution network model corresponding to each image block sequence based on the motion parameters of each image block sequence includes: determining, for each image block sequence, whether a first motion parameter and a second motion parameter of the image block sequence are both smaller than a preset threshold, the first motion parameter being a motion parameter between an image block of the t-th image frame and an image block of the t-1-th image frame, and the second motion parameter being a motion parameter between an image block of the t-th image frame and an image block of the t+1-th image frame; determining a super-resolution network model corresponding to the image block sequence as a first super-resolution network model when the first motion parameter and the second motion parameter are both smaller than the preset threshold; determining whether a third motion parameter and a fourth motion parameter of the image block sequence are both less than a predetermined threshold when the first motion parameter and / or the second motion parameter are equal to or greater than the predetermined threshold, the third motion parameter being a motion parameter between an image block of the t-2th image frame and an image block of the t-1th image frame, and the fourth motion parameter being a motion parameter between an image block of the t+1th image frame and an image block of the t+2th image frame; determining a super-resolution network model corresponding to the image block sequence as a second super-resolution network model when the third motion parameter and the fourth motion parameter are both smaller than the preset threshold; The video super-resolution method of claim 3, further comprising determining a super-resolution network model corresponding to the image block sequence as a third super-resolution network model when the third motion parameter and / or the fourth motion parameter is greater than or equal to the predetermined threshold.

5. the first super-resolution network model is a single-frame super-resolution network model; the second super-resolution network model is used to super-resolve the image block of the t-1th image frame based on the image block of the t-1th image frame, the image block of the t-1th image frame, and the image block of the t+1th image frame; The video super-resolution method of claim 4 , wherein the third super-resolution network model is used to super-resolve an image block of the t th image frame based on all image blocks in a sequence of image blocks.

6. super-resolving the image block of the t-th image frame with the first super-resolution network model, processing the image block of the t-th image frame through a cascade deformable convolutional PCD alignment module to obtain a first feature; processing the first features through a feature fusion module to obtain a second feature, the second feature being a feature obtained by stitching a plurality of the first features in a channel dimension; reconstructing the second feature by a reconstruction module to obtain a first image block; 6. The method of claim 5, further comprising: upsampling the first image block by an upsampling module to obtain a super-resolved image block corresponding to the image block of the t-th image frame.

7. The video super-resolution method of claim 6 , wherein the second feature is obtained by stitching five of the first features in a channel dimension.

8. super-resolving the image block of the t-th image frame with the second super-resolution network model, Processing the image block of the t-1th image frame, the image block of the tth image frame, and the image block of the t+1th image frame by a PCD alignment module to obtain a third feature, the third feature being a feature obtained by stitching the fourth feature, the fifth feature, and the sixth feature in a channel dimension, the fourth feature being a feature obtained by processing the image block of the t-1th image frame and the image block of the tth image frame by the PCD alignment module, the fifth feature being a feature obtained by processing the image block of the tth image frame by the PCD alignment module, and the sixth feature being a feature obtained by processing the image block of the tth image frame and the image block of the t+1th image frame by the PCD alignment module; Processing the third feature through a feature fusion module to obtain a seventh feature, the seventh feature being a feature obtained by stitching the fourth feature, the third feature and the fifth feature in a channel dimension; reconstructing the seventh feature with a reconstruction module to obtain a second image block; and upsampling the second image block by an upsampling module to obtain a super-resolved image block corresponding to the image block of the t-th image frame.

9. super-resolving the image block of the t-th image frame in the image block sequence by the third super-resolution network model, Processing all image blocks in the image block sequence by a PCD alignment module to obtain an eighth feature, the eighth feature being obtained by stitching the ninth feature, the tenth feature, the eleventh feature, the twelfth feature, and the thirteenth feature in a channel dimension, the ninth feature being obtained by processing the image block of the t-2th image frame and the image block of the t-1th image frame by the PCD alignment module, the tenth feature being obtained by processing the image block of the t-1th image frame and the image block of the tth image frame by the PCD alignment module, the eleventh feature being obtained by processing the image block of the tth image frame by the PCD alignment module, the twelfth feature being obtained by processing the image block of the tth image frame and the image block of the t+1th image frame by the PCD alignment module, and the thirteenth feature being obtained by processing the image block of the t+1th image frame and the image block of the t+2th image frame by the PCD alignment module; Processing the eighth feature through a feature fusion module to obtain a fourteenth feature; reconstructing the fourteenth feature by a reconstruction module to obtain a third image block; The video super-resolution method of claim 5, further comprising: upsampling the third image block by an upsampling module to obtain a super-resolved image block corresponding to the image block of the t-th image frame in the image block sequence.

10. Adjacent image blocks among the image blocks obtained by decomposing the t-th image frame and the neighboring image frame have an overlapping area, and generating a super-resolved image frame of the t-th image frame based on each super-resolved image block of the t-th image frame includes: stitching each super-resolved image block of the t-th image frame to generate a stitched image; 2. The video super-resolution method of claim 1, further comprising: setting a pixel value of each pixel in an overlapping area of ​​each super-resolved image block in the stitched image as an average value of pixel values ​​of corresponding pixels in each super-resolved image block to generate a super-resolved image frame of the tth image frame.

11. Decomposing the t-th image frame of a video to be super-resolved into N image blocks can be expressed as: The video super-resolution method of claim 1, comprising: sampling each position of the t-th image frame by a sampling window having a size of one image block, starting from the first pixel of the t-th image frame and sliding with a predetermined step size, so that each sampling area of ​​the sampling window is an image block, and obtaining N image blocks.

12. 1. A video super-resolution device, comprising: an image decomposition module for decomposing a t-th image frame of the video to be super-resolved and a neighboring image frame of the t-th image frame into N image blocks, respectively, where t and N are both positive integers; a sequence generation module for generating a sequence of N image blocks based on image blocks obtained by decomposing the t-th image frame and the neighboring image frames, wherein each image block in the sequence of image blocks is located at the same position in a different image frame; a parameter calculation module for calculating a motion parameter for each image block sequence, the motion parameter for any one image block sequence being used to represent an optical flow between image blocks of adjacent image frames in the image block sequence; a model determination module for determining a super-resolution network model corresponding to each image block sequence based on the motion parameters of each image block sequence; an image super-resolution module for adopting a super-resolution network model corresponding to each image block sequence to super-resolve an image block of the t-th image frame in each image block sequence, to obtain each super-resolved image block of the t-th image frame; an image generation module for generating a super-resolved image frame of the t-th image frame based on each super-resolved image block of the t-th image frame.

13. An electronic device, A memory and a processor are included, the memory is used to store instructions; An electronic device, wherein the processor, when executing the instructions, is adapted to cause the electronic device to implement a method for super-resolution of a video according to any one of claims 1 to 11.

14. A computer-readable storage medium having instructions stored thereon that, when executed by a processor, implements the video super-resolution method of any one of claims 1 to 11.

15. A computer program product, which, when run on a computer, causes the computer to implement the method for video super-resolution according to any one of claims 1 to 11.

16. A computer program comprising: A computer program comprising instructions which, when executed by a processor, cause the processor to carry out a method for super-resolution of a video according to any one of claims 1 to 11.

Citation Information

Patent Citations

  • Device and method for signal processing and recording medium

    JP2001250119A

  • Image processing system and program

    WO2020166596A1