Video super-resolution model construction, video super-resolution imaging method and system

Abstract

A video super-resolution model, video super-resolution imaging method, and system are designed to remove interference from real-world video. They learn the complex mapping relationship between super-resolution and low-resolution video frames in real scenes, and propagate the enhanced backward and forward features in a backward / forward loop, generating video super-resolution results with clear structure and texture, rich details, and visually pleasing visuals. Experimental results on the RealVSR dataset show that RBFFEN's perceptual metrics, SSIM and LPIPS, outperform state-of-the-art video super-resolution methods.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a video super-resolution model construction, a video super-resolution imaging method and a system in real scenes. Background Art

[0002] The video super-resolution task refers to reconstructing high-resolution video frames using multiple low-resolution video frames. Video super-resolution technology is widely used in video surveillance, medical image diagnosis, intelligent security, and distance education. Unlike image super-resolution, video super-resolution can use the temporal and spatial correlation between video sequence frames to increase the details and clarity of the reconstructed high-resolution video. Existing classic video super-resolution methods are all trained on benchmark datasets. The trained models cannot simulate the complex degradation process between super-resolution and low-resolution video frames in real scenes well, and perform poorly for real-scene video super-resolution. Currently, there is little research on real-scene video super-resolution. In 2023, researchers proposed a real-scene video super-resolution model, which achieved good results on real-scene video super-resolution datasets, but still has shortcomings such as unclear structure and texture, lack of high-frequency details, and poor visual effects. Summary of the Invention

[0003] The purpose of the present invention is to provide a video super-resolution model construction, video super-resolution imaging method and system in real scenes, which solves the problems existing in the prior art of unclear structural texture, lack of high-frequency details and poor visual effects of video super-resolution results in real scenes.

[0004] In order to achieve the above technical effects, the technical solution adopted by the present invention is:

[0005] A method for constructing a video super-resolution model in a real scene comprises the following steps: obtaining a low-resolution video frame set, the low-resolution video frame set comprising 2N+1 consecutive low-resolution video frames {x t-N ,...,x t ,...,x t+N}, where x t represents the low-resolution video frame at time t; the low-resolution video frame set is input into the video super-resolution network to train the video super-resolution model in the real scene. The training process is as follows:

[0006] S1, input the low-resolution video frame set into the smoothing module for denoising to obtain a smoothed video frame set in, represents the smoothed video frame at time t;

[0007] S2, input the smoothed video frame set into the feature extraction module for feature extraction, and obtain the video frame feature set {gt-N ,...,g t ,...,g t+N}, where g t Represents the video frame features at time t;

[0008] S3, input the low-resolution video frame set into the first optical flow estimation network for reverse optical flow alignment to obtain the reverse aligned optical flow set {S t+N ,...,S t ,...,S t-N+1}, where S t Represents the reverse alignment optical flow from the video frame at time t to the video frame at time t-1;

[0009] S4, g t+N and g t+N After splicing, input the reverse attention enhancement module to obtain the reverse enhancement feature at time t+N

[0010] S5, the zero optical flow vector, S t+N 、g t+N-1 、 The zero optical flow feature is input into the first convolution alignment module at the same time to obtain the reverse alignment feature at time t+N-1

[0011] S6, will and g t+N-1 After splicing, input the reverse attention enhancement module to obtain the reverse enhancement feature at time t+N-1

[0012] S7, S t+N 、S t+N-1 、g t+N-2 、 and At the same time, input the first convolution alignment module to obtain the reverse alignment feature at time t+N-2

[0013] S8, will After being spliced ​​with gt+N-2, it is input into the reverse attention enhancement module to obtain the reverse enhancement feature at time t+N-2

[0014] S9, let t+Ni=t+Ni-1, repeat S7-S8 to get the reverse enhancement feature set

[0015] S10, input the low-resolution video frame set into the second optical flow estimation network for forward optical flow alignment, and obtain the forward aligned optical flow set {f t-N ,...,f t ,...,ft+N-1}, where f t Represents the forward alignment optical flow from the video frame at time t to the video frame at time t+1;

[0016] S11, Will g t-N and g t-N After splicing, input the positive attention enhancement module to obtain the positive enhancement features at time tN

[0017] S12, the zero optical flow vector, f t-N 、g t-N+1 、 The zero optical flow feature is input into the second convolution alignment module at the same time to obtain the forward alignment feature at time t-N+1

[0018] S13, will and g t-N+1 After splicing, input the positive attention enhancement module to obtain the positive enhancement features at time t-N+1

[0019] S14, f t-N 、、f t-N+1 、g t-N+2 、 and At the same time, the second convolutional alignment module is input to obtain the forward alignment feature at time t-N+2

[0020] S15, will and g t-N+2 After splicing, input the positive attention enhancement module to obtain the positive enhancement features at time t-N+2

[0021] S16, let t-N+i=t-N+i+1, repeat S14-S15, and get the positive enhancement feature set

[0022] S17, for positive enhancement feature set and smooth video frame sets After upsampling respectively, the results are added together to obtain the video super-resolution results.

[0023] Preferably, the smoothing module comprises three convolutional layers connected in series.

[0024] Preferably, the feature extraction module includes two residual blocks connected in series.

[0025] Preferably, the reverse attention enhancement module is composed of 6 first residual channel attention modules RCAB connected in series, and any first residual channel attention module RCAB includes a first residual block, a second residual block, a first channel attention module CAI and a first fully connected layer connected in series in sequence, and the input end of the first residual block is also residually connected to the output end of the second residual block and the output end of the first fully connected layer respectively.

[0026] Preferably, the forward attention enhancement module is composed of 6 second residual channel attention modules RCAB connected in series, and any second residual channel attention module RCAB includes a third residual block, a fourth residual block, a second channel attention module CAI and a second fully connected layer connected in series in sequence, and the input end of the third residual block is also residually connected to the output end of the fourth residual block and the output end of the second fully connected layer.

[0027] Preferably, the first optical flow estimation network and the second optical flow estimation network are both SpyNet networks.

[0028] Preferably, the first convolution alignment module and the second convolution alignment module are both DCA networks.

[0029] Preferably, the value of N is 7.

[0030] A video super-resolution imaging method in a real scene comprises the following steps:

[0031] Step 1: obtain a low-resolution video and segment it to obtain a low-resolution video frame set;

[0032] Step 2: Input the low-resolution video frame set obtained in step 1 into a video super-resolution model in a real scene to obtain a video super-resolution result;

[0033] Among them, the video super-resolution model in the real scene is the video super-resolution model in the real scene constructed by the video super-resolution model construction method in the real scene disclosed in this application.

[0034] A video super-resolution imaging system in real scenes, including an acquisition and preprocessing module and a video super-resolution module;

[0035] The acquisition and preprocessing module is used to acquire low-resolution video and segment it to obtain a low-resolution video frame set;

[0036] The video super-resolution module inputs a low-resolution video frame set into a video super-resolution model in a real scene to obtain the video super-resolution result;

[0037] Among them, the video super-resolution model in the real scene is the video super-resolution model in the real scene constructed by the video super-resolution model construction method in the real scene disclosed in this application.

[0038] The above technical solution has the following beneficial effects:

[0039] (1) The real-world video super-resolution model constructed by the method of the present invention can remove interference from real-world videos, learn the complex mapping relationship between super-resolution and low-resolution video frames in real-world scenarios, and propagate the enhanced backward and forward features in a backward / forward loop to generate video super-resolution results with clear structure and texture, rich details, and visually pleasing visuals. Experimental results on the RealVSR dataset show that RBFFEN's perceptual metrics SSIM and LPIPS lead the state-of-the-art video super-resolution methods.

[0040] (2) The video super-resolution imaging method and system in real scenes of the present invention have a high degree of automation, and the video super-resolution model in real scenes does not require human intervention after training. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 Schematic diagram of the process of the video super-resolution imaging method in real scenes of the present invention;

[0042] Figure 2 This is a plane diagram comparing the perceptual quality of the present invention and the existing video super-resolution method;

[0043] Figure 3 This is the video super-resolution result diagram obtained by the present invention in the RealVSR set test;

[0044] Figure 4 This is a video super-resolution result diagram obtained by testing the present invention on the RealVSR set.

[0045] The present invention will be described in detail below with reference to the accompanying drawings and specific implementation methods. DETAILED DESCRIPTION

[0046] The present invention will be described in detail below with reference to the accompanying drawings and embodiments to facilitate a better understanding of the present invention by those skilled in the art. It should be noted that in the following description, detailed descriptions of known functions and designs will be omitted when they might obscure the main aspects of the present invention.

[0047] Other structures and functions of the method of the present invention are known to those skilled in the art and will not be described in detail to reduce redundancy.

[0048] Example 1

[0049] This embodiment discloses a method for constructing a video super-resolution model in a real scene, including the following steps:

[0050] Obtain a low-resolution video frame set, wherein the low-resolution video frame set includes 2N+1 consecutive low-resolution video frames {x t-N ,...,x t ,...,x t+N}, where x t represents a low-resolution video frame at time t; the low-resolution video frame set is input into a video super-resolution network for training to obtain a video super-resolution model in a real scene. In this embodiment, the value of N is 7;

[0051] The model training process of this embodiment is as follows:

[0052] S1, input the low-resolution video frame set into the smoothing module for denoising to obtain a smoothed video frame set in, represents the smoothed video frame at time t;

[0053] Among them, the smoothing module consists of three convolutional layers connected in series;

[0054] S2, input the smoothed video frame set into the feature extraction module for feature extraction, and obtain the video frame feature set {g t-N ,...,g t ,...,g t+N}, where g t Represents the video frame features at time t;

[0055] Among them, the feature extraction module includes two residual blocks connected in series;

[0056] S3, input the low-resolution video frame set into the first optical flow estimation network for reverse optical flow alignment to obtain the reverse aligned optical flow set {S t+N ,...,S t ,...,S t-N+1}, where S t Represents the reverse alignment optical flow from the video frame at time t to the video frame at time t-1;

[0057] Among them, the first optical flow estimation network in this embodiment is a SpyNet network.

[0058] S4, g t+N and g t+N After splicing, input the reverse attention enhancement module to obtain the reverse enhancement feature at time t+N

[0059] Among them, the reverse attention enhancement module of this embodiment is composed of 6 first residual channel attention modules RCAB connected in series, and any first residual channel attention module RCAB includes a first residual block, a second residual block, a first channel attention module CAI and a first fully connected layer connected in series in sequence, and the input end of the first residual block is further residually connected to the output end of the second residual block and the output end of the first fully connected layer respectively;

[0060] In this embodiment, the first residual block is composed of a first convolutional layer and a first activation layer connected in series, the second residual block is composed of a second convolutional layer and a second activation layer connected in series, and the input end of the first convolutional layer is residually connected to the output end of the second convolutional layer;

[0061] S5, the zero optical flow vector, S t+N 、g t+N-1 、 The zero optical flow feature is input into the first convolution alignment module at the same time to obtain the reverse alignment feature at time t+N-1

[0062] Among them, the first convolution alignment module disclosed in this embodiment is a DCA network;

[0063] S6, will After concatenating with gt+N-1, it is input into the reverse attention enhancement module to obtain the reverse enhancement feature at time t+N-1.

[0064] S7, S t+N 、S t+N-1 、g t+N-2 、 and At the same time, input the first convolution alignment module to obtain the reverse alignment feature at time t+N-2

[0065] S8, will and g t+N-2 After splicing, input the reverse attention enhancement module to obtain the reverse enhancement feature at time t+N-2

[0066] S9, let t+Ni=t+Ni-1, repeat S7-S8 to get the reverse enhancement feature set

[0067] S10, input the low-resolution video frame set into the second optical flow estimation network for forward optical flow alignment, and obtain the forward aligned optical flow set {f t-N ,...,f t ,...,f t+N-1}, where f t Represents the forward alignment optical flow from the video frame at time t to the video frame at time t+1:

[0068] The second optical flow estimation network in this embodiment is a SpyNet network;

[0069] S11, Will g t-N and g t-N After splicing, input the positive attention enhancement module to obtain the positive enhancement features at time tN

[0070] Among them, the forward attention enhancement module of this embodiment is composed of 6 second residual channel attention modules RCAB connected in series, and any of the second residual channel attention modules RCAB includes a third residual block, a fourth residual block, a second channel attention module CAI and a second fully connected layer connected in series in sequence, and the input end of the third residual block is also residually connected with the output end of the fourth residual block and the output end of the second fully connected layer.

[0071] The third residual block of this embodiment is composed of a third convolutional layer and a third activation layer connected in series, the fourth residual block is composed of a fourth convolutional layer and a fourth activation layer connected in series, and the input end of the third convolutional layer is residually connected to the output end of the fourth convolutional layer;

[0072] S12, the zero optical flow vector, f t-N 、g t-N+1 、 The zero optical flow feature is input into the second convolution alignment module at the same time to obtain the forward alignment feature at time t-N+1

[0073] The second convolutional alignment module in this embodiment is a DCA network;

[0074] S13, will and g t-N+1 After splicing, input the positive attention enhancement module to obtain the positive enhancement features at time t-N+1

[0075] S14, f t-N 、、f t-N+1 、g t-N+2 、 and At the same time, the second convolutional alignment module is input to obtain the forward alignment feature at time t-N+2

[0076] S15, will and g t-N+2 After splicing, input the positive attention enhancement module to obtain the positive enhancement features at time t-N+2

[0077] S16, let t-N+i=t-N+i+1, repeat S14-S15, and get the positive enhancement feature set

[0078] S17, for positive enhancement feature set and smooth video frame sets After upsampling respectively, the results are added together to obtain the video super-resolution results.

[0079] Example 2

[0080] This embodiment provides a video super-resolution imaging method in a real scene, including the following steps:

[0081] Step 1: obtain a low-resolution video and segment it to obtain a low-resolution video frame set;

[0082] Step 2: Input the low-resolution video frame set obtained in step 1 into a video super-resolution model in a real scene to obtain a video super-resolution result;

[0083] The real-scene video super-resolution model is a real-scene video super-resolution model constructed by the real-scene video super-resolution model construction method disclosed in Example 1.

[0084] Example 3

[0085] This embodiment also provides a video super-resolution imaging system for real scenes, including an acquisition and preprocessing module and a video super-resolution module;

[0086] The acquisition and preprocessing module is used to acquire low-resolution video and segment it to obtain a low-resolution video frame set;

[0087] The video super-resolution module inputs the low-resolution video frame set into the video super-resolution model under the real scene to obtain the video super-resolution result;

[0088] The real-scene video super-resolution model is a real-scene video super-resolution model constructed by the real-scene video super-resolution model construction method disclosed in Example 1.

[0089] Comparative Example 1

[0090] Table 1 shows the LPIPS, SSIM, and PSNR values ​​of our method on the RealVSR test set with a 2x over-resolution, with red representing the best result and blue representing the suboptimal result. As can be seen from Table 1, our method achieves the best LPIPS and SSIM values, while its PSNR is slightly inferior to the state-of-the-art method.

[0091] Table 1

[0092]

[0093]

[0094] In order to make a more intuitive comparison of perceptual quality, Figure 2 The perceptual quality comparison diagram of the proposed method and the existing video super-resolution method is shown. The perceptual indicators of the proposed method are the best.

[0095] Comparative Example 2

[0096] Figure 3 and Figure 4 The video super-resolution results obtained by testing this method and the existing video super-resolution method on the real scene dataset RealVSR are given. Figure 3 The data is taken from the 472th sequence of the RealVSR dataset. Figure 4 The data is taken from sequence 245 of the RealVSR dataset. It is worth noting that both the LR and HR video frames in the RealVSR dataset are sized at 1024×512. LR frames are equivalently treated as HR frames downsampled by a factor of 2. This method uses bilinear interpolation downsampling to downsample the HR frames to 512×256.

[0097] Depend on Figure 3 It can be seen that the high-rise building area marked by the ellipse can well reflect the significant advantages of this method in recovering structure, texture, high-frequency details and overcoming blur artifacts. Moreover, the edges and structures of white buildings are complete and clear, with few blur artifacts. Compared with other methods, Figure 4 The numbers and letters in the license plate are more continuous, smooth and complete.

[0098] The present application is described with reference to the flowcharts and / or block diagrams of the methods, systems, and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0099] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0100] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

Claims

1. A method for constructing a video super-resolution model in a real scene, characterized by: The following steps are involved: Obtain a low-resolution video frame set, wherein the low-resolution video frame set includes 2N+1 consecutive low-resolution video frames {x t-N ,...,x t ,...,x t+N }, where x t represents a low-resolution video frame at time t; the low-resolution video frame set is input into a video super-resolution network and trained to obtain a video super-resolution model in a real scene. The training process is as follows: S1, input the low-resolution video frame set into the smoothing module for denoising to obtain a smoothed video frame set in, represents the smoothed video frame at time t; S2, input the smoothed video frame set into the feature extraction module for feature extraction, and obtain the video frame feature set {g t-N ,...,g t ,...,g t+N }, where g t Represents the video frame features at time t; S3, input the low-resolution video frame set into the first optical flow estimation network for reverse optical flow alignment to obtain the reverse aligned optical flow set {S t+N ,...,S t ,...,S t-N+1 }, where S t Represents the reverse alignment optical flow from the video frame at time t to the video frame at time t-1; S4, g t+N and g t+N After splicing, input the reverse attention enhancement module to obtain the reverse enhancement feature at time t+N S5, the zero optical flow vector, S t+N 、g t+N-1 、 The zero optical flow feature is input into the first convolution alignment module at the same time to obtain the reverse alignment feature at time t+N-1 S6, will and g t+N-1 After splicing, input the reverse attention enhancement module to obtain the reverse enhancement feature at time t+N-1 S7, S t+N 、S t+N-1 、g t+N-2 、 and At the same time, input the first convolution alignment module to obtain the reverse alignment feature at time t+N-2 S8, will and g t+N-2 After splicing, input the reverse attention enhancement module to obtain the reverse enhancement feature at time t+N-2 S9, let t+Ni=t+Ni-1, repeat S7-S8 to get the reverse enhancement feature set S10, input the low-resolution video frame set into the second optical flow estimation network for forward optical flow alignment, and obtain the forward aligned optical flow set {f t-N ,...,f t ,...,f t+N-1 }, where f t Represents the forward alignment optical flow from the video frame at time t to the video frame at time t+1; S11, Will g t-N and g t-N After splicing, input the positive attention enhancement module to obtain the positive enhancement features at time tN S12, the zero optical flow vector, f t-N 、g t-N+1 、 The zero optical flow feature is input into the second convolution alignment module at the same time to obtain the forward alignment feature at time t-N+1 S13, will and g t-N+1 After splicing, input the positive attention enhancement module to obtain the positive enhancement features at time t-N+1 S14, f t-N 、、f t-N+1 、g t-N+2 、 and At the same time, the second convolutional alignment module is input to obtain the forward alignment feature at time t-N+2 S15, will and g t-N+2 After splicing, input the positive attention enhancement module to obtain the positive enhancement features at time t-N+2 S16, let t-N+i=t-N+i+1, repeat S14-S15, and get the positive enhancement feature set S17, for positive enhancement feature set and smooth video frame sets After upsampling respectively, the results are added together to obtain the video super-resolution results.

2. The method for constructing a video super-resolution model in a real scene according to claim 1, wherein: The smoothing module consists of three convolutional layers connected in series.

3. The method for constructing a video super-resolution model in a real scene according to claim 1, wherein: The feature extraction module includes two residual blocks connected in series.

4. The method for constructing a video super-resolution model in a real scene according to claim 1, wherein: The reverse attention enhancement module is composed of 6 first residual channel attention modules RCAB connected in series, and any of the first residual channel attention modules RCAB includes a first residual block, a second residual block, a first channel attention module CAI and a first fully connected layer connected in series in sequence, and the input end of the first residual block is also residually connected with the output end of the second residual block and the output end of the first fully connected layer respectively.

5. The method for constructing a video super-resolution model in a real scene according to claim 1, wherein: The forward attention enhancement module is composed of 6 second residual channel attention modules RCAB connected in series, and any of the second residual channel attention modules RCAB includes a third residual block, a fourth residual block, a second channel attention module CAI and a second fully connected layer connected in series in sequence. The input end of the third residual block is also residually connected with the output end of the fourth residual block and the output end of the second fully connected layer respectively.

6. The method for constructing a video super-resolution model in a real scene according to claim 1, wherein: The first optical flow estimation network and the second optical flow estimation network are both SpyNet networks.

7. The method for constructing a video super-resolution model in a real scene according to claim 1, wherein: The first convolution alignment module and the second convolution alignment module are both DCA networks.

8. The method for constructing a video super-resolution model in a real scene according to any one of claims 1 to 7, characterized in that: The value of N is 7.

9. A video super-resolution imaging method in a real scene, characterized by: The following steps are involved: Step 1: obtain a low-resolution video and segment it to obtain a low-resolution video frame set; Step 2: Input the low-resolution video frame set obtained in step 1 into a video super-resolution model in a real scene to obtain a video super-resolution result; The real-scene video super-resolution model is a real-scene video super-resolution model constructed by the real-scene video super-resolution model construction method according to any one of claims 1 to 8.

10. A video super-resolution imaging system in real scenes, characterized by: Including acquisition and preprocessing module and video super-resolution module; The acquisition and preprocessing module is used to acquire low-resolution video and segment it to obtain a low-resolution video frame set; The video super-resolution module inputs the low-resolution video frame set into a video super-resolution model under a real scene to obtain a video super-resolution result; The real-scene video super-resolution model is a real-scene video super-resolution model constructed by the real-scene video super-resolution model construction method according to any one of claims 1 to 8.

Images