Video super-resolution model construction method and system and video super-resolution imaging method and system

Abstract

The invention discloses a video super-resolution model construction method and system, and a video super-resolution imaging method and system, which can eliminate interference in a real scene video, learn a complex mapping relation between the super-resolution and low resolution of a video frame in a real scene, circularly propagate enhanced backward features / forward features in a backward / forward manner, and improve the resolution of the video frame in the real scene. And a video super-resolution result with clear structure and texture, rich details and pleasant vision is generated. The experimental result on a real scene video super-resolution RealVSR data set shows that the perception indexes SSIM and LPIPS of the RBFFEN are ahead of the most advanced video super-resolution method.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to video super-resolution model construction in real scenes, and a video super-resolution imaging method and system. Background Art

[0002] The video super-resolution task refers to reconstructing high-resolution video frames using multiple low-resolution video frames. Video super-resolution technology is widely used in video surveillance, medical image diagnosis, intelligent security and distance education. Unlike image super-resolution, video super-resolution can use the temporal and spatial correlation between video sequence frames to increase the details and clarity of reconstructed high-resolution videos. Existing classic video super-resolution methods are all trained on benchmark datasets. The trained models cannot simulate the complex degradation process between super-resolution and low-resolution video frames in real scenes well, and perform poorly for real-scene video super-resolution. At present, there are few studies on real-scene video super-resolution. In 2023, some researchers proposed a real-scene video super-resolution model, which achieved good results on real-scene video super-resolution datasets, but still has shortcomings such as unclear structure and texture, lack of high-frequency details, and poor visual effects. Summary of the invention

[0003] The purpose of the present invention is to provide a video super-resolution model construction, a video super-resolution imaging method and a system in a real scene, which solves the problems existing in the prior art that the structure and texture of the video super-resolution results in the real scene are unclear, lack of high-frequency details, and the visual effect is poor.

[0004] In order to achieve the above technical effects, the technical solution adopted by the present invention is:

[0005] A method for constructing a video super-resolution model in a real scene comprises the following steps: obtaining a low-resolution video frame set, the low-resolution video frame set comprising 2N+1 consecutive low-resolution video frames {x t-N , ..., x t , ..., x t+N}, where x t represents the low-resolution video frame at time t; the low-resolution video frame set is input into the video super-resolution network to train the video super-resolution model in the real scene. The training process is as follows:

[0006] S1, input the low-resolution video frame set into the smoothing module for denoising to obtain a smoothed video frame set in, represents the smoothed video frame at time t;

[0007] S2, input the smoothed video frame set into the feature extraction module for feature extraction, and obtain the video frame feature set {gt-N , ..., g t , ..., g t+N}, where g t Represents the video frame features at time t;

[0008] S3, input the low-resolution video frame set into the first optical flow estimation network for reverse optical flow alignment to obtain the reverse aligned optical flow set {S t+N , ..., S t , ..., S t-N+1}, where S t Represents the reverse alignment optical flow from the video frame at time t to the video frame at time t-1;

[0009] S4, g t+N and g t+N After splicing, input the reverse attention enhancement module to obtain the reverse enhancement features at time t+N

[0010] S5, the zero optical flow vector, S t+N , g t+N-1 , The zero optical flow feature is input into the first convolution alignment module at the same time to obtain the reverse alignment feature at time t+N-1

[0011] S6, will and g t+N-1 After splicing, input the reverse attention enhancement module to obtain the reverse enhancement feature at time t+N-1

[0012] S7, S t+N , S t+N-1 , g t+N-2 , and At the same time, input the first convolution alignment module to obtain the reverse alignment feature at time t+N-2

[0013] S8, After being concatenated with gt+N-2, it is input into the reverse attention enhancement module to obtain the reverse enhancement features at time t+N-2.

[0014] S9, let t+Ni=t+Ni-1, repeat S7-S8, and get the reverse enhancement feature set

[0015] S10, input the low-resolution video frame set into the second optical flow estimation network for forward optical flow alignment, and obtain the forward aligned optical flow set {f t-N , ..., f t , ..., ft+N-1}, where f t Represents the forward alignment optical flow from the video frame at time t to the video frame at time t+1;

[0016] S11, G t-N and g t-N After splicing, input the positive attention enhancement module to obtain the positive enhancement features at time tN

[0017] S12, the zero optical flow vector, f t-N , g t-N+1 , The zero optical flow feature is input into the second convolutional alignment module at the same time to obtain the forward alignment feature at time t-N+1

[0018] S13, and g t-N+1 After splicing, input the positive attention enhancement module to obtain the positive enhancement features at time t-N+1

[0019] S14, f t-N 、、f t-N+1 , g t-N+2 , and At the same time, the second convolutional alignment module is input to obtain the forward alignment feature at time t-N+2

[0020] S15, and g t-N+2 After splicing, input the positive attention enhancement module to obtain the positive enhancement features at time t-N+2

[0021] S16, let t-N+i=t-N+i+1, repeat S14-S15, and get the positive enhancement feature set

[0022] S17, for positive enhancement feature set and smooth video frame sets After upsampling respectively, they are added together to obtain the video super-resolution result.

[0023] Preferably, the smoothing module comprises three convolutional layers connected in series.

[0024] Preferably, the feature extraction module comprises two residual blocks connected in series.

[0025] Preferably, the reverse attention enhancement module is composed of 6 first residual channel attention modules RCAB connected in series, and any first residual channel attention module RCAB includes a first residual block, a second residual block, a first channel attention module CAI and a first fully connected layer connected in series in sequence, and the input end of the first residual block is also residually connected to the output end of the second residual block and the output end of the first fully connected layer, respectively.

[0026] Preferably, the forward attention enhancement module is composed of 6 second residual channel attention modules RCAB connected in series, and any second residual channel attention module RCAB includes a third residual block, a fourth residual block, a second channel attention module CAI and a second fully connected layer connected in series in sequence, and the input end of the third residual block is also residually connected to the output end of the fourth residual block and the output end of the second fully connected layer, respectively.

[0027] Preferably, the first optical flow estimation network and the second optical flow estimation network are both SpyNet networks.

[0028] Preferably, the first convolution alignment module and the second convolution alignment module are both DCA networks.

[0029] Preferably, the value of N is 7.

[0030] A video super-resolution imaging method in a real scene comprises the following steps:

[0031] Step 1, obtaining a low-resolution video and segmenting it to obtain a low-resolution video frame set;

[0032] Step 2, inputting the low-resolution video frame set obtained in step 1 into a video super-resolution model in a real scene to obtain a video super-resolution result;

[0033] Among them, the video super-resolution model in the real scene is the video super-resolution model in the real scene constructed by the video super-resolution model construction method in the real scene disclosed in this application.

[0034] A video super-resolution imaging system in a real scene, comprising an acquisition and preprocessing module and a video super-resolution module;

[0035] The acquisition and preprocessing module is used to acquire low-resolution video and segment it to obtain a low-resolution video frame set;

[0036] The video super-resolution module inputs the low-resolution video frame set into the video super-resolution model in the real scene to obtain the video super-resolution result;

[0037] Among them, the video super-resolution model in the real scene is the video super-resolution model in the real scene constructed by the video super-resolution model construction method in the real scene disclosed in this application.

[0038] The above technical solution has the following beneficial effects:

[0039] (1) The real-scene video super-resolution model constructed by the method of the present invention can remove interference in real-scene videos, learn the complex mapping relationship between super-resolution and low-resolution video frames in real scenes, and propagate the enhanced backward features / forward features in a backward / forward loop to generate video super-resolution results with clear structure and texture, rich details, and visual pleasure. Experimental results on the real-scene video super-resolution RealVSR dataset show that the perceptual indicators SSIM and LPIPS of RBFFEN are ahead of the most advanced video super-resolution methods.

[0040] (2) The video super-resolution imaging method and system in real scenes of the present invention have a high degree of automation, and the video super-resolution model in real scenes does not require manual intervention after training. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a flow chart of the video super-resolution imaging method in a real scene of the present invention;

[0042] Figure 2 is a plane diagram comparing the perceived quality of the present invention and the existing video super-resolution method;

[0043] Figure 3 This is a video super-resolution result diagram obtained by the present invention in the RealVSR set test;

[0044] Figure 4 This is a video super-resolution result diagram obtained by testing the present invention on the RealVSR set.

[0045] The present invention is described in detail below with reference to the accompanying drawings and specific implementation methods. DETAILED DESCRIPTION

[0046] The present invention is described in detail below in conjunction with the accompanying drawings and embodiments so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when the detailed description of known functions and designs may dilute the main content of the present invention, these descriptions will be omitted here.

[0047] Other structures and functions of the method of the present invention are known to those skilled in the art and will not be described in detail to reduce redundancy.

[0048] Example 1

[0049] In this embodiment, a method for constructing a video super-resolution model in a real scene is disclosed, comprising the following steps:

[0050] Obtain a low-resolution video frame set, wherein the low-resolution video frame set includes 2N+1 consecutive low-resolution video frames {x t-N , ..., x t , ..., x t+N}, where x t represents a low-resolution video frame at time t; the low-resolution video frame set is input into a video super-resolution network for training to obtain a video super-resolution model in a real scene, and the value of N in this embodiment is 7;

[0051] The model training process of this embodiment is as follows:

[0052] S1, input the low-resolution video frame set into the smoothing module for denoising to obtain a smoothed video frame set in, represents the smoothed video frame at time t;

[0053] Among them, the smoothing module includes 3 convolutional layers connected in series;

[0054] S2, input the smoothed video frame set into the feature extraction module for feature extraction, and obtain the video frame feature set {g t-N , ..., g t , ..., g t+N}, where g t Represents the video frame features at time t;

[0055] Among them, the feature extraction module includes 2 residual blocks connected in series;

[0056] S3, input the low-resolution video frame set into the first optical flow estimation network for reverse optical flow alignment to obtain the reverse aligned optical flow set {S t+N , ..., S t , ..., S t-N+1}, where S t Represents the reverse alignment optical flow from the video frame at time t to the video frame at time t-1;

[0057] Among them, the first optical flow estimation network of this embodiment is a SpyNet network.

[0058] S4, g t+N and g t+N After splicing, input the reverse attention enhancement module to obtain the reverse enhancement features at time t+N

[0059] Among them, the reverse attention enhancement module of this embodiment is composed of 6 first residual channel attention modules RCAB connected in series, and any first residual channel attention module RCAB includes a first residual block, a second residual block, a first channel attention module CAI and a first fully connected layer connected in series in sequence, and the input end of the first residual block is also residually connected to the output end of the second residual block and the output end of the first fully connected layer respectively;

[0060] Among them, the first residual block of this embodiment is composed of a first convolutional layer and a first activation layer connected in series, the second residual block is composed of a second convolutional layer and a second activation layer connected in series, and the input end of the first convolutional layer is residually connected to the output end of the second convolutional layer;

[0061] S5, the zero optical flow vector, S t+N , g t+N-1 , The zero optical flow feature is input into the first convolution alignment module at the same time to obtain the reverse alignment feature at time t+N-1

[0062] Among them, the first convolution alignment module disclosed in this embodiment is a DCA network;

[0063] S6, will After being concatenated with gt+N-1, it is input into the reverse attention enhancement module to obtain the reverse enhancement feature at time t+N-1.

[0064] S7, S t+N , S t+N-1 , g t+N-2 , and At the same time, input the first convolution alignment module to obtain the reverse alignment feature at time t+N-2

[0065] S8, and g t+N-2 After splicing, input the reverse attention enhancement module to obtain the reverse enhancement features at time t+N-2

[0066] S9, let t+Ni=t+Ni-1, repeat S7-S8, and get the reverse enhancement feature set

[0067] S10, input the low-resolution video frame set into the second optical flow estimation network for forward optical flow alignment, and obtain the forward aligned optical flow set {f t-N , ..., f t , ..., f t+N-1}, where f t Represents the forward alignment optical flow of the video frame at time t to the video frame at time t+1:

[0068] The second optical flow estimation network in this embodiment is a SpyNet network;

[0069] S11, G t-N and g t-N After splicing, input the positive attention enhancement module to obtain the positive enhancement features at time tN

[0070] Among them, the forward attention enhancement module of this embodiment is composed of 6 second residual channel attention modules RCAB connected in series, and any of the second residual channel attention modules RCAB includes a third residual block, a fourth residual block, a second channel attention module CAI and a second fully connected layer connected in series in sequence, and the input end of the third residual block is also residually connected with the output end of the fourth residual block and the output end of the second fully connected layer, respectively.

[0071] Among them, the third residual block of this embodiment is composed of a third convolutional layer and a third activation layer connected in series, the fourth residual block is composed of a fourth convolutional layer and a fourth activation layer connected in series, and the input end of the third convolutional layer is residually connected to the output end of the fourth convolutional layer;

[0072] S12, the zero optical flow vector, f t-N , g t-N+1 , The zero optical flow feature is input into the second convolutional alignment module at the same time to obtain the forward alignment feature at time t-N+1

[0073] Among them, the second convolution alignment module of this embodiment is a DCA network;

[0074] S13, and g t-N+1 After splicing, input the positive attention enhancement module to obtain the positive enhancement features at time t-N+1

[0075] S14, f t-N 、、f t-N+1 , g t-N+2 , and At the same time, the second convolutional alignment module is input to obtain the forward alignment feature at time t-N+2

[0076] S15, and g t-N+2 After splicing, input the positive attention enhancement module to obtain the positive enhancement features at time t-N+2

[0077] S16, let t-N+i=t-N+i+1, repeat S14-S15, and get the positive enhancement feature set

[0078] S17, for positive enhancement feature set and smooth video frame sets After upsampling respectively, they are added together to obtain the video super-resolution result.

[0079] Example 2

[0080] This embodiment provides a video super-resolution imaging method in a real scene, comprising the following steps:

[0081] Step 1, obtaining a low-resolution video and segmenting it to obtain a low-resolution video frame set;

[0082] Step 2, inputting the low-resolution video frame set obtained in step 1 into a video super-resolution model in a real scene to obtain a video super-resolution result;

[0083] The video super-resolution model in the real scene is a video super-resolution model in the real scene constructed by the method for constructing a video super-resolution model in the real scene disclosed in Example 1.

[0084] Example 3

[0085] This embodiment also provides a video super-resolution imaging system in a real scene, including an acquisition and preprocessing module and a video super-resolution module;

[0086] The acquisition and preprocessing module is used to acquire low-resolution video and segment it to obtain a low-resolution video frame set;

[0087] The video super-resolution module inputs the low-resolution video frame set into the video super-resolution model in the real scene to obtain the video super-resolution result;

[0088] The video super-resolution model in the real scene is a video super-resolution model in the real scene constructed by the method for constructing a video super-resolution model in the real scene disclosed in Example 1.

[0089] Comparative Example 1

[0090] Table 1 shows the LPIPS, SSIM, and PSNR values ​​of the method with 2x super-resolution on the RealVSR test set, where red represents the best result and blue represents the second-best result. As can be seen from Table 1, the LPIPS and SSIM values ​​of the method are the best, and the PSNR value is slightly inferior to the most advanced method.

[0091] Table 1

[0092]

[0093]

[0094] In order to make a more intuitive perceptual quality comparison, Figure 2 The perceptual quality comparison diagram of the proposed method and the existing video super-resolution method is shown. The perceptual index of the proposed method reaches the best.

[0095] Comparative Example 2

[0096] Figure 3 and Figure 4 The video super-resolution results of this method and the existing video super-resolution methods tested on the real scene dataset RealVSR are given. Figure 3 The data is taken from the 472th sequence of the RealVSR dataset. Figure 4 The data is taken from the 245th sequence of the RealVSR dataset. It is worth noting that the sizes of the LR and HR video frames in the RealVSR dataset are both 1024×512. The LR video frame is equivalently regarded as the result of 2 times downsampling of the HR video frame. This method uses the bilinear interpolation downsampling method to downsample the HR video frame and change its size to 512×256.

[0097] Depend on Figure 3 It can be seen that the high-rise building area marked by the ellipse can well reflect the significant advantages of this method in recovering structure, texture, high-frequency details and overcoming blur artifacts. Moreover, the edges and structures of white buildings are complete and clear, with few blur artifacts. Compared with other methods, Figure 4 The numbers and letters in the license plate are more continuous, smooth and complete.

[0098] The present application is described with reference to the flowcharts and / or block diagrams of the methods, systems, and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0099] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0100] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

Claims

1. A method for constructing a video super-resolution model in a real scene, characterized in that: The following steps are involved: Obtain a low-resolution video frame set, wherein the low-resolution video frame set includes 2N+1 consecutive low-resolution video frames {x t-N ,...,x t ,...,x t+N }, where x t represents a low-resolution video frame at time t; the low-resolution video frame set is input into and trained into a video super-resolution network to obtain a video super-resolution model in a real scene, and the training process is specifically as follows: S1, input the low-resolution video frame set into the smoothing module for denoising to obtain a smoothed video frame set in, represents the smoothed video frame at time t; S2, input the smoothed video frame set into the feature extraction module for feature extraction, and obtain the video frame feature set {g t-N ,...,g t ,...,g t+N }, where g t Represents the video frame features at time t; S3, input the low-resolution video frame set into the first optical flow estimation network for reverse optical flow alignment to obtain the reverse aligned optical flow set {S t+N ,...,S t ,...,S t-N+1 }, where S t Represents the reverse alignment optical flow from the video frame at time t to the video frame at time t-1; S4, g t+N and g t+N After splicing, input the reverse attention enhancement module to obtain the reverse enhancement features at time t+N S5, the zero optical flow vector, S t+N , g t+N-1 , The zero optical flow feature is input into the first convolution alignment module at the same time to obtain the reverse alignment feature at time t+N-1 S6, will and g t+N-1 After splicing, input the reverse attention enhancement module to obtain the reverse enhancement feature at time t+N-1 S7, S t+N , S t+N-1 , g t+N-2 , and At the same time, input the first convolution alignment module to obtain the reverse alignment feature at time t+N-2 S8, and g t+N-2 After splicing, input the reverse attention enhancement module to obtain the reverse enhancement features at time t+N-2 S9, let t+Ni=t+Ni-1, repeat S7-S8, and get the reverse enhancement feature set S10, input the low-resolution video frame set into the second optical flow estimation network for forward optical flow alignment, and obtain the forward aligned optical flow set {f t-N ,...,f t ,...,f t+N-1 }, where f t Represents the forward alignment optical flow from the video frame at time t to the video frame at time t+1; S11, G t-N and g t-N After splicing, input the positive attention enhancement module to obtain the positive enhancement features at time tN S12, the zero optical flow vector, f t-N , g t-N+1 , The zero optical flow feature is input into the second convolutional alignment module at the same time to obtain the forward alignment feature at time t-N+1 S13, and g t-N+1 After splicing, input the positive attention enhancement module to obtain the positive enhancement features at time t-N+1 S14, f t-N 、、f t-N+1 , g t-N+2 , and At the same time, the second convolutional alignment module is input to obtain the forward alignment feature at time t-N+2 S15, and g t-N+2 After splicing, input the positive attention enhancement module to obtain the positive enhancement features at time t-N+2 S16, let t-N+i=t-N+i+1, repeat S14-S15, and get the positive enhancement feature set S17, for positive enhancement feature set and smooth video frame sets After upsampling respectively, they are added together to obtain the video super-resolution result.

2. The method for constructing a video super-resolution model in a real scene as claimed in claim 1, characterized in that: The smoothing module consists of three convolutional layers connected in series.

3. The method for constructing a video super-resolution model in a real scene as described in claim 1, characterized in that: The feature extraction module includes two residual blocks connected in series.

4. The method for constructing a video super-resolution model in a real scene as claimed in claim 1, characterized in that: The reverse attention enhancement module is composed of 6 first residual channel attention modules RCAB connected in series, and any of the first residual channel attention modules RCAB includes a first residual block, a second residual block, a first channel attention module CAI and a first fully connected layer connected in series in sequence, and the input end of the first residual block is also residually connected to the output end of the second residual block and the output end of the first fully connected layer, respectively.

5. The method for constructing a video super-resolution model in a real scene as claimed in claim 1, characterized in that: The forward attention enhancement module is composed of 6 second residual channel attention modules RCAB connected in series, and any of the second residual channel attention modules RCAB includes a third residual block, a fourth residual block, a second channel attention module CAI and a second fully connected layer connected in series in sequence, and the input end of the third residual block is also residually connected to the output end of the fourth residual block and the output end of the second fully connected layer, respectively.

6. The method for constructing a video super-resolution model in a real scene as claimed in claim 1, characterized in that: The first optical flow estimation network and the second optical flow estimation network are both SpyNet networks.

7. The method for constructing a video super-resolution model in a real scene as claimed in claim 1, characterized in that: The first convolution alignment module and the second convolution alignment module are both DCA networks.

8. The method for constructing a video super-resolution model in a real scene according to any one of claims 1 to 7, characterized in that: The value of N is 7.

9. A video super-resolution imaging method in a real scene, characterized in that: The following steps are involved: Step 1, obtaining a low-resolution video and segmenting it to obtain a low-resolution video frame set; Step 2, inputting the low-resolution video frame set obtained in step 1 into a video super-resolution model in a real scene to obtain a video super-resolution result; Among them, the video super-resolution model in the real scene is a video super-resolution model in the real scene constructed by the video super-resolution model construction method in the real scene described in any one of claims 1-8.

10. A video super-resolution imaging system in a real scene, characterized in that: Including acquisition and preprocessing module and video super-resolution module; The acquisition and preprocessing module is used to acquire low-resolution video and segment it to obtain a low-resolution video frame set; The video super-resolution module inputs the low-resolution video frame set into a video super-resolution model in a real scene to obtain a video super-resolution result; Among them, the video super-resolution model in the real scene is a video super-resolution model in the real scene constructed by the video super-resolution model construction method in the real scene described in any one of claims 1-8.

Images