Deep learning based video super-resolution reconstruction method, system and medium

Abstract

The application relates to a deep learning-based video super-resolution reconstruction method and system and a medium, in particular to the technical field of video processing. The method comprises the following steps: inputting each frame of a to-be-processed video into a super-resolution model to obtain a super-resolution image corresponding to each frame of the to-be-processed video; and obtaining a super-resolution video corresponding to the to-be-processed video according to the super-resolution images corresponding to each frame of the to-be-processed video, wherein the super-resolution model is obtained by training a BasicVSR model with a to-be-trained video as input, a super-resolution video corresponding to the to-be-trained video as output and a minimum frequency loss function as a target; and the forward branch and the backward branch of the BasicVSR model both comprise a GDFN module. The application can improve the quality of high-resolution video images.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to a video super-resolution reconstruction method, system and medium based on deep learning. Background Art

[0002] Resolution is a set of performance parameters used to evaluate the richness of detailed information contained in an image, including temporal resolution, spatial resolution, and color resolution, reflecting the imaging system's ability to actually reflect the detailed information of an object. Compared to low-resolution images, high-resolution images usually contain greater pixel density, richer texture details, and higher reliability. However, in reality, due to the constraints of many factors such as the acquisition equipment and environment, network transmission media and bandwidth, and the video degradation model itself, it is usually not possible to directly obtain an ideal high-resolution image with sharp edges and no block blur. The most direct way to improve image resolution is to improve the optical hardware in the acquisition system. However, since the manufacturing process is difficult to improve significantly and the manufacturing cost is very high, it is often too costly to physically solve the problem of low image resolution.

[0003] Video super-resolution reconstruction technology involves using specific algorithms to restore a given low-resolution image into a corresponding high-resolution video. Compared to image super-resolution, video super-resolution can leverage information from multiple adjacent frames to achieve better super-resolution results. Traditional super-resolution algorithms, such as interpolation, can cause edges in high-resolution video images to blur, resulting in poor results. Summary of the Invention

[0004] The purpose of the present invention is to provide a video super-resolution reconstruction method, system and medium based on deep learning, which can improve the quality of high-resolution video images.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A video super-resolution reconstruction method based on deep learning, comprising:

[0007] Construct a super-resolution model; the super-resolution model is obtained by training a BasicVSR model with an image corresponding to each frame of the video to be trained as input and a super-resolution image corresponding to each frame of the video to be trained as output, with the frequency loss function minimized; the forward branch and the backward branch of the BasicVSR model both include a GDFN module;

[0008] Get the video to be processed;

[0009] Inputting each frame image of the video to be processed into the super-resolution model to obtain a super-resolution image corresponding to each frame image of the video to be processed;

[0010] A super-resolution video corresponding to the video to be processed is obtained according to the super-resolution images corresponding to each frame image of the video to be processed.

[0011] Optionally, the BasicVSR model includes a forward branch, a backward branch and an upsampling branch; the output end of the forward branch and the output end of the backward branch are both connected to the input end of the upsampling branch.

[0012] Optionally, the forward branch includes N forward propagation modules; the backward branch includes N backward propagation modules; the upsampling branch includes N upsampling propagation modules; N is a positive integer greater than 1;

[0013] The first input end of the i-th forward propagation module is connected to the first output end of the i-1-th forward propagation module; the second input end of the i-th forward propagation module is used to input the i-th frame image and the i-1-th frame image of the video to be processed; the first output end of the i-th forward propagation module is connected to the first input end of the i+1-th forward propagation module; the second output end of the i-th forward propagation module is connected to the first input end of the i-th upsampling module;

[0014] The first input end of the i-th backward propagation module is connected to the first output end of the i+1-th backward propagation module; the second input end of the i-th backward propagation module is used to input the i-th frame image and the i-1-th frame image of the video to be processed; the first output end of the i-th backward propagation module is connected to the first input end of the i-1-th backward propagation module; the second output end of the i-th backward propagation module is connected to the second input end of the i-th upsampling module.

[0015] Optionally, the forward propagation module and the backward propagation module both include an optical flow estimation module, a spatial warping module and a depth residual block, and the optical flow estimation module, the spatial warping module, the GDFN module and the depth residual block are connected in sequence.

[0016] Optionally, the frequency loss function is specifically:

[0017]

[0018] in, represents the frequency loss function, represents the image generated by inputting the video to be trained into the BasicVSR model, I represents the super-resolution image corresponding to the video to be trained, ∈ represents the first constant, α represents the second constant, Express Perform a fast Fourier transform, represents the fast Fourier transform of I.

[0019] A deep learning-based video super-resolution reconstruction system comprises:

[0020] A construction module is configured to construct a super-resolution model; the super-resolution model takes images corresponding to frames of a to-be-trained video as input, takes super-resolution images corresponding to the frames of the to-be-trained video as output, and is trained to obtain a BasicVSR model with a frequency loss function as the target;

[0021] An acquisition module is configured to acquire a to-be-processed video;

[0022] A super-resolution image determination module is configured to input images of frames of the to-be-processed video into the super-resolution model to obtain super-resolution images corresponding to the images of the frames of the to-be-processed video;

[0023] A super-resolution video determination module is configured to obtain a super-resolution video corresponding to the to-be-processed video according to the super-resolution images corresponding to the images of the frames of the to-be-processed video.

[0024] Optionally, the BasicVSR model comprises a forward branch, a backward branch, and an up-sampling branch; an output end of the forward branch and an output end of the backward branch are connected to an input end of the up-sampling branch.

[0025] Optionally, the forward branch comprises N forward propagation modules; the backward branch comprises N backward propagation modules; and the up-sampling branch comprises N up-sampling propagation modules; N is a positive integer greater than 1.

[0026] A first input end of an i-th forward propagation module is connected to a first output end of an i-1-th forward propagation module; a second input end of the i-th forward propagation module is configured to input an i-th image and an i-1-th image of the to-be-processed video; a first output end of the i-th forward propagation module is connected to a first input end of an i+1-th forward propagation module; and a second output end of the i-th forward propagation module is connected to a first input end of an i-th up-sampling module.

[0027] A first input end of an i-th backward propagation module is connected to a first output end of an i+1-th backward propagation module; a second input end of the i-th backward propagation module is configured to input an i-th image and an i-1-th image of the to-be-processed video; a first output end of the i-th backward propagation module is connected to a first input end of an i-1-th backward propagation module; and a second output end of the i-th backward propagation module is connected to a second input end of the i-th up-sampling module.

[0028] Optionally, the forward propagation module and the backward propagation module both include an optical flow estimation module, a spatial warping module and a depth residual block, and the optical flow estimation module, the spatial warping module, the GDFN module and the depth residual block are connected in sequence.

[0029] Optionally, a computer-readable storage medium stores a computer program, which, when executed by a processor, implements the deep learning-based video super-resolution reconstruction method as described above.

[0030] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects: the present invention uses the GDFN module to achieve better feature fusion effect, which can improve the quality of high-resolution video images. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0032] Figure 1 A flowchart of a video super-resolution reconstruction method based on deep learning provided by an embodiment of the present invention;

[0033] Figure 2 This is the specific architecture diagram of the BasicVSR model;

[0034] Figure 3 This is the specific structure diagram of the forward propagation module;

[0035] Figure 4 This is the specific structure diagram of the back-propagation module;

[0036] Figure 5 This is the specific structure diagram of the GDFN module;

[0037] Figure 6 This is the specific structural diagram of the video super-resolution system. DETAILED DESCRIPTION

[0038] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0039] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0040] With the rise of deep learning, video super-resolution technology based on deep learning is developing more and more rapidly. The present invention proposes a video super-resolution reconstruction method based on deep learning. The super-resolution model in the present invention uses a recurrent network architecture to propagate information between video frames, and uses a GDFN module to improve the effect of feature fusion. The frequency loss function is added to optimize the network so that the super-resolution model has the advantages of good performance, low parameter count and high computational efficiency.

[0041] The embodiment of the present invention provides a video super-resolution reconstruction method based on deep learning, such as Figure 1 Shown, including:

[0042] Step 101: Construct a super-resolution model; the super-resolution model is obtained by training a BasicVSR model with an image corresponding to each frame of the video to be trained as input and a super-resolution image corresponding to each frame of the video to be trained as output, with the frequency loss function minimized as the goal; the forward branch and the backward branch of the BasicVSR model both include a GDFN module.

[0043] Step 102: Obtain the video to be processed.

[0044] Step 103: Input each frame image of the video to be processed into the super-resolution model to obtain a super-resolution image corresponding to each frame image of the video to be processed.

[0045] Step 104: Obtain a super-resolution video corresponding to the video to be processed according to the super-resolution images corresponding to each frame of the video to be processed.

[0046] In practical applications, the BasicVSR model includes a forward branch, a backward branch, and an upsampling branch; the output end of the forward branch and the output end of the backward branch are both connected to the input end of the upsampling branch.

[0047] In practical applications, such as Figure 2 As shown, the forward branch includes N forward propagation modules; the backward branch includes N backward propagation modules; the upsampling branch includes N upsampling propagation modules; and N is a positive integer greater than 1.

[0048] The first input terminal of the i-th forward propagation module is connected to the first output terminal of the i-1-th forward propagation module, and is used to input the forward propagation features of the i-1-th frame image output by the i-1-th forward propagation module. The second input end of the i-th forward propagation module is used to input the i-th frame image x of the video to be processedi and the i-1th frame image x i-1 The first output terminal of the i-th forward propagation module is connected to the first input terminal of the i+1-th forward propagation module, and is used to output the forward propagation features of the i-th frame image output by the i-th forward propagation module. The second output terminal of the i-th forward propagation module is connected to the first input terminal of the i-th upsampling module

[0049] The first input terminal of the i-th backward propagation module is connected to the first output terminal of the i+1-th backward propagation module, and is used to input the backward propagation features of the i+1-th frame image output by the i+1-th backward propagation module. The second input end of the i-th backward propagation module is used to input the i-th frame image x of the video to be processed i and the i-1th frame image x i-1 The first output terminal of the i-th backward propagation module is connected to the first input terminal of the i-1-th backward propagation module, and is used to output the backward propagation features of the i-th frame image output by the i-th backward propagation module. The second output terminal of the i-th backward propagation module is connected to the second input terminal of the i-th upsampling module for outputting The output end of the i-th upsampling module outputs the super-resolution image hr corresponding to the i-th frame image i .

[0050] In practical applications, such as Figure 3 and Figure 4 As shown, the forward propagation module and the backward propagation module both include an optical flow estimation module, a spatial warping module and a depth residual block, and the optical flow estimation module, the spatial warping module, the GDFN module and the depth residual block are connected in sequence.

[0051] Taking the i-th forward propagation module as an example, the specific workflow of forward propagation is introduced: First, the optical flow estimation module calculates x i-1 with x i Forward optical flow information use right Perform spatial warping alignment to obtain the forward propagation features of the i-1th frame image aligned with the i-th frame image Then through the GDFN module with x i Fusion is performed to obtain the fused features Will Enter the depth residual block to calculate

[0052] Take the i-th back propagation module as an example to introduce the specific workflow of back propagation: first, calculate x i-1 and the back optical flow information of x i Use to perform spatial warping alignment to obtain the back propagation feature of the i+1-th frame image aligned with the i-th frame image After that, the GDFN module is used to fuse and x i to obtain the fused feature Send to the depth residual block to calculate

[0053] After that, fuse and to obtain the final feature map, and then perform upsampling through the pixel-shuffle technology, and then pass through the reconstruction network to obtain the final high-resolution video.

[0054] In actual application, the specific structure of the GDFN module is as shown in Figure 5 The feature fusion module GDFN uses depth-wise convolution to encode information from spatially adjacent pixel positions, which can be used to learn effective feature fusion. After the input features are normalized (Norm), they are evenly divided into two parts by channel, each of which is passed through a 1x1 convolution and a 3x3 convolution. One branch is activated by a GELU activation function, and then multiplied element-wise with the other branch. After a 1x1 convolution is used to restore the channel, the result is added to the original input to obtain the final result.

[0055] In actual application, the present application uses a frequency loss function to help obtain more image detail information. When using some commonly used loss functions, in order to reduce the loss value, the model tends to make the video image smoother. These detail parts often correspond to the high frequency part of the frequency signal, so by reducing the difference in frequency space through the frequency loss function, a clearer and sharper video is obtained. The frequency loss function is specifically:

[0056]

[0057] Wherein, represents the frequency loss function, represents the image generated by inputting the video to be trained into the BasicVSR model, I represents the super-resolution image corresponding to the video to be trained, and represents a first constant, and represents a second constant, represents the fast Fourier transform of ​​​represents the fast Fourier transform of I.

[0058] The embodiment of the present invention further provides a video super-resolution reconstruction system based on deep learning for the above method, including:

[0059] A construction module is used to construct a super-resolution model; the super-resolution model is obtained by training the BasicVSR model with the image corresponding to each frame of the video to be trained as input and the super-resolution image corresponding to each frame of the video to be trained as output, with the frequency loss function minimized as the goal; the forward branch and backward branch of the BasicVSR model both include the GDFN module.

[0060] The acquisition module is used to obtain the video to be processed.

[0061] The super-resolution image determination module is used to input each frame image of the video to be processed into the super-resolution model to obtain the super-resolution image corresponding to each frame image of the video to be processed.

[0062] A super-resolution video determination module is provided; and a super-resolution video corresponding to the video to be processed is obtained according to the super-resolution images corresponding to each frame image of the video to be processed.

[0063] In practical applications, the BasicVSR model includes a forward branch, a backward branch, and an upsampling branch; the output end of the forward branch and the output end of the backward branch are both connected to the input end of the upsampling branch.

[0064] In practical applications, the forward branch includes N forward propagation modules; the backward branch includes N backward propagation modules; the upsampling branch includes N upsampling propagation modules; and N is a positive integer greater than 1.

[0065] The first input end of the i-th forward propagation module is connected to the first output end of the i-1-th forward propagation module; the second input end of the i-th forward propagation module is used to input the i-th frame image and the i-1-th frame image of the video to be processed; the first output end of the i-th forward propagation module is connected to the first input end of the i+1-th forward propagation module; the second output end of the i-th forward propagation module is connected to the first input end of the i-th upsampling module.

[0066] The first input end of the i-th backward propagation module is connected to the first output end of the i+1-th backward propagation module; the second input end of the i-th backward propagation module is used to input the i-th frame image and the i-1-th frame image of the video to be processed; the first output end of the i-th backward propagation module is connected to the first input end of the i-1-th backward propagation module; the second output end of the i-th backward propagation module is connected to the second input end of the i-th upsampling module.

[0067] In practical applications, the forward propagation module and the backward propagation module both include an optical flow estimation module, a spatial warping module and a depth residual block, and the optical flow estimation module, the spatial warping module, the GDFN module and the depth residual block are connected in sequence.

[0068] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the deep learning-based video super-resolution reconstruction method described in the above embodiment.

[0069] The embodiment of the present invention further provides a video super-resolution system:

[0070] In order to better demonstrate the super-resolution performance of the model, the present invention uses the Open Neural Network Exchange (ONNX) to convert the model, so that the video super-resolution task can be performed in an environment where the model dependency library is not installed. Use pyqt to build the system interface. Figure 6 As shown in the figure, select is the video selection button. Click select to select the video file to be processed. OriginalVideo is the original video player, Modified Video is the super-resolution video player, model a, modelb and model c are super-resolution algorithm selection buttons. Press the button to use the corresponding super-resolution algorithm to process the video and play it to get the corresponding video.

[0071] This paper improves the BasicVSR model. Compared with the existing BasicVSR, the model of this paper uses the GDFN module to achieve better feature fusion effect and uses the frequency loss function to reduce the loss of high-frequency components in the super-resolution results.

[0072] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0073] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only intended to help understand the method and core concept of the present invention. At the same time, those skilled in the art will find that the specific implementation methods and application scopes may vary based on the concept of the present invention. In summary, the contents of this specification should not be construed as limiting the present invention.

Claims

1. A video super-resolution reconstruction method based on deep learning, characterized in that: include: Construct a super-resolution model; the super-resolution model is obtained by taking the image corresponding to each frame of the video to be trained as input and the super-resolution image corresponding to each frame of the video to be trained as output, and training the BasicVSR model with the goal of minimizing the frequency loss function; the forward branch and the backward branch of the BasicVSR model both include GDFN modules; the BasicVSR model includes a forward branch, a backward branch and an upsampling branch; the output end of the forward branch and the output end of the backward branch are both connected to the input end of the upsampling branch; the forward branch includes N forward propagation modules; the backward branch includes N backward propagation modules; the upsampling branch includes N upsampling propagation modules; N is a positive integer greater than 1; the forward propagation module and the backward propagation module both include an optical flow estimation module, a spatial warping module and a depth residual block, and the optical flow estimation module, the spatial warping module, the GDFN module and the depth residual block are connected in sequence; Get the video to be processed; Inputting each frame image of the video to be processed into the super-resolution model to obtain a super-resolution image corresponding to each frame image of the video to be processed; A super-resolution video corresponding to the video to be processed is obtained according to the super-resolution images corresponding to each frame image of the video to be processed.

2. The method for video super-resolution reconstruction based on deep learning according to claim 1, characterized in that: The first input end of the i-th forward propagation module is connected to the first output end of the i-1-th forward propagation module; the second input end of the i-th forward propagation module is used to input the i-th frame image and the i-1-th frame image of the video to be processed; the first output end of the i-th forward propagation module is connected to the first input end of the i+1-th forward propagation module; The second output terminal of the i-th forward propagation module is connected to the first input terminal of the i-th upsampling module; The first input end of the i-th backward propagation module is connected to the first output end of the i+1-th backward propagation module; the second input end of the i-th backward propagation module is used to input the i-th frame image and the i-1-th frame image of the video to be processed; the first output end of the i-th backward propagation module is connected to the first input end of the i-1-th backward propagation module; the second output end of the i-th backward propagation module is connected to the second input end of the i-th upsampling module.

3. The method for video super-resolution reconstruction based on deep learning according to claim 1, characterized in that: The frequency loss function is specifically: in, represents the frequency loss function, Indicates that the video to be trained is input into the image generated by the BasicVSR model, represents the super-resolution image corresponding to the video to be trained, represents the first constant, represents the second constant, Express Perform a fast Fourier transform, Express Perform a fast Fourier transform.

4. A video super-resolution reconstruction system based on deep learning, characterized in that: include: A construction module for constructing a super-resolution model; the super-resolution model is obtained by training the BasicVSR model with the image corresponding to each frame of the video to be trained as input and the super-resolution image corresponding to each frame of the video to be trained as output, with the frequency loss function minimized as the goal; the forward branch and the backward branch of the BasicVSR model both include GDFN modules; the BasicVSR model includes a forward branch, a backward branch and an upsampling branch; the output end of the forward branch and the output end of the backward branch are both connected to the input end of the upsampling branch; the forward branch includes N forward propagation modules; the backward branch includes N backward propagation modules; the upsampling branch includes N upsampling propagation modules; N is a positive integer greater than 1; the forward propagation module and the backward propagation module both include an optical flow estimation module, a spatial warping module and a depth residual block, and the optical flow estimation module, the spatial warping module, the GDFN module and the depth residual block are connected in sequence; An acquisition module is used to obtain the video to be processed; A super-resolution image determination module is used to input each frame image of the video to be processed into the super-resolution model to obtain a super-resolution image corresponding to each frame image of the video to be processed; Super-resolution video determination module; A super-resolution video corresponding to the video to be processed is obtained according to the super-resolution images corresponding to each frame image of the video to be processed.

5. The deep learning-based video super-resolution reconstruction system according to claim 4, characterized in that: The first input end of the i-th forward propagation module is connected to the first output end of the i-1-th forward propagation module; the second input end of the i-th forward propagation module is used to input the i-th frame image and the i-1-th frame image of the video to be processed; the first output end of the i-th forward propagation module is connected to the first input end of the i+1-th forward propagation module; The second output terminal of the i-th forward propagation module is connected to the first input terminal of the i-th upsampling module; The first input end of the i-th backward propagation module is connected to the first output end of the i+1-th backward propagation module; the second input end of the i-th backward propagation module is used to input the i-th frame image and the i-1-th frame image of the video to be processed; the first output end of the i-th backward propagation module is connected to the first input end of the i-1-th backward propagation module; the second output end of the i-th backward propagation module is connected to the second input end of the i-th upsampling module.

6. A computer-readable storage medium, characterized in that It stores a computer program, which, when executed by a processor, implements the deep learning-based video super-resolution reconstruction method according to any one of claims 1 to 3.

Images