A scene prior learning method for video super-resolution

Abstract

The present invention discloses a method for learning scene priors in video super-resolution. The present invention uses the attention mechanism of deep learning as a carrier, and creates a scene prior learning module on it. The module includes three branches: a forgetting branch, a memory branch, and an output branch. It enables the network model containing the attention mechanism to continuously learn a corresponding scene prior as the video frame sequence is traversed, and each iteration will utilize the learned scene prior. The present invention can effectively mine and utilize the scene prior information in the video sequence. At the same time, it effectively makes up for the shortcoming of insufficient acquisition of time information of the iterative structural model in the video super-resolution algorithm. To a certain extent, it breaks the limitation of the attention mechanism on the utilization of time information. Only less computing resources are spent to achieve excellent effect and performance improvement.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning, is applicable to video super-resolution algorithms, and specifically relates to a scene priori learning method in video super-resolution. Background Art

[0002] With the rapid development of artificial intelligence and the advent of the information age, people's practical demands for video are increasing. Video super-resolution (VSR) algorithms are widely used in fields such as video restoration, live video broadcasting, and autonomous driving. Video super-resolution (VSR) seeks to reconstruct high-resolution (HR) video sequences from low-resolution (LR) video input. Compared to image super-resolution methods that rely solely on the spatial information of a single image, VSR methods can use additional temporal information from adjacent frames to achieve higher reconstruction quality. Many classic VSR algorithms have been proposed, and based on their execution structure, they can be roughly divided into two categories: recurrent structures and iterative structures.

[0003] The loop structure means that only one frame of the video is input at a time, and all video frames are traversed at once according to the sequence of video frames. The algorithm model also refers to the structure of the recurrent neural network to compensate for the video context information, thereby extracting the temporal information in the frame sequence to enhance the currently processed frame. The advantage of this structure is that it can fully extract the temporal information of the frame sequence, but it also has a significant disadvantage. That is, when computing resources are limited, the context information extraction module occupies too many resources, resulting in the current frame information extraction module being unable to perform effectively and extract sufficient spatial information, making the final effect suboptimal.

[0004] The iterative structure means that each time the n frames of the input video are used as a window, in which the middle frame is generally selected as the center frame, and the information of the center frame is compensated with other neighboring frames in the window, so that temporal and spatial information can be obtained to a certain extent. For an entire video sequence, this window will slide from the initial frame sequence to the last frame, and each window will only output the enhanced center frame, and eventually the entire video sequence can be completely traversed. The advantage of this structure is that it has sufficient resources to extract spatial information, but the extraction of temporal information is limited to the frame sequence of the window size, and cannot cover the contextual information of the entire video sequence. This makes very low the utilization of temporal information, so this structure cannot reach the optimal value. Summary of the Invention

[0005] The purpose of this invention is to address the problem that the iterative structure of video super-resolution algorithms fails to extract sufficient temporal information and neglects the use of global scene prior knowledge. How to enable the iterative structure to better utilize scene prior information to compensate for the lack of temporal information, thereby achieving more effective performance in the original algorithm structure, is a question that requires careful study.

[0006] The present invention uses the attention mechanism of deep learning as a carrier and creates a scene prior learning module on it. This module enables the network model containing the attention mechanism to continuously learn a corresponding scene prior as it traverses the video frame sequence, and each iteration will utilize the learned scene prior.

[0007] The technical solutions adopted by the present invention to solve the technical problems are as follows:

[0008] Step 1: Mount the created scene prior learning module to the attention mechanism of the network model. The training method is as follows:

[0009] HR i , SP i =Net(n*LR,SP i-1 ) (1)

[0010] Loss=criterion(HR i , GT i ) (2)

[0011] Among them, HR i Represents the central frame of the network input, n represents the window size of the input frame; LR represents the input frame, which is a low-resolution initial frame; GT represents the reference high-resolution frame, and i represents the unique identifier of the central frame; in addition to the low-resolution frame sequence of the input window size, the input of the network model also requires the scene prior information SP obtained in the previous round i-1 At the same time, the network model additionally outputs a scene prior information SP learned in the current training round i Because the basic support of the scene prior is that they belong to the same video, the center frame unique identifier i must also be traversed in sequence according to the video frame sequence, and i-1 represents the center frame unique identifier of the previous training round.

[0012] Step 2: Mount the created scene prior learning module to the attention mechanism of the network model. The test method is as follows:

[0013] HR i , SP i =Net(n*LR,SP i-1 ) (3)

[0014] Among them, the testing method is generally similar to the testing method of the network model that does not use the scene prior learning module, and the difference is similar to the difference in training method. The network model has an additional scene prior information of the previous test round as input, and an additional scene prior information learned in this test round as output.

[0015] Step 3. The present invention is a scene prior learning module mounted on the attention mechanism, so the operating structure of the attention mechanism will also change after using the present invention:

[0016] The traditional attention mechanism takes feature maps extracted from one or more adjacent frames as input and generates an attention map with values ​​in the range [0, 1]:

[0017] A=f att (F t，n ) (4)

[0018] Among them f att represents the attention mechanism, F t,n Denotes the feature map extracted from n adjacent frames at time t, and A denotes the output feature map of the attention structure. In order to utilize and learn scene prior information, the present invention improves the attention mechanism as follows:

[0019]

[0020] in, represents the current feature map first extracted in the first frame at time t, and then enhanced by the scene prior learning module. The calculation formula is as follows:

[0021]

[0022] Among them, f spl represents the scene prior learning module, P t and P t-1 Represent the scene prior feature maps obtained at time t and t-1 respectively, represents the enhanced feature map obtained in the first frame at time t-1.

[0023] Step 4: The scene prior learning module consists of three branches: the forgetting branch, the memory branch, and the output branch. The preprocessing work for these three branches is undertaken by three convolutional layers. The work of these three convolutional layers can be summarized as follows:

[0024]

[0025]

[0026]

[0027] Among them F fb 、F rb and F ob Represent the preprocessing result features of the forget branch, memory branch and output branch respectively; C fb 、C rb and C ob Represent the convolutional layer processing functions of these three branches respectively.

[0028] 4-1. The specific processing process for memory branches is as follows:

[0029]

[0030]

[0031]

[0032] The memory branch will detect and fuse the valuable parts of the feature map. Specifically in the memory branch, i t and are two intermediate outputs, The purpose of is to preserve the details of the fusion feature map, i t Plays the role of feature selection, where and Yes and F rb The two feature maps separated by average, σ represents the sigmoid function. t and The result of the dot product is used as the output of the memory branch. represents the output of the memory branch, and ⊙ represents the dot product operation.

[0033] 4-2. The specific processing process for the forgotten branch is as follows:

[0034]

[0035] f t =σ(F fb ) (14)

[0036]

[0037] Among them, the forget branch combines the effective output of the memory branch To update the scene prior information and delete useless information from the previous scene prior information, it plays a filtering role. Specifically, first combine the previous scene prior feature P t-1 and the output of the memory branch Get a temporarily updated scene feature At the same time, the preprocessing result feature F from the forget branch fbGenerate a forget selection feature f t , to remove scene features Useless information, through the sigmoid function, forget the selected feature f t The value of is limited to the range of [0,1], where 0 means completely forgetting the previous information; Finally, the scene features and forget selection feature f t The output of the forgetting branch is obtained by multiplication, which is also the output P of the prior information of this round of scene t .

[0038] 4-3. The specific processing process for the output branch is as follows:

[0039] o t =r(F ob ) (16)

[0040]

[0041] Among them, the output branch is designed to enhance the current feature map in the attention mechanism. t Represents the initial features after simple enhancement by preprocessing convolution, where r represents the LeakyReLU function; in order to better utilize scene features The initial feature o t Scene features with temporary updates Multiply to get the enhanced current feature map Make the current feature More consistent with the contextual feature structure of the entire model.

[0042] The advantages and beneficial results of the method of the present invention are:

[0043] 1. The scene prior learning method of the present invention can effectively mine and utilize scene prior information in video sequences.

[0044] 2. The present invention effectively makes up for the shortcoming of insufficient acquisition of temporal information of the iterative structure model in the video super-resolution algorithm.

[0045] 3. This invention breaks the limitations of the attention mechanism in utilizing temporal information to a certain extent.

[0046] 4. The applicability of this invention is very high, and it has been verified that it can be applied to a variety of attention mechanisms and a variety of video super-resolution network models.

[0047] 5. The present invention can achieve excellent performance improvement with only a small amount of computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a flow chart of operating the iterative structure network model of the present invention;

[0049] Figure 2 It is a specific internal data flow structure diagram of the present invention;

[0050] Figure 3 This is the different mounting methods of the present invention for different types of attention mechanisms;

[0051] Figure 4 It is a data loading method for the iterative structure network model training process of the present invention;

[0052] Figure 5 This is a comparison diagram of the effects of the network model using the present invention and not using the present invention;

[0053] Figure 6 It is a scene prior visualization result diagram of the present invention. DETAILED DESCRIPTION

[0054] The present invention will be further described below with reference to the accompanying drawings and examples.

[0055] like Figure 1 As shown, the present invention uses the attention mechanism of deep learning as a carrier and creates a scene prior learning module on it. This module enables the network model containing the attention mechanism to continuously learn a corresponding scene prior as the video frame sequence is traversed, and the learned scene prior is utilized in each iteration.

[0056] The behaviors of the iterative structure network model without and with the present invention will be different.

[0057] The training method of the network model without using the scene prior learning module is as follows:

[0058] HR i =Net(n*LR)

[0059] Loss=criterion(HR i ,GT i )

[0060] Among them, HR i The term "net" represents the central frame of the network input, which is a high-resolution output; "net" represents the abstract representation of the network model; "n" represents the window size of the input frame; "LR" represents the input frame, which is a low-resolution initial frame; and "GT" represents the reference high-resolution frame, which is the target for network learning. "i" represents the unique identifier of the central frame, a random value. This means that training does not require sequentially traversing the video sequence; instead, K frames of the sliding window size can be randomly selected from a large dataset as input. The GT map corresponding to the central frame is used as the reference for the loss function.

[0061] The testing method of the network model without using the scenario prior learning module is as follows:

[0062] HR i =Net(n*LR)

[0063] During testing, there is no need for backpropagation of the Loss function to modify the network weight parameters. However, for a tested video, it must cover the test results of all frames in the video sequence. Therefore, the displacement identifier of the center frame is usually traversed sequentially, starting from the initial frame of the video to the last frame.

[0064] Furthermore, the training method of the network model using the scenario prior learning module is implemented as follows:

[0065] HR i ,SP i =Net(n*LR,SP i-1 )

[0066] Loss=criterion(HR i ,GT i )

[0067] Compared with the network model training method without using the scene prior learning module:

[0068] ① In addition to the low-resolution frame sequence of the input window size, the input of the network model also needs to input the scene prior information SP obtained in the previous round i-1 .

[0069] ② The network model additionally outputs a scene prior information SP learned in the current training round i .

[0070] ③ Because the basic support of the scene prior is that they belong to the same video, the center frame unique identifier i must also be traversed in sequence according to the video frame sequence, and i-1 represents the center frame unique identifier of the previous training round.

[0071] Furthermore, the testing method of the network model using the scenario prior learning module is implemented as follows:

[0072] HR i ,SP i =Net(n*LR,SP i-1 )

[0073] The testing method is generally similar to the testing method of the network model without using the scene prior learning module, and the difference is similar to the difference in training method. The model has an additional scene prior information from the previous test round as input and an additional scene prior information learned in this test round as output.

[0074] Further, such as Figure 2 As shown, the present invention is a scene prior learning module mounted on the attention mechanism, so the operating structure of the attention mechanism will also change after using the present invention:

[0075] The traditional attention mechanism takes as input feature maps extracted from one or more adjacent frames and generates an attention map with values ​​in the range [0, 1]:

[0076] A=f att (F t,n )

[0077] Among them f att represents the attention mechanism, F t,n Denotes the feature map extracted from n adjacent frames at time t, and A denotes the output feature map of the attention structure. In order to utilize and learn scene prior information, the present invention improves the attention mechanism as follows:

[0078]

[0079] in, represents the current feature map first extracted in the first frame at time t, and then enhanced by the scene prior learning module. The calculation formula is as follows:

[0080]

[0081] Among them, f spl represents the scene prior learning module, P t and P t-1 Represent the scene prior feature maps obtained at time t and t-1 respectively, represents the enhanced feature map obtained in the first frame at time t-1.

[0082] The scene prior learning module is divided into three branches: the forgetting branch, the memory branch, and the output branch. The preprocessing work for these three branches is undertaken by three convolutional layers. The work of these three convolutional layers can be summarized as follows:

[0083]

[0084]

[0085]

[0086] Among them F fb 、F rb and F ob Represent the preprocessing result features of the forget branch, memory branch and output branch respectively; C fb 、Crb and C ob Represent the convolutional layer processing functions of these three branches respectively.

[0087] The specific processing process for memory branches is as follows:

[0088]

[0089]

[0090]

[0091] The memory branch will detect and fuse the valuable parts of the feature map. Specifically, in the memory branch, i t and There are two intermediate outputs. The purpose of is to preserve the details of the fusion feature map, i t Plays the role of feature selection, where and Yes and F rb The two feature maps separated by average, σ represents the sigmoid function. t and The result of the dot product is used as the output of the memory branch. represents the output of the memory branch, and ⊙ represents the dot product operation.

[0092] The specific processing process for the forgotten branch is as follows:

[0093]

[0094] f t =σ(F fb )

[0095]

[0096] The forget branch combines the valid output of the memory branch To update the scene prior information and delete useless information from the previous scene prior information, it plays a filtering role. Specifically, first combine the previous scene prior feature P t-1 and the output of the memory branch Get a temporarily updated scene feature At the same time, the preprocessing result feature F from the forget branch fb Generate a forget selection feature f t , to remove scene features Useless information, through the sigmoid function, forget the selected feature f t The value of is limited to the range of [0,1], where 0 means completely forgetting the previous information; Finally, the scene features and forget selection feature f t The output of the forgetting branch is obtained by multiplication, which is also the output P of the prior information of this round of scene t .

[0097] The specific processing process for the output branch is as follows:

[0098] o t =r(F ob )

[0099]

[0100] The output branch is designed to enhance the current feature map in the attention mechanism. t Represents the initial features after simple enhancement by preprocessing convolution, where r represents the LeakyReLU function; in order to better utilize scene features The initial feature o t Scene features with temporary updates Multiply to get the enhanced current feature map Make the current feature More consistent with the contextual feature structure of the entire model.

[0101] Example:

[0102] like Figure 3 As shown in the figure, the present invention uses four attention structures and their representative video super-resolution network models for verification in the actual experimental process: temporal attention (EDVR, TGA), spatial attention (MuCAN), channel attention (RCAN), and non-local attention (PFNL).

[0103] During the experiment, a unified training method and dataset were adopted. The following data results all used the REDS dataset as the training set, the REDS4 dataset as the test set, and PSNR and SSIM as performance reference indicators.

[0104] The verification results of the EDVR model are as follows, where Base represents the original network model without using the present invention; SPL represents the network model using the present invention; nF represents the number of input frames; the unit of PSNR is dB, and the performance of PSNR and SSIM indicators are proportional to the numerical value; the unit of parameter quantity is M; FLOPs represents the number of floating-point operations per second, and the unit is G.

[0105]

[0106]

[0107] Through experimental data and Figure 4The visualization of the SPL module shows that the use of the SPL module significantly improves the performance of the baseline model across all input frames. The scene prior introduced by the SPL module effectively improves performance while consuming very little computational resources. The SPL module is particularly effective for models with fewer input frames. For example, the introduction of the SPL module improves the PSNR and SSIM of the EDVR_L2 model by 0.09dB and 0.0024, respectively.

[0108] Verification results for other models

[0109]

[0110] like Figure 5 and 6 As shown, Figure 5 This is a comparison chart of the effects of the network model using the present invention and not using the present invention. Figure 6 This is a visualization of the scenario priors presented in this paper. Experimental data from other selected models show that the introduction of the SPL module improves the performance of all selected methods. This further demonstrates the strong applicability of the SPL module to various VSR methods.

Claims

1. A scene prior learning method for video super-resolution, characterized by: This method uses the attention mechanism of deep learning as a carrier and creates a scene prior learning module on it. This module enables the network model containing the attention mechanism to continuously learn a corresponding scene prior as it traverses the video frame sequence, and each iteration will utilize the learned scene prior. The method specifically includes the following steps: Step 1: Mount the created scene prior learning module to the attention mechanism of the network model. The training method is as follows: HR i ,SP i =Net(n*LR,SP i-1 ) (1) Loss=criterion(HR i ,GT i ) (2) Among them, HR i Represents the central frame of the network input, n represents the window size of the input frame; LR represents the input frame, which is a low-resolution initial frame; GT represents the reference high-resolution frame, and i represents the unique identifier of the central frame; in addition to the low-resolution frame sequence of the input window size, the input of the network model also requires the scene prior information SP obtained in the previous round i-1 ; At the same time, the network model additionally outputs a scene prior information SP learned in the current training round i Because the basic support of the scene prior is that they belong to the same video, the center frame unique identifier i must also be traversed in sequence according to the video frame sequence, and i-1 represents the center frame unique identifier of the previous training round; Step 2: Mount the created scene prior learning module to the attention mechanism of the network model. The test method is as follows: HR i ,SP i =Net(n*LR,SP i-1 ) (3) The network model has an additional scene prior information from the previous test round as input, and an additional scene prior information learned in this test round as output; Step 3: After the scene prior learning module is installed, the operational structure of the attention mechanism is improved as follows: Among them, f att represents the attention mechanism, F t,n represents the feature map extracted from n adjacent frames at time t, A represents the output feature map of the attention structure, represents the current feature map first extracted in the first frame at time t, and then enhanced by the scene prior learning module. The calculation formula is as follows: Among them, f spl represents the scene prior learning module, P t and P t-1 Represent the scene prior feature maps obtained at time t and t-1 respectively, represents the enhanced feature map obtained in the first frame at time t-1; Step 4: Design the scenario prior learning module, which includes three branches: forgetting branch, memory branch and output branch.

2. The scene prior learning method in video super-resolution according to claim 1, characterized in that The preprocessing work of the three branches is undertaken by three convolutional layers. The work of these three convolutional layers can be summarized as follows: Among them F fb 、F rb and F ob Represent the preprocessing result features of the forget branch, memory branch and output branch respectively; C fb 、C rb and C ob Represent the convolutional layer processing functions of these three branches respectively.

3. The scene prior learning method in video super-resolution according to claim 2, characterized in that The specific processing process of the memory branch is as follows: Specifically in the memory branch, i t and are two intermediate outputs, The purpose of is to preserve the details of the fusion feature map, i t Plays the role of feature selection, where and It's F rb The two feature maps separated by average, σ represents the sigmoid function; i t and The result of the dot product is used as the output of the memory branch. represents the output of the memory branch, and ⊙ represents the dot product operation.

4. The scene prior learning method in video super-resolution according to claim 3, characterized in that The specific processing process of the forgotten branch is as follows: f t =σ(F fb ) (13) Among them, the forget branch combines the effective output of the memory branch To update the scene prior information and delete useless information from the previous scene prior information; first combine the previous scene prior feature P t-1 and the output of the memory branch Get a temporarily updated scene feature At the same time, the preprocessing result feature F from the forget branch fb Generate a forget selection feature f t , to remove scene features Useless information, through the sigmoid function, forget the selected feature f t The value of is limited to the range of [0,1], where 0 means completely forgetting the previous information; Finally, the scene features and forget selection feature f t The output of the forgetting branch is obtained by multiplication, which is also the output P of the prior information of this round of scene t .

5. The scene prior learning method in video super-resolution according to claim 4, characterized in that The specific processing process of the output branch is as follows: o t =r(F ob ) (15) Among them, the output branch is designed to enhance the current feature map in the attention mechanism; t Represents the initial features after simple enhancement by preprocessing convolution, where r represents the LeakyReLU function; in order to better utilize scene features The initial feature o t Scene features with temporary updates Multiply to get the enhanced current feature map Make the current feature More consistent with the contextual feature structure of the entire model.

Images