A deep network video denoising method based on model constraints and program product

Abstract

The present invention belongs to a video denoising method. In order to solve the technical problem that the denoising effect of the current video denoising method based on deep learning does not completely exceed the traditional model-based method, and the quality of the denoised video cannot be effectively guaranteed, a deep network video denoising method and program product based on model constraints are provided. For video denoising, a new model-guided video denoising network is proposed as an observation model. The video denoising network describes the relationship between adjacent noise frames, and can effectively restore clean video frames through maximum a posteriori probability estimation. The iterative steps can also be effectively calculated. Compared with the traditional observation model, the new observation model in the present invention greatly reduces the computational complexity by realizing effective iterative steps, and can effectively improve the quality of the denoised video.

Description

Technical Field

[0001] The present invention relates to a video denoising method, and in particular to a deep network video denoising method based on model constraints and a program product. Background Art

[0002] Although the performance of imaging sensors continues to improve, noise is still inevitably introduced in the digital imaging process, which not only reduces the image and video quality, but also has an adverse effect on subsequent image processing tasks.

[0003] In order to improve the visual effect, researchers at home and abroad have proposed many image and video denoising algorithms. For single image denoising, the existing methods are mainly divided into model-based methods and deep learning-based methods. Model-based denoising methods are highly flexible, but have high computational complexity and are difficult to jointly optimize the parameters in the algorithm. With the rise of deep learning, convolutional neural networks (CNNs) have also been used for image denoising. By learning end-to-end mapping functions, CNNs can infer corresponding high-quality images from input low-quality images. They have gradually become the current mainstream image denoising methods.

[0004] Compared with image denoising, video denoising is more challenging due to the complex motion of objects between frames. Video image sequences have strong temporal correlation, so how to exploit the temporal correlation between adjacent frames is the key to video denoising. At present, most video denoising algorithms use motion compensation of previous and next frames of the video to reduce redundancy and expand the image denoising algorithm to multi-frame denoising, such as the paper "Video Denoising by Sparse 3D Transform-Domain Collaborative Filtering" published by K. Dabov et al. in 2017 and the paper "Video Denoising, Deblocking, and Enhancement Through Separable 4-D Nonlocal Spatiotemporal Transforms" published by M. Maggioni et al. in 2012 in "IEEE Transactions on Image Processing A Publication of the IEEE Signal Processing Society" are all based on this method. However, the success of CNN-based image denoisers has also promoted the development of deep learning-based video denoising methods, which first align noisy frames and then use CNN to process the aligned noisy frames for denoising. In their paper “Optical Flow Estimation using a Spatial Pyramid Network” published in IEEE in 2017, Ranjan et al. proposed using an optical flow estimation network for motion compensation. In their paper “Kernel-predicting Convolutional Networks for Denoising Monte Carlo Renderings” published in ACM Transactions on Graphics in 2017, Bako et al. proposed using a kernel prediction network for inter-frame motion compensation and then using CNN to eliminate the noise of aligned frames.

[0005] Although most existing deep learning-based methods perform denoising by directly mapping low-quality video sequences to the required high-quality images, the observation model that characterizes the video degradation process has been largely ignored, and the neural network is regarded as a black box, and the denoising performance is improved by continuously deepening and widening the network structure, that is, the construction of motion compensation and denoisers is heuristic and has poor interpretability. This heuristic network model has no rules to follow, resulting in the video denoising effect based on deep learning not completely surpassing the traditional method, and the quality of the denoised video cannot be effectively guaranteed. Summary of the invention

[0006] The present invention aims to solve the technical problem that the current video denoising method based on deep learning has not completely surpassed the traditional model-based method in denoising effect and the quality of the denoised video cannot be effectively guaranteed. A deep network video denoising method and program product based on model constraints are provided.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] A deep network video denoising method based on model constraints, which is special in that it includes the following steps:

[0009] S1, constructing a model-guided video denoising network, wherein the video denoising network includes a multi-frame video grouping fusion module, a single-frame video denoising module and a reconstruction solution module; the multi-frame video grouping fusion module includes a first fusion module and a second fusion module, and the single-frame video denoising module includes at least one denoising submodule;

[0010] S2, obtain the video denoising network parameters

[0011] S2.1, divide a frame of video into a-2 groups of images, each group of images contains 3 frames of video; where a is an odd number greater than or equal to 3;

[0012] S2.2, inputting a-2 groups of images into a-2 first fusion modules in sequence, obtaining a-2 preliminary fusion results corresponding to the first fusion modules, and then fusing the a-2 preliminary fusion results through the second fusion module to obtain a fused single-frame video image;

[0013] The first fusion module and the second fusion module are Unet networks with the same network structure, and the downsampling in the network structure is implemented by a convolution layer with a stride of 2, and parameters are shared between a-2 of the first fusion modules;

[0014] S2.3, subjecting the fused single-frame video image to denoising processing by each denoising submodule in turn to obtain a preliminary denoising result; the denoising submodule is a Unet network including five encoding blocks and four decoding blocks;

[0015] S2.4, inputting the preliminary denoising result into a reconstruction solution module, and obtaining a reconstructed denoising result through iterative calculation and solution; the reconstruction solution module is provided with a solution method for the objective function;

[0016] S2.5, according to the reconstructed denoising result and the target optimization function, the video denoising network parameters are obtained;

[0017] S3, training the video denoising network through the video data set, optimizing the parameters of the video denoising network, and obtaining the trained video denoising network;

[0018] S4, inputting the video to be denoised into the trained video denoising network to obtain the denoised video.

[0019] Further, in step S2.2, the network structure of the first fusion module includes an input layer and an output layer, and a first convolution layer, a second convolution layer, a third convolution layer, a first activation function layer, a fourth convolution layer, a second activation function layer, a fifth convolution layer, a third activation function layer, a sixth convolution layer, a first down-sampling layer, a fourth activation function layer, a seventh convolution layer, a fifth activation function layer, an eighth convolution layer, a sixth activation function layer, a ninth convolution layer, a seventh activation function layer, a tenth convolution layer, a second down-sampling layer, an eighth activation function layer, an eleventh convolution layer, a ninth activation function layer, a twelfth convolution layer, a tenth activation function layer, a thirteenth convolution layer, an eleventh activation function layer, a fourteenth convolution layer, a third down-sampling layer, a twelfth activation function layer, a fifteenth convolution layer, a thirteenth activation function layer, a sixteenth convolution layer, a fourteenth activation function layer, a seventeenth convolution layer, a fifteenth activation function layer, an eighteenth convolution layer, a fourth down-sampling layer, a sixteenth activation function layer, and a nineteenth convolution layer. convolution layer, the seventeenth activation function layer, the twentieth convolution layer, the eighteenth activation function layer, the twenty-first convolution layer, the nineteenth activation function layer, the twenty-second convolution layer, the first upsampling layer, the twenty-third convolution layer, the twentieth activation function layer, the twenty-fourth convolution layer, the twenty-first activation function layer, the twenty-fifth convolution layer, the twenty-second activation function layer, the twenty-sixth convolution layer, the second upsampling layer, the twenty-seventh convolution layer, the twenty-third activation function layer, the twenty-eighth convolution layer, the twenty-fourth activation function layer, the twenty-ninth convolution layer, the twenty-fifth activation function layer, the thirtieth convolution layer, the third upsampling layer, the thirty-first convolution layer, the twenty-sixth activation function layer, the thirty-second convolution layer, the twenty-seventh activation function layer, the thirty-third convolution layer, the twenty-eighth activation function layer, the thirty-fourth convolution layer, the fourth upsampling layer, the thirty-fifth convolution layer, the twenty-ninth activation function layer, the thirty-sixth convolution layer, the thirtieth activation function layer, the thirty-seventh convolution layer, the thirty-first activation function layer, the thirty-eighth convolution layer and the thirty-ninth convolution layer.

[0020] Furthermore, the number of input channels of the first convolutional layer is 9, the number of output channels is 64, and the convolution step size is 1;

[0021] The number of input channels of the second convolutional layer to the twenty-second convolutional layer is 64, the number of output channels is 64, and the convolution step size is 1;

[0022] The number of input channels of the 23rd convolutional layer, the 27th convolutional layer, the 31st convolutional layer, and the 35th convolutional layer are all 128, the number of output channels are all 64, and the convolution step size is all 1;

[0023] The number of input channels of the 24th to 26th convolutional layers is 64, the number of output channels is 64, and the convolution step size is 1;

[0024] The number of input channels of the 28th to 30th convolutional layers is 64, the number of output channels is 64, and the convolution step size is 1;

[0025] The number of input channels of the thirty-second convolutional layer to the thirty-fourth convolutional layer is 64, the number of output channels is 64, and the convolution step size is 1;

[0026] The number of input channels of the thirty-sixth convolutional layer to the thirty-eighth convolutional layer is 64, the number of output channels is 64, and the convolution step size is 1;

[0027] The number of input channels of the thirty-ninth convolutional layer is 64, the number of output channels is 3, and the convolution step size is 1;

[0028] The number of input channels from the first downsampling layer to the fourth downsampling layer is 64, the number of output channels is 64, and the convolution step size is 2;

[0029] The number of input channels from the first upsampling layer to the fourth upsampling layer is 64, the number of output channels is 64, and the deconvolution step size is 2;

[0030] The first activation function layer to the thirty-first activation function layer all use the Relu activation function.

[0031] Further, in step S2.3, the network structure of the denoising submodule includes an input layer and an output layer during the first iterative calculation, and a fortieth convolutional layer, a forty-first convolutional layer, a forty-third convolutional layer, a thirty-second activation function layer, a forty-fourth convolutional layer, a thirty-third activation function layer, a forty-fifth convolutional layer, a thirty-fourth activation function layer, a forty-sixth convolutional layer, a fifth downsampling layer, a thirty-fifth activation function layer, a forty-seventh convolutional layer, a thirty-sixth activation function layer, a forty-eighth convolutional layer, and a third convolutional layer connected layer by layer from the input layer to the output layer. The seventeenth activation function layer, the forty-ninth convolution layer, the thirty-eighth activation function layer, the fiftieth convolution layer, the sixth downsampling layer, the thirty-ninth activation function layer, the fifty-first convolution layer, the fortieth activation function layer, the fifty-second convolution layer, the forty-first activation function layer, the fifty-third convolution layer, the forty-sixth activation function layer, the fifty-fourth convolution layer, the seventh downsampling layer, the forty-third activation function layer, the fifty-fifth convolution layer, the forty-fourth activation function layer, the fifty-sixth convolution layer, the forty-fifth activation function layer, the fifty-seventh convolution layer, the forty-sixth activation function layer, the The fifty-eighth convolution layer, the eighth downsampling layer, the forty-seventh activation function layer, the fifty-ninth convolution layer, the forty-eighth activation function layer, the sixtieth convolution layer, the forty-ninth activation function layer, the sixty-first convolution layer, the fiftieth activation function layer, the sixty-second convolution layer, the fifth upsampling layer, the sixty-third convolution layer, the fifty-first activation function layer, the sixty-fourth convolution layer, the fifty-second activation function layer, the sixty-fifth convolution layer, the fifty-third activation function layer, the sixty-sixth convolution layer, the sixth upsampling layer, the sixty-seventh convolution layer, the fifty-fourth activation function layer, the sixty-eighth ... convolution layer, the fifty-fifth activation function layer, the sixty-ninth convolution layer, the fifty-sixth activation function layer, the seventieth convolution layer, the seventh upsampling layer, the seventy-first convolution layer, the fifty-seventh activation function layer, the seventy-second convolution layer, the fifty-eighth activation function layer, the seventy-third convolution layer, the fifty-ninth activation function layer, the seventy-fourth convolution layer, the eighth upsampling layer, the seventy-fifth convolution layer, the sixtieth activation function layer, the seventy-sixth convolution layer, the sixty-first activation function layer, the seventy-seventh convolution layer, the sixty-second activation function layer, the seventy-eighth convolution layer and the seventy-ninth convolution layer;

[0032] Starting from the second iterative calculation, a 42nd convolutional layer is included between the 41st convolutional layer and the 43rd convolutional layer.

[0033] Furthermore, the number of input channels of the 40th convolutional layer is 3, the number of output channels is 64, and the convolution step size is 1;

[0034] The number of input channels of the 41st convolutional layer is 64, the number of output channels is 64, and the convolution step size is 1;

[0035] The number of input channels of the 42nd convolutional layer is 64 multiplied by the number of iterations, the number of output channels is 64, and the convolution step size is 1;

[0036] The number of input channels of the 43rd convolutional layer to the 62nd convolutional layer is 64, the number of output channels is 64, and the convolution step size is 1;

[0037] The number of input channels of the sixty-third convolutional layer, the sixty-seventh convolutional layer, the seventy-first convolutional layer, and the seventy-fifth convolutional layer are all 128, the number of output channels are all 64, and the convolution step size is 1;

[0038] The number of input channels of the 64th convolutional layer to the 66th convolutional layer is 64, the number of output channels is 64, and the convolution step size is 1;

[0039] The number of input channels of the 68th convolutional layer to the 70th convolutional layer is 64, the number of output channels is 64, and the convolution step size is 1;

[0040] The number of input channels of the 72nd convolutional layer to the 74th convolutional layer is 64, the number of output channels is 64, and the convolution step size is 1;

[0041] The number of input channels of the 76th convolutional layer to the 78th convolutional layer is 64, the number of output channels is 64, and the convolution step size is 1;

[0042] The number of input channels of the 79th convolutional layer is 64 multiplied by the number of iterations, the number of output channels is 64, and the convolution step size is 1;

[0043] The number of input channels of the 80th convolutional layer is 64 multiplied by the number of iterations, the number of output channels is 64, and the convolution step size is 1;

[0044] The number of input channels of the 81st convolutional layer is 64, the number of output channels is 3, and the convolution step size is 1;

[0045] The number of input channels of the fifth downsampling layer to the eighth downsampling layer is 64, the number of output channels is 64, and the convolution step size is 2;

[0046] The number of input channels of the fifth upsampling layer to the eighth upsampling layer is 64, the number of output channels is 64, and the deconvolution step size is 2;

[0047] The 32nd to 62nd activation function layers all use the Relu activation function.

[0048] Furthermore, in step S2.4, the objective function is solved by the following formula:

[0049]

[0050] in, is the Gaussian noise variance, x (t+1)The output result of the t+1th iteration calculation of the reconstruction solution module is: v (t+1) is the preliminary denoising result output by the denoising submodule after the t+1th iteration calculation, λ is the penalty factor of the regularization term, which is obtained through network learning, J is the maximum value of the video frame number, and W j Represents the denoised video frame x with sequence number j j to x j The transformation matrix corresponding to the intermediate frame x, y j Represents x j Adjacent noisy video frames.

[0051] Furthermore, in step S2.5, the target optimization function is specifically:

[0052]

[0053] Where Θ represents the video denoising network parameters, M is the maximum batch size, F(·; Θ) is the reconstructed denoising result, and x i is the reference frame image that needs to be restored through the adjacent frame degraded image, y i,j It represents the degraded image with batch size i and video frame number j.

[0054] Furthermore, in step S3, the video denoising network is trained using the video data set as follows:

[0055] S3.a, part of the sequences in the Vimeo-90K video dataset are used as training sets, and the other part of the sequences are used as test sets, where the number of sequences in the training set is greater than the number of sequences in the test set;

[0056] S3.b, input the training set into the video denoising network and execute step S2; wherein, when executing step S2.5, the target optimization function minimizes the function value of the target optimization function using the Adam optimization algorithm, and β is set in the Adam optimization algorithm. 1 =0.9,β 2 =0.999,ε=10 -8 , the learning rate is 10 -4 , the parameters of each layer of the semantic segmentation network are updated through gradient back propagation to complete the training.

[0057] Furthermore, step S3 also includes testing the video denoising network after completing the training:

[0058] Annotating the test set to obtain an annotated test set;

[0059] The test set is input into the trained video denoising network, the back propagation of the video denoising network is turned off, steps S2.1 to S2.4 are executed, and the execution results are compared with the labeled test set to obtain the video denoising network detection performance results.

[0060] The present invention also provides a computer program product, including a computer program, which is special in that when the program is executed by a processor, it implements the steps of the above-mentioned deep network video denoising method based on model constraints.

[0061] Compared with the prior art, the present invention has the following beneficial effects:

[0062] 1. The present invention is based on a model-constrained deep network video denoising method. For video denoising, a new model-guided video denoising network is proposed as an observation model. The video denoising network describes the relationship between adjacent noise frames, and can effectively restore clean video frames through maximum a posteriori probability estimation. The iterative steps can also be effectively calculated. Compared with the traditional observation model, the new observation model in the present invention greatly reduces the computational complexity by implementing effective iterative steps, and can effectively improve the quality of the denoised video.

[0063] 2. The present invention combines the observation model, the CNN-based image denoising module and the motion compensation module, and can be implemented as a deep neural network. The network simulates the video denoising process of the observation model to form a simple and effective denoising architecture. At the same time, after testing, it is concluded that the video denoising effect of the present invention in the RGB domain is significantly better than the traditional denoising method based on the model and the heuristic method based on deep learning, and can reach the industry-leading level. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 For the present invention Figure 1 A flowchart of an embodiment of a deep network video denoising method based on model constraints;

[0065] Figure 2 A schematic diagram of the network structure of a model-guided video denoising network according to an embodiment of the present invention;

[0066] Figure 3 This is a network structure diagram of the first fusion module in the multi-frame video grouping fusion module in an embodiment of the present invention;

[0067] Figure 4 A network structure diagram of a denoising submodule in a single-frame video denoising module in an embodiment of the present invention;

[0068] Figure 5A set of comparison diagrams of denoising simulation results in an embodiment of the present invention, wherein (a) is the original image, (b) is the simulation result diagram using the best existing method, and (c) is the simulation result diagram using the method of the present invention;

[0069] Figure 6 1 is another set of comparison diagrams of denoising simulation results in an embodiment of the present invention, wherein (a) is the original image, (b) is the simulation result diagram using the best existing method, and (c) is the simulation result diagram using the method of the present invention. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0071] The purpose of the present invention is to design a model-guided video denoising method to overcome the problems of high computational complexity of existing model-based traditional video denoising methods, poor interpretability of deep learning-based video denoising methods, and low motion compensation efficiency of traditional optical flow alignment methods.

[0072] like Figure 1 As shown, the specific implementation of the present invention is as follows:

[0073] Step 1: Obtain a training sample set.

[0074] The video dataset used is Vimeo-90K, which consists of 91,701 seven-frame color video sequences. 84,765 sequences are selected as training sets, and the resolution of the video frames is 448*256. The remaining 6,936 seven-frame sequences of Vimeo-90K are used as test sets.

[0075] Step 2: Establish the observation equation.

[0076] Based on the relationship between adjacent frames, a new observation equation is proposed as follows:

[0077]

[0078] Among them, x represents a clear video frame, y represents the corresponding noisy video frame, and n j is the additive noise of the video frame number j, y j and x j Represent adjacent noisy and clear video frames, y j =x j +n j , Wj Indicated by x j The transformation matrix to x, W j Any video frame alignment operator can be used, such as global alignment or video frame alignment operator based on optical flow estimation or deep neural network. In the present invention, a multi-frame video grouping fusion module is selected to implement it. and n j obey the same Gaussian distribution.

[0079] Step 3: Derive the objective function and propose an iterative solution.

[0080] From the perspective of the Bayesian formula, using the maximum a posteriori probability estimation method, the present invention can estimate the clear frame by maximizing the following objective function:

[0081]

[0082] Where logP(y j |x) is the Gaussian data likelihood term, P(x) is the prior distribution of the clean image, logP(x|{y j}) is the posterior distribution of the clean image, and the expression is shown in the following formula:

[0083]

[0084] Among them, σ n,j is the Gaussian noise variance. logP(x) is a priori. In order to utilize the deep neural network, the present invention introduces a denoising prior based on the deep convolutional neural network, namely:

[0085] P(x)∝exp(-λ||xv|| 1 ),where v=f(x)

[0086] λ is a constant, f(x) is a single-frame denoising network, and v is an additional variable introduced by modeling the clean image x prior through the network. Substituting the Gaussian data likelihood term and the denoising prior term into the MAP formula (the formula for maximum a posteriori probability estimation) can obtain a new objective function, as shown below:

[0087]

[0088] The above formula is an optimization problem involving both the first norm and the second norm, which can be solved using soft thresholding. The solution formula is as follows:

[0089]

[0090] v (t+1) =f(x (t) )

[0091] in, x (t+1) is the output result of the t+1th iteration calculation of the reconstruction solution module, v (t+1) is the preliminary denoising result output by the denoising submodule after the t+1th iteration, and J is the maximum value of the video frame number. It can be seen from the above formula that both the x-subproblem and the v-subproblem can be effectively solved, and It can be calculated by some efficient video frame motion alignment methods.

[0092] Step 4: Build a model-guided video denoising network.

[0093] The model-guided video denoising network consists of a multi-frame video grouping fusion module, a single-frame video denoising module, and a reconstruction solution module. Figure 2 A schematic diagram of an example of a model-guided video denoising network is shown, and this example is used as an example for explanation below.

[0094] 4.1) Construct a multi-frame video grouping fusion module, which includes a first fusion module and a second fusion module. The network structure of the first fusion module is as follows: Figure 3 shown.

[0095] The multi-frame video grouping fusion module is a multi-stage structure. Here we take the input video frame number of 5 as an example, which are y 0 ,y 1 ,y 2 ,y 3 and 4 , and then extended to more odd frames with more than 5 frames, or 3 frames. When the input is 5 frames, the 5 frames can be divided into three groups as the input of Fusion1, and the outputs of the three Fusion1s (equivalent to the first fusion module) are used as the input of Fusion2 (equivalent to the second fusion module). Fusion2 outputs the final fusion result, such as Figure 3 , for the first Fusion1, enter y 0 ,y 1 ,y 2 , and get the fused output The three Fusion1s share parameters, and Fusion1 and Fusion2 have the same network structure. Specifically, Fusion1 and Fusion2 are both improved Unet networks. Compared with the original Unet network, in the present invention, the downsampling operation is not implemented by a pooling layer, but by a convolutional layer with a stride of 2. Due to the input of three frames of images, the input channel of the first convolutional layer is the number of channels of a single-frame input image 3 multiplied by the number of input images 3, which is equal to 9 channels.

[0096] The network structure of Fusion1 and Fusion2 includes input layer → first convolution layer → second convolution layer → third convolution layer → first activation function layer → fourth convolution layer → second activation function layer → fifth convolution layer → third activation function layer → sixth convolution layer → first down-sampling layer → fourth activation function layer → seventh convolution layer → fifth activation function layer → eighth convolution layer → sixth activation function layer → ninth convolution layer → seventh activation function layer → tenth convolution layer → second down-sampling layer → eighth activation function layer → eleventh convolution layer → ninth activation function layer → twelfth convolution layer → tenth activation function layer → thirteenth convolution layer → eleventh activation function layer → fourteenth convolution layer → third down-sampling layer → twelfth activation function layer → fifteenth convolution layer → thirteenth activation function layer → sixteenth convolution layer → fourteenth activation function layer → seventeenth convolution layer → fifteenth activation function layer → eighteenth convolution layer → fourth down-sampling layer → sixteenth activation function layer → nineteenth convolution layer → seventeenth activation function layer → twentieth convolution layer The first convolution layer → the eighteenth activation function layer → the twenty-first convolution layer → the nineteenth activation function layer → the twenty-second convolution layer → the first upsampling layer → the twenty-third convolution layer → the twentieth activation function layer → the twenty-fourth convolution layer → the twenty-first activation function layer → the twenty-fifth convolution layer → the twenty-second activation function layer → the twenty-sixth convolution layer → the second upsampling layer → the twenty-seventh convolution layer → the twenty-third activation function layer → the twenty-eighth convolution layer → the twenty-fourth activation function layer → the twenty-ninth convolution layer → the twenty-fifth activation function layer → the thirtieth convolution layer → the third upsampling layer → the thirty-first convolution layer → the twenty-sixth activation function layer → the thirty-second convolution layer → the twenty-seventh activation function layer → the thirty-third convolution layer → the twenty-eighth activation function layer → the thirty-fourth convolution layer → the fourth upsampling layer → the thirty-fifth convolution layer → the twenty-ninth activation function layer → the thirty-sixth convolution layer → the thirtieth activation function layer → the thirty-seventh convolution layer → the thirty-first activation function layer → the thirty-eighth convolution layer → the thirty-ninth convolution layer → the output layer. Its output is the fusion result of three input video frames. The network structure of Fusion1 and Fusion2 of the multi-frame video grouping fusion module is the same.

[0097] The parameters of each layer are as follows:

[0098] The input layer is three adjacent video frame images;

[0099] The number of input channels of the first convolutional layer is 9, the number of output channels is 64, and the convolution step size is 1;

[0100] The number of input channels from the second convolutional layer to the twenty-second convolutional layer is 64, the number of output channels is 64, and the convolution step size is 1;

[0101] The number of input channels of the 23rd, 27th, 31st, and 35th convolutional layers is 128, the number of output channels is 64, and the convolution step size is 1;

[0102] The number of input channels from the 24th to the 26th convolutional layers is 64, the number of output channels is 64, and the convolution step size is 1;

[0103] The number of input channels from the 28th to the 30th convolutional layers is 64, the number of output channels is 64, and the convolution step size is 1;

[0104] The number of input channels from the 32nd to the 34th convolutional layers is 64, the number of output channels is 64, and the convolution step size is 1;

[0105] The number of input channels from the 36th to the 38th convolutional layers is 64, the number of output channels is 64, and the convolution step size is 1;

[0106] The number of input channels of the 39th convolutional layer is 64, the number of output channels is 3, and the convolution step size is 1;

[0107] The number of input channels from the first downsampling layer to the fourth downsampling layer is 64, the number of output channels is 64, and the convolution step size is 2;

[0108] The number of input channels from the first upsampling layer to the fourth upsampling layer is 64, the number of output channels is 64, and the deconvolution step size is 2;

[0109] The first activation function layer to the thirty-first activation function layer all use the Relu activation function.

[0110] The output layer is a single-frame video image after the three frames are fused.

[0111] 4.2) Construct a single-frame video denoising module. The network structure and parameters of this module are as follows. Figure 4 :

[0112] The single-frame video denoising module includes at least one denoising submodule, and the denoising submodule uses the Unet network, which is composed of 5 encoding blocks (Encoder Block) and 4 decoding blocks (Decoder Block). Each encoding block and decoding block is composed of 4 convolutional layers. Except for the first layer of the decoding block, the convolution kernel size is 1*1, and the remaining convolution kernel sizes are 3*3, and there is a relu function between each convolution layer to activate the features. Except for the last encoding block, each encoding block has an additional convolution layer with a step size of 2 at the end, which is used for downsampling operation. Similarly, all decoding blocks have a deconvolution layer with a step size of 2 at the front, which is used for upsampling operation, that is, each Unet has 4 downsampling and 4 upsampling operations. Inspired by Densenet, the present invention connects the features of the previous denoising submodule with the subsequent denoising submodule to form a dense connection, that is, the features of the first encoding block and the last decoding block in each Unet are connected to the corresponding parts of the subsequent Unet.

[0113] The network structure of the denoising submodule includes input layer → 40th convolution layer → 41st convolution layer → 42nd convolution layer → 43rd convolution layer → 32nd activation function layer → 44th convolution layer → 33rd activation function layer → 45th convolution layer → 34th activation function layer → 46th convolution layer → 5th down-sampling layer → 35th activation function layer → 47th convolution layer → 36th activation function layer → 48th convolution layer → 37th activation function layer → 49th convolution layer → 38th activation function layer → 50th convolution layer → 6th down-sampling layer → 39th activation function layer → 51st convolution layer → 40th activation function layer → 52nd convolution layer → 41st activation function layer → 53rd convolution layer → 46th activation function layer → 54th convolution layer → 7th down-sampling layer → 43rd activation function layer → 55th convolution layer → 44th activation function layer → 56th convolution layer → 45th activation function layer → 57th convolution layer → 46th activation function layer → 58th convolution layer → 8th down-sampling layer → 47th activation function layer The network consists of the following layers: the fifth convolution layer → the forty-eighth activation function layer → the sixtieth convolution layer → the forty-ninth activation function layer → the sixty-first convolution layer → the fiftieth activation function layer → the sixty-second convolution layer → the fifth upsampling layer → the sixty-third convolution layer → the fifty-first activation function layer → the sixty-fourth convolution layer → the fifty-second activation function layer → the sixty-fifth convolution layer → the fifty-third activation function layer → the sixty-sixth convolution layer → the sixth upsampling layer → the sixty-seventh convolution layer → the fifty-fourth activation function layer → the sixty-eighth convolution layer → the fifty-fifth activation function layer → the sixty-ninth convolution layer → the fifty-sixth activation function layer → the seventy-seventh convolution layer → the seventh upsampling layer → the seventy-first convolution layer → the fifty-seventh activation function layer → the seventy-second convolution layer → the fifty-eighth activation function layer → the seventy-third convolution layer → the fifty-ninth activation function layer → the seventy-fourth convolution layer → the eighth upsampling layer → the seventy-fifth convolution layer → the sixty-first activation function layer → the seventy-sixth convolution layer → the sixty-first activation function layer → the seventy-seventh convolution layer → the sixty-second activation function layer → the seventy-eighth convolution layer → the seventy-ninth convolution layer → the output layer.

[0114] The parameters of each layer are as follows:

[0115] The input layer is the reconstructed output of the previous denoiser. If the input layer is in the first iteration, the input is the output of the multi-frame video grouping fusion module.

[0116] The number of input channels of the 40th convolutional layer is 3, the number of output channels is 64, and the convolution step size is 1;

[0117] The number of input channels of the 41st convolutional layer is 64, the number of output channels is 64, and the convolution step size is 1;

[0118] If the 42nd convolutional layer is the first iteration, this layer does not exist. In the subsequent iterations, the number of input channels is 64*the number of iterations, the number of output channels is 64, and the convolution step size is 1;

[0119] The number of input channels from the 43rd convolutional layer to the 62nd convolutional layer is 64, the number of output channels is 64, and the convolution step size is 1;

[0120] The number of input channels of the 63rd convolutional layer, the 67th convolutional layer, the 71st convolutional layer, and the 75th convolutional layer are all 128, the number of output channels are all 64, and the convolution step size is 1;

[0121] The number of input channels from the 64th convolutional layer to the 66th convolutional layer is 64, the number of output channels is 64, and the convolution step size is 1;

[0122] The number of input channels from the 68th convolutional layer to the 70th convolutional layer is 64, the number of output channels is 64, and the convolution step size is 1;

[0123] The number of input channels from the 72nd to the 74th convolutional layer is 64, the number of output channels is 64, and the convolution step size is 1;

[0124] The number of input channels from the 76th convolutional layer to the 78th convolutional layer is 64, the number of output channels is 64, and the convolution step size is 1;

[0125] The number of input channels of the 79th convolutional layer is 64 times the number of iterations, the number of output channels is 64, and the convolution step size is 1;

[0126] The number of input channels of the 80th convolutional layer is 64 times the number of iterations, the number of output channels is 64, and the convolution step size is 1;

[0127] The number of input channels of the 81st convolutional layer is 64, the number of output channels is 3, and the convolution step size is 1;

[0128] The number of input channels from the fifth downsampling layer to the eighth downsampling layer is 64, the number of output channels is 64, and the convolution step size is 2;

[0129] The number of input channels from the fifth upsampling layer to the eighth upsampling layer is 64, the number of output channels is 64, and the deconvolution step size is 2;

[0130] The 33rd to 63rd activation function layers all use the Relu activation function;

[0131] The output result of the output layer is then processed by the reconstruction solution module. The specific processing method in the reconstruction solution module is to derive the reconstructed denoising output according to the formula of the objective function solution method in step three.

[0132] 4.3) Construct the target optimization function:

[0133] The objective optimization function of the video denoising task is as follows:

[0134]

[0135] Among them, Θ represents the video denoising network parameters, x i Indicates that the image y needs to be degraded through adjacent frames i,j The restored reference frame image, F(·; Θ) is the reconstructed denoising result.

[0136] Step 5: Input the training data set, update the network parameters through multiple iterations of end-to-end learning, optimize the objective function of the video denoising network, and obtain the trained video denoising network. Then, send the test data set to the trained video denoising network to obtain the test results.

[0137] 5.1) Input training data set, iterate and update network parameters:

[0138] All the training image data in the training data set are input, and then passed through the multi-frame video grouping fusion module, the single-frame image denoising module and the reconstruction module. The optimization function is minimized by using the Adam optimization algorithm, and the parameters are set to β 1 =0.9,β 2 =0.999,ε=10 -8 , the learning rate is set to 10 -4 , gradient back propagation is performed to update the parameters of each layer of the semantic segmentation network. Different training times are set according to different experimental configurations. The iterative calculation is performed until the preset number of times is reached. At the end of the training, the trained video denoising network is obtained.

[0139] 5.2) Input the test data set to test network performance:

[0140] Input all the test data in the test data set, load the trained video denoising network, turn off the network back propagation, and then pass the test image data through the multi-frame video grouping fusion module, the single-frame image denoising module and the reconstruction module in turn to obtain the input intermediate frame denoising result, compare it with the annotation of the test set, and obtain the detection performance results of the video denoising algorithm on the test set.

[0141] Then the video to be denoised can be input, and denoising processing can be performed through the model-guided video denoising network of the present invention.

[0142] The video denoising method proposed by Tassano et al. in "Towards Real-Time Deep Video Denoising Without Flow Estimation" published in "IEEE" in 2020 is the best existing video denoising method. Among them, the optical flow calculation of the video is improved, mainly using a U-shaped network with an encoder-decoder. Because it has the function of alignment within the receptive field, it saves time when calculating the optical flow, so that when similar performance is obtained with other methods, it has lower time complexity. Since the present invention is not entirely a heuristic model, the observation model is introduced during the design, and the prior is designed, which can further reduce the computational complexity and further reduce the running time compared to the traditional heuristic method.

[0143] Experimental results show that the quality index and actual effect of video restoration are better than the above methods.

[0144] In order to compare the performance of the present invention with the best existing technology, the performance indicators of the two technologies were tested on the same test data set. In the simulation experiment, the peak signal-to-noise ratio (PSNR) and the structural similarity (SSIM) were used as the objective quantitative evaluation indicators. Assuming that the target enhanced image is I label , the dehazed image enhanced by the natural enhancement network G is I output , the length and width of the image are W and H respectively. The peak signal-to-noise ratio PSNR can be expressed as:

[0145]

[0146]

[0147] Among them, max represents the maximum pixel value of the image, which is generally in the range of [0,255], and MSE represents the target enhanced image I label And the image I after model enhancement output The mean square error between the two images is used to measure the error between the corresponding pixels in the two images. The unit is dB. The larger the value of this indicator, the smaller the difference between the two images being evaluated.

[0148] The formula of structural similarity SSIM(x,y) is expressed as:

[0149]

[0150] where μ x For I label The mean value, σ x For I label Variance, similarly μ y For I output The mean value, σy For I output Variance, σ xy For I label and I output The covariance of 1 =(k 1 L) 2 and c 2 =(k 2 L) 2 are two constants to avoid the denominator being zero. L represents the dynamic range of the pixel. The default constant k 1 = 0.01 and k 2 =0.03, the value range of structural similarity SSIM is [0,1]. The larger the SSIM value, the smaller the image details and distortion.

[0151] The test results shown in Table 1 are obtained:

[0152] Table 1 Comparison of test results of the best existing method and the method of the present invention under the same data set

[0153]

[0154] It can be seen from Table 1 that the denoising result of the method of the present invention has higher PSNR and SSIM values, indicating that the video image denoised by the method of the present invention has higher definition.

[0155] In addition, if Figure 5 As shown, a simulation result of the best existing method and the method of the present invention is shown. Figure 5 (a) is the original image, (b) is the simulation result of the best method, the peak signal-to-noise ratio PSNR is 37.9dB, (c) is the simulation result of the method of the present invention, the peak signal-to-noise ratio PSNR is 40.22dB. Figure 6 As shown, another simulation result of the best existing method and the method of the present invention is shown. Figure 6 (a) is the original image, (b) is the simulation result of the best existing method, with a peak signal-to-noise ratio (PSNR) of 34.53 dB, and (c) is the simulation result of the method of the present invention, with a peak signal-to-noise ratio (PSNR) of 35.08 dB.

[0156] In addition, the invented model-constrained deep network video denoising method can also form a computer program product, which includes a computer program that implements the steps of the model-constrained deep network video denoising method when executed by a processor.

[0157] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A deep network video denoising method based on model constraints, characterized in that: The following steps are involved: S1, constructing a model-guided video denoising network, wherein the video denoising network includes a multi-frame video grouping fusion module, a single-frame video denoising module and a reconstruction solution module; the multi-frame video grouping fusion module includes a first fusion module and a second fusion module, and the single-frame video denoising module includes at least one denoising submodule; S2, obtain the video denoising network parameters S2.1, divide a frame of video into a-2 groups of images, each group of images contains 3 adjacent frames; where a is an odd number greater than or equal to 3; S2.2, inputting a-2 groups of images into a-2 first fusion modules in sequence, obtaining a-2 preliminary fusion results corresponding to the first fusion modules, and then fusing the a-2 preliminary fusion results through the second fusion module to obtain a fused single-frame video image; The first fusion module and the second fusion module are Unet networks with the same network structure, and the downsampling in the network structure is implemented by a convolution layer with a stride of 2, and parameters are shared between a-2 of the first fusion modules; S2.3, subjecting the fused single-frame video image to denoising processing by each denoising submodule in turn to obtain a preliminary denoising result; the denoising submodule is a Unet network including five encoding blocks and four decoding blocks; S2.4, input the preliminary denoising result into the reconstruction solution module, and obtain the reconstructed denoising result through iterative calculation and solution; the reconstruction solution module is provided with a solution method of the objective function; the solution method of the objective function is specifically solved by the following formula: in, is the Gaussian noise variance, x (t+1) The output result of the t+1th iteration calculation of the reconstruction solution module is: v (t+1) is the preliminary denoising result output by the denoising submodule after the t+1th iteration, λ is the penalty factor of the regularization term, J is the maximum value of the video frame number, and W j Represents the denoised video frame x with sequence number j j to x j The transformation matrix corresponding to the intermediate frame x, y j Represents x j Adjacent noisy video frames; S2.5, according to the reconstructed denoising result and the target optimization function, the video denoising network parameters are obtained; S3, training the video denoising network through the video data set, optimizing the parameters of the video denoising network, and obtaining the trained video denoising network; S4, inputting the video to be denoised into the trained video denoising network to obtain the denoised video.

2. A deep network video denoising method based on model constraints as claimed in claim 1, characterized in that: In step S2.2, the network structure of the first fusion module includes an input layer and an output layer, and a first convolution layer, a second convolution layer, a third convolution layer, a first activation function layer, a fourth convolution layer, a second activation function layer, a fifth convolution layer, a third activation function layer, a sixth convolution layer, a first down-sampling layer, a fourth activation function layer, a seventh convolution layer, a fifth activation function layer, an eighth convolution layer, a sixth activation function layer, a ninth convolution layer, a seventh activation function layer, a tenth convolution layer, a second down-sampling layer, an eighth activation function layer, an eleventh convolution layer, a ninth activation function layer, a twelfth convolution layer, a tenth activation function layer, a thirteenth convolution layer, an eleventh activation function layer, a fourteenth convolution layer, a third down-sampling layer, a twelfth activation function layer, a fifteenth convolution layer, a thirteenth activation function layer, a sixteenth convolution layer, a fourteenth activation function layer, a seventeenth convolution layer, a fifteenth activation function layer, an eighteenth convolution layer, a fourth down-sampling layer, a sixteenth activation function layer, a nineteenth convolution layer, The seventeenth activation function layer, the twentieth convolution layer, the eighteenth activation function layer, the twenty-first convolution layer, the nineteenth activation function layer, the twenty-second convolution layer, the first upsampling layer, the twenty-third convolution layer, the twentieth activation function layer, the twenty-fourth convolution layer, the twenty-first activation function layer, the twenty-fifth convolution layer, the twenty-second activation function layer, the twenty-sixth convolution layer, the second upsampling layer, the twenty-seventh convolution layer, the twenty-third activation function layer, the twenty-eighth convolution layer, the twenty-fourth activation function layer, the twenty-ninth convolution layer, the twenty-fifth activation function layer, the thirtieth convolution layer, the third upsampling layer, the thirty-first convolution layer, the twenty-sixth activation function layer, the thirty-second convolution layer, the twenty-seventh activation function layer, the thirty-third convolution layer, the twenty-eighth activation function layer, the thirty-fourth convolution layer, the fourth upsampling layer, the thirty-fifth convolution layer, the twenty-ninth activation function layer, the thirty-sixth convolution layer, the thirtieth activation function layer, the thirty-seventh convolution layer, the thirty-first activation function layer, the thirty-eighth convolution layer and the thirty-ninth convolution layer.

3. A deep network video denoising method based on model constraints as claimed in claim 2, characterized in that: The number of input channels of the first convolutional layer is 9, the number of output channels is 64, and the convolution step size is 1; The number of input channels of the second convolutional layer to the twenty-second convolutional layer is 64, the number of output channels is 64, and the convolution step size is 1; The number of input channels of the 23rd convolutional layer, the 27th convolutional layer, the 31st convolutional layer, and the 35th convolutional layer are all 128, the number of output channels are all 64, and the convolution step size is all 1; The number of input channels of the 24th to 26th convolutional layers is 64, the number of output channels is 64, and the convolution step size is 1; The number of input channels of the 28th to 30th convolutional layers is 64, the number of output channels is 64, and the convolution step size is 1; The number of input channels of the thirty-second convolutional layer to the thirty-fourth convolutional layer is 64, the number of output channels is 64, and the convolution step size is 1; The number of input channels of the thirty-sixth convolutional layer to the thirty-eighth convolutional layer is 64, the number of output channels is 64, and the convolution step size is 1; The number of input channels of the thirty-ninth convolutional layer is 64, the number of output channels is 3, and the convolution step size is 1; The number of input channels from the first downsampling layer to the fourth downsampling layer is 64, the number of output channels is 64, and the convolution step size is 2; The number of input channels from the first upsampling layer to the fourth upsampling layer is 64, the number of output channels is 64, and the deconvolution step size is 2; The first activation function layer to the thirty-first activation function layer all use the Relu activation function.

4. A deep network video denoising method based on model constraints as described in any one of claims 1 to 3, characterized in that: In step S2.3, the network structure of the denoising submodule includes an input layer and an output layer during the first iterative calculation, and a fortieth convolutional layer, a forty-first convolutional layer, a forty-third convolutional layer, a thirty-second activation function layer, a forty-fourth convolutional layer, a thirty-third activation function layer, a forty-fifth convolutional layer, a thirty-fourth activation function layer, a forty-sixth convolutional layer, a fifth downsampling layer, a thirty-fifth activation function layer, a forty-seventh convolutional layer, a thirty-sixth activation function layer, a forty-eighth convolutional layer, and a thirty-seventh activation function layer connected layer by layer from the input layer to the output layer. function layer, the forty-ninth convolution layer, the thirty-eighth activation function layer, the fiftieth convolution layer, the sixth downsampling layer, the thirty-ninth activation function layer, the fifty-first convolution layer, the fortieth activation function layer, the fifty-second convolution layer, the forty-first activation function layer, the fifty-third convolution layer, the forty-sixth activation function layer, the fifty-fourth convolution layer, the seventh downsampling layer, the forty-third activation function layer, the fifty-fifth convolution layer, the forty-fourth activation function layer, the fifty-sixth convolution layer, the forty-fifth activation function layer, the fifty-seventh convolution layer, the forty-sixth activation function layer, the fifty-eighth convolution layer Convolution layer, eighth down-sampling layer, forty-seventh activation function layer, fifty-ninth convolution layer, forty-eighth activation function layer, sixtieth convolution layer, forty-ninth activation function layer, sixty-first convolution layer, fiftieth activation function layer, sixty-second convolution layer, fifth up-sampling layer, sixty-third convolution layer, fifty-first activation function layer, sixty-fourth convolution layer, fifty-second activation function layer, sixty-fifth convolution layer, fifty-third activation function layer, sixty-sixth convolution layer, sixth up-sampling layer, sixty-seventh convolution layer, fifty-fourth activation function layer, sixty-eighth convolution layer layer, the fifty-fifth activation function layer, the sixty-ninth convolution layer, the fifty-sixth activation function layer, the seventieth convolution layer, the seventh upsampling layer, the seventy-first convolution layer, the fifty-seventh activation function layer, the seventy-second convolution layer, the fifty-eighth activation function layer, the seventy-third convolution layer, the fifty-ninth activation function layer, the seventy-fourth convolution layer, the eighth upsampling layer, the seventy-fifth convolution layer, the sixtieth activation function layer, the seventy-sixth convolution layer, the sixty-first activation function layer, the seventy-seventh convolution layer, the sixty-second activation function layer, the seventy-eighth convolution layer and the seventy-ninth convolution layer; Starting from the second iterative calculation, a 42nd convolutional layer is included between the 41st convolutional layer and the 43rd convolutional layer.

5. A deep network video denoising method based on model constraints as claimed in claim 4, characterized in that: The number of input channels of the 40th convolutional layer is 3, the number of output channels is 64, and the convolution step size is 1; The number of input channels of the 41st convolutional layer is 64, the number of output channels is 64, and the convolution step size is 1; The number of input channels of the 42nd convolutional layer is 64 multiplied by the number of iterations, the number of output channels is 64, and the convolution step size is 1; The number of input channels of the 43rd convolutional layer to the 62nd convolutional layer is 64, the number of output channels is 64, and the convolution step size is 1; The number of input channels of the sixty-third convolutional layer, the sixty-seventh convolutional layer, the seventy-first convolutional layer, and the seventy-fifth convolutional layer are all 128, the number of output channels are all 64, and the convolution step size is 1; The number of input channels of the 64th convolutional layer to the 66th convolutional layer is 64, the number of output channels is 64, and the convolution step size is 1; The number of input channels of the 68th convolutional layer to the 70th convolutional layer is 64, the number of output channels is 64, and the convolution step size is 1; The number of input channels of the 72nd convolutional layer to the 74th convolutional layer is 64, the number of output channels is 64, and the convolution step size is 1; The number of input channels of the 76th convolutional layer to the 78th convolutional layer is 64, the number of output channels is 64, and the convolution step size is 1; The number of input channels of the 79th convolutional layer is 64 multiplied by the number of iterations, the number of output channels is 64, and the convolution step size is 1; The number of input channels of the fifth downsampling layer to the eighth downsampling layer is 64, the number of output channels is 64, and the convolution step size is 2; The number of input channels of the fifth upsampling layer to the eighth upsampling layer is 64, the number of output channels is 64, and the deconvolution step size is 2; The 32nd to 62nd activation function layers all use the Relu activation function.

6. A deep network video denoising method based on model constraints as claimed in claim 5, characterized in that: In step S2.5, the target optimization function is specifically: Where Θ represents the video denoising network parameters, M is the maximum batch size, F(·; Θ) is the reconstructed denoising result, and x i is the reference frame image that needs to be restored through the adjacent frame degraded image, y i,j It represents the degraded image with batch size i and video frame number j.

7. A deep network video denoising method based on model constraints as claimed in claim 6, characterized in that: Step S3, the video denoising network is trained by using the video data set as follows: S3.a, part of the sequences in the Vimeo-90K video dataset are used as training sets, and the other part of the sequences are used as test sets, where the number of sequences in the training set is greater than the number of sequences in the test set; S3.b, input the training set into the video denoising network and execute step S2; wherein, when executing step S2.5, the objective optimization function minimizes the function value of the objective optimization function using the Adam optimization algorithm, and the Adam optimization algorithm is set to β1 = 0.9, β2 = 0.999, ε = 10 -8 , the learning rate is 10 -4 , the parameters of each layer of the semantic segmentation network are updated through gradient back propagation to complete the training.

8. A deep network video denoising method based on model constraints as claimed in claim 7, characterized in that: Step S3 also includes testing the video denoising network after completing the training: Annotating the test set to obtain an annotated test set; The test set is input into the trained video denoising network, the back propagation of the video denoising network is turned off, steps S2.1 to S2.4 are executed, and the execution results are compared with the labeled test set to obtain the video denoising network detection performance results.

9. A computer program product, comprising a computer program, characterized in that: When the program is executed by a processor, the steps of a deep network video denoising method based on model constraints described in any one of claims 1 to 8 are implemented.

Images