Video deinterlacing and super-resolution method based on recurrent Unet network

Abstract

The present invention discloses a method for video deinterlacing and super-resolution based on a cyclic Unet network. The method of the present invention is a multi-frame input and multi-frame output adaptive video deinterlacing based on a cyclic Unet neural network. Each time, the current two adjacent frames of data and the two-field deinterlacing prediction data predicted in the previous time are input, and the above data are input into the Unet neural network, and the two-field deinterlacing data predicted in the current time and the deinterlacing data restored in combination with the input data are output. If the input is standard definition data and the output requires high-definition data, the deinterlacing data is input into the SRnet (super-resolution restoration network) to output high-definition deinterlacing data. If no data conversion is required, the deinterlacing data is directly output. The cyclic Unet neural network of the present invention has a powerful video deinterlacing restoration capability, and can adaptively process interlaced data of different scenes without scene judgment to obtain an ideal deinterlacing effect.

Description

Technical Field

[0001] The invention belongs to the technical field of video image processing, and in particular relates to a video de-interlacing and super-resolution method based on a cyclic Unet network. Background Art

[0002] When transmitting television signals, due to the limitation of transmission bandwidth, the television signals are transmitted using interlaced scanning technology. That is, the television frame is divided into two television fields and scanned separately. The field composed of odd-numbered scanning lines is called the odd field (or top field), and the field composed of even-numbered scanning lines is called the even field (or bottom field). The odd and even fields are interlaced to form a television frame. In actual transmission, there are two scenarios: top field priority and bottom field priority. Taking top field priority as an example, the transmitted interlaced frame data comes from the top field data of the current frame and the bottom field data of the next frame.

[0003] Interlaced scanning technology transmission saves transmission bandwidth, but also brings some problems. First, since each frame transmitted is composed of the top field of the previous frame and the bottom field of the current frame (or the bottom field of the previous frame and the top field of the current frame), the vertical clarity of the interlaced scanned image is reduced. In addition, interlaced scanning will also cause obvious inter-field flickering, moving objects are prone to feathering effects, and nearly horizontal straight lines are prone to edge jagged effects, etc. In order to solve the above problems caused by field effects, deinterlacing technology has been developed. For example: direct merging technology of odd and even fields for still images, field copy merging technology and interpolation point technology that only retains a single field, field fusion technology combined with odd and even fields, deinterlacing technology based on motion compensation, etc. However, most of the above deinterlacing methods are effective for a specific type of interlaced data, and the overall deinterlacing effect is not good. The method based on scene judgment integrates multiple methods mentioned above, but this method is easy to cause deinterlacing errors due to scene misjudgment, resulting in poor deinterlacing effect. The de-interlacing technology based on motion compensation has a relatively good de-interlacing effect among the above methods, but this method requires large hardware resources, resulting in high product costs.

[0004] Deep neural networks have made breakthrough progress in various fields, and their technology has been extended to many related fields. Compared with other traditional machine learning methods, neural network models have stronger fitting and representation capabilities. Unet neural network is a lightweight network structure and has been proven to be an effective feature extraction network in the field of image processing. RNN (Recurrent Neural Network, RNN) neural network has the ability to connect previous information to the current task and can be used to complement information between previous and next frames. Summary of the invention

[0005] The purpose of the present invention is to provide a video deinterlacing and super-resolution method based on a cyclic Unet network in view of the shortcomings of the prior art, so as to solve the problem that the prior art adopts different deinterlacing processing means to deal with different scenes, and the restored video quality is not ideal due to inaccurate scene judgment.

[0006] The present invention includes a de-interlacing task and a super-resolution task, wherein the de-interlacing task uses a Unet neural network and the super-resolution task uses an SRnet neural network; specifically:

[0007] Step (1) constructs a Unet neural network; the Unet neural network includes a feature extraction module, a feature fusion module, and a feature recovery module.

[0008] Furthermore, the feature extraction module receives input data of the Unet neural network, including a convolution block and three residual stacks; the first convolution block includes two 3×3 standard convolutions and two nonlinear mapping layers prelu; each residual stack includes two residual submodules, each residual submodule includes a 1×1 standard convolution, a 3×3 separable convolution, and a nonlinear mapping layer prelu;

[0009] The feature fusion module performs feature fusion on the low-dimensional features and high-dimensional features output by the feature extraction module, and includes three residual stacks, each residual stack includes two residual submodules, and each residual submodule includes a 1×1 standard convolution, a 3×3 separable convolution, a nonlinear mapping layer prelu, and a 2×2 deconvolution;

[0010] The feature recovery module recovers the de-interlaced data from the feature fused data as the output of the Unet neural network, including two 3×1 standard convolutions and one 3×3 standard convolution.

[0011] Step (2) constructs an SRnet neural network; the SRnet neural network is a super-resolution restoration network, including two convolution blocks and two residual stacks.

[0012] Furthermore, the first convolution block includes a 3×3 standard convolution, two nonlinear mapping layers prelu, and a 3×3 deconvolution, and the second convolution block includes a 3×3 standard convolution and a nonlinear mapping layer prelu; each residual stack includes two residual sub-modules, and each residual sub-module includes a 1×1 standard convolution, a 3×3 separable convolution, and a nonlinear mapping layer prelu.

[0013] Step (3) jointly training the Unet neural network and the SRnet neural network;

[0014] First, the training data of the Unet neural network and the SRnet neural network are generated. The training data covers the application scenarios of deinterlacing as much as possible (such as subtitles, moving targets, still images, nearly horizontal straight line data, etc.).

[0015] Unet neural network predicts data DF corresponding to label data LF, loss function constrained by deinterlacing effect SRnet neural network prediction data SF corresponds to label data LS, loss function of super-resolution effect constraint Γ1 represents the L1 norm, Γ2 represents the L2 norm, represents the first-order gradient operator, and the constraint factor is set to 0.1≤α≤0.5.

[0016] The total loss function of joint training is Loos=Loos1+βLoos2, and the constraint factor is set as 0.5≤β≤1.0.

[0017] The output of the Unet neural network is used to constrain the video deinterlacing effect, and the output of the SRnet neural network is used to constrain the video super-resolution effect. Both use the L1+L2 norm constraint, and L2 is added to ensure the smoothness of the output image. After the joint training of Loos meets the requirements, the trained Unet neural network and SRnet neural network are obtained.

[0018] Step (4) extracts the input first frame data F1 into two field data, namely bottom field data F 1,b and top field data F 1,t ; The second frame data F2 is extracted into the bottom field data F 2,b and top field data F 2,t ; F 1,b 、F 1,t 、F 2,b 、F 2,t and two field data initialized to 0 as the first input of the Unet neural network; the Unet neural network outputs the top field prediction field data f corresponding to the bottom field data of the first frame 1,t , the bottom field prediction field data corresponding to the top field data of the second frame f 2,b , the first frame bottom field de-interlaced frame data DF1, the second frame top field de-interlaced frame data DF2.

[0019] Step (5) extracts the input second frame data F2 into bottom field data F 2,b and top field data F 2,t ; Extract the input third frame data F3 into two field data, namely bottom field data F 3,b and top field data F 3,t ; And so on, the input nth frame data F n Extract bottom field data F n,b and top field data Fn,t , the n+1th frame data F n+1 Extract bottom field data F n+1,b and top field data F n+1,t , frame number n = 3, 4, ..., N, N is the total number of input frames; F n,b 、F n,t 、F n+1,b 、F n+1,t and f n-1,t 、f n,b As the nth input of the Unet neural network; the Unet neural network outputs the top field prediction field data corresponding to the bottom field data of the nth frame f n,t , the bottom field prediction field data corresponding to the top field data of the n+1th frame f n+1,b , nth frame bottom field de-interlaced frame data DF 2n-1 , the n+1th frame top field de-interlaced frame data DF 2n ; The specific method is:

[0020] F n,b In the frame extraction direction, a 3×3 convolution interpolation function μ1 is used to obtain the intermediate variable D1 2n-1 ,D1 2n-1 =μ1(F n,b ), and then obtain the de-interlaced intermediate data D3 2n-1 =D1 2n-1 +D2 2n-1 , D2 2n-1 It is the prediction residual result of the 3×1 standard convolution output in the feature recovery module;

[0021] D3 2n-1 The top field prediction data f corresponding to the bottom field data of the nth frame is obtained through 3×3 standard convolution in the feature recovery module n,t =conv(D3 2n-1 ), conv(·) represents 3×3 smooth convolution;

[0022] F n+1,t In the frame extraction direction, another 3×3 convolution interpolation function μ2 is used to obtain the intermediate variable D4 2n ,D4 2n =μ2(F n+1,t ), and then obtain the de-interlaced intermediate data D6 2n =D4 2n +D5 2n , D5 2n It is the prediction residual result of the 3×1 standard convolution output in the feature recovery module;

[0023] D6 2n The bottom field prediction data f corresponding to the top field data of the n+1th frame is obtained through 3×3 standard convolution in the feature recovery module n+1,b=conv(D6 2n ), conv(·) represents 3×3 smooth convolution;

[0024] F n,b and f n,t According to the top and bottom field interlaced data format, synthesize DF 2n-1 ; F n+1,t and f n+1,b According to the top and bottom field interlaced data format, synthesize DF 2n .

[0025] Step (6) DF 2n-1 and DF 2n Input SRnet neural network to get the enlarged high-definition data SF 2n-1 and SF 2n , output high-definition data, frame number n = 1, 2, ..., N; SF 2n-1 =S1 2n-1 +S2 2n-1 , S.F. 2n =S1 2n +S2 2n , intermediate variable S1 2n-1 =μ(DF 2n-1 ), S1 2n =μ(DF 2n ), μ is the amplification function, S2 2n-1 and S2 2n It is the second residual stack output of the SRnet neural network.

[0026] The method of the present invention draws on the RNN concept, and the proposed recurrent Unet neural network can fully explore the information between the previous and next frames, as well as the frame itself, and finally obtain an ideal deinterlacing restoration effect. In addition, according to the actual application requirements, super-resolution restoration processing is added after deinterlacing, which can be selected according to actual needs. The video deinterlacing method proposed in the present invention not only has a good deinterlacing effect, but also has low computing power and is easy to deploy in hardware. The present invention is aimed at actual needs. If standard definition data is processed, after video deinterlacing, the standard definition needs to be converted into high-definition data output. Therefore, the method of the present invention adds image magnification processing (SRnet) after deinterlacing. This operation can be selected according to actual needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a flow chart of an embodiment of the method of the present invention;

[0028] Figure 2 It is a schematic diagram of Unet and SRnet neural networks. DETAILED DESCRIPTION

[0029] The present invention is described in detail below with the help of embodiments in conjunction with the accompanying drawings. However, it should be noted that unless otherwise specified, the numerical expressions, letter expressions and numerical values ​​involved in the embodiments do not limit the scope of the present invention, and these embodiments are provided only to make it easier for relevant personnel to understand the present invention. The technical methods known to technicians in the relevant fields are not described in detail. However, under certain conditions, the technology described in the present invention should be regarded as part of the specification.

[0030] The video deinterlacing and super-resolution method based on the cyclic Unet network includes deinterlacing tasks and super-resolution tasks. The deinterlacing task uses the Unet neural network, and the super-resolution task uses the SRnet neural network. The specific process is as follows Figure 1 As shown, Unet and SRnet neural networks are as follows Figure 2 As shown:

[0031] Step (1) constructing a Unet neural network; the Unet neural network includes a feature extraction module, a feature fusion module, and a feature recovery module.

[0032] The feature extraction module receives the input data of the Unet neural network, including a convolution block and three residual stacks; the first convolution block includes two 3×3 standard convolutions and two non-linear mapping layers prelu; each residual stack includes two residual sub-modules, each residual sub-module includes a 1×1 standard convolution, a 3×3 separable convolution, and a non-linear mapping layer prelu.

[0033] The feature fusion module performs feature fusion on the low-dimensional features and high-dimensional features output by the feature extraction module, including three residual stacks, each residual stack includes two residual sub-modules, and each residual sub-module includes a 1×1 standard convolution, a 3×3 separable convolution, a nonlinear mapping layer prelu, and a 2×2 deconvolution.

[0034] The feature recovery module recovers the de-interlaced data after feature fusion as the output of the Unet neural network, including two 3×1 standard convolutions and one 3×3 standard convolution.

[0035] Step (2) constructs an SRnet neural network; the SRnet neural network is a super-resolution restoration network, including two convolution blocks and two residual stacks. The first convolution block includes a 3×3 standard convolution, two nonlinear mapping layers prelu, and a 3×3 deconvolution, and the second convolution block includes a 3×3 standard convolution and a nonlinear mapping layer prelu; each residual stack includes two residual submodules, and each residual submodule includes a 1×1 standard convolution, a 3×3 separable convolution, and a nonlinear mapping layer prelu.

[0036] Step (3) jointly train the Unet neural network and the SRnet neural network.

[0037] First, the training data of the Unet neural network and the SRnet neural network are generated. The training data covers the application scenarios of deinterlacing as much as possible (such as subtitles, moving targets, still images, nearly horizontal straight line data, etc.).

[0038] Unet neural network predicts data DF corresponding to label data LF, loss function constrained by deinterlacing effect SRnet neural network prediction data SF corresponds to label data LS, loss function of super-resolution effect constraint Γ1 represents the L1 norm, Γ2 represents the L2 norm, represents a first-order gradient operator, and the constraint factor is set to 0.1≤α≤0.5. In this embodiment, α=0.3.

[0039] The total loss function of the joint training is Loos=Loos1+βLoos2, and the constraint factor is set as 0.5≤β≤1.0, and β=0.6 in this embodiment.

[0040] The output of the Unet neural network is used to constrain the video deinterlacing effect, and the output of the SRnet neural network is used to constrain the video super-resolution effect. Both use the L1+L2 norm constraint, and L2 is added to ensure the smoothness of the output image. After the joint training of Loos meets the requirements, the trained Unet neural network and SRnet neural network are obtained.

[0041] Step (4) extracts the input first frame data F1 into two field data, namely bottom field data F 1,b and top field data F 1,t ; The second frame data F2 is extracted into the bottom field data F 2,b and top field data F 2,t ; F 1,b 、F 1,t 、F 2,b 、F 2,t and two field data initialized to 0 as the first input of the Unet neural network; the Unet neural network outputs the top field prediction field data f corresponding to the bottom field data of the first frame 1,t , the bottom field prediction field data corresponding to the top field data of the second frame f 2,b , the first frame bottom field de-interlaced frame data DF1, the second frame top field de-interlaced frame data DF2.

[0042] Step (5) extracts the input second frame data F2 into bottom field data F 2,b and top field data F 2,t ; Extract the input third frame data F3 into two field data, namely bottom field data F 3,b and top field data F 3,t ; And so on, the input nth frame data Fn Extract bottom field data F n,b and top field data F n,t , the n+1th frame data F n+1 Extract bottom field data F n+1,b and top field data F n+1,t , frame number n = 3, 4, ..., N, N is the total number of input frames; F n,b 、F n,t 、F n+1,b 、F n+1,t and f n-1,t 、f n,b As the nth input of the Unet neural network; the Unet neural network outputs the top field prediction field data corresponding to the bottom field data of the nth frame f n,t , the bottom field prediction field data corresponding to the top field data of the n+1th frame f n+1,b , nth frame bottom field de-interlaced frame data DF 2n-1 , the n+1th frame top field de-interlaced frame data DF 2n ; The specific method is:

[0043] F n,b In the frame extraction direction, a 3×3 convolution interpolation function μ1 is used to obtain the intermediate variable D1 2n-1 ,D1 2n-1 =μ1(F n,b ), and then obtain the de-interlaced intermediate data D3 2n-1 =D1 2n-1 +D2 2n-1 , D2 2n-1 It is the prediction residual result of the 3×1 standard convolution output in the feature recovery module;

[0044] D3 2n-1 The top field prediction data f corresponding to the bottom field data of the nth frame is obtained through 3×3 standard convolution in the feature recovery module n,t =conv(D3 2n-1 ), conv(·) represents 3×3 smooth convolution;

[0045] F n+1,t In the frame extraction direction, another 3×3 convolution interpolation function μ2 is used to obtain the intermediate variable D4 2n ,D4 2n =μ2(F n+1,t ), and then obtain the de-interlaced intermediate data D6 2n =D4 2n +D5 2n , D5 2n It is the prediction residual result of the 3×1 standard convolution output in the feature recovery module;

[0046] D6 2nThe bottom field prediction data f corresponding to the top field data of the n+1th frame is obtained through 3×3 standard convolution in the feature recovery module n+1,b =conv(D6 2n ), conv(·) represents 3×3 smooth convolution;

[0047] F n,b and f n,t According to the top and bottom field interlaced data format, synthesize DF 2n-1 ; F n+1,t and f n+1,b According to the top and bottom field interlaced data format, synthesize DF 2n .

[0048] Step (6) DF 2n-1 and DF 2n Input SRnet neural network to get the enlarged high-definition data SF 2n-1 and SF 2n , output high-definition data, frame number n = 1, 2, ..., N; SF 2n-1 =S1 2n-1 +S2 2n-1 , S.F. 2n =S1 2n +S2 2n , intermediate variable S1 2n-1 =μ(DF 2n-1 ), S1 2n =μ(DF 2n ), μ is the amplification function, S2 2n-1 and S2 2n It is the second residual stack output of the SRnet neural network.

Claims

1. A video deinterlacing and super-resolution method based on a cyclic Unet network, including a deinterlacing task and a super-resolution task, wherein the deinterlacing task uses a Unet neural network and the super-resolution task uses an SRnet neural network; the characteristics are: Step (1) constructing a Unet neural network; the Unet neural network includes a feature extraction module, a feature fusion module, and a feature recovery module; Step (2) constructing an SRnet neural network; the SRnet neural network is a super-resolution restoration network, including two convolution blocks and two residual stacks; Step (3) jointly training the Unet neural network and the SRnet neural network; First, generate training data for Unet neural network and SRnet neural network; The Unet neural network predicts that the data DF corresponds to the label data LF, and the loss function constrained by the de-interlacing effect is Loos1 = Γ2(DF, LF) + αΓ1(▽(DF), ▽(LF)); The SRnet neural network predicts that the data SF corresponds to the label data LS, and the loss function constrained by the super-resolution effect is Loos2 = Γ2(SF, LS) + αΓ1(▽(SF), ▽(LS)); Γ1 represents the L1 norm, Γ2 represents the L2 norm, ▽ represents the first-order gradient operator, the total loss function of joint training is Loos=Loos1+βLoos2, α and β are the set constraint factors; After the joint training of Loos meets the requirements, the trained Unet neural network and SRnet neural network are obtained; Step (4) extracts the input first frame data F1 into two field data, namely bottom field data F 1,b and top field data F 1,t ; The second frame data F2 is extracted into the bottom field data F 2,b and top field data F 2,t ; F 1,b 、F 1,t 、F 2,b 、F 2,t and two field data initialized to 0 as the first input of the Unet neural network; the Unet neural network outputs the top field prediction field data f corresponding to the bottom field data of the first frame 1,t , the bottom field prediction field data corresponding to the top field data of the second frame f 2,b , the first frame bottom field de-interlaced frame data DF1, the second frame top field de-interlaced frame data DF2; Step (5) extracts the input second frame data F2 into bottom field data F 2,b and top field data F 2,t ; Extract the input third frame data F3 into two field data, namely bottom field data F 3,b and top field data F 3,t ; And so on, the input nth frame data F n Extract bottom field data F n,b and top field data F n,t , the n+1th frame data F n+1 Extract bottom field data F n+1,b and top field data F n+1,t , frame number n = 3, 4, ..., N, N is the total number of input frames; F n,b 、F n,t 、F n+1,b 、F n+1,t and f n-1,t 、f n,b As the nth input of the Unet neural network; the Unet neural network outputs the top field prediction field data corresponding to the bottom field data of the nth frame f n,t , the bottom field prediction field data corresponding to the top field data of the n+1th frame f n+1,b , nth frame bottom field de-interlaced frame data DF 2n-1 , the n+1th frame top field de-interlaced frame data DF 2n ; Step (6) DF 2n-1 and DF 2n Input SRnet neural network to get the enlarged high-definition data SF 2n-1 and SF 2n , output high-definition data, frame number n = 1, 2, ..., N; SF 2n-1 =S1 2n-1 +S2 2n-1 , S.F. 2n =S1 2n +S2 2n , intermediate variable S1 2n-1 =μ(DF 2n-1 ), S1 2n =μ(DF 2n ), μ is the amplification function, S2 2n-1 and S2 2n It is the second residual stack output of the SRnet neural network.

2. The video deinterlacing and super-resolution method based on a cyclic Unet network as claimed in claim 1, characterized in that: The feature extraction module receives input data of the Unet neural network, and includes a convolution block and three residual stacks; the convolution block includes two 3×3 standard convolutions and two nonlinear mapping layers prelu; each residual stack includes two residual submodules, and each residual submodule includes a 1×1 standard convolution, a 3×3 separable convolution, and a nonlinear mapping layer prelu; The feature fusion module performs feature fusion on the low-dimensional features and high-dimensional features output by the feature extraction module, and includes three residual stacks, each residual stack includes two residual submodules, and each residual submodule includes a 1×1 standard convolution, a 3×3 separable convolution, a nonlinear mapping layer prelu, and a 2×2 deconvolution; The feature recovery module recovers the de-interlaced data from the feature fused data as the output of the Unet neural network, including two 3×1 standard convolutions and one 3×3 standard convolution.

3. The video deinterlacing and super-resolution method based on a cyclic Unet network as claimed in claim 2, characterized in that: The specific method of step (5) is: F n,b In the frame extraction direction, a 3×3 convolution interpolation function μ1 is used to obtain the intermediate variable D1 2n-1 ,D1 2n-1 =μ1(F n,b ), and then obtain the de-interlaced intermediate data D3 2n-1 =D1 2n-1 +D2 2n-1 , D2 2n-1 It is the prediction residual result of the 3×1 standard convolution output in the feature recovery module; D3 2n-1 The top field prediction data f corresponding to the bottom field data of the nth frame is obtained through 3×3 standard convolution in the feature recovery module n,t =conv(D3 2n-1 ), conv(·) represents 3×3 smooth convolution; F n+1,t In the frame extraction direction, another 3×3 convolution interpolation function μ2 is used to obtain the intermediate variable D4 2n ,D4 2n =μ2(F n+1,t ), and then obtain the de-interlaced intermediate data D6 2n =D4 2n +D5 2n , D5 2n It is the prediction residual result of the 3×1 standard convolution output in the feature recovery module; D6 2n The bottom field prediction data f corresponding to the top field data of the n+1th frame is obtained through 3×3 standard convolution in the feature recovery module n+1,b =conv(D6 2n ), conv(·) represents 3×3 smooth convolution; F n,b and f n,t According to the top and bottom field interlaced data format, synthesize DF 2n-1 ; F n+1,t and f n+1,b According to the top and bottom field interlaced data format, synthesize DF 2n .

4. The video deinterlacing and super-resolution method based on a cyclic Unet network as claimed in claim 1, characterized in that: The first convolution block of the SRnet neural network includes a 3×3 standard convolution, two nonlinear mapping layers prelu, and a 3×3 deconvolution, and the second convolution block includes a 3×3 standard convolution and a nonlinear mapping layer prelu; each residual stack includes two residual submodules, and each residual submodule includes a 1×1 standard convolution, a 3×3 separable convolution, and a nonlinear mapping layer prelu.

5. The video deinterlacing and super-resolution method based on cyclic Unet network as claimed in claim 1, characterized in that: The constraint factors are 0.1≤α≤0.5, 0.5≤β≤1.0.

Images