A mask-based deep learning video moiré removal method

Abstract

The present invention relates to a mask-based deep learning video moiré removal method, comprising the following steps: step S1: obtaining paired images of high-definition images and moiré images, and enhancing the data to construct an image data set; step S2: constructing a two-stage U-Net network model, and training based on the image data set to obtain an image moiré removal model; step S3: inputting the image of the video to be processed frame by frame into the image moiré removal model to obtain an image with moiré removed; step S4: using the original video image frame and the image frame with moiré removed as input, obtaining an output time-domain aligned video image frame through the convolutional neural network, merging the output time-domain aligned video image frames, and finally obtaining a complete video with time-domain consistency. The invention can more simply achieve time-domain consistency and obtain a video with better moiré removal effect.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to a mask-based deep learning video moiré removal method. Background Art

[0002] Moiré is an interference pattern generated when two spatial frequencies are close. The interference pattern is an irregular stripe, a moiré image. Due to its irregular shape, different colors, and wide frequency distribution range, it is difficult to remove. At present, there are two main methods for removing moiré. One is the traditional method. Liu proposed a low-polynomial and sparse matrix decomposition method to remove moiré patterns on high-frequency images. This method uses the frequency and spatial characteristics of moiré images to process and can only remove high-frequency moiré. Some low-frequency moiré is difficult to remove and easily causes the image to be too smooth and the details to be blurred. Another method is to use a neural network for supervised learning. By learning the mapping relationship between the moiré image and the clean image, a clean image is finally output. With Sun et al. proposing the use of convolutional neural networks to remove moiré, deep learning methods have begun to be widely used in the field of moiré removal. Gao et al. proposed a multi-scale feature framework to enhance semantic information, but the network structure of their proposed method is not strong, the feature expression ability is weak, and the features of the input image cannot be fully extracted. Bolin et al. proposed a moiré removal method based on a deep convolutional network that can effectively remove most moiré, but they only use low resolution as input and can only process moiré on a single frequency band. Although various existing attempts have achieved certain results, they cannot be well applied in actual projects.

[0003] At present, many image algorithms perform well when processing a single image, but directly applying them to videos often encounters the problem of time domain inconsistency, resulting in flickering in the video. In order to improve the consistency of the time domain, most researchers have designed dedicated algorithms to improve the consistency of the time domain for different video processing tasks. For example, the solution proposed by Lei is for video colorization, while the solution proposed by Shocher is for video super-resolution, which is not universal. Bonneel et al. proposed to improve this problem by minimizing the distance between the output and processed videos in the gradient domain and the distance between two consecutive output frames, but this is based on the assumption that the output and processed videos are similar in the gradient domain, which is impossible in reality. In addition, many scholars have tried to use convolutional neural networks to improve the problem of time domain inconsistency, but their performance will gradually decrease as training progresses. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a mask-based deep learning video moiré removal method, which can more simply achieve temporal consistency and obtain a video with better moiré removal effect.

[0005] To achieve the above object, the present invention adopts the following technical solution:

[0006] A mask-based deep learning video moiré removal method comprises the following steps:

[0007] Step S1: Obtain paired images of high-definition images and moiré images, enhance the data, and construct an image dataset;

[0008] Step S2: construct a two-stage U-Net network model and train it based on the image data set to obtain an image moiré removal model;

[0009] Step S3: inputting the to-be-processed video frame by frame into the image moiré removal model to obtain an image with moiré removed;

[0010] Step S4: The original video image frame and the video image frame with moiré removed obtained by the image moiré removal model are taken as input, and the output time-domain aligned video image frame is obtained through the convolutional neural network, and finally the output time-domain aligned video image frame is synthesized to obtain a complete video with consistent time domain.

[0011] Furthermore, the data enhancement includes cropping and rotating the image.

[0012] Furthermore, the two-stage U-Net network model includes a mask generation model and a moiré removal network model.

[0013] Further, the model for generating the mask includes an encoder part and a decoder part;

[0014] The encoder part inputs the moiré image in the training set into the encoder network to perform feature extraction, and finally obtains the mask image. The specific steps are as follows:

[0015] The feature extraction stage contains three convolutional layers, which can be expressed by the following formula:

[0016] Downsampling:

[0017] input=conv_R{conv_R(I moire )} (1)

[0018] F1=conv_R{conv_R{DOWN(input)}} (2)

[0019] F2=conv{conv_R{conv_R{DOWN(F1)}}} (3)

[0020] Im=I clear -I moire(4)

[0021] Among them, Im represents the difference between the high-definition image and the moiré image, conv_R represents a convolutional layer plus an activation function RELU, conv represents a convolutional layer, DOWN represents downsampling, and F represents the output result obtained in the feature extraction stage. The moiré image learns deep features through different degrees of convolution.

[0022] Upsampling:

[0023] F3=conv{conv_R{conv_R{UP(F2)+F1}}} (5)

[0024] I mask =conv_R{conv_R{UP(F3)+input}} (6)

[0025] The decoder part consists of upsampling. After learning, downsampling is restored to the original image size through upsampling. Upsampling is achieved by deconvolution, and finally I is obtained. mask Output image, trans represents deconvolution; the upsampling part connects the output of the corresponding layer of the downsampling with the input of the upsampling through skipconnection.

[0026] Furthermore, the de-moiré network model adopts a multi-resolution de-moiré network structure to construct a bottom-up structure to fuse fine features into coarse features, specifically:

[0027] Downsampling:

[0028] input=conv_R{conv_R{C(I moire , I mask )}} (7)

[0029] S1=conv_R{conv_R{DOWN(input)}} (8)

[0030] S2=conv_R{conv_R{DOWN(S1)}} (9)

[0031] S3=conv{conv_R{conv_R{DOWN(S2)}}} (10)

[0032] Upsampling:

[0033] S4=conv{conv_R{conv_R{UP(S3)+S2}}} (11)

[0034] S5=conv{conv_R{conv_R{UP(S4)+S1+S2}}} (12)

[0035] output=conv_R{conv_R{UP(S5)+input+S2+S1}}} (13)

[0036] Among them, S represents the feature map extracted at each stage.

[0037] Furthermore, the model loss function of generating the mask is expressed as:

[0038] L mask =L2(I mask ,I output ) (14)

[0039] The loss function of the de-moiré network model is expressed as:

[0040] L output =L2(I clear ,I output ) (15)

[0041] Both networks use the L2 loss function to perform point-to-point loss calculations on images.

[0042] Furthermore, the model training adopts a progressive training strategy, dividing the training process into multiple sub-training cycles, and performing training in the sub-training cycles in sequence. The training starts with a large learning rate, and the learning rate is gradually reduced after each sub-training cycle until the iteration is completed.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] The present invention can more simply achieve temporal consistency and obtain a video with better moiré removal effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a network structure for generating a mask in one embodiment of the present invention;

[0046] Figure 2 is an effect diagram of generating a mask in one embodiment of the present invention;

[0047] Figure 3 is a comparison diagram of the generated mask in one embodiment of the present invention;

[0048] Figure 4 It is a moiré-removing network structure in one embodiment of the present invention;

[0049] Figure 5 is a diagram of predicted mask effect in one embodiment of the present invention;

[0050] Figure 6 This is a diagram showing the effect of removing moiré patterns in one embodiment of the present invention;

[0051] Figure 7 This is a comparison diagram of removing moiré patterns in one embodiment of the present invention;

[0052] Figure 8 This is a video de-moiré network structure in one embodiment of the present invention. DETAILED DESCRIPTION

[0053] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0054] The present invention provides a mask-based deep learning video moiré removal method, comprising the following steps:

[0055] Step S1: Obtain paired images of high-definition images and moiré images, enhance the data, and construct an image dataset;

[0056] Step S2: construct a two-stage U-Net network model and train it based on the image data set to obtain an image moiré removal model;

[0057] Step S3: inputting the to-be-processed video frame by frame into the image moiré removal model to obtain an image with moiré removed;

[0058] Step S4: The original video image frame and the image frame with moiré removed obtained by the moiré removal model are used as input, and the output time-domain aligned video image frame is obtained through the convolutional neural network. Finally, the output time-domain aligned video image frame is synthesized to obtain a complete video with final time-domain consistency.

[0059] In this embodiment, a large number of I moire , I clear Paired color images, by cropping the images. Get a lot of augmented image data. Calculate their difference to locate the moiré pattern and get the mask image I mask , the original high-definition image I clear and moiré image I moire Use the image as the network input, train the network to learn to generate the mask image, and save the model of generating the mask.

[0060] Preferably, the model for generating the mask includes an encoder part and a decoder part:

[0061] In the encoder part, in the mask network, the encoder part inputs the moiré image in the training set into the encoder network to perform the feature extraction phase, obtains the feature map, and finally obtains the image close to the real mask after optimization. The specific steps are as follows:

[0062] The feature extraction stage contains three convolutional layers, which can be expressed by the following formula:

[0063] Downsampling:

[0064] input=conv_R{conv_R(I moire )} (1)

[0065] F1=conv_R{conv_R{DOWN(input)}} (2)

[0066] F2=conv{conv_R{conv_R{DOWN(F1)}}} (3)

[0067] I m =I clear -I moire (4)

[0068] Among them, I m It represents the difference between the high-definition image and the moiré image. conv_R represents a convolutional layer plus an activation function RELU. conv represents a convolutional layer. DOWN represents downsampling. F represents the output result obtained in the feature extraction stage. The moiré image learns deep features through different degrees of convolution.

[0069] Upsampling:

[0070] F3=conv{conv_R{conv_R{UP(F2)+F1}}} (5)

[0071] I mask =conv_R{conv_R{UP(F3)+input}} (6)

[0072] The decoder part consists of upsampling. After learning, downsampling is restored to the original image size through upsampling. Upsampling is achieved by deconvolution, and finally I is obtained. mask Output image, trans represents deconvolution; the upsampling part connects the output of the corresponding layer of downsampling with the input of upsampling through skipconnection. The generation network of the entire mask is as follows Figure 1 As shown in the figure, the upsampling part connects the output of the downsampling corresponding layer with the upsampling input through skip connection, which can effectively avoid gradient explosion and enhance information by retaining the original features.

[0073] In this embodiment, after the mask model is obtained in step 1, a de-moiré network is introduced to predict the mask of the moiré image. The de-moiré network model adopts a multi-resolution de-moiré network structure, which includes three parallel branches. Each branch learns the original resolution. It can effectively remove moiré patterns in different frequency bands and construct a bottom-up structure to fuse fine features into coarse features to make full use of image information. The output feature map is restored to the original resolution by upsampling at the end of each branch. The model structure is as follows Figure 4 As shown, the structure of each part can be expressed by the following formula:

[0074] Downsampling:

[0075] input=conv_R{conv_R{C(I moire , I mask )}} (7)

[0076] S1=conv_R{conv_R{DOWN(input)}} (8)

[0077] S2=conv_R{conv_R{DOWN(S1)}} (9)

[0078] S3=conv{conv_R{conv_R{DOWN(S2)}}} (10)

[0079] Upsampling:

[0080] S4=conv{conv_R{conv_R{UP(S3)+S2}}} (11)

[0081] S5=conv{conv_R{conv_R{UP(S4)+S1+S2}}} (12)

[0082] output=conv_R{conv_R{UP(S5)+input+S2+S1}}} (13)

[0083] Among them, S represents the feature map extracted at each stage. The network makes full use of skip-connection to connect feature maps between different frequency bands. First, it can prevent gradient explosion, and second, it adds the feature maps output by different branches, making full use of the moiré features of different frequency bands to enhance image information.

[0084] Preferably, in this embodiment, the model for generating the mask uses I clear with I moire The difference I m Compare it with the color image output by the network to make the output continuously fit I m , the loss function is expressed as:

[0085] L mask =L2(I mask ,I output ) (14)

[0086] The L2 loss function used here is also called the minimum square error, which is to minimize the sum of the squares of the difference between the mask image and the moire image. Its formula is as follows:

[0087]

[0088] I represents each pixel. Since the image is an RGB color image, when calculating the loss function, it is necessary to calculate the loss value of each pixel in the three channels of the image separately.

[0089] The de-moiré network model uses the output de-moiré image I output With the clean image I clear For comparison, the loss function is expressed as:

[0090] L output =L2(I clear ,I output ) (15)

[0091] Both networks use the L2 loss function to perform point-to-point loss calculations on images.

[0092] Preferably, in this embodiment, the model training adopts a progressive training strategy, and the training process is divided into multiple sub-training cycles. The training starts with a large learning rate. In the early stage of training, the learning rate is large and the learning is relatively fast. However, as the learning progresses, we hope that the curve will swing in an area near the minimum value, rather than swinging around the minimum value by a large margin. Therefore, we gradually reduce the learning rate after each iteration of a training cycle, so that the curve can swing in a small area until the iteration is completed.

[0093] Embodiment 1:

[0094] In this embodiment, a method using video depth prior is used to solve the problem of time domain inconsistency. Because in the process of using video frames to train the convolutional network, the network prediction output of the corresponding image blocks between different video frames tends to be consistent, so the original video image frame and the video frame processed by the image de-moiré model can be input into a convolutional network to achieve the purpose of stabilizing the time domain.

[0095] In this embodiment, a simple framework is adopted to utilize the deep video prior and train a convolutional neural network on the video to obtain a moiré-free video. Figure 8 As shown in the figure, a single input video frame I t The corresponding pre-processed frame P is obtained by the image processing algorithm f t , use a convolutional neural network g to imitate the image algorithm f to get the output frame Q t.

[0096] Unlike previous methods, our training only requires a single video without a training set, and each iteration uses a single frame to optimize it:

[0097]

[0098] Among them, L data represents g(I t ;θ) and P t The distance (here we use L1 distance), when near And the training is stopped before the flicker overfitting. Based on this structure, the problem of time domain inconsistency can be alleviated.

[0099] The L1 distance is expressed as:

[0100]

[0101] Where p represents that the image is a P-dimensional vector.

[0102] We use Perceptual Loss as the loss function. The loss function can be expressed as:

[0103]

[0104] Among them, j represents the jth layer of the network, C j H j W j represents the size of the feature map of the jth layer, and φ represents the network.

[0105] The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made according to the scope of the patent application of the present invention should fall within the scope of the present invention.

Claims

1. A mask-based deep learning video moiré removal method, characterized in that: The following steps are involved: Step S1: Obtain paired images of high-definition images and moiré images, enhance the data, and construct an image dataset; Step S2: construct a two-stage U-Net network model and train it based on the image data set to obtain an image moiré removal model; Step S3: inputting the to-be-processed video frame by frame into the image moiré removal model to obtain an image with moiré removed; Step S4: using the original video image frame and the image frame with moiré removed obtained by the above-mentioned image moiré removal model as input, obtaining the output time-domain aligned video image frame through a deep convolutional neural network, synthesizing the output time-domain aligned video image frame, and finally obtaining a complete video with consistent time domain; The two-stage U-Net network model includes a mask generation model and a moiré removal network model; The model for generating the mask includes an encoder part and a decoder part; The encoder part inputs the moiré image in the training set into the encoder network to perform feature extraction, and finally obtains the mask image. The specific steps are as follows: The feature extraction stage contains three convolutional layers, which are represented by the following formulas: Downsampling: input=conv_R{conv_R(I moire )} (1) F1=conv_R{conv_R{DOWN(input)}} (2) F2=conv{conv_R{conv_R{DOWN(F1)}}} (3) Im=I clear -I moire (4) Among them, Im represents the difference between the high-definition image and the moiré image, conv_R represents a convolution layer plus an activation function RELU, conv represents a convolution layer, DOWN represents downsampling, and F represents the output result obtained in the feature extraction stage. The moiré image learns deep features through different degrees of convolution; Upsampling: F3=conv{conv_R{conv_R{UP(F2)+F1}}} (5) I mask =conv_R{conv_R{UP(F3)+input}} (6) The decoder part consists of upsampling. After learning, the downsampling is restored to the original image size through upsampling. The upsampling is realized by deconvolution, and finally I is obtained. mask Output image, trans represents deconvolution; the upsampling part connects the output of the corresponding layer of downsampling with the input of upsampling through skipconnection; The de-moiré network model adopts a multi-resolution de-moiré network structure to construct a bottom-up structure to fuse fine features into coarse features, specifically: Downsampling: input=conv_R{conv_R{C(I moire ,I mask )}} (7) S1=conv_R{conv_R{DOWN(input)}} (8) S2=conv_R{conv_R{DOWN(S1)}} (9) S3=conv{conv_R{conv_R{DOWN(S2)}}} (10) Upsampling: S4=conv{conv_R{conv_R{UP(S3)+S2}}} (11) S5=conv{conv_R{conv_R{UP(S4)+S1+S2}}} (12) output=conv_R{conv_R{UP(S5)+input+S2+S1}}} (13) Among them, S represents the feature map extracted at each stage; The step S4 specifically includes: inputting a single video frame I t The corresponding pre-processed frame P is obtained by removing the moiré pattern model from the image t , the convolutional neural network g is used to obtain the output frame Q t ; The deep convolutional neural network g used to align the temporal domain between video image frames is a U-NET image transformation network; wherein the U-NET network is an encoder-decoder architecture, including three downsampling and three upsampling operations, forming a U-shaped structure; wherein the downsampling operation is implemented by a maximum pooling layer, and the upsampling operation is implemented by a transposed convolution; Training is done on a single video without a training set, and each iteration uses a single frame to optimize it: Among them, L data represents g(I t ;θ) and P t The distance, when near And stop training before overfitting occurs.

2. The mask-based deep learning video de-moiré method according to claim 1, characterized in that: Data augmentation includes cropping and rotating the images.

3. The mask-based deep learning video de-moiré method according to claim 2, characterized in that: The model loss function for generating the mask is expressed as: L mask =L2(I mask ,I output ) (14) The loss function of the de-moiré network model is expressed as: L output =L2(I clear ,I output ) (15) Both networks use the L2 loss function to perform point-to-point loss calculations on images.

4. The mask-based deep learning video de-moiré method according to claim 1, characterized in that: The model training adopts a progressive training strategy, which divides the training process into multiple sub-training cycles, and conducts training in each sub-training cycle in sequence. The training starts with a preset large learning rate, and the learning rate is gradually reduced after each sub-training cycle until the iteration is completed.

Images