Unsupervised image moire removing method based on deep learning

Abstract

The invention discloses an unsupervised image moire pattern removing method based on deep learning. The method comprises the following steps: firstly, preparing a moire pattern image and a clear image which do not need to be paired; an unsupervised network model of image moire removal is constructed, and the model is based on a generative adversarial network architecture, is combined with a comparative learning method, and comprises a generator, a discriminator and a content encoder. And then carrying out unsupervised training on the constructed image moire removal network model. According to the method, the mapping relation from the moire image to the clear image can be learned from the non-paired data set, and the tedious work of collecting paired images in the real world is avoided. Experiments show that the method can achieve a good moire removal effect on a non-paired data set.

Description

Technical Field

[0001] The present invention belongs to the field of image moiré removal based on deep learning technology, and specifically relates to an unsupervised image moiré removal method based on deep learning. Background Art

[0002] Moiré is an interfering pattern that appears on an image due to the optical difference between the image acquisition device (such as a camera, scanner, etc.) and the display device (such as a display). It usually appears as regular light and dark stripes or ripples in the image, which will damage the quality and clarity of the image and seriously affect many downstream tasks based on images, such as target detection, image segmentation, etc.

[0003] In today's world, digital screens are everywhere, and mobile phone cameras have become a very frequently used device in people's daily life. Mobile phones are needed to take pictures in many occasions, and it is inevitable to take pictures of electronic screens, such as meeting records, etc. However, the photos taken will be more or less disturbed by moiré and unclear, which seriously affects the quality of the picture. Moiré problems occur in many fields, such as photography, printing, texture analysis, etc. With the development of deep learning, people have found that deep learning performs well in the field of image processing, and its application in image moiré removal is also possible. Deep learning can use its powerful feature learning ability and pattern recognition ability to reduce or eliminate the influence of moiré. At present, many researchers have also used deep learning methods to study image moiré removal tasks and have achieved certain results, but most of the research is based on supervised learning and requires a large number of paired training data sets. Considering that photos taken in real life cannot have paired data sets for model training, in order to remove moiré from moiré images taken in reality, the present invention proposes an unsupervised moiré removal method based on deep learning. Summary of the invention

[0004] Technical problem to be solved: In view of the defect that the existing de-moiré model requires a large number of paired training data sets for training, while paired data sets are difficult to obtain in real life, the present invention provides an unsupervised image de-moiré method based on deep learning. An unsupervised image de-moiré method without paired training data sets is proposed. The method can learn the mapping relationship from the moiré image to the clear image from the unpaired data set, thereby effectively solving the limitation that the existing method requires a large number of paired data sets.

[0005] Implementation steps: An unsupervised image de-moiré method based on deep learning, the basic steps are:

[0006] Step 1: Prepare a moiré image and a clear image that do not require pairing.

[0007] Step 2: Build an unsupervised network model for image moiré removal.

[0008] The network model is based on the generative adversarial network architecture and combines contrastive learning methods. The network model includes a generator, a discriminator, and a content encoder.

[0009] Step 3: Perform unsupervised training on the constructed image de-moiré network model using the image dataset obtained in step 1.

[0010] Step 4: The generator in the trained image demoiré network receives the verification dataset image that needs to be demoiré processed, and outputs the image after the demoiré processing is completed.

[0011] Furthermore, the specific structure of the generator: The generator consists of two parts, namely the encoder module and the decoder module. A skip connection is used between the encoder module and the decoder module to avoid gradient disappearance.

[0012] Specific structure of the encoder module: The encoder module first uses 7×7 convolution and 2x bilinear downsampling operations to encode the input moiré image into a low-resolution feature map. Then, 5×5 convolution and 2x bilinear downsampling operations are used again to further reduce the feature map resolution. Finally, 5 residual blocks are used to extract more complex and deeper features in the low-resolution space.

[0013] Specific structure of the decoder module: The decoder module accepts the deep output features from the encoder module. It first uses 4 residual blocks to process the features from the encoder module, then uses 2x bilinear upsampling and 3×3 convolution to gradually restore the image features, and finally uses 2x bilinear upsampling and 3×3 convolution again to restore the image features to the original image size and complete the clear image output.

[0014] Furthermore, the specific structure of the discriminator is as follows:

[0015] PatchGAN is used as the discriminator, which consists of three convolutional layers with a stride of 2, two convolutional layers with a stride of 1, and finally an average pooling layer.

[0016] Furthermore, the specific structure of the content encoder is as follows:

[0017] The content encoder consists of a 3×3 convolutional layer and two residual blocks, which are used to extract the content features of the input image and the output image to ensure the consistency of the output content with the input content.

[0018] Furthermore, the specific method of step 3 is as follows:

[0019] The network model is trained by first inputting the moiré image I prepared in step 1 input ; Then, the desired image I after de-moiré processing is obtained through the generator in the designed network model G ; Finally, the contrast loss, content loss, generator loss and discriminator loss functions are used to continuously optimize the model output after de-moiré processing image I G , making it gradually approach the clear image I in style clear , far away from the moiré image I input .

[0020] During the training process, the loss function L consists of four parts: contrast loss, content loss, generator loss, and discriminator loss.

[0021] Specifically, the contrast loss is constructed as follows:

[0022] From the input image I input Randomly select N+1 patches from the output image I G Randomly select a corresponding patch from . The two corresponding patches are represented as positive samples and anchor points, I input The other N patches in are represented as negative samples. Then, a contrast loss function is used to maximize the difference between positive samples and anchor points (I input and I G Specifically, firstly, the anchor point (I G The patches in ), positive samples and N negative samples are mapped to k-dimensional vectors using two layers of MLP and are represented as v, v respectively. + and v - . After that, a cross entropy loss is established:

[0023]

[0024] Among them, sim(u,v) represents the cosine similarity between u and v, τ represents the temperature parameter, which is set to 0.07, represents the nth negative sample.

[0025] At the same time, the input image I is also extracted input And the output image I G The deep features after different layers of the encoder module are used to construct the contrast loss. The features after the first convolutional layer, the second convolutional layer, the first residual block, and the fifth residual block in the encoder module are extracted to construct the multi-layer contrast loss, which are expressed as L2(v,v + ,v - ), L3(v,v + ,v -), L4(v,v + ,v - ), L5(v,v + ,v - ).

[0026] Therefore, the final contrast loss is:

[0027] L contra =L1(v,v + ,v - )+L2(v,v + ,v - )+L3(v,v + ,v - )+L4(v,v + ,v - )+L5(v,v + ,v - )

[0028] Specifically, the content loss is constructed as follows:

[0029] L con =||E c (I G )-E c (I input )||1

[0030] Among them, E c represents the content encoder and ||*||1 represents the L1 loss.

[0031] Specifically, the generator loss is as follows:

[0032] L G =(D(I G )-1) 2

[0033] Where D represents the discriminator.

[0034] Specifically, the discriminator loss is as follows:

[0035] L D =(D(I G )) 2 +(D(I clear )-1) 2

[0036] Among them, I clear Represents an unpaired true clear image.

[0037] Furthermore, the specific method of step 4 is as follows:

[0038] Load the weights of the generator in the image de-moiré network model trained in step 3 and update the parameters in the model. moire As input data, it is passed into the network model, and the output image I is obtained after the generator removes the moiré pattern. G .

[0039] The beneficial effects of the present invention are as follows:

[0040] The present invention innovatively proposes that the mapping relationship from moiré images to clear images can also be learned from unpaired datasets, thus avoiding the tedious work of collecting paired images from the real world. Experiments show that the present invention can achieve good de-moiré effects on unpaired datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a flow chart of an embodiment of the present invention.

[0042] Figure 2 This is a network structure diagram of unsupervised image moiré removal based on deep learning in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0044] The present invention first carries out the following definitions and descriptions:

[0045] I moire : Moiré image

[0046] I clear : Real unpaired clear images

[0047] I G : Output image of the de-moiré network model

[0048] Implementation steps: Figure 1 As shown, the present invention proposes an unsupervised image de-moiré method based on deep learning, and its basic steps are:

[0049] Step 1: Prepare a moiré image and a clear image that do not require pairing.

[0050] Obtain the image dataset required for network training in step 2. In order to make a better quantitative comparison, this embodiment uses the training dataset in the UHDM dataset as the training dataset for unsupervised training, and tests on the test set in the UHDM dataset. Note that although the UHDM dataset is a paired dataset, we use unpaired data in the training process.

[0051] Download the public dataset UHDM dataset, which contains 5000 sets of 4K images of different scenes. Each set of images contains two images, namely, the moiré image I and the moiré image I contaminated by moiré. moire and the original clear image I clear . 4500 sets of images in the 5000 sets of images are used as training data sets, and the other 500 sets of images are used as verification data sets. In the unsupervised training process of the present invention, we use 4500 sets of images in the training data set as training data, but we use unpaired data in the training process.

[0052] Step 2: Build an unsupervised network model for image moiré removal.

[0053] The network model is based on the generative adversarial network architecture and combines contrastive learning methods. The network model includes a generator, a discriminator, and a content encoder.

[0054] The specific structure of the generator: The generator consists of two parts, the encoder module and the decoder module. A skip connection is used between the encoder module and the decoder module to avoid gradient disappearance.

[0055] Specific structure of the encoder module: The encoder module first uses 7×7 convolution and 2x bilinear downsampling operations to encode the input moiré image into a low-resolution feature map. Then, 5×5 convolution and 2x bilinear downsampling operations are used again to further reduce the feature map resolution. Finally, 5 residual blocks are used to extract more complex and deeper features in the low-resolution space.

[0056] Specific structure of the decoder module: The decoder module accepts the deep output features from the encoder module. It first uses 4 residual blocks to process the features from the encoder module, then uses 2x bilinear upsampling and 3×3 convolution to gradually restore the image features, and finally uses 2x bilinear upsampling and 3×3 convolution again to restore the image features to the original image size and complete the clear image output.

[0057] The specific structure of the discriminator is as follows:

[0058] PatchGAN is used as the discriminator, which consists of three convolutional layers with a span of 2, two convolutional layers with a span of 1, and finally an average pooling layer. It can reduce the parameters of the network and perform faster than other traditional discriminators. The role of the discriminator is to judge whether a given image is a real clean image or a fake image generated by the generator, thereby guiding the generator to generate more realistic images.

[0059] The specific structure of the content encoder is as follows:

[0060] The content encoder consists of a 3×3 convolutional layer and two residual blocks, which are used to extract the content features of the input image and the output image to ensure the consistency of the output content with the input content.

[0061] Step 3: Perform unsupervised training on the constructed image de-moiré network model using the image dataset obtained in step 1. The specific method is as follows:

[0062] The network model is trained by first inputting the moiré image I prepared in step 1 input ; Then, the desired image I after de-moiré processing is obtained through the generator in the designed network model G ; Finally, the contrast loss, content loss, generator loss and discriminator loss functions are used to continuously optimize the model output after de-moiré processing image I G , making it gradually approach the clear image I in style clear , far away from the moiré image I input .

[0063] During the training process, the loss function L consists of four parts: contrast loss, content loss, generator loss, and discriminator loss.

[0064] Specifically, the contrast loss is constructed as follows:

[0065] From the input image I input Randomly select N+1 patches from the output image I G A corresponding patch is randomly selected from Figure 2 ). The two corresponding patches are represented as positive samples and anchor points, I input The other N patches in are represented as negative samples. Then, a contrast loss function is used to maximize the difference between positive samples and anchor points (I input and I G Specifically, firstly, the anchor point (I G The patches in ), positive samples and N negative samples are mapped to k-dimensional vectors using two layers of MLP and are represented as v, v respectively. + and v -. Afterwards, a cross entropy loss is established:

[0066]

[0067] Among them, sim(u,v) represents the cosine similarity between u and v, τ represents the temperature parameter, which is set to 0.07, represents the nth negative sample.

[0068] At the same time, the input image I is also extracted input And the output image I G The deep features after different layers of the encoder module are used to construct the contrast loss. Here we extract the features after the first convolutional layer, the second convolutional layer, the first residual block and the fifth residual block in the encoder module to construct the multi-layer contrast loss (see Figure 2 ), respectively expressed as L2(v,v + ,v - ), L3(v,v + ,v - ), L4(v,v + ,v - ), L5(v,v + ,v - ).

[0069] Therefore, the final contrast loss is:

[0070] L contra =L1(v,v + ,v - )+L2(v,v + ,v - )+L3(v,v + ,v - )+L4(v,v + ,v - )+L5(v,v + ,v - )

[0071] Specifically, the content loss is constructed as follows:

[0072] L con =||E c (I G )-E c (I input )||1

[0073] Among them, E c represents the content encoder and ||*||1 represents the L1 loss.

[0074] Specifically, the generator loss is as follows:

[0075] LG =(D(I G )-1) 2

[0076] Where D represents the discriminator.

[0077] Specifically, the discriminator loss is as follows:

[0078] L D =(D(I G )) 2 +(D(I clear )-1) 2

[0079] Among them, I clear Represents an unpaired true clear image.

[0080] Step 4: The generator in the trained image demoiré network receives the verification dataset image that needs to be demoiré processed, and outputs the image after the demoiré processing is completed. The specific method is as follows:

[0081] Load the weights of the generator in the image de-moiré network model trained in step 3 and update the parameters in the model. moire As input data, it is passed into the network model, and the output image I is obtained after the generator removes the moiré pattern. G .

[0082]

[0083] Table 1

[0084] We conducted comparative experiments on the method of the present invention and other related works on the UHDM dataset. The experimental results are shown in Table 1. From the data results in Table 1, it can be seen that the moiré removal effect of the present invention on the UHDM dataset is better than that of other existing methods, which shows the effectiveness of the present invention.

[0085] The above contents are further detailed descriptions of the present invention in combination with specific / preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, they can also make several substitutions or modifications to these described embodiments without departing from the concept of the present invention, and these substitutions or modifications should be regarded as belonging to the protection scope of the present invention.

[0086] Parts of the present invention that are not described in detail belong to the well-known technologies of those skilled in the art.

Claims

1. An unsupervised image moiré removal method based on deep learning, characterized in that: Here are the steps: Step 1: Prepare a moiré image and a clear image that do not require pairing; Step 2: Construct an unsupervised network model for image moiré removal; The network model is based on the generative adversarial network architecture and combines contrastive learning methods; the network model includes a generator, a discriminator, and a content encoder; Step 3: Perform unsupervised training on the constructed image de-moiré network model using the image dataset obtained in step 1; Step 4: The generator in the trained image demoiré network receives the verification dataset image that needs to be demoiré processed, and outputs the image after the demoiré processing is completed.

2. The unsupervised image de-moiré method based on deep learning according to claim 1, characterized in that: The specific structure of the generator: The generator consists of two parts, namely the encoder module and the decoder module. The jump connection is used between the encoder module and the decoder module to avoid gradient disappearance. Specific structure of the encoder module: The encoder module first uses 7×7 convolution and 2x bilinear downsampling operations to encode the input moiré image into a low-resolution feature map; then, 5×5 convolution and 2x bilinear downsampling operations are used again to further reduce the resolution of the feature map; finally, 5 residual blocks are used to extract more complex and deeper features in the low-resolution space; Specific structure of the decoder module: The decoder module accepts the deep output features from the encoder module. It first uses 4 residual blocks to process the features from the encoder module, then uses 2x bilinear upsampling and 3×3 convolution to gradually restore the image features, and finally uses 2x bilinear upsampling and 3×3 convolution again to restore the image features to the original image size and complete the clear image output.

3. The unsupervised image de-moiré method based on deep learning according to claim 2, characterized in that: The specific structure of the discriminator is as follows: PatchGAN is used as the discriminator, which consists of three convolutional layers with a stride of 2, two convolutional layers with a stride of 1, and finally an average pooling layer.

4. The unsupervised image de-moiré method based on deep learning according to claim 3, characterized in that: The specific structure of the content encoder is as follows: The content encoder consists of a 3×3 convolutional layer and two residual blocks, which are used to extract the content features of the input image and the output image to ensure the consistency of the output content with the input content.

5. The unsupervised image de-moiré method based on deep learning according to claim 1, characterized in that: Step 3: The network model is trained by first inputting the moiré image I prepared in step 1 input ; Then, the desired image I after de-moiré processing is obtained through the generator in the designed network model G Finally, the contrast loss, content loss, generator loss, and discriminator loss functions are used to continuously optimize the model output of the de-moiré processed image I G , making it gradually approach the clear image I in style clear , far away from the moiré image I input ; During the training process, the loss function L consists of four parts: contrast loss, content loss, generator loss, and discriminator loss; Specifically, the contrast loss is constructed as follows: From the input image I input Randomly select N+1 patches from the output image I G Randomly select a corresponding patch from the two corresponding patches; denote the two corresponding patches as positive samples and anchor points, I input The other N patches in are represented as negative samples. Then, a contrastive loss function is used to maximize the mutual information between the positive sample and the anchor point. Specifically, the anchor point, positive sample and N negative samples are first mapped to a k-dimensional vector using a two-layer MLP and represented as v, v respectively. + and v - ; After that, establish a cross entropy loss: Among them, sim(u,v) represents the cosine similarity between u and v, τ represents the temperature parameter, which is set to 0.07, Represents the nth negative sample; At the same time, the input image I is also extracted input And the output image I G The deep features after different layers of the encoder module are used to construct the contrast loss. The features after the first convolutional layer, the second convolutional layer, the first residual block, and the fifth residual block in the encoder module are extracted to construct the multi-layer contrast loss, which are expressed as L2(v,v + ,v - ), L3(v,v + ,v - ), L4(v,v + ,v - ), L5(v,v + ,v - ); Therefore, the final contrast loss is: L contra =L1(v,v + ,v - )+L2(v,v + ,v - )+L3(v,v + ,v - )+L4(v,v + ,v - )+L5(v,v + ,v - ) Specifically, the content loss is constructed as follows: THE con =||And c (THE G )-AND c (THE input )||1 Among them, E c represents the content encoder, ||*||1 represents the L1 loss; Specifically, the generator loss is as follows: L G =(D(I G )-1) 2 Where D represents the discriminator; Specifically, the discriminator loss is as follows: L D =(D(I G )) 2 +(D(I clear )-1) 2 Among them, I clear Represents an unpaired true clear image.

6. The unsupervised image de-moiré method based on deep learning according to claim 5, characterized in that: Step 4: Load the weights of the generator in the image de-moiré network model trained in step 3 and update the parameters in the model; secondly, the moiré image I of the validation dataset in step 1 moire As input data, it is passed into the network model, and the output image I is obtained after the generator removes the moiré pattern. G .

Images