An image moiré removal method based on RGB channel trilateral compensation

Abstract

The application discloses an image moire removal method based on RGB channel three-edge compensation, first acquires an image data set required for network training, then constructs an image moire removal network model based on RGB channel three-edge compensation, and trains the constructed image moire removal network model through the obtained image data set; finally, the trained model receives an image requiring moire removal processing, outputs the image after completing the moire removal processing. The application innovatively proposes to use different channel branches to extract image information under different channels, learn moire information under different scales from each channel respectively, remove the moire information, and perform color recovery, use a multi-channel feature fusion module to fuse image features under different channels, so that each channel branch can use the features of other branches to improve the features of the current scale branch, and improve the ability of the model to remove moire and recover color.

Description

Technical Field

[0001] The present invention relates to an image moiré removal method based on RGB channel three-sided compensation, and in particular to the field of image moiré removal effect processing based on deep learning technology. Background Art

[0002] Moiré is an interfering pattern that appears in images due to optical differences between image acquisition devices (such as cameras and scanners) and display devices (such as monitors). It typically appears as regular bright and dark stripes or ripples in the image, and can cause color shifts and distortions, impairing image quality and clarity. It also severely impacts many downstream image-based tasks, such as object detection and image segmentation.

[0003] In today's world, digital screens are ubiquitous, and mobile phone cameras are a frequently used device. Mobile phones are used for photography in many situations, including taking photos of electronic screens, such as meeting minutes. However, these photos often suffer from moiré artifacts that can cause blur and severely impact image quality. Moiré artifacts are a common problem in many fields, including photography, printing, and texture analysis.

[0004] With the development of deep learning, researchers have discovered its remarkable performance in image processing, and its application in image moiré removal is also a possibility. Deep learning can leverage its powerful feature learning and pattern recognition capabilities to reduce or eliminate the effects of moiré. Many researchers have used deep learning methods to study image moiré removal, and have achieved some success. However, a common shortcoming of most studies is the lack of exploration of color restoration in moiré-infected images. While some studies have achieved good results in removing streaks from moiré-infected images, the problem of image color distortion remains unresolved. Therefore, the ability to simultaneously remove moiré and restore image color remains a challenge. Summary of the Invention

[0005] Technical problem to be solved: Since existing de-moiré technology cannot restore the color of moiré images well, the present invention proposes an image de-moiré method based on RGB channel three-sided compensation, which can remove image moiré while restoring the original color of the image to the greatest extent, solving the problem of color distortion.

[0006] Implementation steps: This paper proposes an image moiré removal method based on RGB channel three-sided compensation, the basic steps of which are:

[0007] Step 1: Dataset preparation;

[0008] Obtain the image dataset required for network training in step 2.

[0009] Step 2: Construct an image de-moiré network model based on RGB channel trilateral compensation.

[0010] The network model consists of an encoder module with three RGB channel branches, a multi-channel feature fusion module, and a decoder module for image restoration. The three RGB channel branches are divided into R channel branch, G channel branch, and B channel branch, each with an encoder module. The three branches encode features of the moiré image from different color channels to obtain the color information of the moiré image. The multi-channel feature fusion module accepts the image encoding features from the R, G, and B channel branches and adjusts the weights of the features of different color channels to correct the image color information. The decoder module for image restoration accepts the RGB image features integrated by the multi-channel feature fusion module and gradually decodes them into a clear image.

[0011] Step 3: Train the constructed image de-moiré network model using the image dataset obtained in step 1;

[0012] Step 4: The trained image demoiré network model receives the validation dataset image that needs to be demoiré processed and outputs the image after completing the demoiré processing.

[0013] Furthermore, the specific method of step 1 is as follows:

[0014] Download the public dataset UHDM dataset, which contains 5000 sets of 4K images of different scenes. Each set of images contains two images, namely, moiré images I and II contaminated by moiré patterns. moire and the original clear image I clear . 4500 of the 5000 sets of images are used as training data sets, and the other 500 sets of images are used as validation data sets. moire As the input image data in the network model training process, the original clear image I clear As the network model training process used to predict the image I pred Reference image for comparison.

[0015] Furthermore, the specific structure of the encoder module is as follows: each encoder module contains three encoding layers (encoder level). After the first two encoder levels, bilinear interpolation is used to downsample the features by a factor of 2, and then pass through the next encoder level.

[0016] The specific structure of each encoder level: input feature Xu nAfter a residual dense block (RDB) module for feature extraction, the output feature X is obtained out , output feature X out The feature size and number of channels remain the same as the input feature X in Similarly, the output features are downsampled and transferred to the next encoder level, and on the other hand, they are transferred to the encoder level corresponding to the encoder module through skip connections as input features. The specific connection method is as follows Figure 1 shown.

[0017] Furthermore, the specific structure of the RDB module is as follows: the input features are connected to the features before convolution in the channel dimension after undergoing dilated convolution with a convolution kernel size of 3*3 and ReLU activation function, and then the above process is repeated 4 times. The dilation rates of the 5 dilated convolutions are 1, 2, 3, 2, and 1 respectively to expand the receptive field. Then, a 1*1 convolution is performed to obtain a feature with the same size and number of channels as the input feature. Finally, the feature is linked to the input feature through a residual to obtain the output feature.

[0018] Furthermore, the specific structure of the multi-channel feature fusion module is as follows: the input of the multi-channel feature fusion module is the output features of three encoders from different branches. Inside the multi-channel feature fusion module, the three input features (i.e., the output features of the third encoder level of each channel branch encoder module) are first concatenated in the channel dimension to form a feature 3C×H×W containing different channel branches. Then, global average pooling is used in the spatial dimension to obtain a 3C×1×1 feature. The weights are distributed through the multi-layer perceptron (MLP) module, and the sigmoid function is used for weight normalization. Finally, the normalized weights are multiplied by the 3C×H×W feature containing different channel branches to obtain the output feature after weight redistribution.

[0019] Furthermore, the specific structure of the decoder module: The structure of the decoder module corresponds to that of the encoder module and is also divided into three decoder levels. The difference is that the first two decoder levels of the decoder need to be upsampled by 2 times after processing the features, which is exactly the opposite of the encoder level. Each decoder level receives the output features from the corresponding encoder level through a skip connection. The internal processing of each decoder level is the same as that of the encoder level. The input feature X in After a residual dense block module (RDB) for feature extraction, the output feature X of the decoder level is obtainedout The input and output differ slightly. The input features of the first decoder level are the output features after weight redistribution after processing by the multi-channel feature fusion module. The input features of the second decoder level are the output features of the three branches corresponding to the second encoder level, concatenated with the output features from the first decoder level along the channel dimension. The input features of the third decoder level are the output features of the three branches corresponding to the first encoder level, concatenated with the output features from the second decoder level along the channel dimension. In addition, the output of each decoder level undergoes a 3*3 convolution and pixel unshuffle to obtain a three-channel RGB image, i.e., a de-moiréd image. This image is then compared with the real clear image for loss calculation to supervise model training.

[0020] Furthermore, the specific structure of the complete network: the network model contains branches with three different channels. The first branch is the R channel of the input image, which undergoes pixel shuffle and then flows through a 3*3 convolution and ReLU activation function, and then is processed by the encoder module. The second branch is the G channel of the input image, which undergoes pixel shuffle and then flows through a 3*3 convolution and ReLU activation function, and then is processed by the encoder module. The third branch is the B channel of the input image, which undergoes pixel shuffle and then flows through a 3*3 convolution and ReLU activation function, and then is processed by the encoder module. After the three branches are processed by the encoder module, the corresponding features are input into the multi-channel feature fusion module for feature fusion. The fused features are restored by the decoder module, and finally a clear image is obtained.

[0021] Furthermore, the specific method of step 3 is as follows:

[0022] The training method of the network model is to first input the moiré image I prepared in step 1 moire Then, the desired image I after de-moiré processing is obtained through the designed network model pred Finally, the loss function is used to continuously optimize the model output after de-moiré processing of the image I pred , making it gradually similar to the real clear image I in the dataset prepared in step 1 clear .

[0023] During the training process, the loss function L uses the L1 function combined with the improved Sobel loss function ASL as the loss function. Compared with the existing Sobel filter, the improved Sobel loss function provides two additional 45-degree filters. The loss function is specifically expressed as:

[0024] Loss=L1(I pred , I clear )+k*ASL(I pred , I clear )

[0025] Where k is a model hyperparameter used to balance the L1 loss function and the Sobel loss function. The training adopts a multi-supervision strategy to supervise the progressive output of the decoder module, which is specifically expressed as:

[0026] Loss_total=Loss(I pred1 , I clear1 )+Loss(I pred2 , I clear2 )+Loss(I pred3 , I clear3 )

[0027] Among them I pred1 , I pred2 , I pred3 They are the predicted images of the three decoder levels of the decoder module, I clear1 , I clear2 , I clear3 They are real clear images of corresponding sizes respectively.

[0028] Furthermore, the specific method of step 4 is as follows:

[0029] Load the image de-moiré network model weights trained in step 3 and update the parameters in the model. e The input data is passed into the network model, and the input data passes through the encoder and decoder in turn to obtain the model output image I after de-moiré processing. pred3 ,Note that when verifying the model, it is only necessary to obtain the maximum image output by the model and compare it with the real clear image, that is, I pred3 .

[0030] The beneficial effects of the present invention are as follows:

[0031] This invention innovatively proposes using different channel branches to extract image information from different channels. It then learns moiré information at different scales from each channel, removes it, and performs color restoration. It also innovatively proposes using a multi-channel feature fusion module to fuse image features from different channels. This allows each channel branch to draw on the features of other branches to improve the features of the branch at its own scale, improving the model's ability to remove moiré and restore color. This invention can process ultra-high-definition moiré images with excellent results. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a schematic diagram of the network structure for image moiré removal based on RGB channel trilateral compensation according to an embodiment of the present invention;

[0033] Figure 2 Schematic diagram of the residual dense block structure according to an embodiment of the present invention;

[0034] Figure 3 Schematic diagram of the structure of a multi-channel feature fusion module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0036] The present invention first carries out the following definitions and descriptions:

[0037] I moire : Moiré images

[0038] I clear : Real clear pictures

[0039] I pred : Output image of the de-moiré network model

[0040] Implementation steps: This paper proposes an image moiré removal method based on RGB channel three-sided compensation, the basic steps of which are:

[0041] Step 1: Dataset preparation;

[0042] Get the image dataset required for network training in step 2. The specific method is as follows:

[0043] Download the public dataset UHDM dataset, which contains 5000 sets of 4K images of different scenes. Each set of images contains two images, namely, moiré images I and II contaminated by moiré patterns. moire and the original clear image I clear. 4500 of the 5000 sets of images are used as training data sets, and the other 500 sets of images are used as validation data sets. moire As the input image data in the network model training process, the original clear image I clear As the network model training process used to predict the image I pred Reference image for comparison.

[0044] Step 2: Build an image de-moiré network model based on RGB channel trilateral compensation.

[0045] The network model consists of an encoder module with three RGB channel branches, a multi-channel feature fusion module, and a decoder module for image restoration. The three RGB channel branches are divided into R channel branch, G channel branch, and B channel branch, each with an encoder module. The three branches encode features of the moiré image from different color channels to obtain the color information of the moiré image. The multi-channel feature fusion module accepts the image encoding features from the R, G, and B channel branches and adjusts the weights of the features of different color channels to correct the image color information. The decoder module for image restoration accepts the RGB image features integrated by the multi-channel feature fusion module and gradually decodes them into a clear image.

[0046] Specific structure of the encoder module: Each encoder module contains three encoding layers (encoder level). After the first two encoder levels, bilinear interpolation is used to downsample the features by 2 times, and then pass through the next encoder level.

[0047] The specific structure of each encoder level: input feature X in After a residual dense block (RDB) module for feature extraction, the output feature X is obtained out , output feature X out The feature size and number of channels remain the same as the input feature X in Similarly, the output features are downsampled and transferred to the next encoder level, and on the other hand, they are transferred to the encoder level corresponding to the encoder module through skip connections as input features. The specific connection method is as follows Figure 1 shown.

[0048] like Figure 2As shown in the figure, the specific structure of the RDB module is as follows: the input features are connected to the features before convolution in the channel dimension after undergoing dilated convolution with a convolution kernel size of 3*3 and ReLU activation function. Then, the above process is repeated 4 times, and the dilation rates of the 5 dilated convolutions are 1, 2, 3, 2, and 1 respectively to expand the receptive field. Then, a 1*1 convolution is performed to obtain a feature with the same size and number of channels as the input feature. Finally, the feature is linked with the input feature through a residual to obtain the output feature.

[0049] like Figure 3 As shown in the figure, the specific structure of the multi-channel feature fusion module is as follows: the input of the multi-channel feature fusion module is the output features of three encoders from different branches. Inside the multi-channel feature fusion module, the three input features (i.e., the output features of the third encoder level of each channel branch encoder module) are first concatenated in the channel dimension to form a feature 3C×H×W containing different channel branches. Then, global average pooling is used in the spatial dimension to obtain a 3C×1×1 feature. The weights are distributed through the multi-layer perceptron (MLP) module, and the weights are normalized using the sigmoid function. Finally, the normalized weights are multiplied by the 3C×H×W feature containing different channel branches to obtain the output feature after weight redistribution.

[0050] Specific structure of decoder module: The structure of decoder module is similar to that of encoder module, and is also divided into three decoder levels. The difference is that the first two decoder levels of decoder need to be upsampled by 2 times after processing the features, which is exactly the opposite of the encoder level. Each decoder level receives the output features from the corresponding encoder level through skip connection. The internal processing of each decoder level is the same as that of the encoder level. The input feature X in After a residual dense block module (RDB) for feature extraction, the output feature X of the decoder level is obtained out, with only slight differences between the input and output. The input features of the first decoder level are the features of the output features of the three branches corresponding to the third encoder level processed by the multi-channel feature fusion module. The input features of the second decoder level are the features of the output features of the three branches corresponding to the second encoder level concatenated with the output features from the first decoder level along the channel dimension. The input features of the third decoder level are the features of the output features of the three branches corresponding to the first encoder level concatenated with the output features from the second decoder level along the channel dimension. In addition, the output of each decoder level will undergo a 3*3 convolution and pixel unshuffle to obtain a three-channel RGB image, that is, an image that has been de-moiréed. This image is then compared with the real clear image for loss calculation to supervise model training.

[0051] The specific structure of the complete network: The network model contains three branches with different channels. The first branch is the R channel of the input image, which undergoes pixel shuffle, then flows through a 3*3 convolution and ReLU activation function, and is then processed by the encoder module. The second branch is the G channel of the input image, which undergoes pixel shuffle, then flows through a 3*3 convolution and ReLU activation function, and is then processed by the encoder module. The third branch is the B channel of the input image, which undergoes pixel shuffle, then flows through a 3*3 convolution and ReLU activation function, and is then processed by the encoder module. After the three branches are processed by the encoder module, the corresponding features are fused through the multi-channel feature fusion module. The fused features are restored by the decoder module, and finally a clear image is obtained.

[0052] Step 3: Use the image dataset obtained in step 1 to train the constructed image de-moiré network model. The specific method is as follows:

[0053] The training method of the network model is to first input the moiré image I prepared in step 1 moire Then, the desired image I after de-moiré processing is obtained through the designed network model pred Finally, the loss function is used to continuously optimize the model output after de-moiré processing of the image I pred , making it gradually similar to the real clear image I in the dataset prepared in step 1 clear .

[0054] During the training process, the loss function L uses the L1 function combined with the improved Sobel loss function ASL as the loss function. Compared with the existing Sobel filter, the improved Sobel loss function provides two additional 45-degree filters. The loss function is specifically expressed as:

[0055] Loss=L1(I pred , I clear )+k*ASL(I pred , I clear )

[0056] Where k is a model hyperparameter used to balance the L1 loss function and the Sobel loss function. The training adopts a multi-supervision strategy to supervise the progressive output of the decoder module, which is specifically expressed as:

[0057] Loss_total=Loss(I pred1 , I clear1 )+Loss(I pred2 , I clear2 )+Loss(I pred3 , I clear3 )

[0058] Among them I pred1 , I pred2 , I pred3 They are the predicted images of the three decoder levels of the decoder module, I clear1 , I clear2 , I clear3 They are real clear images of corresponding sizes respectively.

[0059] Step 4: The trained image demoiré network model receives the validation dataset images that need to be demoiré processed and outputs the images after demoiré processing. The specific method is as follows:

[0060] Load the image de-moiré network model weights trained in step 3 and update the parameters in the model. moire The input data is passed into the network model, and the input data passes through the encoder and decoder in turn to obtain the model output image I after de-moiré processing. pred3 ,Note that when verifying the model, it is only necessary to obtain the maximum image output by the model and compare it with the real clear image, that is, I pred3 .

[0061]

[0062] Table 1

[0063] We conducted comparative experiments on the UHDM dataset using the method of the present invention and other related works. The experimental results are shown in Table 1. From the data results in Table 1, it can be seen that the moiré removal effect of the present invention on the UHDM dataset is better than that of other existing methods, demonstrating the effectiveness of the present invention.

[0064] The above description is a further detailed description of the present invention in conjunction with specific / preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. Those skilled in the art of the present invention may make various substitutions or modifications to the described embodiments without departing from the scope of the present invention, and such substitutions or modifications should be considered to fall within the scope of protection of the present invention.

[0065] Parts of the present invention that are not described in detail belong to the common knowledge of those skilled in the art.

Claims

1. An image moiré removal method based on RGB channel trilateral compensation, characterized in that: The basic steps are: Step 1: Dataset preparation; Obtain the image dataset required for network training in step 2; Step 2: Construct an image moiré removal network model based on RGB channel trilateral compensation; The network model includes an encoder module with RGB three-channel branches, a multi-channel feature fusion module, and a decoder module; The RGB three-channel branches are R channel branch, G channel branch and B channel branch. Each branch has an encoder module. The three branches encode the features of the moiré image from different color channels to obtain the color information of the moiré image. The multi-channel feature fusion module accepts image encoding features from the R, G, and B channel branches and is used to adjust the weights of different color channel features to correct the color information of the image; The decoder module receives the RGB image features output by the multi-channel feature fusion module and gradually decodes a clear image. Step 3: Train the constructed image de-moiré network model using the image dataset obtained in step 1; The training method of the network model is to first input the moiré image I prepared in step 1 moire Then, the desired image I after de-moiré processing is obtained through the designed network model pred Finally, the loss function is used to continuously optimize the model output after de-moiré processing of the image I pred , making it gradually similar to the real clear image I in the dataset prepared in step 1 clear ; During the training process, the loss function L uses the L1 function combined with the improved Sobel loss function ASL as the loss function. Compared with the existing Sobel filter, the improved Sobel loss function provides two additional filters at 45 degrees. The loss function is specifically expressed as: Loss=L1(I pred ,I clear )+k*ASL(I pred ,I clear ) Where k is a model hyperparameter used to balance the L1 loss function and the Sobel loss function; the training adopts a multi-supervision strategy to supervise the progressive output of the decoder module, which is specifically expressed as: Loss_total=Loss(I pred1 ,I clear1 )+Loss(I pred2 ,I clear2 ) +Loss(I pred3 ,I clear3 ) Among them I pred1 , I pred2 , I pred3 They are the predicted images of the three decoder levels of the decoder module, I clear1 , I clear2 , I clear3 They are real clear images of corresponding sizes respectively; Step 4: The trained image demoiré network model receives the validation dataset image that needs to be demoiré processed and outputs the image after completing the demoiré processing.

2. The image moiré removal method based on RGB channel trilateral compensation according to claim 1, characterized in that: Step 1: Download the public dataset UHDM dataset, which contains 5000 sets of 4K images of different scenes. Each set of images contains two images, namely, moiré images I and II contaminated by moiré patterns. moire and the original clear image I clear ; 4500 of the 5000 sets of images are used as training data sets, and the other 500 sets of images are used as verification data sets; moire As the input image data in the network model training process, the original clear image I clear As the network model training process used to predict the image I pred Reference image for comparison.

3. The image moiré removal method based on RGB channel trilateral compensation according to claim 1, characterized in that: Specific structure of the encoder module: Each encoder module contains three encoding layers. After the first two encoder levels, bilinear interpolation is used to downsample the features by a factor of 2, and then pass through the next encoder level; The specific structure of each encoder level: input feature X in After a residual dense block RDB module for feature extraction, the output feature X is obtained out , output feature X out The feature size and number of channels remain the same as the input feature X in Similarly, the output features are downsampled and transmitted to the next encoder level, and on the other hand, they are transmitted to the encoder level corresponding to the encoder module through skip connections as input features.

4. The image moiré removal method based on RGB channel trilateral compensation according to claim 3, characterized in that: The specific structure of the RDB module is as follows: the input features are connected to the features before convolution in the channel dimension after undergoing dilated convolution with a convolution kernel size of 3*3 and ReLU activation function. Then, the above process is repeated 4 times, and the dilation rates of the 5 dilated convolutions are 1, 2, 3, 2, and 1 respectively to expand the receptive field. Then, a 1*1 convolution is performed to obtain a feature with the same size and number of channels as the input feature. Finally, the feature is linked with the input feature through a residual to obtain the output feature.

5. The image moiré removal method based on RGB channel three-side compensation according to claim 3 or 4, characterized in that: The specific structure of the multi-channel feature fusion module: The input of the multi-channel feature fusion module is the output features of three encoders from different branches; Inside the multi-channel feature fusion module, the three input features are first concatenated in the channel dimension to form a feature 3C×H×W containing different channel branches. Then, global average pooling is used in the spatial dimension to obtain a 3C×1×1 feature. The weights are distributed through the multi-layer perceptron (MLP) module, and the sigmoid function is used for weight normalization. Finally, the normalized weights are multiplied by the feature 3C×H×W containing different channel branches to obtain the output feature after weight redistribution.

6. The image moiré removal method based on RGB channel trilateral compensation according to claim 5, characterized in that: Specific structure of the decoder module: The structure of the decoder module corresponds to that of the encoder module and is also divided into three decoder levels. The difference is that the first two decoder levels of the decoder need to be upsampled by 2 times after processing the features. Each decoder level receives the output features from the corresponding encoder level through a skip connection; the internal processing of each decoder level is the same as that of the encoder level, and the input feature X in After a residual dense block module for feature extraction, the decoder level output feature X is obtained out The input features of the first decoder level are the output features after weight redistribution after processing by the multi-channel feature fusion module. The input features of the second decoder level are the features after the output features of the three branches corresponding to the second encoder level are connected with the output features from the first decoder level in the channel dimension. The input features of the third decoder level are the features after the output features of the three branches corresponding to the first encoder level are connected with the output features from the second decoder level in the channel dimension. In addition, the output of each decoder level will undergo a 3*3 convolution and pixel inverse recombination to obtain a three-channel RGB image, that is, the image after de-moiré processing.

7. The image moiré removal method based on RGB channel trilateral compensation according to claim 1, characterized in that: The specific structure of the complete network: The network model contains three branches with different channels. The first branch is the R channel of the input image, which flows through a 3*3 convolution and ReLU activation function after pixel reorganization, and then is processed by the encoder module; the second branch is the G channel of the input image, which flows through a 3*3 convolution and ReLU activation function after pixel reorganization, and then is processed by the encoder module; the third branch is the B channel of the input image, which flows through a 3*3 convolution and ReLU activation function after pixel reorganization, and then is processed by the encoder module; after the three branches are processed by the encoder module respectively, the corresponding features are input into the multi-channel feature fusion module for feature fusion, and the fused features are restored by the decoder module to finally obtain a clear image.

8. The image moiré removal method based on RGB channel trilateral compensation according to claim 1, characterized in that: Step 4: Load the image de-moiré network model weights trained in step 3 and update the parameters in the model; secondly, the moiré image I of the validation dataset in step 1 is moire The input data is passed into the network model, and the input data passes through the encoder and decoder in turn to obtain the model output image I after de-moiré processing. pred3 .

Images