Method for removing moire and super-resolution of combined image

Abstract

The invention discloses a method for removing moire and super-division of a combined image. The method comprises the following steps: constructing a data set; constructing a network model for removing moire and super-division of the combined picture based on loss information compensation; training a joint image moire removal and super-division network model based on loss information compensation through the training data set, verifying the model through the verification data set, and completing the construction of the model; and inputting the verified model into a picture needing moire removal processing to obtain a final result. Meanwhile, loss information of picture features in processing is utilized, loss information is extracted from original unprocessed pictures, the loss information in the picture processing process is effectively supplemented, and the loss information of the pictures in the processing process is reduced to the maximum extent. According to the method, through an innovative technical means, the picture quality is remarkably improved, and the method has wide application prospects and market value.

Description

Technical Field

[0001] The invention relates to the field of image processing, and in particular to a method for combining image moiré removal and super-resolution. Background Art

[0002] With the continuous development of science and technology, it has become a normal practice for mobile phone cameras to support ultra-high-resolution picture shooting. In our work and study process, we need to take pictures to record important text materials in many occasions (such as meeting minutes, etc.). However, the photos taken are often disturbed by moiré and unclear, which seriously affects the quality of the pictures. With the continuous development of deep learning, people have found that deep learning performs well in the field of image processing, so it can also be applied to image moiré removal. Deep learning can be used to eliminate or reduce the impact of moiré by virtue of its powerful feature learning and pattern recognition capabilities.

[0003] Then, existing methods for removing moiré generally have some problems: (1) Existing methods only remove moiré noise, but ignore the fact that low-resolution images can no longer meet the requirements; (2) Existing methods lack the supplement of some non-moiré information lost in the moiré processing process, so it may cause some important information to be mistakenly removed while removing moiré; (3) At the same time, existing research on removing text scene images is all based on synthetic, low-resolution datasets, so it is unknown whether existing methods can effectively remove moiré from super-resolution text images, because higher-resolution images have more complex moiré, wider coverage, and higher computational cost. Therefore, methods for joint image moiré removal and image super-resolution still need to be explored. Summary of the invention

[0004] In order to solve the above-mentioned problem of low precision in removing moiré patterns, the present invention provides a method for combining image moiré removal and super-resolution.

[0005] In order to implement the above technical solution, the details are as follows:

[0006] S1. Build a dataset;

[0007] Since there is no suitable ultra-high resolution text image dataset, the present invention constructs a UHDTM dataset, which contains 4500 groups of 4k text image groups; each group of text image groups contains two images, namely: the image I contaminated by moiré M And the original clear picture I C ;

[0008] S2. Construct a network model for joint image de-moiré and super-resolution based on loss information compensation;

[0009] The model includes: a downsampling operation, a feature encoding operation, a first network flow, an upsampling operation, a second network flow and a feature decoding operation;

[0010] The first network flow and the second network flow respectively include: a moire global removal module, a moire noise perception and loss information compensation module, and a multi-scale feature dynamic fusion module; the input image first passes through the moire global removal module and then the moire noise perception and loss information compensation module, and finally inputs the output of the multi-scale feature dynamic fusion module;

[0011] The build process is as follows:

[0012] S2.1. Take a moiré-contaminated image I M Feed the model as the original input image;

[0013] Image contaminated by moiré I M is a 3×H×W picture;

[0014] S2.2, downsampling the original input image;

[0015] The original input image after downsampling becomes a Pictures;

[0016] S2.3, feature encoding the downsampled image;

[0017] In this embodiment, the feature-encoded picture becomes a Features;

[0018] S2.4, input the feature-encoded image into the first network flow, and input the downsampled image into the moiré noise perception and loss information compensation module in the first network flow; the output result is input into the first network flow again after the feedback callback mechanism to obtain the final output of the first network flow; the details are as follows:

[0019] S2.4.1. The feature-encoded image is input into the global moiré removal module;

[0020] Specifically, the feature-encoded image input first passes through the dense block in the global moiré removal module, and the output of the dense block is used as the input of the compression and excitation module. The output of the compression and excitation module is again matrix-added with the feature-encoded image, and then input into the residual block to obtain the output of the global moiré removal module.

[0021] The dense block includes: convolution operation, ReLU activation function and feature splicing operation; the dense block used in the present invention performs three convolution operations, ReLU activation functions and feature splicing operations;

[0022] The residual block includes: a dilated convolution operation, a ReLU activation function, and a matrix addition operation; the residual block used in the present invention performs three dilated convolution operations, ReLU activation functions, and matrix addition operations;

[0023] S2.4.2, input the output of the global moiré removal module and the downsampled image into the moiré noise perception and loss information compensation module;

[0024] The moiré noise perception and loss information module is divided into two layers: the upper layer is the moiré noise perception layer; the lower layer is the loss information compensation layer;

[0025] The moiré noise perception and loss information module has two inputs, the first is the output of the global moiré removal module, and the other is the downsampled image;

[0026] The upper moiré noise perception layer processes the input image through the following operations: convolution operation, matrix subtraction operation, convolution operation, global average pooling operation, two convolution and ReLU activation function operations, one convolution and Sigmoid activation function operation, and then performs matrix multiplication operation with the input image. The matrix subtraction operation specifically inputs the downsampled image into the upper layer and performs matrix subtraction operation with the image input from the upper layer.

[0027] At the same time, the image input from the upper layer will also enter the lower layer, undergoing global average pooling, two convolutions and ReLU activation function operations, and one convolution and Sigmoid activation function operation;

[0028] The lower loss information compensation layer passes the input image through: a convolution operation, a matrix multiplication operation with the upper input image entering the lower layer, and then a residual block;

[0029] At the same time, the downsampled image will be input into the upper layer to perform matrix subtraction operation with the image input into the upper layer;

[0030] The outputs of the upper and lower layers are output features after feature concatenation and feature optimization operations;

[0031] Feature splicing is splicing along the channel dimension;

[0032] The feature optimization operation is to process the image through three convolutions and ReLU activation functions.

[0033] S2.4.3, dynamically integrate the output and input multi-scale features of the moiré noise perception and loss information module into the module;

[0034] Specifically, the output of the moiré noise perception and loss information module first passes through a dense block to obtain feature X1, and then downsamples feature X1 twice to obtain feature X2 and feature X3; features X1, X2, and X3 pass through a dense block respectively, and feature X1 passes through a dense block to obtain feature Y1; feature X2 passes through a dense block and then undergoes an upsampling operation again to obtain feature Y2; feature X3 passes through a dense block and then undergoes two upsampling operations to obtain feature Y3; at this time, the scales of Y1, Y2, and Y3 are consistent;

[0035] Furthermore, the output of the moiré noise perception and loss information module is added to the Y1, Y2 and Y3 matrices to obtain the feature F 1 、F 2 and F 3 ; The feature F 1 、F 2 and F 3 Feature concatenation is performed to obtain feature F; feature F undergoes feature concatenation, global average pooling, two convolutions and ReLU activation function operations, one convolution and Sigmoid activation function operation, and feature separation to obtain feature weight W, which is then divided into w according to the channel. 1 、w 2 and w 3 ; Combine features Y1, Y2 and Y3 with weight w 1 、w 2 and w 3 Matrix multiplication gives the output w 1out 、w 2out and w 3out ; Finally, the output will be w 1out 、w 2out and w 3out Perform matrix addition operation with the output of the moiré noise perception and loss information module to obtain the output of the multi-scale feature dynamic integration module;

[0036] S2.5. The final output of the first network flow is input into the second network flow after upsampling, and the original input image is input into the moiré noise perception and loss information compensation module in the second network flow. The output result is input into the second network flow again after the feedback callback mechanism to obtain the final output of the second network flow. The details are as follows:

[0037] S2.5.1, the final output of the first network stream is input into the global moiré removal module after upsampling;

[0038] S2.5.2, inputting the output of the global moiré removal module into the moiré noise perception and loss information module;

[0039] The moiré noise perception and loss information module is divided into two layers: the upper layer is the moiré noise perception layer; the lower layer is the loss information compensation layer;

[0040] The moiré noise perception and loss information module has two inputs, the first is the output of the global moiré removal module, and the other is the original input image;

[0041] The upper moiré noise perception layer processes the input image through the following operations: convolution operation, matrix subtraction operation, convolution operation, global average pooling operation, two convolution and ReLU activation function operations, one convolution and Sigmoid activation function operation, and then performs matrix multiplication operation with the input image. The matrix subtraction operation specifically inputs the downsampled image into the upper layer and performs matrix subtraction operation with the image input from the upper layer.

[0042] At the same time, the image input from the upper layer will also enter the lower layer, undergoing global average pooling, two convolutions and ReLU activation function operations, and one convolution and Sigmoid activation function operation;

[0043] The lower loss information compensation layer passes the input image through: a convolution operation, a matrix multiplication operation with the upper input image entering the lower layer, and then a residual block;

[0044] At the same time, the downsampled image will be input into the upper layer to perform matrix subtraction operation with the image input into the upper layer;

[0045] The outputs of the upper and lower layers are output features after feature concatenation and feature optimization operations;

[0046] Feature splicing is splicing along the channel dimension;

[0047] The feature optimization operation is to process the image through three convolutions and ReLU activation functions.

[0048] S2.5.3, dynamically integrate the output and input multi-scale features of the moiré noise perception and loss information module into the module;

[0049] S2.6, the final output of the second network stream is input into the feature decoding module to obtain the output of the network model for joint image de-moiré and super-resolution based on loss information compensation;

[0050] S3, training a network model of joint image de-moiré and image super-resolution based on loss information compensation by using the constructed data set, and evaluating the model by using a test set to complete the training of the model;

[0051] S4. Input the image to be processed for moiré removal into the trained model to obtain the final output result (an image without moiré).

[0052] Beneficial Effects of the Invention

[0053] The present invention greatly improves the image moiré removal and image super-resolution capabilities through global image moiré removal, moiré noise perception and loss information supplementation, multi-scale feature dynamic fusion and image super-resolution reconstruction.

[0054] In the global moiré removal stage of images, the present invention combines dense blocks, residual blocks and dilated convolutions to improve the receptive field of the model, thereby performing global preliminary removal of larger moiré patterns. Next, an image generated by the absolute difference between an unprocessed image with moiré noise and an image being processed is combined through a spatial attention module to enhance the model's perception of the moiré position, thereby accurately and effectively removing the moiré patterns.

[0055] The present invention uses the known condition that the image being processed has lost information, and uses the image being processed through a spatial attention module to generate a feature weight containing the position of information lost during the image processing process, and then uses the weight to accurately extract the lost information from the unprocessed image, thereby compensating for the information lost during the image processing process; in the multi-scale feature dynamic fusion stage, the features are processed at multiple sizes to simultaneously remove moiré patterns of various sizes, while capturing information at different levels, paving the way for better subsequent image super-resolution. Finally, through image super-resolution reconstruction, image super-resolution reconstruction is achieved by reducing the number of feature accesses and increasing the size of the features. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is a flow chart of the method of the present invention;

[0057] Figure 2 is a network structure diagram of the present invention;

[0058] Figure 3 A global moiré removal module of the present invention;

[0059] Figure 4 The moiré noise perception and loss information compensation module of the present invention;

[0060] Figure 5 It is the multi-size feature dynamic fusion module of the present invention. DETAILED DESCRIPTION

[0061] The present invention is further described in detail below in conjunction with specific embodiments.

[0062] A method for joint image moiré removal and super-resolution comprises the following steps:

[0063] S1. Build a dataset;

[0064] In this embodiment, since there is no suitable ultra-high resolution text image dataset, the present invention constructs a UHDTM dataset, which includes 4500 groups of 4k text image groups; each group of text image groups includes two images, namely: the image I contaminated by moiré and the image II contaminated by moiré. M and original clear pictures;

[0065] Furthermore, 4000 of the 4500 sets of images are used as training data sets, and the other 500 sets of images are used as validation data sets; M As the input image data in the network model training process, the original clear image is used as the reference image for comparison with the model prediction image in the network model training process.

[0066] S2. Construct a network model for joint image de-moiré and super-resolution based on loss information compensation;

[0067] The model includes: a downsampling operation, a feature encoding operation, a first network flow, an upsampling operation, a second network flow and a feature decoding operation;

[0068] The first network flow and the second network flow respectively include: a moire global removal module, a moire noise perception and loss information compensation module, and a multi-scale feature dynamic fusion module; the input image first passes through the moire global removal module and then the moire noise perception and loss information compensation module, and finally inputs the output of the multi-scale feature dynamic fusion module;

[0069] The build process is as follows:

[0070] S2.1. Take a moiré-contaminated image I M Feed the model as the original input image;

[0071] Image contaminated by moiré I M is a 3×H×W picture;

[0072] S2.2, downsampling the original input image;

[0073] The original input image after downsampling becomes a Pictures;

[0074] S2.3, feature encoding the downsampled image;

[0075] In this embodiment, the feature-encoded picture becomes a Features;

[0076] S2.4, input the feature-encoded image into the first network flow, and input the downsampled image into the moiré noise perception and loss information compensation module in the first network flow; the output result is input into the first network flow again after the feedback callback mechanism to obtain the final output of the first network flow; the details are as follows:

[0077] S2.4.1. The feature-encoded image is input into the global moiré removal module;

[0078] Specifically, the feature-encoded image input first passes through the dense block in the global moiré removal module, and the output of the dense block is used as the input of the compression and excitation module. The output of the compression and excitation module is again matrix-added with the feature-encoded image, and then input into the residual block to obtain the output of the global moiré removal module.

[0079] The dense block includes: convolution operation, ReLU activation function and feature splicing operation; the dense block used in the present invention performs three convolution operations, ReLU activation functions and feature splicing operations;

[0080] The residual block includes: a dilated convolution operation, a ReLU activation function, and a matrix addition operation; the residual block used in the present invention performs three dilated convolution operations, ReLU activation functions, and matrix addition operations;

[0081] S2.4.2, input the output of the global moiré removal module and the downsampled image into the moiré noise perception and loss information compensation module;

[0082] The moiré noise perception and loss information module is divided into two layers: the upper layer is the moiré noise perception layer; the lower layer is the loss information compensation layer;

[0083] The moiré noise perception and loss information module has two inputs, the first is the output of the global moiré removal module, and the other is the downsampled image;

[0084] The upper moiré noise perception layer processes the input image through the following operations: convolution operation, matrix subtraction operation, convolution operation, global average pooling operation, two convolution and ReLU activation function operations, one convolution and Sigmoid activation function operation, and then performs matrix multiplication operation with the input image. The matrix subtraction operation specifically inputs the downsampled image into the upper layer and performs matrix subtraction operation with the image input from the upper layer.

[0085] At the same time, the image input from the upper layer will also enter the lower layer, undergoing global average pooling, two convolutions and ReLU activation function operations, and one convolution and Sigmoid activation function operation;

[0086] The lower loss information compensation layer passes the input image through: a convolution operation, a matrix multiplication operation with the upper input image entering the lower layer, and then a residual block;

[0087] At the same time, the downsampled image will be input into the upper layer to perform matrix subtraction operation with the image input into the upper layer;

[0088] The outputs of the upper and lower layers are output features after feature concatenation and feature optimization operations;

[0089] Feature splicing is splicing along the channel dimension;

[0090] The feature optimization operation is to process the image through three convolutions and ReLU activation functions.

[0091] S2.4.3, dynamically integrate the output and input multi-scale features of the moiré noise perception and loss information module into the module;

[0092] Specifically, the output of the moiré noise perception and loss information module first passes through a dense block to obtain feature X1, and then downsamples feature X1 twice to obtain feature X2 and feature X3; features X1, X2, and X3 pass through a dense block respectively, and feature X1 passes through a dense block to obtain feature Y1; feature X2 passes through a dense block and then undergoes an upsampling operation again to obtain feature Y2; feature X3 passes through a dense block and then undergoes two upsampling operations to obtain feature Y3; at this time, the scales of Y1, Y2, and Y3 are consistent;

[0093] Furthermore, the output of the moiré noise perception and loss information module is added to the Y1, Y2 and Y3 matrices to obtain the feature F 1 、F 2 and F 3 ; The feature F 1 、F 2 and F 3 Feature concatenation is performed to obtain feature F; feature F undergoes feature concatenation, global average pooling, two convolutions and ReLU activation function operations, one convolution and Sigmoid activation function operation, and feature separation to obtain feature weight W, which is then divided into w according to the channel. 1 、w 2 and w 3 ; Combine features Y1, Y2 and Y3 with weight w 1 、w 2 and w 3 Matrix multiplication gives the output w 1out 、w 2out and w 3out ; Finally, the output will be w 1out 、w 2out and w 3outPerform matrix addition operation with the output of the moiré noise perception and loss information module to obtain the output of the multi-scale feature dynamic integration module;

[0094] S2.5. The final output of the first network flow is input into the second network flow after upsampling, and the original input image is input into the moiré noise perception and loss information compensation module in the second network flow. The output result is input into the second network flow again after the feedback callback mechanism to obtain the final output of the second network flow. The details are as follows:

[0095] S2.5.1, the final output of the first network stream is input into the global moiré removal module after upsampling;

[0096] S2.5.2, inputting the output of the global moiré removal module into the moiré noise perception and loss information module;

[0097] The moiré noise perception and loss information module is divided into two layers: the upper layer is the moiré noise perception layer; the lower layer is the loss information compensation layer;

[0098] The moiré noise perception and loss information module has two inputs, the first is the output of the global moiré removal module, and the other is the original input image;

[0099] The upper moiré noise perception layer processes the input image through the following operations: convolution operation, matrix subtraction operation, convolution operation, global average pooling operation, two convolution and ReLU activation function operations, one convolution and Sigmoid activation function operation, and then performs matrix multiplication operation with the input image. The matrix subtraction operation specifically inputs the downsampled image into the upper layer and performs matrix subtraction operation with the image input from the upper layer.

[0100] At the same time, the image input from the upper layer will also enter the lower layer, undergoing global average pooling, two convolutions and ReLU activation function operations, and one convolution and Sigmoid activation function operation;

[0101] The lower loss information compensation layer passes the input image through: a convolution operation, a matrix multiplication operation with the upper input image entering the lower layer, and then a residual block;

[0102] At the same time, the downsampled image will be input into the upper layer to perform matrix subtraction operation with the image input into the upper layer;

[0103] The outputs of the upper and lower layers are output features after feature concatenation and feature optimization operations;

[0104] Feature splicing is splicing along the channel dimension;

[0105] The feature optimization operation is to process the image through three convolutions and ReLU activation functions.

[0106] S2.5.3, dynamically integrate the output and input multi-scale features of the moiré noise perception and loss information module into the module;

[0107] S2.6, the final output of the second network stream is input into the feature decoding module to obtain the output of the network model for joint image de-moiré and super-resolution based on loss information compensation;

[0108] S3, training a network model of joint image de-moiré and image super-resolution based on loss information compensation by using the constructed data set, and evaluating the model by using a test set to complete the training of the model;

[0109] S4. Input the image to be processed for moiré removal into the trained model to obtain the final output result (an image without moiré).

[0110] Specifically, a comparison is made on the UHDTM dataset with different combinations of moiré removal models and image super-resolution models, as shown in Table 1;

[0111] Table 1 Comparison of scores of different models

[0112]

[0113] In the table, PSRN is the peak signal-to-noise ratio, and the higher the PSRN value, the better the effect; SSIM is the structural similarity index, and the higher the SSIM value, the better the effect; LPIPS is the image similarity metric, and the lower the LPIPS value, the better the effect. Experiments have shown that the present invention is better than other existing methods in removing moiré on the UHDTM dataset, which shows the effectiveness of the present invention.

Claims

1. A method for joint image moiré removal and super-resolution, characterized in that: The following steps are involved: S1. Build a dataset. S2. Construct a network model for joint image de-moiré and super-resolution based on loss information compensation; S3, training a network model of joint image de-moiré and image super-resolution based on loss information compensation by using the constructed data set, and evaluating the model by using a test set to complete the training of the model; S4. Input the image to be processed for moiré removal into the trained model to obtain the final output result.

2. The method for joint image demoiré and super-resolution according to claim 1, characterized in that: The steps of constructing a network model for joint image de-moiré and super-resolution based on loss information compensation are as follows: S2.

1. Take a moiré-contaminated image I M Feed the model as the original input image; S2.2, downsampling the original input image; S2.3, feature encoding the downsampled image; S2.4, input the feature-encoded image into the first network flow, and input the downsampled image into the moiré noise perception and loss information compensation module in the first network flow; the output result is input into the first network flow again after the feedback callback mechanism to obtain the final output of the first network flow; S2.

5. The final output of the first network stream is input into the second network stream after upsampling, and the original input image is input into the moiré noise perception and loss information compensation module in the second network stream at the same time; the output result is input into the second network stream again after the feedback callback mechanism to obtain the final output of the second network stream; S2.

6. The final output of the second network stream is input into the feature decoding module to obtain the output of the network model for joint image de-moiré and super-resolution based on loss information compensation.

3. The method for joint image demoiré and super-resolution according to claim 1, characterized in that: The steps of inputting the feature-encoded picture into the first network flow, and inputting the downsampled picture into the moiré noise perception and loss information compensation module in the first network flow; and inputting the output result into the first network flow again after the feedback callback mechanism to obtain the final output of the first network flow are as follows: S2.4.

1. The feature-encoded image is input into the global moiré removal module; The feature-encoded image input first passes through the dense block in the global moiré removal module, and the output of the dense block is used as the input of the compression and excitation module. The output of the compression and excitation module is again added to the feature-encoded image by matrix addition operation, and then input into the residual block to obtain the output of the global moiré removal module; The dense block includes: convolution operation, ReLU activation function and feature splicing operation; the dense block used in the present invention performs three convolution operations, ReLU activation functions and feature splicing operations; The residual block includes: a dilated convolution operation, a ReLU activation function, and a matrix addition operation; the residual block used in the present invention performs three dilated convolution operations, ReLU activation functions, and matrix addition operations; S2.4.2, input the output of the global moiré removal module and the downsampled image into the moiré noise perception and loss information compensation module; The moiré noise perception and loss information module is divided into two layers: the upper layer is the moiré noise perception layer; the lower layer is the loss information compensation layer; The moiré noise perception and loss information module has two inputs, the first is the output of the global moiré removal module, and the other is the downsampled image; The upper moiré noise perception layer processes the input image through the following operations: convolution operation, matrix subtraction operation, convolution operation, global average pooling operation, two convolution and ReLU activation function operations, one convolution and Sigmoid activation function operation, and then performs matrix multiplication operation with the input image. The matrix subtraction operation specifically inputs the downsampled image into the upper layer and performs matrix subtraction operation with the image input from the upper layer. At the same time, the image input from the upper layer will also enter the lower layer, undergoing global average pooling, two convolutions and ReLU activation function operations, and one convolution and Sigmoid activation function operation; The lower loss information compensation layer passes the input image through: a convolution operation, a matrix multiplication operation with the upper input image entering the lower layer, and then a residual block; At the same time, the downsampled image will be input into the upper layer to perform matrix subtraction operation with the image input into the upper layer; The outputs of the upper and lower layers are output features after feature concatenation and feature optimization operations; Feature splicing is splicing along the channel dimension; The feature optimization operation is to process the image through three convolutions and ReLU activation functions. S2.4.3, dynamically integrate the output and input multi-scale features of the moiré noise perception and loss information module into the module; The output of the moiré noise perception and loss information module first passes through a dense block to obtain feature X1, and then downsamples feature X1 twice to obtain feature X2 and feature X3; features X1, X2, and X3 pass through a dense block respectively, and feature X1 passes through a dense block to obtain feature Y1; feature X2 passes through a dense block and then undergoes an upsampling operation again to obtain feature Y2; feature X3 passes through a dense block and then undergoes two upsampling operations to obtain feature Y3; at this time, the scales of Y1, Y2, and Y3 are consistent; The output of the moiré noise perception and loss information module is added to the Y1, Y2 and Y3 matrices to obtain features F1, F2 and F3; the features F1, F2 and F3 are concatenated to obtain feature F; feature F is sequentially subjected to feature concatenation, global average pooling, two convolutions and ReLU activation function operations, one convolution and Sigmoid activation function operation, and feature separation operations to obtain feature weight W, which is divided into w1, w2 and w3 according to the channel; features Y1, Y2 and Y3 are multiplied by the weight w1, w2 and w3 matrices to obtain output w 1out 、w 2out and w 3out ; Finally, the output will be w 1out 、w 2out and w 3out Perform matrix addition operation with the output of the moiré noise perception and loss information module to obtain the output of the multi-scale feature dynamic integration module.

4. The method for joint image demoiré and super-resolution according to claim 1, characterized in that: The final output of the first network flow is input into the second network flow after upsampling, and the original input image is input into the moiré noise perception and loss information compensation module in the second network flow; the output result is input into the second network flow again after the feedback callback mechanism to obtain the final output of the second network flow. The steps are as follows: S2.5.1, the final output of the first network stream is input into the global moiré removal module after upsampling; S2.5.2, inputting the output of the global moiré removal module into the moiré noise perception and loss information module; The moiré noise perception and loss information module is divided into two layers: the upper layer is the moiré noise perception layer; the lower layer is the loss information compensation layer; The moiré noise perception and loss information module has two inputs, the first is the output of the global moiré removal module, and the other is the original input image; The upper moiré noise perception layer processes the input image through the following operations: convolution operation, matrix subtraction operation, convolution operation, global average pooling operation, two convolution and ReLU activation function operations, one convolution and Sigmoid activation function operation, and then performs matrix multiplication operation with the input image. The matrix subtraction operation specifically inputs the downsampled image into the upper layer and performs matrix subtraction operation with the image input from the upper layer. At the same time, the image input from the upper layer will also enter the lower layer, undergoing global average pooling, two convolutions and ReLU activation function operations, and one convolution and Sigmoid activation function operation; The lower loss information compensation layer passes the input image through: a convolution operation, a matrix multiplication operation with the upper input image entering the lower layer, and then a residual block; At the same time, the downsampled image will be input into the upper layer to perform matrix subtraction operation with the image input into the upper layer; The outputs of the upper and lower layers are output features after feature concatenation and feature optimization operations; Feature splicing is splicing along the channel dimension; The feature optimization operation is to process the image through three convolutions and ReLU activation functions. S2.5.

3. The output and input multi-scale features of the moiré noise perception and loss information module are dynamically integrated into the module to obtain the final output of the second network flow.

Images