Moire removal method, computer equipment and storage medium

Abstract

The invention discloses a moire removal method, computer equipment and a storage medium. The moire removal method comprises the following steps: preprocessing to obtain standardized data; inputting the standardized data into N discrete wavelet transform (DWT) for frequency domain decomposition; adjusting the weight parameters through space attention and channel attention to obtain feature data; performing inverse discrete wavelet transform (IDWT) on the Nth feature data to obtain Nth reconstructed data; the Nth reconstruction data and the (N-1) th feature data are fused, and reconstruction is carried out through residual error convolution modules RCB and IDWT to obtain the (N-1) th reconstruction data; step-by-step fusion reconstruction is carried out until fusion with the first feature data is carried out, and final reconstruction data is obtained through RCB and IDWT; and carrying out convolution Conv operation on the final reconstruction data to obtain moire-removed image data. The method has the advantages that the cross-domain processing capacity is improved, the difference utilization rate of low-frequency and high-frequency characteristics is improved, the global information modeling capacity is improved, and moire patterns are effectively removed.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and image processing, and in particular to a moiré removal method, a computer device, and a storage medium. Background Art

[0002] When shooting with a device like a digital camera, the camera sensor's frequency may approach the spatial frequency of the object being photographed. This causes the camera's photosensitive element (usually a CMOS or CCD) to overlap with the luminous portion of the object being photographed, resulting in moiré patterns that affect image quality. Factors such as shooting angle, lighting conditions, and object shape can affect the appearance of moiré patterns. Images contaminated by moiré patterns often contain rich color information, and their complex textures and colors can blend with the original image's tones, making image restoration more difficult.

[0003] The existing technology has some technical defects in moiré removal:

[0004] (1) Limited frequency perception range and insufficient fusion of multi-band features. Although existing methods based on multi-scale features and parallel branches have expanded the frequency perception range, they are not adaptable enough to the frequency distribution of complex moiré patterns. The ability to handle strong noise in the low-frequency region is limited, and the detailed features of the high-frequency sub-bands are not fully explored. Traditional discrete Fourier transform (DFT) and discrete cosine transform (DCT) cannot achieve joint analysis in the time and frequency domains, resulting in insufficient time domain positioning accuracy of high-frequency moiré features. Although wavelet transform achieves multi-band decomposition, existing methods do not fully utilize the differences in low-frequency / high-frequency sub-band features, and there is a lack of adaptive mechanism for feature fusion between sub-bands.

[0005] (2) Lack of cross-domain processing capabilities. Existing models are mostly designed for a single image domain (RAW or RGB) and do not consider the amplification effect of the ISP process on moiré. When processing only the RGB domain, moiré noise is already enhanced by the ISP process, making repair significantly more difficult. Although RAW domain methods retain the original sensor information, they are not compatible with conventional RGB image processing requirements and lack a dual-domain collaborative feature representation learning framework.

[0006] (3) The ability to model global information is weak. Traditional CNN methods are limited by local receptive fields and have difficulty modeling global color shifts (such as large-area color block distortion) and structural damage (such as text breakage and line distortion) caused by moiré patterns. The existing attention mechanism does not explicitly introduce frequency domain prior guidance and lacks the ability to dynamically distinguish the importance of frequency bands, making it difficult to balance high-frequency detail restoration with low-frequency color restoration. Summary of the Invention

[0007] In order to solve the problems of limited frequency perception range, insufficient fusion of multi-band features, lack of cross-domain processing capability and weak global information modeling capability in the process of moiré removal, the present invention proposes a moiré removal method, which aims to improve the cross-domain processing capability by pre-processing standardized data for different image data types, decompose the moiré image into multiple frequency sub-bands based on multiple serial discrete wavelet transforms (DWTs), improve the differential utilization of low-frequency and high-frequency features, and introduce spatial attention mechanism and channel attention mechanism to process the high-frequency and low-frequency features after discrete wavelet transform respectively. At the same time, a cross-frequency interaction module is used to realize the interaction between high-frequency and low-frequency information, improve the global information modeling capability, and effectively remove moiré.

[0008] The present invention discloses a moiré removal method, comprising the following steps:

[0009] Step S1, acquiring image data, and preprocessing the image data to obtain standardized data with a consistent number of channels based on the image data type;

[0010] Step S2, inputting the standardized data into a decomposition network consisting of N serial discrete wavelet transforms (DWTs) for frequency domain decomposition;

[0011] Step S3, based on each discrete wavelet transform (DWT) frequency domain decomposition result, the spatial attention mechanism and channel attention mechanism in the corresponding frequency domain attention module (FAT) are used to adjust the low-frequency and high-frequency weight parameters in each decomposition result to obtain feature data;

[0012] Step S4, reconstructing the Nth feature data by inverse discrete wavelet transform IDWT to obtain corresponding Nth reconstructed data;

[0013] Step S5, fusing the N-th reconstructed data with the N-1-th feature data, and reconstructing the fused result sequentially through the residual convolution module RCB and the inverse discrete wavelet transform IDWT to obtain the corresponding N-1-th reconstructed data;

[0014] Step S6, repeating step S5, fusing and reconstructing step by step until the final reconstructed data is obtained after fusion with the first feature data and then reconstructed by the residual convolution module RCB and the inverse discrete wavelet transform IDWT;

[0015] Step S7: performing a convolution operation on the final reconstructed data, adjusting the number of channels to obtain image data with moiré patterns removed.

[0016] In a preferred embodiment, step S1 specifically includes the following steps:

[0017] Get image data;

[0018] If the image data type is a RAW domain image, splitting the image data into 4-channel image data based on the arrangement characteristics of the RAW domain image data;

[0019] If the image data type is an RGB domain image, copy any color channel data in the image data and combine it with the original three color channel data to form four-channel image data;

[0020] Based on the 4-channel image data, features are extracted through a convolution operation to obtain standardized data.

[0021] In a preferred embodiment, any of the discrete wavelet transforms DWT is used to decompose the input data into a low-frequency sub-band LL and three high-frequency sub-bands LH, HL and HH, which is specifically implemented by the following calculation method:

[0022]

[0023] Among them, F LL is the low-frequency subband LL filter kernel, F LH is the high frequency subband LH filter kernel, F HL is the high frequency subband HL filter kernel; F HH is the high frequency subband HH filter kernel.

[0024] In a preferred embodiment, the frequency domain attention module FAT includes a first normalization layer for receiving the low-frequency sub-band LL and performing a normalization operation, a second normalization layer for receiving the three high-frequency sub-bands LH, HL, and HH and performing a normalization operation, a first CCR module connected to the output end of the first normalization layer for obtaining the normalized query Q, key K, and value V of the low-frequency sub-band LL, a second CCR module connected to the output end of the second normalization layer for obtaining the normalized query Q, key K, and value V of the three high-frequency sub-bands LH, HL, and HH, a low-frequency processing module LFPM connected to the output end of the first CCR module, a high-frequency processing module HFPM connected to the output end of the second CCR module, a cross-frequency fusion module CFIM connected to the output end of the low-frequency processing module LFPM and the output end of the high-frequency path module HFPM, a third normalization layer, and a multi-layer perceptron MLP connected to the output end of the third normalization layer;

[0025] The output end of the cross-frequency fusion module CFIM establishes a residual connection with the input end of the first normalization layer and the output end of the second normalization layer, and the result is input into the third normalization layer for normalization operation. The multi-layer perceptron MLP is used to perform a nonlinear transformation on the normalized result of the third normalization layer, and establish a residual connection between the output end and the output end of the third normalization layer to obtain the feature data.

[0026] In a preferred embodiment, the first CCR module and the second CCR module adopt the same structure, including a convolution layer Conv, a block layer Chunk, and a reshape layer Reshape connected in sequence;

[0027] The convolution layer is used to perform a convolution layer operation on the input data to extract features, and the chunk layer Chunk is used to divide the extracted results into 3 sub-blocks along the channel dimension. The reshape layer Reshape is used to adjust the dimensions of the 3 sub-blocks to generate the query Q, key K and value V of the input data.

[0028] In a preferred embodiment, the low-frequency processing module LFPM is used to query Q, key K and value V based on the normalized low-frequency subband LL, adjust weight parameters through spatial and channel attention mechanisms, restore low-frequency structure, and correct color distribution, which is achieved by the following calculation method:

[0029]

[0030] Where d is the quotient of the number of channels and the number of attention heads;

[0031] The high frequency processing module HFPM is used to query Q, key K and value V based on the normalization of the three high frequency sub-bands LH, HL and HH, adjust the weight parameters through the spatial and channel attention mechanism, suppress high frequency noise and enhance texture details, and is implemented using the following calculation method:

[0032]

[0033] Where d is the quotient of the number of channels and the number of attention heads.

[0034] In a preferred embodiment, the cross-frequency fusion module CFIM is used to perform cross-frequency fusion on the output result of the low-frequency processing module LFPM and the output result of the high-frequency processing module HFPM, specifically comprising the following steps:

[0035] The output of the low frequency processing module LFPM and the output of the high frequency processing module HFPM are spliced along the channel dimension, and the following calculation method is used to achieve this:

[0036] I 0 =Conv(concate(I Low +I High ))

[0037] Among them, I Low is the output result of the low frequency processing module LFPM, I High The output result of the high frequency processing module HFPM, 1 0 is the splicing result;

[0038] The splicing results are pooled in the channel domain and spatial domain respectively, and the channel domain pooling result and the spatial domain pooling result are summed and input into the Sigmoid activation function to obtain the fusion result. The following calculation method is used to achieve this:

[0039] I 1 =Sigmoid(ChPool(I 0 )+SpPool(I 0 ))

[0040] Among them, ChPool() is channel domain pooling, and SpPool() is spatial domain pooling.

[0041] In a preferred embodiment, N is 3, and the standardized data is sequentially transformed through a first discrete wavelet transform DWT, a second discrete wavelet transform DWT, and a third discrete wavelet transform DWT;

[0042] The decomposition result of the third discrete wavelet transform DWT is sequentially passed through the corresponding residual convolution module RCB and the frequency domain attention module FAT to obtain third feature data, and the third feature data is reconstructed through the inverse discrete wavelet transform IDWT to obtain third reconstructed data;

[0043] The decomposition result of the second discrete wavelet transform DWT is sequentially passed through the corresponding residual convolution module RCB and the frequency domain attention module FAT to obtain second feature data, and the second feature data is fused with the third reconstructed data, and the fused result is sequentially passed through the corresponding residual convolution module RCB and the inverse discrete wavelet transform IDWT to reconstruct the second reconstructed data;

[0044] The decomposition result of the first discrete wavelet transform DWT is used to obtain the first feature data through the corresponding frequency domain attention module FAT, and the first feature data is fused with the second reconstructed data, and the fused result is reconstructed in turn through the corresponding residual convolution module RCB and the inverse discrete wavelet transform IDWT to obtain the final reconstructed data.

[0045] The present invention also provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for executing any one of the above methods.

[0046] The present invention also provides a storage medium including a computer program / instruction, which implements any one of the above moiré removal methods when executed by a processor.

[0047] The moiré removal method, computer device, and storage medium disclosed in the present invention have the following beneficial effects compared with the prior art:

[0048] The present invention discloses a moiré removal method, which includes acquiring image data and, based on the image data type, preprocessing to obtain standardized data with a consistent number of channels. By preprocessing, the number of channels of different types of image data is unified, thereby improving cross-domain compatibility and processing capabilities. This method solves the limitations of existing moiré removal methods designed for a single domain (such as a RAW domain or an RGB domain). At the same time, standardizing the data is beneficial to removing noise interference from the sensor itself.

[0049] The standardized data input is decomposed in the frequency domain by a decomposition network composed of N serial discrete wavelet transforms (DWTs). On the one hand, the discrete wavelet transform (DWT) decomposes the low-frequency subbands step by step to generate multi-scale frequency domain features, covering the spectrum from low frequency to high frequency, solving the problem that traditional methods are not adaptable enough to the frequency distribution of complex moiré. On the other hand, the discrete wavelet transform (DWT) provides both time domain positioning and frequency domain separation capabilities, accurately capturing the periodic interference position of high-frequency moiré, and making up for the defect that global frequency domain methods such as discrete Fourier transform (DFT) / discrete cosine transform (DCT) cannot locate spatial information.

[0050] For each discrete wavelet transform DWT frequency domain decomposition result, the spatial attention mechanism and channel attention mechanism in the corresponding frequency domain attention module FAT are used to adjust the low-frequency and high-frequency weight parameters of each decomposition result to obtain feature data. The spatial attention mechanism and channel attention mechanism can be used to adaptively adjust the weight parameters according to the specific conditions of the low-frequency and high-frequency sub-band characteristics in each decomposition result. This enables the network to fully focus on the strong noise in the low-frequency area and the key detail features of the high-frequency sub-band, effectively solving the problem of insufficient utilization of the low-frequency / high-frequency sub-band feature differences in existing methods, and realizing the adaptive fusion of the features of each sub-band.

[0051] The Nth feature data is reconstructed using the inverse discrete wavelet transform (IDWT) to obtain the corresponding Nth reconstructed data. After completing multi-level frequency domain decomposition and feature adjustment, the highest-level feature data is reconstructed back into the spatial domain using the IDWT to obtain preliminary reconstructed data. This process provides intermediate results for subsequent level-by-level fusion and further reconstruction, enabling the network to gradually transform the advantages of frequency domain processing into improved image quality in the spatial domain.

[0052] The N-th reconstructed data is fused with the N-1-th feature data, and the fused results are reconstructed in turn through the residual convolution module RCB and the inverse discrete wavelet transform IDWT to obtain the corresponding N-1-th reconstructed data, realizing the interaction and integration of features at different levels. This cross-level feature fusion can make full use of the information obtained in the frequency domain decomposition and feature adjustment process at all levels, enhance the network's understanding of the relationship between different frequency components of the image, and help to more comprehensively remove moiré patterns. The introduction of the residual convolution module RCB, through the residual connection, enables information to cross multiple levels more directly during the transmission process, avoiding the loss and attenuation of information in the deep network, which not only helps to improve the stability and convergence speed of network training, but also better retains the important features of the image at different processing stages, further improving the effect of moiré removal.

[0053] The features are fused and reconstructed step by step until they are fused with the first feature data, and then reconstructed in sequence through the residual convolution module RCB and the inverse discrete wavelet transform IDWT to obtain the final reconstructed data. On the one hand, through the process of step-by-step fusion and reconstruction, the features at all levels can fully interact and fuse, continuously optimizing the reconstruction quality of the image. This deep fusion method can fully utilize the complementarity between features of different frequency bands, effectively solving the problem of insufficient multi-band feature fusion in traditional methods, so that the final reconstructed image can better preserve the original structure and color information of the image while removing moiré. On the other hand, during the entire step-by-step fusion and reconstruction process, the network can gradually capture the global information of the image. After multiple feature fusion and reconstruction operations, the network can comprehensively analyze and repair the global color shift and structural damage caused by moiré, overcoming the shortcomings of traditional CNN methods limited by local receptive fields and significantly improving the global information modeling ability. The final reconstructed data is subjected to a convolution operation Conv, and the number of channels is adjusted to obtain the image data after moiré removal. The image details are further fine-tuned to further improve the quality of the image after moiré removal. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is a technical implementation flow chart of a first embodiment of a moiré removal method of the present invention;

[0055] Figure 2 Schematic diagram of the connection structure of discrete wavelet transform DWT, frequency domain attention module FAT, inverse discrete wavelet transform IDWT and residual convolution module RCB in embodiment 1 of a moiré removal method of the present invention;

[0056] Figure 3 Schematic diagram of a moiré removal method according to embodiment 1 of the present invention, wherein the discrete wavelet transform DWT decomposes input data into a low-frequency sub-band LL and three high-frequency sub-bands LH, HL and HH;

[0057] Figure 4 Schematic diagram of the structure of a frequency domain attention module FAT in a first embodiment of a moiré removal method of the present invention;

[0058] Figure 5 Schematic diagram of the structure of a low frequency processing module LFPM in a first embodiment of a moiré removal method of the present invention;

[0059] Figure 6 FIG. 1 is a structural diagram of a high-frequency processing module HFPM in a first embodiment of a moiré removal method of the present invention. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0061] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0062] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein in the specification of this application are intended only to describe specific embodiments and are not intended to limit this application. The term "or / and" as used herein includes any and all combinations of one or more of the associated listed items.

[0063] In addition, in the present invention, descriptions such as "first" and "second" are only used for descriptive purposes and do not specifically refer to the order or sequence, nor are they used to limit the present invention. They are only used to distinguish components or operations described with the same technical terms, and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one such feature. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in this field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0064] Example 1

[0065] A moiré removal method of this embodiment is as follows: Figure 1 and Figure 2 As shown, it at least includes the following steps:

[0066] Step S1, obtain image data, and based on the image data type, preprocess to obtain standardized data with a consistent number of channels. By preprocessing, the number of channels of different types of image data is unified, cross-domain compatibility and processing capabilities are improved, and the limitations of existing moiré removal methods designed for a single domain (such as RAW domain or RGB domain) are solved. At the same time, standardized data is conducive to removing noise interference from the sensor itself.

[0067] Step S1 specifically includes the following steps:

[0068] Step S101: Acquire image data.

[0069] Step S102 : If the image data type is a RAW domain image, the image data is split based on the arrangement characteristics of the RAW domain image data to obtain 4-channel image data.

[0070] To better understand step S102, an example is given below:

[0071] For RAW domain images, due to the use of the Bayer pattern, its color arrangement follows a specific pattern. The most common is the RGGB format, as shown below:

[0072]

[0073] This arrangement means that for every 2×2 pixel area on the image sensor, only one color channel is captured: red (R), green (G1), green (G2), and blue (B). Green pixels appear more frequently to enhance brightness details. Based on the Bayer pattern arrangement, the four color components are extracted and rearranged into four channels to form a unified 4-channel image. This is achieved using the following calculation method:

[0074] I(x, y) = [I RAW (2x,2y),I RAW (2x,2y+1),I RAW (2x+1,2y),I RAW (2x+1, 2y+1)].

[0075] Step S103: If the image data type is an RGB domain image, any color channel data in the image data is copied and combined with the original three color channel data to form four-channel image data.

[0076] To better understand step S103, an example is given below:

[0077] For RGB domain images, each pixel contains complete red, green, and blue color information, as shown below:

[0078]

[0079] In order to keep the structure of the RGB image consistent with that of the RAW image, in this embodiment, the green channel is copied to construct 4-channel image data, which is specifically achieved by the following calculation method:

[0080] I(x, y) = [I R (x, y), I G (x, y), I G (x, y), I B (x, y)].

[0081] Step S104: extract features based on the 4-channel image data through a convolution operation to obtain standardized data.

[0082] In step S2, the standardized data is input into a decomposition network composed of N serial discrete wavelet transforms (DWTs) for frequency domain decomposition. On the one hand, the discrete wavelet transform (DWT) decomposes the low-frequency subbands step by step to generate multi-scale frequency domain features, covering the spectrum from low frequency to high frequency, which solves the problem that traditional methods are not adaptable enough to the frequency distribution of complex moiré. On the other hand, the discrete wavelet transform (DWT) provides both time domain positioning and frequency domain separation capabilities, accurately capturing the periodic interference position of high-frequency moiré, and making up for the defect that global frequency domain methods such as discrete Fourier transform (DFT) / discrete cosine transform (DCT) cannot locate spatial information.

[0083] In this embodiment, any discrete wavelet transform DWT is used to decompose the input data into a low-frequency sub-band LL and three high-frequency sub-bands LH, HL and HH, which is specifically implemented using the following calculation method:

[0084]

[0085] Among them, F LL is the low-frequency subband LL filter kernel, F LH is the high frequency subband LH filter kernel, F HL is the high frequency subband HL filter kernel; F HH is the high frequency subband HH filter kernel.

[0086] Use discrete wavelet transform to transform input data from spatial domain to frequency domain. Figure 3As shown, the discrete wavelet transform decomposes the image into a low-frequency subband (LL) and three high-frequency subbands (LH, HL, and HH), corresponding to horizontal, vertical, and diagonal edge information, respectively. This decomposition not only helps capture the structural details and texture features of the image, but also enhances the network's perception in the frequency domain, enabling it to more effectively separate and process interfering signals such as moiré. The LL subband retains the image's primary structural information and overall outline, reflecting the image's low-frequency components; the LH subband captures horizontal edge information; the HL subband reflects vertical edges and texture; and the HH subband focuses on high-frequency variations in the image's diagonal direction, containing the richest texture details and local interference.

[0087] Step S3, based on each discrete wavelet transform DWT frequency domain decomposition result, the spatial attention mechanism and channel attention mechanism in the corresponding frequency domain attention module FAT are used to adjust the low-frequency and high-frequency weight parameters in each decomposition result to obtain feature data. The spatial attention mechanism and channel attention mechanism can be used to adaptively adjust the weight parameters according to the specific conditions of the low-frequency and high-frequency sub-band features in each decomposition result. This enables the network to fully focus on the strong noise in the low-frequency area and the key detail features of the high-frequency sub-band, effectively solving the problem of insufficient utilization of the low-frequency / high-frequency sub-band feature differences in existing methods, and realizing the adaptive fusion of the features of each sub-band.

[0088] The frequency domain attention module FAT introduces a modeling mechanism based on the Transformer architecture to improve the network's ability to understand and process complex interference in the image. Figure 4 As shown, the frequency domain attention module FAT includes a first normalization layer for receiving a low-frequency sub-band LL and performing a normalization operation, a second normalization layer for receiving three high-frequency sub-bands LH, HL and HH and performing a normalization operation, a first CCR module connected to the output end of the first normalization layer for obtaining the normalized query Q, key K and value V of the low-frequency sub-band LL, a second CCR module connected to the output end of the second normalization layer for obtaining the normalized query Q, key K and value V of the three high-frequency sub-bands LH, HL and HH, a low-frequency processing module LFPM connected to the output end of the first CCR module, a high-frequency processing module HFPM connected to the output end of the second CCR module, a cross-frequency fusion module CFIM connected to the output end of the low-frequency processing module LFPM and the output end of the high-frequency path module HFPM, a third normalization layer, and a multi-layer perceptron MLP connected to the output end of the third normalization layer.

[0089] The output end of the cross-frequency fusion module CFIM establishes a residual connection with the input end of the first normalization layer and the output end of the second normalization layer, and the result is input into the third normalization layer for normalization operation. The multi-layer perceptron MLP is used to perform nonlinear transformation on the normalized result of the third normalization layer, and establishes a residual connection between the output end and the output end of the third normalization layer to obtain feature data.

[0090] In this embodiment, the first CCR module and the second CCR module adopt the same structure, including a convolutional layer Conv, a block layer Chunk, and a reshape layer Reshape connected in sequence;

[0091] The convolution layer is used to perform convolution operations on the input data to extract features and divide the extracted results into chunks. The chunk layer is used to split the extracted results into three sub-blocks along the channel dimension. The reshape layer is used to adjust the dimensions of the three sub-blocks to generate the query Q, key K, and value V of the input data.

[0092] The low-frequency processing module LFPM is used to query Q, key K, and value V based on the normalized low-frequency subband LL. It adjusts the weight parameters through the spatial and channel attention mechanism, restores the low-frequency structure, and corrects the color distribution. This is achieved using the following calculation method:

[0093]

[0094] Where d is the quotient of the number of channels and the number of attention heads;

[0095] The high-frequency processing module HFPM is used to query Q, key K, and value V based on the normalization of the three high-frequency sub-bands LH, HL, and HH. It adjusts the weight parameters through the spatial and channel attention mechanism to suppress high-frequency noise and enhance texture details. This is achieved using the following calculation method:

[0096]

[0097] Where d is the quotient of the number of channels and the number of attention heads.

[0098] In this embodiment, the low frequency processing module LFPM and the high frequency processing module HFPM are as follows: Figure 5 and 6 As shown in the figure, the query Q, key K, and value V are linearly transformed by the linear transformation layer R after the low-frequency subband LL is normalized to adjust the dimension. The query Q, key K, and value V are then multiplied to achieve information interaction and fusion between different features. The intermediate S can be regarded as an adaptive weight matrix generated based on the feature information of Q and K. By fusing information across different channels and locations, LFPM can model the global information of the image to a certain extent.

[0099] In this embodiment, the cross-frequency fusion module CFIM is used to perform cross-frequency fusion on the output result of the low-frequency processing module LFPM and the output result of the high-frequency processing module HFPM, which specifically includes the following steps:

[0100] The output of the low-frequency processing module LFPM and the output of the high-frequency processing module HFPM are spliced along the channel dimension using the following calculation method:

[0101] I 0 =Conv(concate(I Low +I High ))

[0102] Among them, I Low is the output result of the low frequency processing module LFPM, I High is the output result of the high frequency processing module HFPM, I 0 is the splicing result;

[0103] The splicing results are pooled in the channel domain and spatial domain respectively, and the channel domain pooling result and the spatial domain pooling result are summed and input into the Sigmoid activation function to obtain the fusion result. The following calculation method is used to achieve this:

[0104] I 1 =Sigmoid(ChPool(I 0 )+SpPool(I 0 ))

[0105] Among them, ChPool() is channel domain pooling, and SpPool() is spatial domain pooling.

[0106] In step S4, the Nth feature data is reconstructed using the inverse discrete wavelet transform (IDWT) to obtain the corresponding Nth reconstructed data. After completing the multi-level frequency domain decomposition and feature adjustment, the highest-level feature data is reconstructed back to the spatial domain using the IDWT to obtain preliminary reconstructed data. This process provides intermediate results for subsequent step-by-step fusion and further reconstruction, enabling the network to gradually transform the advantages of frequency domain processing into improved image quality in the spatial domain.

[0107] In step S5, the N-th reconstructed data is fused with the N-1-th feature data, and the fused results are sequentially reconstructed through the residual convolution module RCB and the inverse discrete wavelet transform IDWT to obtain the corresponding N-1-th reconstructed data, thereby realizing the interaction and integration of features at different levels. This cross-level feature fusion can make full use of the information obtained in the frequency domain decomposition and feature adjustment process at all levels, enhance the network's understanding of the relationship between different frequency components of the image, and help to more comprehensively remove moiré patterns. The introduction of the residual convolution module RCB, through the residual connection, enables information to cross multiple levels more directly during the transmission process, avoiding the loss and attenuation of information in the deep network, which not only helps to improve the stability and convergence speed of network training, but also better retains the important features of the image at different processing stages, further improving the effect of moiré removal.

[0108] In this embodiment, Figure 1 As shown in the figure, in the residual convolution module (RCB), the input data enters the first convolution (Conv) for preliminary feature extraction. After the first convolution, the data enters the first ReLU activation function to introduce nonlinearity. It then enters the second convolution and the second ReLU activation function in sequence to further extract and transform features. The result of the third convolution and the second ReLU activation function are added to the final output.

[0109] Step S6, repeat step S5, fuse and reconstruct step by step until it is fused with the first feature data and then reconstructed in sequence through the residual convolution module RCB and the inverse discrete wavelet transform IDWT to obtain the final reconstructed data. On the one hand, through the process of step-by-step fusion and reconstruction, the features at all levels can fully interact and fuse, and the reconstruction quality of the image is continuously optimized. This deep fusion method can make full use of the complementarity between features of different frequency bands, and effectively solve the problem of insufficient fusion of multi-band features in traditional methods, so that the final reconstructed image can better retain the original structure and color information of the image while removing the moiré. On the other hand, in the entire step-by-step fusion and reconstruction process, the network can gradually capture the global information of the image. After multiple feature fusion and reconstruction operations, the network can comprehensively analyze and repair the global color shift and structural damage caused by the moiré, overcoming the shortcomings of the traditional CNN method limited by the local receptive field, and significantly improving the global information modeling capability.

[0110] In step S7, a convolution operation is performed on the final reconstructed data to adjust the number of channels to obtain image data with moiré removed. Further fine-tuning of image details is performed to further improve the quality of the image after moiré removal.

[0111] In this embodiment, N is 3, and the normalized data is sequentially transformed through the first discrete wavelet transform DWT, the second discrete wavelet transform DWT, and the third discrete wavelet transform DWT;

[0112] The decomposition result of the third discrete wavelet transform DWT is sequentially passed through the corresponding residual convolution module RCB and the frequency domain attention module FAT to obtain the third feature data, and the third feature data is reconstructed through the inverse discrete wavelet transform IDWT to obtain the third reconstructed data;

[0113] The decomposition result of the second discrete wavelet transform DWT is sequentially passed through the corresponding residual convolution module RCB and the frequency domain attention module FAT to obtain the second feature data, and the second feature data is fused with the third reconstructed data, and the fused result is sequentially passed through the corresponding residual convolution module RCB and the inverse discrete wavelet transform IDWT to reconstruct the second reconstructed data;

[0114] The decomposition result of the first discrete wavelet transform DWT is used to obtain the first feature data through the corresponding frequency domain attention module FAT, and the first feature data is fused with the second reconstructed data, and the fused result is reconstructed through the corresponding residual convolution module RCB and inverse discrete wavelet transform IDWT in turn to obtain the final reconstructed data.

[0115] It should be noted that the residual convolution module RCB, the frequency domain attention module FAT, and the convolution Conv parameters require pre-training to determine the optimal weight parameters. In this embodiment, the optimal weight parameters are loaded into each functional module, and PSNR, SSIM, and LPIPS are used as evaluation indicators. As shown in Table 1, when the input is an RGB image, the moiré removal method of this embodiment achieves the best performance in all three indicators: PSNR, SSIM, and LPIPS.

[0116] When the input image format is RAW, as shown in Table 2, there are no corresponding LPIPS results for RAW images, as LPIPS only applies to three-channel images. These results demonstrate that the image data obtained after the moiré removal method of this embodiment has smaller pixel differences than the original image data, and the image noise level is lower. In terms of structural similarity and image differences perceived by the human eye, the moiré-removed image is visually closer to the original image.

[0117] Table 1. Quantitative comparison of the moiré removal method of this embodiment with other methods in the RGB domain on the TMM22 dataset

[0118]

[0119] Table 2. Quantitative comparison of the moiré removal method of this example with other methods in the RAW domain on the TMM22 dataset

[0120]

[0121] Example 2

[0122] This embodiment further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for executing any one of the above methods.

[0123] This embodiment further provides a storage medium including a computer program / instruction, which, when executed by a processor, implements any of the above-described moiré removal methods.

[0124] In short, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A moiré removal method, characterized in that: The steps include: Step S1, acquiring image data, and preprocessing the image data to obtain standardized data with a consistent number of channels based on the image data type; Step S2, inputting the standardized data into a decomposition network consisting of N serial discrete wavelet transforms (DWTs) for frequency domain decomposition; Step S3, based on each discrete wavelet transform (DWT) frequency domain decomposition result, the spatial attention mechanism and channel attention mechanism in the corresponding frequency domain attention module (FAT) are used to adjust the low-frequency and high-frequency weight parameters in each decomposition result to obtain feature data; Step S4, reconstructing the Nth feature data by inverse discrete wavelet transform IDWT to obtain corresponding Nth reconstructed data; Step S5, fusing the N-th reconstructed data with the N-1-th feature data, and reconstructing the fused result sequentially through the residual convolution module RCB and the inverse discrete wavelet transform IDWT to obtain the corresponding N-1-th reconstructed data; Step S6, repeating step S5, fusing and reconstructing step by step until the final reconstructed data is obtained after fusion with the first feature data and then reconstructed by the residual convolution module RCB and the inverse discrete wavelet transform IDWT; Step S7: performing a convolution operation on the final reconstructed data, adjusting the number of channels to obtain image data with moiré removed.

2. A moiré removal method according to claim 1, characterized in that: The step S1 specifically includes the following steps: Get image data; If the image data type is a RAW domain image, splitting the image data into 4-channel image data based on the arrangement characteristics of the RAW domain image data; If the image data type is an RGB domain image, copy any color channel data in the image data and combine it with the original three color channel data to form four-channel image data; Based on the 4-channel image data, features are extracted through a convolution operation to obtain standardized data.

3. A moiré removal method according to claim 1 or 2, characterized in that: Any of the discrete wavelet transforms DWT is used to decompose the input data into a low-frequency sub-band LL and three high-frequency sub-bands LH, HL and HH, which is specifically implemented using the following calculation method: Among them, F LL is the low-frequency subband LL filter kernel, F LH is the high frequency subband LH filter kernel, F HL is the high frequency subband HL filter kernel; F HH is the high frequency subband HH filter kernel.

4. The moiré removal method according to claim 3, wherein: The frequency domain attention module FAT includes a first normalization layer for receiving the low-frequency sub-band LL and performing a normalization operation, a second normalization layer for receiving the three high-frequency sub-bands LH, HL and HH and performing a normalization operation, a first CCR module connected to the output end of the first normalization layer for obtaining the normalized query Q, key K and value V of the low-frequency sub-band LL, a second CCR module connected to the output end of the second normalization layer for obtaining the normalized query Q, key K and value V of the three high-frequency sub-bands LH, HL and HH, a low-frequency processing module LFPM connected to the output end of the first CCR module, a high-frequency processing module HFPM connected to the output end of the second CCR module, a cross-frequency fusion module CFIM connected to the output end of the low-frequency processing module LFPM and the output end of the high-frequency path module HFPM, a third normalization layer, and a multi-layer perceptron MLP connected to the output end of the third normalization layer; The output end of the cross-frequency fusion module CFIM establishes a residual connection with the input end of the first normalization layer and the output end of the second normalization layer, and the result is input into the third normalization layer for normalization operation. The multi-layer perceptron MLP is used to perform a nonlinear transformation on the normalized result of the third normalization layer, and establish a residual connection between the output end and the output end of the third normalization layer to obtain the feature data.

5. The moiré removal method according to claim 4, wherein: The first CCR module and the second CCR module adopt the same structure, including a convolution layer Conv, a block layer Chunk and a reshape layer Reshape connected in sequence; The convolution layer is used to perform a convolution layer operation on the input data to extract features, and the chunk layer Chunk is used to divide the extracted results into 3 sub-blocks along the channel dimension. The reshape layer Reshape is used to adjust the dimensions of the 3 sub-blocks to generate the query Q, key K and value V of the input data.

6. A moiré removal method according to claim 4 or 5, characterized in that: The low-frequency processing module LFPM is used to query Q, key K and value V based on the normalized low-frequency subband LL, adjust the weight parameters through the spatial and channel attention mechanism, restore the low-frequency structure, and correct the color distribution. This is achieved using the following calculation method: Where d is the quotient of the number of channels and the number of attention heads; The high frequency processing module HFPM is used to query Q, key K and value V based on the normalization of the three high frequency sub-bands LH, HL and HH, adjust the weight parameters through the spatial and channel attention mechanism, suppress high frequency noise and enhance texture details, and is implemented using the following calculation method: Where d is the quotient of the number of channels and the number of attention heads.

7. A moiré removal method according to claim 4 or 5, characterized in that: The cross-frequency fusion module CFIM is used to perform cross-frequency fusion on the output result of the low-frequency processing module LFPM and the output result of the high-frequency processing module HFPM, and specifically includes the following steps: The output of the low frequency processing module LFPM and the output of the high frequency processing module HFPM are spliced along the channel dimension, and the following calculation method is used to achieve this: I 0 =Conv(concate(I Low +I High )) Among them, I Low is the output result of the low frequency processing module LFPM, I High The output result of the high frequency processing module HFPM, 1 0 is the splicing result; The splicing results are pooled in the channel domain and spatial domain respectively, and the channel domain pooling result and the spatial domain pooling result are summed and input into the Sigmoid activation function to obtain the fusion result. The following calculation method is used to achieve this: I 1 =Sigmoid(ChPool(I 0 )+SpPool(I 0 )) Among them, ChPool() is channel domain pooling, and SpPool() is spatial domain pooling.

8. A moiré removal method according to any one of claims 1, 2, 4 or 5, characterized in that: The N is 3, and the standardized data is sequentially transformed through a first discrete wavelet transform DWT, a second discrete wavelet transform DWT, and a third discrete wavelet transform DWT; The decomposition result of the third discrete wavelet transform DWT is sequentially passed through the corresponding residual convolution module RCB and the frequency domain attention module FAT to obtain third feature data, and the third feature data is reconstructed through the inverse discrete wavelet transform IDWT to obtain third reconstructed data; The decomposition result of the second discrete wavelet transform DWT is sequentially passed through the corresponding residual convolution module RCB and the frequency domain attention module FAT to obtain second feature data, and the second feature data is fused with the third reconstructed data, and the fused result is sequentially passed through the corresponding residual convolution module RCB and the inverse discrete wavelet transform IDWT to reconstruct the second reconstructed data; The decomposition result of the first discrete wavelet transform DWT is used to obtain the first feature data through the corresponding frequency domain attention module FAT, and the first feature data is fused with the second reconstructed data, and the fused result is reconstructed in turn through the corresponding residual convolution module RCB and the inverse discrete wavelet transform IDWT to obtain the final reconstructed data.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the method according to any one of claims 1 to 8.

10. A storage medium comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, a moiré removal method according to any one of claims 1 to 8 is implemented.

Images