Moire pattern removing method and system based on generative AI model

Abstract

The invention relates to a moire removal method and system based on a generative AI model. Comprising the following steps: S1, receiving a to-be-processed moire-containing image, and taking the to-be-processed moire-containing image as a reference image; s2, extracting reference features of at least one level from the reference image by using a feature extraction network; s3, initializing a noise image, and iteratively de-noising the noise image in a plurality of preset time steps; and S4, in the de-noising processing of each time step, inputting the noise image of the current time step, the time step code and the reference feature into a diffusion generation model based on a Transform architecture. According to the invention, the moire-containing image is taken as the reference image, and the multi-scale feature extraction is utilized to guide the diffusion generation model, so that powerful priori knowledge of the pre-training generation model can be fully utilized, and a clean image with consistent content and without moire can be regenerated.

Description

Technical Field

[0001] This invention relates to the field of image processing, and in particular to a method and system for removing moiré patterns based on a generative AI model. Background Technology

[0002] Moiré patterns are a type of visual artifact that occurs during digital imaging when the frequency of the sensor pixel grid overlaps with the periodic texture of the object being photographed (such as a screen or textile). It manifests as irregular colored ripples or stripes, severely degrading the visual quality and readability of images. Moiré patterns have become a pressing technical challenge in numerous scenarios, including screen content capture, industrial online inspection, professional photography, and document scanning.

[0003] Existing moiré removal techniques can be mainly divided into traditional methods and deep learning-based methods.

[0004] Traditional methods mainly include frequency domain filtering, spatial domain filtering, and image decomposition. Frequency domain filtering methods (such as notch filtering) attempt to identify and remove specific frequency components corresponding to moiré patterns in the Fourier domain. These methods are effective in processing moiré patterns with obvious, single-frequency characteristics, but their core assumption—that moiré patterns are completely separable from image content in the frequency domain—often does not hold true in real-world scenarios. The spectrum of moiré patterns often overlaps with the spectrum of the image's natural texture, making the filtering process prone to damaging image details, producing ringing effects, and exhibiting poor adaptability to complex and varied moiré patterns.

[0005] Deep learning-based methods, especially those based on convolutional neural networks (CNNs), have made significant progress in recent years. From early multi-scale networks (such as DMCNN and MopNet) to more recent frequency and spatial domain joint processing networks (such as FSD-Net) and more efficient architectures (such as MZNet), these methods, through end-to-end learning, far outperform traditional methods in terms of performance. However, existing deep learning methods still face the following challenges: 1. The contradiction between detail preservation and moiré pattern removal: CNN models, while removing moiré patterns, can easily lead to over-smoothing of image textures, losing high-frequency details and making the recovered image less realistic and natural. 2. Insufficient generalization ability to complex patterns: Most existing models are trained on specific datasets. When encountering moiré patterns not seen in the training set (such as combinations of different frequencies, directions, and shapes), the removal effect will significantly decrease. 3. High computational complexity: To achieve ideal results, models usually require a huge number of parameters and computational resources (such as tens of terabytes of MACs), making it difficult to achieve real-time processing on devices with limited computing resources (such as smartphones and edge computing terminals). 4. Heavy reliance on paired data: Most supervised learning methods require large amounts of precisely aligned “moiré-clean image” data pairs for training, while building high-quality, large-scale real datasets is extremely costly.

[0006] Therefore, designing a moiré removal technique that can completely remove various complex moiré patterns, retain the texture details and realism of the original image to the greatest extent, and take into account both computational efficiency and generalization ability is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0007] In view of this, the present invention aims to propose a method and system for moiré pattern removal based on a generative AI model, in order to solve the problems in the prior art.

[0008] To achieve the above objectives, the technical solution of the present invention is implemented as follows: This invention proposes a moiré pattern removal method based on a generative AI model, comprising the following steps: S1, Receive an image containing moiré patterns to be processed and use it as a reference image; S2, using a feature extraction network, extract at least one level of reference features from the reference image; S3, initialize a noisy image, and iteratively denoise the noisy image within a preset number of time steps; S4, In the denoising process at each time step, the noise image of the current time step, the time step encoding, and the reference features are input together into a diffusion generation model based on the Transformer architecture, and the diffusion generation model is guided to predict noise through the reference features; S5, update the noisy image according to the predicted noise until the last time step is completed, and obtain a target image with moiré removed.

[0009] Furthermore, in S5, the image update performs the following operations: in, The noisy image at the current time step. For the updated noisy image, , , For predefined noise scheduling parameters, The noise predicted by the diffusion generation model. It is standard Gaussian noise.

[0010] Furthermore, the reference feature is a multi-scale feature, including at least two of the following: high-level semantic features, mid-level texture features, and low-level edge features.

[0011] Furthermore, in the denoising process at different time steps, the guiding weights of reference features at different scales for noise prediction are dynamically adjusted.

[0012] Furthermore, the diffusion generation model fuses the reference features with the noisy image information at the current time step through a cross-attention mechanism.

[0013] Furthermore, the method also includes a training step, which employs a composite loss function, including at least two of pixel-level loss, perceptual-level loss, frequency domain loss, and structural similarity loss.

[0014] Furthermore, the composite loss function is obtained through the following calculation: in, , , , These are the weighting coefficients corresponding to each loss term. For pixel-level loss, In order to perceive loss, For frequency domain loss, This is the structural similarity loss.

[0015] Furthermore, the pixel-level loss is obtained through the following calculation: in, It can be 1 or 2; The total number of pixels in the image. To predict images for the model, For a truly clean image; The perceived loss is obtained through the following calculation: in, For pre-training the VGG network Feature extractor of the layer; The selected number of VGG layers, typically 5. The weights are for the corresponding layers; The frequency domain loss is obtained through calculation: FFT(·) is the Fast Fourier Transform, which transforms the image from the spatial domain to the frequency domain. The Frobenius norm is used to calculate the overall difference in spectral amplitude. HPF(·) is a high-pass filter used to extract high-frequency components. These are high-frequency weighting coefficients; The structural similarity loss is obtained through the following calculation: in, For image The local mean, For local standard deviation, For local covariance, , It is the stability constant.

[0016] Furthermore, the method also includes a parameter fine-tuning step: injecting a low-rank adaptation module into the pre-trained diffusion generation model, and fine-tuning the parameters of the low-rank adaptation module using only a small amount of moiré image data from a specific domain, while keeping the original parameters of the diffusion generation model unchanged.

[0017] This invention also proposes a moiré pattern removal system based on a generative AI model to implement the above method, comprising: The input module is used to receive images containing moiré patterns; A feature extraction module, connected to the input module, is used to extract multi-scale reference features from the moiré image; The diffusion generation module, connected to the feature extraction module, includes a diffusion generation model based on the Transformer architecture, used to perform a condition-guided diffusion denoising generation process; The output module, connected to the diffusion generation module, is used to output the target image after removing moiré patterns.

[0018] Compared with the prior art, the present invention has the following advantages: In this invention, by using an image containing moiré patterns as a reference image and leveraging multi-scale feature extraction to guide a diffusion generation model, the powerful prior knowledge of the pre-trained generation model can be fully utilized to regenerate a clean image with identical content but no moiré patterns. This avoids the texture smoothing and detail loss problems common in traditional restoration methods, resulting in a more visually natural and realistic image.

[0019] This invention does not rely on the recognition and repair of specific moiré patterns, but guides the generation process through the content of a reference image. Therefore, it also exhibits good robustness to unseen moiré types (such as different frequencies, directions, and color combinations) and has strong cross-scene adaptability. Attached Figure Description

[0020] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a schematic diagram of the overall process of the method of the present invention. Detailed Implementation

[0021] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0022] In the description of this invention, it should be noted that the terms "upper," "lower," "inner," and "back," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0023] Furthermore, in the description of this invention, unless otherwise explicitly defined, the terms "installation," "connection," "linking," and "connector" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention in light of the specific circumstances.

[0024] The following will refer to the appendix. Figure 1 The present invention will be described in detail with reference to the embodiments.

[0025] Example 1 Overall, this embodiment discloses a moiré pattern removal method based on a generative AI model, including the following steps: S1, Receive an image containing moiré patterns to be processed and use it as a reference image; S2, using a feature extraction network, extract at least one level of reference features from the reference image; S3, initialize a noisy image, and iteratively denoise the noisy image within a preset number of time steps; S4, In the denoising process at each time step, the noise image of the current time step, the time step encoding, and the reference features are input together into a diffusion generation model based on the Transformer architecture, and the diffusion generation model is guided to predict noise through the reference features; S5, update the noisy image according to the predicted noise until the last time step is completed, and obtain a target image with moiré removed.

[0026] In this embodiment, by using an image containing moiré patterns as a reference image and leveraging multi-scale feature extraction to guide a diffusion generation model, the powerful prior knowledge of the pre-trained generation model can be fully utilized to regenerate a clean image with identical content but no moiré patterns. This avoids the texture smoothing and detail loss problems common in traditional restoration methods, resulting in a more visually natural and realistic image.

[0027] This invention does not rely on the recognition and repair of specific moiré patterns, but guides the generation process through the content of a reference image. Therefore, it also exhibits good robustness to unseen moiré types (such as different frequencies, directions, and color combinations) and has strong cross-scene adaptability.

[0028] By introducing an efficient diffusion model based on Transformer, the computational complexity is significantly reduced while maintaining high image quality, enabling the method to achieve near real-time processing in resource-constrained environments such as mobile devices, thus expanding its practical application scenarios.

[0029] In S5, the image update performs the following operations: in, The noisy image at the current time step. For the updated noisy image, , , For predefined noise scheduling parameters, The noise predicted by the diffusion generation model. It is standard Gaussian noise.

[0030] It should be noted that, in this embodiment, the reference feature is a multi-scale feature, including at least two of the following: high-level semantic features, mid-level texture features, and low-level edge features.

[0031] By extracting and fusing multi-scale features such as high-level semantics, mid-level texture, and low-level edges, hierarchical and refined guidance can be achieved during image generation. High-level semantic features ensure the correct overall layout and main content of the generated image; mid-level texture features are responsible for restoring the material and delicate texture of the object's surface; and low-level edge features ensure clear outlines and sharp details. This synergistic effect ensures that the generated image is highly consistent with the reference image in terms of content, texture, and structure.

[0032] It should be further explained that in the denoising process at different time steps, the guiding weights of reference features at different scales for noise prediction are dynamically adjusted.

[0033] Using all features evenly throughout the process may lead to errors in content generation due to detail interference in the early stages, or loss of detail due to insufficient semantic information in the later stages. For example, in the early stages, increase the weight of high-level semantic features to ensure the generation of correct image content; in the later stages, increase the weight of low-level edge and texture features to ensure richer content generation.

[0034] Based on the above settings, the diffusion generation model fuses the reference features with the noisy image information at the current time step through a cross-attention mechanism. This ensures that the generated image maintains a high degree of consistency with the reference image not only in terms of global structure but also in local details, effectively avoiding content misalignment or information conflicts that may result from simple feature splicing or addition.

[0035] The method also includes a training step that employs a composite loss function, which includes at least two of pixel-level loss, perceptual-level loss, frequency domain loss, and structural similarity loss.

[0036] By fusing pixel-level, perceptual-level, frequency domain, and structural similarity losses, this composite loss function constructs a multi-dimensional supervision system. It drives the model not only to pursue accurate pixel value reproduction (pixel loss) but also to emphasize the visual naturalness and realism of the generated results (perceptual loss), while specifically suppressing moiré-related frequency domain artifacts (frequency domain loss) and maintaining the structural integrity of the image (SSIM loss). This collaborative optimization mechanism effectively avoids problems such as over-smoothing and texture distortion caused by the model optimizing a single metric (such as PSNR).

[0037] The composite loss function is obtained through the following calculation: in, , , , These are the weighting coefficients corresponding to each loss term. For pixel-level loss, In order to perceive loss, For frequency domain loss, This is the structural similarity loss.

[0038] The pixel-level loss is obtained through the following calculation: in, It can be 1 or 2; The total number of pixels in the image. To predict images for the model, For a truly clean image; The perceived loss is calculated as follows: in, For pre-training the VGG network Feature extractor of the layer; The selected number of VGG layers, typically 5. The weights are for the corresponding layers; Frequency domain loss is obtained through calculation: FFT(·) is the Fast Fourier Transform, which transforms the image from the spatial domain to the frequency domain. The Frobenius norm is used to calculate the overall difference in spectral amplitude. HPF(·) is a high-pass filter used to extract high-frequency components. These are high-frequency weighting coefficients; The structural similarity loss is obtained through the following calculation: in, For image The local mean, For local standard deviation, For local covariance, , It is the stability constant.

[0039] In addition, the method includes a parameter-efficient fine-tuning step: a low-rank adaptation module is injected into the pre-trained diffusion generation model, and the parameters of the low-rank adaptation module are fine-tuned using only a small amount of moiré image data from a specific domain, while keeping the original parameters of the diffusion generation model unchanged.

[0040] This allows the model to quickly adapt to unique moiré patterns generated by new materials, new devices, or new scenarios (such as specific models of displays or special textile fabrics), avoiding the huge computational resources and data costs required to train a large model from scratch.

[0041] Example 2 This embodiment is a further extension of Embodiment 1 described above. This embodiment proposes a moiré pattern removal system based on a generative AI model to implement the method proposed in the above embodiment. It includes: an input module for receiving an image containing moiré patterns; The feature extraction module, connected to the input module, is used to extract multi-scale reference features from the moiré image; The diffusion generation module, connected to the feature extraction module, contains a diffusion generation model based on the Transformer architecture, used to perform a condition-guided diffusion denoising generation process; The output module, connected to the diffusion generation module, is used to output the target image after removing moiré patterns.

[0042] The modules are seamlessly integrated, forming a complete automated processing pipeline. Users only need to input a moiré image, and the system can automatically complete the entire complex process from multi-scale feature analysis and conditional guided denoising to the final high-quality image generation, without manual intervention. It is easy to operate, and the output is stable and reliable, greatly improving processing efficiency and user experience.

[0043] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for removing moiré patterns based on a generative AI model, characterized in that: Includes the following steps: S1, Receive an image containing moiré patterns to be processed and use it as a reference image; S2, using a feature extraction network, extract at least one level of reference features from the reference image; S3, initialize a noisy image, and iteratively denoise the noisy image within a preset number of time steps; S4, In the denoising process at each time step, the noise image of the current time step, the time step encoding, and the reference features are input together into a diffusion generation model based on the Transformer architecture, and the diffusion generation model is guided to predict noise through the reference features; S5, update the noisy image according to the predicted noise until the last time step is completed, and obtain a target image with moiré removed.

2. The moiré pattern removal method based on a generative AI model according to claim 1, characterized in that: In S5, the image update performs the following operation: in, The noisy image at the current time step. For the updated noisy image, , , For predefined noise scheduling parameters, The noise predicted by the diffusion generation model. It is standard Gaussian noise.

3. The moiré pattern removal method based on a generative AI model according to claim 1, characterized in that: The reference features are multi-scale features, including at least two of the following: high-level semantic features, mid-level texture features, and low-level edge features.

4. The moiré pattern removal method based on a generative AI model according to claim 1, characterized in that: In the denoising process at different time steps, the guiding weights of reference features at different scales for noise prediction are dynamically adjusted.

5. The moiré pattern removal method based on a generative AI model according to claim 1, characterized in that: The diffusion generation model fuses the reference features with the noisy image information at the current time step through a cross-attention mechanism.

6. The moiré pattern removal method based on a generative AI model according to claim 1, characterized in that: The method further includes a training step, which employs a composite loss function, including at least two of pixel-level loss, perceptual-level loss, frequency domain loss, and structural similarity loss.

7. The moiré pattern removal method based on a generative AI model according to claim 6, characterized in that: The composite loss function is obtained through the following calculation: in, , , , These are the weighting coefficients corresponding to each loss term. For pixel-level loss, In order to perceive loss, For frequency domain loss, This is the structural similarity loss.

8. The moiré pattern removal method based on a generative AI model according to claim 7, characterized in that: The pixel-level loss is obtained through the following calculation: in, It can be 1 or 2; The total number of pixels in the image. To predict images for the model, For a truly clean image; The perceived loss is obtained through the following calculation: in, For pre-training the VGG network Feature extractor of the layer; The selected VGG layer number is 5. The weights are for the corresponding layers; The frequency domain loss is obtained through calculation: FFT(·) is the Fast Fourier Transform, which transforms the image from the spatial domain to the frequency domain. The Frobenius norm is used to calculate the overall difference in spectral amplitude. HPF(·) is a high-pass filter used to extract high-frequency components. These are high-frequency weighting coefficients; The structural similarity loss is obtained through the following calculation: in, For image The local mean, For local standard deviation, For local covariance, , It is the stability constant.

9. The moiré pattern removal method based on a generative AI model according to claim 1, characterized in that: The method further includes a parameter fine-tuning step: injecting a low-rank adaptation module into the pre-trained diffusion generation model, and fine-tuning the parameters of the low-rank adaptation module using only a small amount of moiré image data from a specific domain, while keeping the original parameters of the diffusion generation model unchanged.

10. A moiré pattern removal system based on a generative AI model, used to implement the method as described in any one of claims 1 to 8, characterized in that, include: The input module is used to receive images containing moiré patterns; A feature extraction module, connected to the input module, is used to extract multi-scale reference features from the moiré image; The diffusion generation module, connected to the feature extraction module, includes a diffusion generation model based on the Transformer architecture, used to perform a condition-guided diffusion denoising generation process; The output module, connected to the diffusion generation module, is used to output the target image after removing moiré patterns.

Images