Image moire removing method, system and equipment based on perception guiding and depth generation prior, and medium

Abstract

The invention discloses an image moire removing method, system and device based on perception guidance and depth generation prior, and a medium, and the method comprises the following steps: providing an original training data set, and enriching training samples through a data enhancement strategy; a coarse-grained network is used for carrying out primary processing on moire patterns in the image, and the visual influence of the moire patterns is reduced; a diffusion network based on wavelet sampling is adopted, and the relationship between moire removal and image detail reservation is balanced by combining wavelet transform and discrete cosine transform; and refining and reconstructing the image by using a decoder network to obtain a final moire-removed image. According to the method, the image quality can be remarkably improved, and the details of the image can be better reserved, so that the image after moire removal is more natural and real; according to the method, the perception characteristics of a human vision system on image quality are fully considered, and prior information of the deep generation network is introduced, so that the moire removing precision and stability are enhanced.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an image moiré removal method, system, device and medium based on perception guidance and depth generation prior. Background Art

[0002] In recent years, with the rapid development of digital sensing technology, the convenience and efficiency of image acquisition have been greatly improved, especially in the fields of mobile photography, security monitoring, medical imaging, etc. High-quality images have become a common demand of people. Clear and delicate image quality not only helps to enhance the beauty of photographic works, but also improves the accuracy of recognition in security monitoring, and even has a decisive significance in medical diagnosis. However, the improvement of image quality faces many challenges. Due to the limitation of sensor resolution and the difference in pixel area, images are often troubled by problems such as moiré, which affects the visual effect of the image and may even cause the loss of key details. Moiré is a special texture interference caused by the grid effect between the sensor and the object, which can greatly affect the clarity of the image in severe cases. Therefore, the development of a special de-moiré algorithm has become the key to improving image quality, which can effectively repair the interference problem in the image and ensure higher image quality.

[0003] Moiré is a common artifact in the digital imaging process, especially in natural images or complex textures. It is often distributed in a complex and unpredictable way, which seriously affects the visual quality of the image and the processing effect of downstream tasks. Although the existing image de-moiré technology has made some progress, it still faces many challenges. First, moiré has a very wide frequency band, which makes it difficult for existing methods to effectively remove moiré in different frequency bands. Second, the spatial frequency of moiré is complex and diverse, and it is difficult for existing technologies to accurately process different types of moiré. At present, the mainstream de-moiré methods usually rely on pixel-level comparison with real clean images for optimization. Although this method can achieve certain effects in specific scenarios, it often shows limitations when processing images containing complex moiré, and it is difficult to balance the relationship between detail preservation and artifact removal. In addition, most of the existing de-moiré methods are based on small models, which limits their ability to learn rich prior knowledge from large-scale data. Because these methods still have "ill-posed problems", the images after de-moiré often appear "over-smoothed", resulting in the loss of image details, affecting the actual use effect. Summary of the invention

[0004] The purpose of the present invention is to provide an image de-moiré method, system, device and medium based on perception guidance and deep generative priors. Unlike traditional small models, the diffusion model can learn the distribution and manifold of natural images from a large amount of external data, thereby more effectively removing complex moiré. The present invention can not only significantly improve the image quality, but also better preserve the details of the image, making the de-moiré image more natural and realistic; the method fully considers the perceptual characteristics of the human visual system for image quality, and introduces the prior information of the deep generative network to enhance the accuracy and stability of the de-moiré processing.

[0005] To achieve the above purpose, the technical solution provided by the present invention is:

[0006] A first aspect of the present application provides an image moiré removal method based on perception guidance and depth generation prior, characterized in that it comprises the following steps:

[0007] Provide original training data sets and enrich training samples through data enhancement strategies;

[0008] Use a coarse-grained network to preliminarily process the moiré in the image to reduce the visual impact of the moiré;

[0009] A diffusion network based on wavelet sampling is used to balance the relationship between moiré removal and image detail preservation by combining wavelet transform and discrete cosine transform.

[0010] The decoder network is used for image refinement and reconstruction to obtain the final de-moiréd image.

[0011] To optimize the above technical solutions, the specific measures taken also include:

[0012] The providing of the original training data set and enriching the training samples through the data enhancement strategy includes: rotating, flipping, and scaling the images in the original training data set to generate an extended training data set.

[0013] Furthermore, the use of a coarse-grained network to perform preliminary processing on the moiré in the image includes: a deep learning framework based on a convolutional neural network, using an adaptive multi-scale de-moiré network, performing downsampling operations at multiple scales, extracting image features at different resolutions, and independently capturing specific frequency features of the moiré at each scale, and comprehensively covering the wide spectrum characteristics of the moiré by fusing specific frequency features at different scales.

[0014] Furthermore, an adaptive multi-spectral coding technique is introduced into the adaptive multi-scale de-moiré network to adaptively adjust the statistical characteristics of the image, and the frequency characteristics in the image are dynamically encoded to help the network automatically identify and adapt to the frequency pattern of the moiré; a variety of neural networks are used in the adaptive multi-scale de-moiré network to improve the de-moiré effect.

[0015] Furthermore, the use of a coarse-grained network to perform preliminary processing on the moiré patterns in the image also includes introducing a VAE encoder network, and adopting a pre-trained encoder strategy and freezing encoder parameters to reduce computational overhead by mapping the image to a low-dimensional latent space to cope with large-scale image data processing for moiré removal.

[0016] Furthermore, the use of the diffusion network based on wavelet sampling, by combining wavelet transform and discrete cosine transform, balances the relationship between moiré removal and image detail retention, specifically: utilizing the multi-scale frequency decomposition characteristics of wavelet transform, decomposing the image into different frequency components, so as to accurately identify and eliminate moiré, while maintaining the overall structure and detail information of the image;

[0017] Among them, the diffusion process in the diffusion network is as follows:

[0018] Suppose the image after preliminary moiré removal is y 0 , during the diffusion process, the image is converted into pure noise by gradually adding noise to it, which can be expressed as:

[0019]

[0020] Among them, y t is the image of step t, ∈ t is the noise that follows a normal distribution, During the diffusion process, the intensity of the noise is determined by a predefined planning factor α t Control, as t increases, the image is gradually covered by noise; when t = T, the image is completely transformed into noise,

[0021] By deriving the diffusion process, any time step y t With the original image y 0 The relationship between:

[0022]

[0023] in, is the cumulative product of the planning factors of the previous t steps, that is The noise is combined with the original image through this formula to generate the intermediate image y in the diffusion process t , providing a basis for the subsequent denoising and restoration process;

[0024] The reverse diffusion process in a diffusion network is as follows:

[0025] The inverse diffusion process gradually restores the image completely covered by noise to the original data distribution through recursive denoising steps; the inverse diffusion process uses a learnable denoising model ∈ θ (x t ,t) to achieve, the formula is as follows:

[0026]

[0027] in, represents the reverse diffusion process, ∈ θ (x t ,t) is the denoising function learned during the training process, which is used to denoise the image at each time step t; through recursive application, the noise in the image is gradually reduced, generating a more natural image and removing moiré artifacts.

[0028] Furthermore, the conditional diffusion function is introduced into the diffusion network To guide the denoising and restoration of the image in the inverse diffusion process, the conditional diffusion function The formula is as follows:

[0029]

[0030] Among them, LPF represents the low-pass filter function, which aims to separate the high-frequency information from the low-frequency information in the image to control the information transmission in the diffusion process; if too much high-frequency information is transmitted, the generated image may become blurred; on the contrary, if insufficient low-frequency information is transmitted, the semantic details of the image will not be fully preserved.

[0031] Furthermore, the wavelet transform process includes:

[0032] Decomposition of input image: Assume that the input image is I, which is decomposed into low-frequency components L by Haar wavelet transform I and high frequency component H I , the formula is as follows:

[0033]

[0034] Among them, L I Represents the low-frequency components of the image, mainly including the overall outline and large-scale structure of the image; H I Represents the high-frequency components of the image, including the texture and detail information of the image;

[0035] Decomposition of diffusion model features: The intermediate feature image F generated by the diffusion model is also decomposed by Haar wavelet to obtain its low-frequency component L F and high frequency component H F, the formula is as follows:

[0036]

[0037] Using a low-frequency replacement strategy, the low-frequency component L of the input image is replaced I Replaced by the low-frequency component L generated by the diffusion model F , to ensure that the global structure of the image can be refined and optimized by the diffusion model, the process is as follows:

[0038] Low-frequency replacement: By replacing the image L I The low-frequency component of is replaced by the low-frequency component L of the diffusion characteristic F , to achieve global structural optimization in the process of removing moiré patterns, the formula is as follows:

[0039] (L I ,H I )→(L F ,H I )

[0040] Reconstruct the image: After completing the low-frequency replacement, recombining the low-frequency and high-frequency components of the image, and reconstructing the image through inverse wavelet transform to obtain the image after de-moiré , the formula is as follows:

[0041]

[0042] Furthermore, the decoder network is used for image refinement and reconstruction to obtain the final de-moiré image, specifically: restoring details that may be lost due to the de-moiré process, maintaining the structural consistency and naturalness of the image; the decoder network structure includes: a multi-scale feature input layer, a gradually refined convolutional layer, a skip connection mechanism, and feature fusion and multi-layer aggregation.

[0043] A second aspect of the present application provides an image de-moiré system based on perception guidance and depth generation prior, comprising:

[0044] The data preprocessing module is used to provide the original training data set and enrich the training samples through data enhancement strategies;

[0045] A coarse-grained network module is used to perform preliminary processing on the moiré patterns in the image using a coarse-grained network to reduce the visual impact of the moiré patterns.

[0046] A diffusion network module based on wavelet sampling is used to balance the relationship between moiré removal and image detail preservation by using a diffusion network based on wavelet sampling and combining wavelet transform and discrete cosine transform;

[0047] A decoder network module is used to use the decoder network for image refinement and reconstruction to obtain a final de-moiré image;

[0048] The coarse-grained network module is based on a deep learning framework of a convolutional neural network and uses an adaptive multi-scale de-moiré network module, wherein the adaptive multi-scale de-moiré network module includes an adaptive multi-spectrum encoding module and a de-moiré module;

[0049] The adaptive multi-spectral encoding module is used to adaptively adjust the statistical characteristics of the image, and helps the network automatically identify and adapt to the frequency pattern of the moiré by dynamically encoding the frequency characteristics in the image;

[0050] The moiré module includes: an improved residual block, which is used to introduce a feature transfer mechanism, alleviate the gradient vanishing problem in the deep network through jump connections, and retain more detail information in the deep layer of the network; a large core attention module, which is used to use large core convolution and self-attention mechanisms to capture long-distance dependencies in the image over a large range, helping the network to better understand the global information in the image; a multi-layer perceptron module, which is used to model complex feature spaces through the connection of multiple layers of neurons, perform nonlinear transformations on the features of the network output, and enhance the network's expressiveness.

[0051] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the image demoiré method based on perceptual guidance and depth generation prior as described above is implemented.

[0052] A fourth aspect of the present application provides a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the image de-moiré method based on perceptual guidance and depth generation prior as described above.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] The present invention adopts data enhancement strategies in the training stage, including image rotation, flipping and scaling, to improve the generalization ability of the model when processing different types of moiré patterns, generate more diverse training data sets, ensure that all images are uniformly processed to the same size, and ensure data consistency. The prior art often relies only on raw data for training, resulting in poor performance of the model when encountering complex or non-standard moiré scenes. The present invention greatly enriches the training samples through data enhancement strategies, so that the model can have stronger recognition and processing capabilities when facing moiré patterns of different angles, directions and scales, improve the generalization of the model, especially in complex and changeable scenes. At the same time, standardized processing improves training efficiency, ensuring that the model converges faster and has higher stability.

[0055] The present invention first performs preliminary removal of significant moiré in the image through a coarse-grained network module, greatly reducing the visual impact of the moiré and reducing the computational complexity of subsequent fine processing. Traditional moiré removal methods usually directly process the entire image, resulting in high complexity and slow processing speed. The present invention can quickly eliminate large-area, obvious moiré noise by first performing coarse-grained moiré removal, laying the foundation for subsequent more refined moiré removal processes. This layered processing method reduces the amount of subsequent calculations and improves the efficiency of the overall system. At the same time, it enables the fine network to perform efficient processing based on a cleaner image, thereby improving the final effect of the image.

[0056] The present invention uses an adaptive multi-scale moiré removal network AMSDM, which extracts image features at different frequencies through multi-scale downsampling. Combined with an adaptive spectrum coding module, it can effectively capture the multi-frequency characteristics of moiré. The generation of moiré is the result of frequency interference. Moiré of different frequencies behaves differently in images of different resolutions. Compared with the traditional single-scale processing model, the AMSDM network can process moiré features of different frequencies at multiple scales, ensuring that the model can effectively remove moiré from low frequency to high frequency. In addition, AMSDM also has the ability to adaptively adjust image frequency characteristics, so that the model can automatically adapt to different moiré patterns, thereby achieving better removal effect. This multi-scale network design greatly improves the flexibility and accuracy of the model in moiré removal tasks.

[0057] The present invention adopts a variational autoencoder (VAE) network, which is used to compress high-dimensional images into a low-dimensional latent space, reducing the consumption of computing resources while maintaining effective extraction of the main features of the image. In traditional de-moiré tasks, the computational burden of processing high-dimensional image data is extremely high, and the introduction of VAE fundamentally reduces the computational complexity by compressing the image into a latent space, especially making the processing of high-frequency information more efficient. In the latent space, the model can complete the de-moiré operation with fewer resources, and can effectively retain details during the image reconstruction process, greatly improving the processing efficiency of the de-moiré task, and is particularly suitable for large-scale image processing tasks or resource-constrained scenarios.

[0058] The present invention proposes a diffusion network that combines wavelet transform and discrete cosine transform (DCT), which, through multi-scale processing and frequency decomposition, can remove moiré while retaining the detail information of the image. Traditional moiré removal methods usually find it difficult to remove moiré and retain details at the same time. The present invention, by introducing a low-pass filter that combines wavelet and DCT, can effectively separate high-frequency and low-frequency information, maintain the structure and details of the image while eliminating moiré artifacts, and balance the relationship between moiré removal and image quality through step-by-step frequency processing, generating a clearer and more natural image. This multi-scale, multi-frequency processing strategy significantly improves the performance of the model in complex scenes.

[0059] The invention can make the image moiré removal task more practical and accurate in complex scenes. It not only solves the shortcomings of the existing technology, but also significantly improves the processing speed, removal effect and computing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 : A flowchart of the image moiré removal method based on perception guidance and depth generation prior of the present invention.

[0061] Figure 2 : A schematic diagram of the structure of the image moiré removal system based on perception guidance and depth generation priors of the present invention.

[0062] Figure 3 : Schematic diagram of the intermediate steps in the diffusion process of the present invention. DETAILED DESCRIPTION

[0063] The above contents of the present invention are further described in detail below in the form of embodiments, but this should not be understood as the scope of the above subject matter of the present invention being limited to the following embodiments, and all technologies realized based on the above contents of the present invention belong to the scope of the present invention.

[0064] Some of the terms involved in the solution of the present invention are explained as follows:

[0065] Ill-posed: In image restoration tasks, "ill-posed" problems refer to problems for which the existence, uniqueness, and stability of mathematical solutions cannot be guaranteed. Image restoration is usually an inverse problem, that is, restoring the complete image information from a partially observed image. Due to information loss in the damaged area, nonlinear damage processes, etc., the restoration results may have multiple reasonable solutions or be very sensitive to initial conditions. In order to solve these ill-posed problems, strategies such as introducing prior knowledge, using deep learning models, multi-scale processing methods, and data enhancement are usually used to improve the model's restoration effect and the stability of the solution.

[0066] Oversmoothing: refers to the excessive blurring of image details and textures in image processing or restoration tasks, causing the restoration results to appear too smooth and lack details.

[0067] VAE encoder: The VAE encoder is a key component in the variational autoencoder (VAE) and is responsible for mapping the input data to a probability distribution in the latent space. It extracts the features of the input data through a neural network and outputs the mean and variance vectors of the latent space, which define the latent distribution of the data. Subsequently, through the reparameterization technique, the VAE encoder can sample latent variables from this distribution to generate new samples similar to the input data. The VAE encoder is not only used to compress data, but also provides a basis for generative models to achieve data generation by learning the distribution of the latent space.

[0068] Diffusion Models: Diffusion Models are a type of generative model that generates complex data through a step-by-step denoising process from noise to data. Its working principle includes two stages: forward diffusion and reverse generation. In the forward diffusion process, noise is gradually added to the real data; in the reverse generation process, the model gradually reconstructs the data through denoising. Compared with other generative models, diffusion models have high generation quality and stable training, and are widely used in image, audio and other fields. Although the generation process is slow, it performs particularly well in high-resolution image generation, becoming a cutting-edge technology in generative modeling.

[0069] DCT: Discrete cosine transform (DCT) is a mathematical transform that converts spatial or time domain signals into frequency domain representation. It is widely used in signal processing and image compression. DCT can effectively compress energy by decomposing the signal into a linear combination of a series of cosine functions, concentrating most of the important information in the low-frequency part, thereby reducing redundant data. In the field of image processing, DCT is the core step of compression algorithms such as JPEG, achieving efficient compression by removing high-frequency details. The characteristics of DCT include energy compression, decorrelation, and orthogonality, making it a common tool for processing image and video data.

[0070] Wavelet transform: Wavelet transform is a signal processing tool used to convert signals from the time or space domain to the frequency domain while retaining local information. Unlike Fourier transform, wavelet transform can provide localized analysis in both time and frequency, and is suitable for processing non-stationary signals. Its multi-resolution analysis characteristics enable it to capture the details and overall trends of signals at different scales. Wavelet transform is widely used in image compression, denoising, signal analysis and other fields. It can achieve compression and denoising effects by removing high-frequency noise, and is an important technology in modern signal processing.

[0071] TIP dataset: The first large-scale moiré dataset. This dataset is large in size, containing a total of 135,000 pairs of images. Each pair of images consists of two parts, one is the image contaminated by moiré, and the other is the corresponding reference image that is not contaminated by moiré.

[0072] Haar wavelet: Haar wavelet is a simple and efficient discrete wavelet transform that decomposes the signal into low-frequency and high-frequency components by weighted averaging and difference. Its calculation process is simple and can effectively capture the edge and mutation information in the signal. It is suitable for applications such as image compression, denoising and edge detection. When processing images, Haar wavelet can decompose the image into frequency sub-bands of different scales, thereby performing multi-resolution analysis.

[0073] In one embodiment of the present invention, a method for removing moiré from an image based on perception guidance and depth generation prior is proposed. Figure 1 As shown, the following steps are included:

[0074] Provide original training data sets and enrich training samples through data enhancement strategies;

[0075] Use a coarse-grained network to preliminarily process the moiré in the image to reduce the visual impact of the moiré;

[0076] A diffusion network based on wavelet sampling is used to balance the relationship between moiré removal and image detail preservation by combining wavelet transform and discrete cosine transform.

[0077] The decoder network is used for image refinement and reconstruction to obtain the final de-moiréd image.

[0078] Data preprocessing includes the training phase: an original training data set is provided to the model, denoted as D. In order to enhance the generalization ability of the model in complex scenes and ensure that it can cope with moiré interference of different types and intensities, a variety of data enhancement strategies are designed. The combination of these strategies not only enriches the training samples, but also enhances the robustness of the model; Inference phase: the design goal of the data preprocessing module in the inference phase is to efficiently process the input image and maximize the use of hardware resources to ensure a balance between the calculation speed of the inference phase and the output image quality.

[0079] The network structure of the present application consists of three core components: a coarse-grained network, a diffusion network based on wavelet sampling, and a decoder network. Each module works together, layer by layer, to gradually improve the effect of removing moiré.

[0080] Coarse-grained network module

[0081] The present invention proposes a new image de-moiré solution, the core of which is a multi-stage, multi-module de-moiré network architecture. First, a coarse-grained network module is used to perform preliminary processing on the significant moiré in the image. This module removes the more obvious large-area moiré noise in the image in a fast and effective way. The coarse-grained network module is designed to significantly reduce the visual impact of moiré, provide better input conditions for subsequent more sophisticated de-moiré processes, and reduce the complexity of subsequent calculations. The design of this module fully considers the multi-scale and multi-spectral characteristics of moiré, as well as how to achieve fast and efficient removal in complex visual scenes. The various sub-modules of the network module and their functional implementation will be described in detail below.

[0082] Multi-scale feature extraction network

[0083] The generation of moiré is a complex physical phenomenon, which involves the frequency interference between the image sensor and the fine texture. Therefore, the key to removing moiré lies in how to effectively deal with the interference characteristics of different frequencies. To this end, the present invention designs a multi-scale feature extraction network to deal with the multi-frequency characteristics of moiré by constructing a feature pyramid.

[0084] The network is based on a deep learning framework called Convolutional Neural Network (CNN), and specifically uses an Adaptive Multiscale Moiré Removal Network (AMSDM). AMSDM extracts image features at different resolutions by downsampling at multiple scales. At each scale, the network is able to independently capture specific frequency characteristics of moiré. By fusing these features at different scales, the network can fully cover the wide spectrum characteristics of moiré, from low to high frequencies, ensuring that no matter how the frequency of moiré changes, it can be effectively removed.

[0085] To further enhance the de-moiré effect, the AMSDM network introduces a key component - the adaptive multi-spectral coding module. This module helps the network automatically identify and adapt to the frequency pattern of moiré by dynamically encoding the frequency features in the image, thereby achieving more intelligent de-moiré processing. Specifically, this module can adaptively adjust the statistical characteristics of the image (such as mean, variance, etc.) to flexibly deal with moiré problems of different frequencies in different scenes and texture types. Especially in dynamic scenes, such as videos or images with moving objects, AMSDM exhibits excellent de-moiré capabilities and can handle moiré interference caused by texture changes in real time.

[0086] In addition to the multi-scale network design, AMSDM also includes a dedicated de-moiré module that combines a variety of advanced neural network technologies to further improve the de-moiré effect.

[0087] Encoder Network

[0088] In order to better solve the complex feature extraction and reconstruction problems in the process of image moiré removal, the present invention adopts an encoder network based on variational autoencoder (VAE) as one of the core components. The introduction of the encoder network not only improves the model's ability to represent image features, but also greatly reduces the model's demand for hardware resources during training and inference, optimizing the efficiency and scalability of the entire system.

[0089] In traditional image de-moiré tasks, the network usually needs to process high-dimensional image data, and the processing of such high-dimensional data often brings a huge computational burden. Especially in high-definition image processing tasks, how to improve computational efficiency without sacrificing image quality has become a problem to be solved in model design. As a classic generative model, VAE performs well in image compression and reconstruction tasks, and can effectively reduce computational overhead by mapping images to a low-dimensional latent space (Latent Space). Therefore, the present invention introduces a VAE encoder network to cope with the challenges of large-scale image data processing that frequently occur in de-moiré tasks.

[0090] Diffusion Network Based on Wavelet Sampling

[0091] The existence of moiré is due to the superposition of high-frequency interference, which is especially obvious when fine textures or grid patterns appear in the image. This interference will produce obvious stripes or ripples visually, which will have an adverse effect on the overall quality of the image. Traditional methods for removing moiré usually reduce interference through spatial filtering or frequency domain transformation, but these methods often find it difficult to retain sufficient image details while removing moiré. Therefore, the present invention introduces a diffusion network based on wavelet sampling, and provides a new image moiré removal scheme by combining wavelet transform and discrete cosine transform (DCT), which aims to balance the relationship between moiré removal and image detail retention.

[0092] Denoising Diffusion Model

[0093] In recent years, diffusion models, as a new generative model framework, have gradually gained wide attention and application in the field of computer vision. It learns the distribution characteristics of data by simulating the process of gradually adding noise to the data, thereby effectively generating high-quality images in the denoising process. In the task of moiré removal, the diffusion model shows great potential. The present invention designs a conditional network based on the diffusion process, and combines wavelet and DCT transforms for image enhancement and moiré removal.

[0094] The core idea of ​​the diffusion model is to gradually transform the image into pure noise through multi-step noise addition, and then gradually restore the true distribution of the image in the reverse denoising process. The entire diffusion process is controlled by a low-pass filter network to ensure that high-frequency moiré artifacts are removed while retaining low-frequency semantic information and structural features.

[0095] Wavelet-based sampling method

[0096] In order to solve the distribution problem of image moiré artifacts in different frequency domains, the present invention proposes a multi-scale sampling method based on wavelet transform. As a common image artifact, moiré is mainly concentrated in high-frequency information, causing interference and distortion in detail areas. Therefore, it is difficult to completely eliminate moiré by traditional moiré removal methods alone. Based on this, the present invention utilizes the multi-scale frequency decomposition characteristics of wavelet transform to decompose the image into different frequency components to accurately identify and eliminate moiré while maintaining the overall structure and detail information of the image.

[0097] Decoder Network

[0098] In the process of removing image moiré, even if the diffusion model and wavelet sampling network have significantly removed most of the artifacts, the generated image often still has insufficient details and blur. This phenomenon is mainly due to the fact that the initially generated image features lose some high-frequency information during the reconstruction process, resulting in the inability to restore the detail quality that matches the real clear image. Therefore, the present invention proposes a decoder network for the refinement and reconstruction of the final image to obtain a high-quality de-moiré image.

[0099] The decoder network is designed to further refine the demoiré image through multi-level feature fusion and reconstruction. Its main task is to restore high-frequency details that may be lost in the demoiré process while maintaining the structural consistency and naturalness of the image. The decoder network not only needs to comprehensively process multi-scale information, but also ensure that the generated image features have good perceptual quality and meet the needs of the human visual system.

[0100] In another embodiment, the present invention provides an image de-moiré system based on perception guidance and depth generation prior, comprising:

[0101] The data preprocessing module is used to provide the original training data set and enrich the training samples through data enhancement strategies;

[0102] A coarse-grained network module is used to perform preliminary processing on the moiré patterns in the image using a coarse-grained network to reduce the visual impact of the moiré patterns.

[0103] A diffusion network module based on wavelet sampling is used to balance the relationship between moiré removal and image detail preservation by using a diffusion network based on wavelet sampling and combining wavelet transform and discrete cosine transform;

[0104] The decoder network module is used to use the decoder network for image refinement and reconstruction to obtain the final de-moiré image.

[0105] In another embodiment, the present invention proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the image demoirification method based on perceptual guidance and depth generation prior as described above is implemented.

[0106] In another embodiment, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the image demoiré method based on perception guidance and depth generation prior as described above.

[0107] The technical solution of the present invention is further described in detail below in conjunction with specific implementation methods:

[0108] The overall framework of the network model of the present invention is as follows Figure 2 As shown. In the following embodiments, the model is trained on the TIP moiré training dataset and evaluated on its test set. The training set in the TIP moiré dataset contains 135,000 real-world moiré images and their corresponding high-resolution images. The dataset used in this embodiment is the test set in the TIP moiré dataset, which contains 100 images and is specifically used to evaluate the performance of the model. Through experiments on the test set, the effectiveness and performance of the model in removing moiré are verified.

[0109] The specific embodiments of the present invention are as follows:

[0110] Data preprocessing module

[0111] During the training phase, the model receives the original training data set D. In order to improve the generalization ability of the model in complex scenarios and ensure that it can effectively deal with diverse moiré interference, a series of data enhancement strategies are designed. These strategies not only enrich the training samples, but also enhance the robustness of the model to different types of moiré. The specific implementation is as follows:

[0112] Image Rotation:

[0113] Each image is randomly rotated from 0 to 360 degrees. By generating images from different perspectives, the model can learn how moiré patterns appear at various angles. This enables the model to better adapt to and process images at different angles in practical applications.

[0114] Image flip:

[0115] Based on the original data, horizontal and vertical flipping operations are implemented. By generating new samples, the model is able to identify and remove the characteristics of moiré in different directions. This flipping enhances the diversity of the data and helps the model grasp the characteristics of multi-directional moiré.

[0116] Scale expansion:

[0117] In order to simulate the changes of moiré patterns at different resolutions and scaling ratios, the original image is randomly scaled. This enables the model to better capture the changing patterns of moiré patterns at different sizes and resolutions, thereby improving its ability to handle multi-scale moiré patterns.

[0118] After these data augmentation operations, an extended training dataset D1 is generated. On this basis, all images are further processed to the same size to ensure the consistency of the training data. This standardization process not only improves the training efficiency, but also ensures the stability of the model under different input conditions. Through the combination of data augmentation and standardization, the model can converge faster during training and show excellent generalization performance.

[0119] The design goal of the inference stage is to achieve the best balance between inference speed and output image quality by efficiently processing input images and making the most of existing hardware resources. Specifically, the data preprocessing module in the inference stage adopts a series of optimization strategies to improve the overall performance and response speed of the system. The process includes the following key steps:

[0120] Image reading and normalization:

[0121] First, the data preprocessing module reads the input moiré image from an external device or storage system. The read image will undergo a series of preprocessing operations according to the model's requirements, including normalizing the image pixel values. Through normalization, the data range of the input image is kept consistent with the training phase, thereby avoiding data bias from affecting the model's performance. In addition, the module also supports color space conversion or format adjustment of input images in different formats to ensure that they meet the format and size requirements of the model input layer, providing standardized data input for subsequent reasoning processes.

[0122] Precision conversion:

[0123] One of the core optimization measures in the inference stage is to use FP16 (16-bit floating point) precision for calculations. Compared with the traditional FP32 (32-bit floating point) precision, FP16 precision can maintain high inference accuracy while significantly reducing computing resource consumption. By reducing the occupancy of video memory and reducing the computing load, the application of FP16 greatly improves the inference speed. This precision conversion is particularly suitable for resource-constrained hardware platforms, such as mobile devices, embedded systems, etc. In these scenarios, the application of FP16 not only improves the inference efficiency, but also optimizes the utilization of hardware resources.

[0124] Dynamically resize input images:

[0125] The input image size and resolution during the inference phase may vary greatly. To ensure the flexibility of model processing, the data preprocessing module has the ability to dynamically adjust the image input. The module automatically performs operations such as scaling and cropping based on the size and resolution of the actual input image to ensure that the image size meets the input requirements of the model. At the same time, the dynamic adjustment process ensures that the model maintains consistent de-moiré effects when processing images of different resolutions, thereby improving the generalization ability of the model.

[0126] Batch processing function:

[0127] To further improve the efficiency of inference, the data preprocessing module supports batch image processing. For large-scale inference tasks, the module can process multiple image samples at a time and automatically perform batch normalization, precision conversion and other operations. This batch processing capability not only improves the inference speed, but also enhances the stability of the model under high-load environments, ensuring that the system can continuously output high-quality inference results in complex scenarios.

[0128] Through the above multiple optimization designs, the data preprocessing module effectively reduces resource consumption during the inference stage and significantly improves the inference speed and processing capabilities. These designs enable the system to quickly and accurately process images with moiré patterns and provide high-quality input data for the de-moiré network, thus showing high practicality in real-time application scenarios. In particular, the module demonstrates outstanding performance advantages in tasks that require high response speed and image processing accuracy.

[0129] Coarse-grained network module

[0130] The present invention proposes a new image de-moiré solution, the core of which is a multi-stage, multi-module de-moiré network architecture. First, a coarse-grained network module is used to perform preliminary processing on the significant moiré in the image. This module removes the more obvious large-area moiré noise in the image in a fast and effective way. The coarse-grained network module is designed to significantly reduce the visual impact of moiré, provide better input conditions for subsequent more sophisticated de-moiré processes, and reduce the complexity of subsequent calculations. The design of this module fully considers the multi-scale and multi-spectral characteristics of moiré, as well as how to achieve fast and efficient removal in complex visual scenes. The various sub-modules of the network module and their functional implementation will be described in detail below.

[0131] Multi-scale feature extraction network

[0132] The generation of moiré is a complex physical phenomenon, which involves the frequency interference between the image sensor and the fine texture. Therefore, the key to removing moiré lies in how to effectively deal with the interference characteristics of different frequencies. To this end, the present invention designs a multi-scale feature extraction network to deal with the multi-frequency characteristics of moiré by constructing a feature pyramid.

[0133] The network is based on the deep learning framework of convolutional neural networks (CNNs), and specifically uses the adaptive multi-scale moiré removal network (AMSDM) that I proposed earlier. AMSDM extracts image features at different resolutions by downsampling at multiple scales. At each scale, the network is able to independently capture the specific frequency characteristics of moiré. By fusing these features at different scales, the network can fully cover the wide spectrum characteristics of moiré, from low frequency to high frequency, ensuring that no matter how the frequency of moiré changes, it can be effectively removed.

[0134] To further enhance the de-moiré effect, the AMSDM network introduces a key component - the adaptive multi-spectral coding module. This module helps the network automatically identify and adapt to the frequency pattern of moiré by dynamically encoding the frequency features in the image, thereby achieving more intelligent de-moiré processing. Specifically, this module can adaptively adjust the statistical characteristics of the image (such as mean, variance, etc.) to flexibly deal with moiré problems of different frequencies in different scenes and texture types. Especially in dynamic scenes, such as videos or images with moving objects, AMSDM exhibits excellent de-moiré capabilities and can handle moiré interference caused by texture changes in real time.

[0135] In addition to the multi-scale network design, AMSDM also includes a dedicated de-moiré module that combines a variety of advanced neural network technologies to further improve the de-moiré effect. The specific implementation details are as follows:

[0136] Improved residual block (CNB): Compared with the traditional residual block, the CNB module introduces a stronger feature transfer mechanism, which can retain more detailed information in the deep layer of the network and prevent information loss. The CNB module effectively alleviates the gradient vanishing problem in the deep network through skip connection, ensuring that the model parameters can be stably optimized during the training process.

[0137] Large Kernel Attention Module (LKA): This module uses large kernel convolution and self-attention mechanism to capture long-range dependencies in images over a large range. The characteristics of moiré often span a large spatial area, so the LKA module can help the network better understand the global information in the image, thereby more effectively removing wide-area moiré.

[0138] Multilayer Perceptron Module (MLP): The MLP module is mainly responsible for further nonlinear transformation of the features output by the network to enhance the network's expressiveness. Through the connection of multiple layers of neurons, MLP can model complex feature spaces, especially showing strong feature extraction capabilities in high-dimensional spaces.

[0139] In order to ensure the effect on high-resolution images, the network adopts a progressive upsampling strategy. This strategy avoids the problem of high-frequency information loss caused by one-time upsampling by gradually increasing the image resolution. Through the staged image restoration process, the network can effectively retain the details in the image and ensure that the output image is clear and natural.

[0140] Encoder Network

[0141] In order to better solve the complex feature extraction and reconstruction problems in the process of image moiré removal, the present invention adopts an encoder network based on variational autoencoder (VAE) as one of the core components. The introduction of the encoder network not only improves the model's ability to represent image features, but also greatly reduces the model's demand for hardware resources during training and inference, optimizing the efficiency and scalability of the entire system.

[0142] In traditional image de-moiré tasks, the network usually needs to process high-dimensional image data, and the processing of such high-dimensional data often brings a huge computational burden. Especially in high-definition image processing tasks, how to improve computational efficiency without sacrificing image quality has become an urgent problem to be solved in model design. VAE, as a classic generative model, performs well in image compression and reconstruction tasks, and can effectively reduce computational overhead by mapping images to a low-dimensional latent space (Latent Space). Therefore, the present invention introduces a VAE encoder network to cope with the challenges of large-scale image data processing that frequently occur in de-moiré tasks. The specific implementation details are as follows:

[0143] Encoder network design and implementation

[0144] The core idea of ​​VAE is to map the input image to a latent space vector through an encoder, which represents the main information or key features in the image. The dimension of the latent space is much smaller than that of the original image, so it can significantly compress the data. In specific implementation, the encoder network uses a multi-layer convolutional neural network (CNN) to extract and compress the image layer by layer. In this process, the network learns the important patterns in the image, filters out redundant information, and converts the key information of the image into a low-dimensional vector representation.

[0145] The encoder designed in the present invention is based on the architecture of variational autoencoder (VAE). Unlike traditional autoencoders, VAE introduces the concept of probability distribution. By learning the probability distribution of input data (usually assumed to be Gaussian distribution), more diverse and more generalizable potential representations can be generated. This design is particularly important in the task of moiré removal because the formation and expression of moiré are highly random and diverse. Through the potential space of VAE, the encoder can better capture the variability and complexity of moiré images, providing a richer and more accurate representation for the subsequent moiré removal process.

[0146] In practice, the encoder is designed with a multi-layer convolutional network, where each layer of convolution gradually extracts different levels of features of the image through the convolution kernel. These features cover multi-dimensional information in the image, such as edges, textures, and colors, from local details to global structures. After each layer of convolution, batch normalization and activation functions (such as ReLU) are used to further optimize the convergence speed and feature extraction capabilities of the model. Finally, the high-dimensional feature map after multiple layers of convolution is converted into a low-dimensional vector in the latent space.

[0147] Pre-training and freezing mechanism

[0148] The present invention adopts the strategy of pre-training the encoder, that is, when using the de-moiré model, the parameters of the encoder part are trained in advance on a large-scale image dataset. During the training phase of the de-moiré task, the parameters of the encoder are frozen, that is, they will not be updated with the training process of the de-moiré network. The advantage of this is that the encoder already has a powerful feature extraction capability and can efficiently compress the input image into the latent space, ensuring that the subsequent diffusion model is executed in a smaller computing space, thereby significantly reducing the consumption of computing resources.

[0149] Another advantage of freezing the encoder parameters is that it can reduce noise interference caused by random gradient changes during training. The stability of the encoder allows the de-moiré network to focus on learning how to remove moiré without being affected by the uncertainty of the image compression part. In addition, the pre-trained encoder has been trained on a large number of images from different scenes and has good generalization ability, which also provides better adaptability for the de-moiré model of the present invention and can be efficiently processed in a variety of complex scenes.

[0150] Diffusion and Moiré Removal in Latent Space

[0151] Through the VAE encoder, the present invention realizes the conversion from high-dimensional image space to low-dimensional latent space. Then, all de-moiré and diffusion operations are performed in this compact latent space. Compared with processing in the original image pixel space, the operation of the latent space greatly reduces the complexity of calculation, especially in the processing of high-frequency information of the image. The diffusion and denoising process of the de-moiré network in the latent space can be understood as smoothing the noise and irregular texture in the latent vector, thereby ensuring that the reconstructed image is clearer and more natural.

[0152] After the demoiré process is completed, the demoiré data in the latent space needs to be reconverted into an image through a decoder. The present invention uses a pre-trained decoder corresponding to the encoder to remap the latent vector back to the pixel space to generate the final demoiré image. Unlike traditional demoiré methods, the decoding process of VAE has a powerful image reconstruction capability, which can retain the detailed information of the image while ensuring the naturalness and consistency of the image. The pre-trained decoder has rich visual experience through learning from a large amount of image data, and can perform well in generating high-quality images.

[0153] The combination of the encoder network and the diffusion model proposed in the present invention significantly reduces the computational resource consumption in the image de-moiré task. Since the encoder can compress large-scale image data into low-dimensional latent vectors, the core calculations of the network are mostly completed in the latent space, which greatly reduces the amount of calculation. Compared with traditional de-moiré methods, the solution of the present invention has obvious advantages in computational efficiency and processing speed, and is particularly suitable for large-scale image processing tasks or resource-constrained scenarios.

[0154] Diffusion Network Based on Wavelet Sampling

[0155] The existence of moiré is due to the superposition of high-frequency interference, which is especially obvious when fine textures or grid patterns appear in the image. This interference will produce obvious stripes or ripples visually, which will have an adverse effect on the overall quality of the image. Traditional methods for removing moiré usually reduce interference through spatial filtering or frequency domain transformation, but these methods often find it difficult to retain sufficient image details while removing moiré. Therefore, the present invention introduces a diffusion network based on wavelet sampling, and provides a new image moiré removal scheme by combining wavelet transform and discrete cosine transform (DCT), which aims to balance the relationship between moiré removal and image detail retention. The schematic diagram of the intermediate steps of the diffusion process is shown in Figure 3 shown.

[0156] Denoising Diffusion Model

[0157] In recent years, diffusion models, as a new generative model framework, have gradually gained wide attention and application in the field of computer vision. It learns the distribution characteristics of data by simulating the process of gradually adding noise to the data, thereby effectively generating high-quality images in the denoising process. In the task of moiré removal, the diffusion model shows great potential. The present invention designs a conditional network based on the diffusion process, and combines wavelet and DCT transforms for image enhancement and moiré removal.

[0158] The core idea of ​​the diffusion model is to gradually transform the image into pure noise through multi-step noise addition, and then gradually restore the true distribution of the image in the reverse denoising process. The entire diffusion process is controlled by a low-pass filter network to ensure that high-frequency moiré artifacts are removed while retaining low-frequency semantic information and structural features. The specific implementation details are as follows:

[0159] Diffusion process

[0160] Suppose the image after preliminary moiré removal is y 0 , during the diffusion process, the image is converted into pure noise by gradually adding noise to it. This process can be expressed as:

[0161]

[0162] Among them, y t is the image of step t, ∈ t is the noise that follows a normal distribution, During the diffusion process, the intensity of the noise is determined by a predefined planning factor α t Control, as t increases, the image is gradually covered by noise. When t = T, the image is completely transformed into noise, that is,

[0163] Next, by derivation, we can conclude that at any time step y in the diffusion process t With the original image y 0 The relationship between:

[0164]

[0165] in, is the cumulative product of the planning factors of the previous t steps, that is Through this formula, the noise can be combined with the original image to generate the intermediate image y in the diffusion process t , providing a basis for the subsequent denoising and restoration process. Through this gradual noise addition method, the diffusion model can learn the data distribution characteristics in the gradual addition of noise.

[0166] Reverse diffusion process

[0167] The core of the diffusion model lies in its inverse diffusion process, which aims to gradually restore the image completely covered by noise to the original data distribution through recursive denoising steps. The inverse diffusion process is achieved through a learnable denoising model ∈ θ (x t ,t) to achieve, the formula is as follows:

[0168]

[0169] in, represents the reverse diffusion process, ∈ θ (x t ,t) is the denoising function learned during the training process and is used to denoise the image at each time step t. By recursively applying this process, the noise in the image is gradually reduced, and finally a high-quality, moiré-free image is restored. This inverse diffusion process allows the model to generate more natural images and effectively remove moiré artifacts, especially in detail processing. Since the image after the initial moiré removal may have the problem of over-smoothing, the image is further optimized through the generation prior of the diffusion model to improve the detail processing ability of the moiré removal and ultimately generate a higher quality image.

[0170] Conditional diffusion sampling and low-pass filter design

[0171] In order to ensure that the diffusion process does not lose important semantic information while removing moiré, the concept of conditional diffusion is introduced. By designing the conditional function, the denoising and restoration of the image can be better guided in the inverse diffusion process. The conditional function can be written as:

[0172]

[0173] Among them, LPF represents the low-pass filter function, which is designed to separate the high-frequency information from the low-frequency information in the image to control the information transfer in the diffusion process. If too much high-frequency information is transferred, the generated image may become blurred; on the contrary, if insufficient low-frequency information is transferred, the semantic details of the image will not be fully preserved. Therefore, designing a suitable low-pass filter is crucial to the performance of the conditional diffusion process.

[0174] Low-pass filter combining wavelet and DCT

[0175] In order to better realize the transmission of low-frequency information and the removal of high-frequency artifacts, the present invention designs a low-pass filter network based on wavelet and discrete cosine transform (DCT). Wavelet transform can decompose the image into sub-bands of different scales and extract the edge and detail information therein, while DCT further removes unnecessary high-frequency components through frequency domain compression. The advantage of this combination is that wavelet transform provides the ability of multi-resolution analysis, so that local features in the image can be processed more carefully, while DCT can improve the efficiency of frequency domain representation, thereby retaining more image details while removing moiré. The low-pass filter network based on wavelet and DCT can effectively enhance the performance of the diffusion model in the task of removing moiré and generate clearer and more natural images.

[0176] Coarse-to-fine multi-scale processing

[0177] The low-pass filter network in the present invention adopts a coarse-to-fine multi-scale processing strategy, which ensures that the image details at each scale can be fully processed by gradually decomposing the high-frequency and low-frequency information in the image. In the process of removing moiré, the model first decomposes the image at multiple scales through wavelet transform, and then applies the diffusion process at each scale to process high-frequency artifacts. Finally, the features in the frequency domain are optimized by DCT to generate a high-quality image without moiré. This multi-scale processing method allows the model to remove moiré at both the global and local levels, which not only improves the effect of removing moiré, but also better preserves the semantic structure and visual details of the image.

[0178] By introducing a diffusion model based on wavelet and DCT transform, the present invention provides a new image de-moiré scheme, which can not only effectively remove moiré artifacts in images, but also retain the semantic information and details of the image while removing moiré, thereby improving the overall quality of the image.

[0179] Wavelet-based sampling method

[0180] In order to solve the distribution problem of image moiré artifacts in different frequency domains, the present invention proposes a multi-scale sampling method based on wavelet transform. As a common image artifact, moiré is mainly concentrated in high-frequency information, causing interference and distortion in detail areas. Therefore, it is difficult to completely eliminate moiré by traditional moiré removal methods alone. Based on this, the present invention utilizes the multi-scale frequency decomposition characteristics of wavelet transform to decompose the image into different frequency components to accurately identify and eliminate moiré while maintaining the overall structure and detail information of the image. The specific implementation details are as follows:

[0181] The Principle and Implementation of Wavelet Transform

[0182] Wavelet transform is a signal processing tool that can provide both time domain and frequency domain information. Unlike Fourier transform, it can decompose images at multiple scales and resolutions, and is suitable for processing image signals with local features. Specifically, Haar wavelet is the wavelet transform method used in the present invention, which is characterized by high computational efficiency, the ability to quickly decompose and reconstruct images, and is particularly suitable for real-time image processing tasks.

[0183] In the present invention, the input image is first decomposed by Haar wavelet to obtain low-frequency components and high-frequency components. The low-frequency components contain the global structural information of the image, while the high-frequency components contain the detailed features and texture information in the image. Through this decomposition, moiré artifacts can be better identified and processed in the high-frequency components.

[0184] The specific implementation steps are as follows:

[0185] Decomposition of input image: Assume that the input image is I, which is decomposed into low-frequency components L by Haar wavelet transform I and high frequency component H I , the formula is as follows:

[0186]

[0187] Among them, L I Represents the low-frequency components of the image, mainly including the overall outline and large-scale structure of the image; H I Represents the high-frequency components of the image, including the texture and detail information of the image.

[0188] Decomposition of diffusion model features: The intermediate feature image F generated by the diffusion model is also decomposed by Haar wavelet to obtain its low-frequency component L F and high frequency component H F , the formula is as follows:

[0189]

[0190] This allows the original image and the features generated by the diffusion model to be accurately distinguished and processed in the frequency domain.

[0191] Low-frequency replacement strategy

[0192] In order to improve the removal effect of moiré artifacts, the present invention designs a strategy based on low-frequency replacement. Specifically, the low-frequency component L of the input image is replaced by I Replaced by the low-frequency component L generated by the diffusion model F To ensure that the global structure of the image can be refined and optimized through the diffusion model, thus enhancing the effect of removing moiré

[0193] The implementation process of this strategy is as follows:

[0194] Low-frequency replacement: By replacing the image L I The low-frequency component of is replaced by the low-frequency component L of the diffusion characteristic F , which can achieve global structural optimization in the process of removing moiré. The formula is as follows:

[0195] (L I ,H I )→(L F ,H I )

[0196] This replacement ensures that the global information of the image is enhanced while the high-frequency part still retains the image details.

[0197] Reconstruct the image: After completing the low-frequency replacement, recombining the low-frequency and high-frequency components of the image, and reconstructing the image through inverse wavelet transform to obtain the image after de-moiré , the formula is as follows:

[0198]

[0199] In this way, the details and global information of the image are effectively preserved and optimized.

[0200] Combination of wavelet and diffusion models

[0201] The present invention designs a wavelet-based conditional diffusion sampling method, which combines wavelet transform with the step-by-step denoising process of the diffusion model, and can more accurately control the denoising process in different frequency domains. The diffusion model is a generative model based on step-by-step noise addition and denoising. In the present invention, the diffusion model is used to generate moiré-free image features, while the wavelet transform is used to separate and replace frequency components in the image, thereby enhancing the denoising effect.

[0202] The specific diffusion sampling process is as follows:

[0203] Stepwise noise addition: During the diffusion process, the low-frequency and high-frequency components of the image are processed separately. Suppose that in the diffusion sampling of the tth step, the input image x t The frequency component of is calculated by the following formula:

[0204]

[0205] Among them, ε t is the noise added at time step t, the low-frequency component L y It is a feature generated by the diffusion model, which ensures that the image after de-moiré remains globally consistent.

[0206] Stepwise denoising: In the reverse process of diffusion, the generated Gaussian noise is transformed into a high-quality image close to the natural image distribution through stepwise denoising. The final generated image not only removes the moiré artifacts, but also maintains the natural visual effect.

[0207] Multi-scale low-frequency filter based on wavelet

[0208] To further improve the de-moiré effect, the present invention proposes a wavelet-based multi-scale low-frequency filter. The filter combines wavelet transform and discrete cosine transform (DCT) to refine image features in the frequency domain. Wavelet transform is used to capture the global structure and high-frequency details of an image, while DCT can more effectively compress and represent the frequency components of an image.

[0209] Specifically, the low-frequency filter designed by the present invention can adaptively adjust the filtering strength to ensure that the key details and semantic information of the image are not lost while removing the moiré artifacts. This adjustable multi-scale filtering mechanism can adapt to the characteristics of different images and provide a more flexible de-moiré effect.

[0210] Decoder Network

[0211] In the process of removing image moiré, even if the diffusion model and wavelet sampling network have significantly removed most of the artifacts, the generated image often still has insufficient details and blur. This phenomenon is mainly due to the fact that the initially generated image features lose some high-frequency information during the reconstruction process, resulting in the inability to restore the detail quality that matches the real clear image. Therefore, the present invention proposes a decoder network for the refinement and reconstruction of the final image to obtain a high-quality de-moiré image.

[0212] The decoder network is designed to further refine the demoiréd image through multi-level feature fusion and reconstruction. Its main task is to restore high-frequency details that may be lost during the demoiré process while maintaining the structural consistency and naturalness of the image. The decoder network not only needs to comprehensively process multi-scale information, but also ensure that the generated image features have good perceptual quality and meet the requirements of the human visual system. The specific implementation details are as follows:

[0213] Structure of the decoder network

[0214] The decoder network structure proposed in the present invention includes the following key modules:

[0215] Multi-scale feature input layer: This module is responsible for receiving multi-resolution feature inputs processed by the aforementioned diffusion model and wavelet sampling network. These feature inputs retain the global structure and local detail information of the image, and will generate high-quality images after further processing by the decoder.

[0216] Gradually refined convolutional layers: In each decoding layer, stacked convolution operations are used to refine features and extract more precise image details. Each convolutional layer is used with ReLU or Leaky ReLU activation functions to ensure that the model maintains the edge and texture information of the image when extracting nonlinear features.

[0217] Skip connection mechanism: In order to solve the problem of feature information loss and degradation in deep networks, the decoder network uses skip connections. This mechanism directly connects the feature map in the initial encoding process with the feature map in the decoding process, ensuring that image information at different levels can be transmitted losslessly, especially the high-frequency details at low levels are better preserved.

[0218] Feature fusion and multi-layer aggregation: In the final stage of the decoder network, a multi-level feature aggregation module is designed, which fuses multi-scale features to keep the decoded image consistent at different scales. Multiple convolution operations and feature weighting strategies are used in the aggregation process to balance high-resolution image details and low-resolution global information.

[0219] Scaling addition and feature reconstruction

[0220] Based on multi-resolution feature aggregation, the present invention also introduces a scaled addition mechanism (ScaledAddition) for processing and fusing features from different resolutions. The core idea of ​​this mechanism is to dynamically adjust the weight of each feature map during feature aggregation to ensure that low-resolution features can provide sufficient global information, while high-resolution features can enhance image details. Specifically, the process of scaled addition is as follows:

[0221]

[0222] Among them, F final is the high-quality image feature finally generated, F i is the multi-scale feature of the i-th layer, is the scaling weight of the features of this layer. Through this scaling addition strategy, the network can balance the presentation of global information and local details, so that the decoded image has both high-frequency details and remains consistent overall.

[0223] Training strategy and loss function

[0224] In order to ensure that the decoder network has sufficient robustness and visual quality when generating clear de-moiré images, the present invention uses L1-based pixel loss to ensure that the decoded image has the smallest difference from the real clear image at the pixel level. This loss function can effectively ensure the accuracy of the de-moiré image, and its calculation formula is as follows:

[0225] L1=|I gt -I gen | 1

[0226] Among them, L1 represents L1 pixel loss, I gt Represents a clear original image, I gen Represents the resulting de-moiréd image.

[0227] In addition, since the pixel loss gives the same weight to all pixels in the image, this usually leads to more attention to the low-frequency part. Because moiré patterns usually have multi-bands, a scaling wavelet loss is proposed here to pay more attention to multi-band information. The calculation formula is as follows:

[0228] Lwav=∑|W Fb (I gt )-W Fb (I gen )| 1

[0229] Where Lwav represents the scaled wavelet loss proposed by the present invention, WFb represents the scaled wavelet function.

[0230] In the specific training process, the network weights need to be initialized first to ensure that the model starts learning from a random or preset state. Then, the preprocessed training set D1 is used to train Figure 2 The network structure shown in is used for training. During the training process, each batch of data will update the network weight parameters through the process of forward propagation and back propagation. Through the role of the optimizer, the model will gradually adjust the weights to reduce the value of the loss function, thereby improving the prediction accuracy and generalization ability of the model. The optimization goal is to gradually minimize the loss function, thereby improving the performance of the model on the training set and preparing for the subsequent verification and testing phases.

[0231] Reasoning results

[0232] When an image X with moiré is input, the image can be reconstructed using the trained model to obtain a high-definition moiré-free image X. clear The effect of the present invention can be verified by the following experiments:

[0233] First, we perform image de-moiré processing on the test set in the TIP moiré dataset to obtain high-definition images after de-moiré. Then, we compare these processed images with the original high-definition images without moiré, and evaluate the effect of the model by calculating the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM). The specific formula is as follows:

[0234]

[0235] Where n = 8, and

[0236]

[0237] The de-moiré effect of the present invention is compared with other existing de-moiré methods on the same TIP dataset. The experimental results are summarized in Table 1. As can be seen from Table 1, the model of the present invention has achieved satisfactory results in both peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM), outperforming other existing methods, verifying the superior performance of the present invention in image de-moiré tasks.

[0238] Table 1 Performance comparison of various methods on the TIP dataset

[0239]

[0240] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any technician familiar with the profession, without departing from the scope of the technical solution of the present invention, according to the technical essence of the present invention, any simple modification, equivalent replacement and improvement made to the above embodiment still falls within the protection scope of the technical solution of the present invention.

Claims

1. An image de-moiré method based on perception-guided and depth-generated priors, characterized in that: The following steps are involved: Provide original training data sets and enrich training samples through data enhancement strategies; Use a coarse-grained network to preliminarily process the moiré in the image to reduce the visual impact of the moiré; A diffusion network based on wavelet sampling is used to balance the relationship between moiré removal and image detail preservation by combining wavelet transform and discrete cosine transform. The decoder network is used for image refinement and reconstruction to obtain the final de-moiréd image.

2. The image de-moiré method based on perception guidance and depth generation prior according to claim 1, characterized in that: The use of a coarse-grained network to perform preliminary processing on the moiré patterns in the image includes: a deep learning framework based on a convolutional neural network, using an adaptive multi-scale de-moiré network, performing downsampling operations at multiple scales, extracting image features at different resolutions, independently capturing specific frequency features of the moiré patterns at each scale, and fusing specific frequency features at different scales.

3. The image de-moiré method based on perception guidance and depth generation prior according to claim 2, characterized in that: In the adaptive multi-scale de-moiré network, adaptive multi-spectral coding technology is introduced to adaptively adjust the statistical characteristics of the image, and the frequency characteristics in the dynamic coding image are used to help the network automatically identify and adapt to the frequency pattern of the moiré; in the adaptive multi-scale de-moiré network, a variety of neural networks are used to improve the de-moiré effect; the use of a coarse-grained network to perform preliminary processing on the moiré in the image also includes introducing a VAE encoder network to map the image to a low-dimensional latent space, and pre-training the encoder and freezing the encoder parameters.

4. The image de-moiré method based on perception guidance and depth generation prior according to claim 1, characterized in that: The diffusion network based on wavelet sampling is used to balance the relationship between moiré removal and image detail retention by combining wavelet transform and discrete cosine transform. Specifically, the image is decomposed into different frequency components by utilizing the multi-scale frequency decomposition characteristics of wavelet transform to accurately identify and eliminate moiré while maintaining the overall structure and detail information of the image. Among them, the diffusion process in the diffusion network is as follows: Assume that the image after preliminary moiré removal is y0. During the diffusion process, the image is converted into pure noise by gradually adding noise to it. This process is expressed as: Among them, y t is the image of step t, ∈ t is the noise that follows a normal distribution, During the diffusion process, the intensity of the noise is determined by a predefined planning factor α t Control, as t increases, the image is gradually covered by noise; when t = T, the image is completely transformed into noise, By deriving the diffusion process, any time step y t The relationship between the original image y0: in, is the cumulative product of the planning factors of the previous t steps, that is The noise is combined with the original image through this formula to generate the intermediate image y in the diffusion process t , providing a basis for the subsequent denoising and restoration process; The reverse diffusion process in a diffusion network is as follows: The inverse diffusion process gradually restores the image completely covered by noise to the original data distribution through recursive denoising steps; the inverse diffusion process uses a learnable denoising model ∈ θ (x t ,t) to achieve, the formula is as follows: in, represents the reverse diffusion process, ∈ θ (x t ,t) is the denoising function learned during the training process, which is used to denoise the image at each time step t; through recursive application, the noise in the image is gradually reduced, generating a more natural image and removing moiré artifacts; Introducing conditional diffusion function in diffusion network To guide the denoising and restoration of the image in the inverse diffusion process, the conditional diffusion function The formula is as follows: Among them, LPF represents a low-pass filter function, which is intended to separate high-frequency information from low-frequency information in the image to control the information transfer in the diffusion process.

5. The image de-moiré method based on perception guidance and depth generation prior according to claim 4, characterized in that: The process of wavelet transform includes: Decomposition of input image: Assume that the input image is I, which is decomposed into low-frequency components L by Haar wavelet transform I and high frequency component H I , the formula is as follows: Among them, L I Represents the low-frequency components of the image, mainly including the overall outline and large-scale structure of the image; H I Represents the high-frequency components of the image, including the texture and detail information of the image; Decomposition of diffusion model features: The intermediate feature image F generated by the diffusion model is also decomposed by Haar wavelet to obtain its low-frequency component L F and high frequency component H F , the formula is as follows: Using a low-frequency replacement strategy, the low-frequency component L of the input image is replaced I Replaced by the low-frequency component L generated by the diffusion model F , to ensure that the global structure of the image can be refined and optimized by the diffusion model, the process is as follows: Low-frequency replacement: By replacing the image L I The low-frequency component of is replaced by the low-frequency component L of the diffusion characteristic F , to achieve global structural optimization in the process of removing moiré patterns, the formula is as follows: (L I ,H I )→(L F ,H I ) Reconstruct the image: After completing the low-frequency replacement, recombining the low-frequency and high-frequency components of the image, and reconstructing the image through inverse wavelet transform to obtain the image after de-moiré , the formula is as follows:

6. The image de-moiré method based on perception guidance and depth generation prior according to claim 1, characterized in that: The decoder network is used for image refinement and reconstruction to obtain the final de-moiré image, specifically: restore the details that may be lost in the de-moiré process, and maintain the structural consistency and naturalness of the image; the decoder network structure includes: a multi-scale feature input layer, a gradually refined convolutional layer, a skip connection mechanism, and feature fusion and multi-layer aggregation.

7. An image moiré removal system based on perception-guided and depth-generated priors, characterized in that: include: The data preprocessing module is used to provide the original training data set and enrich the training samples through data enhancement strategies; A coarse-grained network module is used to perform preliminary processing on the moiré patterns in the image using a coarse-grained network to reduce the visual impact of the moiré patterns. A diffusion network module based on wavelet sampling is used to balance the relationship between moiré removal and image detail preservation by using a diffusion network based on wavelet sampling and combining wavelet transform and discrete cosine transform; A decoder network module is used to use the decoder network for image refinement and reconstruction to obtain a final de-moiré image; The system is used to perform the image demoiré method based on perception guidance and depth generation prior as described in any one of claims 1 to 6.

8. The image de-moiré system based on perception guidance and depth generation prior according to claim 7, characterized in that: The coarse-grained network module is based on a deep learning framework of a convolutional neural network and uses an adaptive multi-scale de-moiré network module, wherein the adaptive multi-scale de-moiré network module includes an adaptive multi-spectrum encoding module and a de-moiré module; The adaptive multi-spectral encoding module is used to adaptively adjust the statistical characteristics of the image, and helps the network automatically identify and adapt to the frequency pattern of the moiré by dynamically encoding the frequency characteristics in the image; The moiré module includes: an improved residual block for introducing a feature transfer mechanism to alleviate the gradient vanishing problem in a deep network through skip connections; The large-core attention module is used to capture long-distance dependencies in images over a large range using large-core convolution and self-attention mechanisms. The multi-layer perceptron module is used to model complex feature spaces through the connection of multiple layers of neurons and perform nonlinear transformations on the features of the network output.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the image de-moiré method based on perception guidance and depth generation prior as described in any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the image de-moiré method based on perception guidance and depth generation prior according to any one of claims 1 to 6.

Images