Image denoising method, system and device based on multi-scale complementary learning and medium
By decomposing the image into sub-images at different spatial frequencies, denoising the noise process and fusing the detailed features, the problem of insufficient details during image denoising in the prior art is solved, and better image detail information retention and noise removal effects are achieved.
Patent Information
- Application Number
- CN202510110367.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-23
- Publication Date
- 2025-05-13
AI Technical Summary
When processing complex images, the prior art tends to remove noise and detail information together, resulting in insufficient image details after denoising.
By decomposing the image into sub-images at different spatial frequencies, denoising process is performed separately, the image scale detail characteristics of the image are directly learned, and the noise scale detail characteristics are indirectly learned by learning the noise distribution, the learning results of the two branches are fused, and the fused denoising sub-image details characteristics are obtained, and the fused denoising sub-image details are added to the base layer of the original image to obtain the final denoising image.
Effectively retain the detailed information of the image while removing noise, improving the retention of the detailed information of the image and adapting to the processing needs of complex images.
Smart Images
Figure CN119991490A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to an image denoising method, system, device and medium based on multi-scale complementary learning. Background Art
[0002] Image processing is an important research area in computer vision. Image color distortion, edge blur, noise, etc. will have unpredictable effects on image information extraction. Image recognition effects may be seriously affected, resulting in image recognition errors, classification failures, and other issues. During image acquisition and transmission, due to the influence of image devices or external factors, the acquired images often contain certain noise pollution. Noisy images will directly affect the performance of the visual system in image processing and understanding.
[0003] In image processing, image denoising is an important preprocessing step. The purpose of image denoising is to restore a clean image from a distorted image while retaining the details of the original image. Image denoising has become an indispensable step in the field of computer vision processing and is of great significance for improving image quality and subsequent processing effects.
[0004] Traditional image denoising methods, such as filter denoising and deep learning denoising, can remove noise to a certain extent, but they often do not work well when processing complex images. They tend to remove noise and detail information together in complex information of the image, resulting in insufficient details in the denoised image. Summary of the invention
[0005] The purpose of the present invention is to address the deficiencies of the above-mentioned prior art and provide an image denoising method, system, device and medium based on multi-scale complementary learning, so as to solve the problem that the prior art can remove noise to a certain extent, but the effect is often poor when processing complex images. It is easy to remove noise and detail information together in the complex information of the image, resulting in insufficient details in the denoised image.
[0006] The present invention specifically provides the following technical solutions: An image denoising method based on multi-scale complementary learning comprises the following steps: Obtaining an original image to be denoised, and decomposing the original image into sub-images at different spatial frequencies; wherein the sub-images with spatial frequencies lower than a threshold represent the base layer of the original image; De-noising is performed on each sub-image respectively, and the image scale detail features of the image are directly learned from the denoised sub-image, and the noise scale detail features of the image are indirectly learned by learning the noise distribution of the denoised sub-image. The image scale detail features and the noise scale detail features are fused to obtain the fused denoised sub-image detail features; The fused denoised sub-image detail features are added to the base layer in the original image to obtain the final denoised image.
[0007] Preferably, the denoising process is performed on each sub-image separately, including: performing denoising process on each sub-image separately using one or more denoising processes selected from filter-based denoising, deep learning-based denoising and statistical model-based denoising.
[0008] Preferably, the denoising based on the statistical model includes: Convert the sub-image to an image matrix; The image matrix is decomposed using the low-rank matrix approximation method, and the basic features are retained to remove the singular values corresponding to the noise; The image matrix is reconstructed using the retained base layer features, and the reconstructed image matrix is converted into a sub-image.
[0009] Preferably, the step of adding the fused denoised sub-image detail features to the base layer in the original image to obtain the final denoised image comprises the following steps: Obtain the added image; Inputting the added image into a convolutional neural network (CNN) to extract abstract features of the added image layer by layer; the abstract features include features that gradually transition from image edges and texture information to semantic information; Based on the abstract features, the image is gradually restored through deconvolution or upsampling operations to obtain the final denoised image.
[0010] Preferably, when the original image is decomposed into sub-images at different spatial frequencies, a method of decomposing the original image data includes Laplace pyramid decomposition, wavelet transform decomposition and Fourier transform decomposition methods.
[0011] The present invention provides an image denoising system based on multi-scale complementary learning, comprising: An acquisition module is used to obtain the original image data to be denoised and decompose the original image into sub-images at different spatial frequencies; wherein the sub-image with a spatial frequency lower than a threshold represents the base layer of the original image; A denoising module is used to perform denoising on each sub-image, directly learn the image scale detail features of the image from the denoised sub-image, and indirectly learn the noise scale detail features of the image by learning the noise distribution of the denoised sub-image, and fuse the image scale detail features and the noise scale detail features to obtain the fused denoised sub-image detail features; The reconstruction module is used to add the fused denoised sub-image detail features to the base layer in the original image to obtain the final denoised image.
[0012] The present invention provides a computer device, comprising a memory and a processor, wherein a program is stored in the memory, and when the program is executed by the processor, the processor executes the steps of the above-mentioned image denoising method based on multi-scale complementary learning.
[0013] The present invention provides a storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps of the above-mentioned image denoising method based on multi-scale complementary learning are implemented.
[0014] Compared with the prior art, the present invention has the following significant advantages: The present invention decomposes an image into different spatial frequencies, distinguishes the base layer and details of the image by the spatial frequency, and then uses two branches to directly learn the image detail features, and indirectly learns the sub-image detail features by the noise distribution of the sub-image, and fuses the detail features learned by the two branches, which helps to remove noise while retaining the useful information in the sub-image, and adds the fused denoised sub-image detail features to the base layer of the original image, while maintaining the main features of the original image, the noise of the detail part is greatly removed, thereby improving the retention degree of the image detail information and being able to better meet the processing requirements of complex images. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 The figure is a flow chart of an image denoising method based on multi-scale complementary learning according to the present invention. DETAILED DESCRIPTION
[0016] The following is a clear and complete description of the technical solutions of the embodiments of the present invention in conjunction with the drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0017] like Figure 1 As shown, an image denoising method based on multi-scale complementary learning of the present invention specifically includes the following steps: Step S1: Acquire the original image data to be denoised, and decompose the original image into sub-images at different spatial frequencies; wherein the sub-images with spatial frequencies lower than a threshold represent the base layer of the original image, and the sub-images with spatial frequencies higher than the threshold represent the detail layer.
[0018] Methods for decomposing raw image data include Laplace pyramid decomposition, wavelet transform decomposition and Fourier transform decomposition methods.
[0019] Step S2: De-noising is performed on each sub-image respectively, and the image scale detail features of the image are directly learned from the denoised sub-image, and the noise scale detail features of the image are indirectly learned by learning the noise distribution of the denoised sub-image, and the image scale detail features and the noise scale detail features are fused to obtain the fused denoised sub-image detail features.
[0020] De-noising is performed on each sub-image separately, including: using one or more denoising methods selected from filter-based denoising, deep learning-based denoising, and statistical model-based denoising to perform denoising on each sub-image separately.
[0021] Among them, filter-based denoising is used, including: Use median filtering for image denoising; use adaptive Wiener filtering for image denoising; use Gaussian low-pass filtering for image denoising; use frequency domain or time domain filtering for image denoising.
[0022] Median filter: Select window size: First determine an appropriate filter window size (such as 3x3, 5x5, etc.).
[0023] Sorting and replacement: For each pixel in the sub-image, sort the pixel values in the surrounding window, and then replace the original pixel value with the sorted median.
[0024] Traversal processing: Repeat the above steps for all pixels in the sub-image to complete denoising.
[0025] Adaptive Wiener Filtering: Local variance estimation: Calculate the local variance of each pixel in the sub-image to assess the noise level.
[0026] Adjust filter output: The output of the filter is adjusted according to the local variance. The larger the local variance, the stronger the smoothing effect.
[0027] Apply Filter: Apply the adjusted filter to the sub-image to complete the denoising.
[0028] Gaussian low-pass filtering: Determine Gaussian kernel: Select a Gaussian kernel (a two-dimensional Gaussian function) whose standard deviation determines the smoothness of the filter.
[0029] Frequency or time domain filtering: Frequency domain filtering: The sub-image is converted to the frequency domain (e.g., by Fourier transform), then filtered using a Gaussian kernel, and finally back to the spatial domain by inverse transform.
[0030] Temporal filtering: Apply a Gaussian kernel directly in the spatial domain, traverse the sub-image through the template, and replace the center pixel value with the weighted average within the template.
[0031] Among them, deep learning-based denoising and statistical model-based denoising are used, including: The denoising based on convolutional neural network CNN includes: Preprocess the sub-images. Perform necessary preprocessing on the sub-images, such as format conversion, normalization, etc.
[0032] Construct a CNN model including convolutional layers, pooling layers, fully connected layers, etc. to extract image features and denoise.
[0033] The model is trained through supervised learning using the preprocessed noisy sub-image as input and the clean sub-image as the target output.
[0034] The sub-image to be denoised is input into the trained CNN model to obtain the denoised image.
[0035] Denoising based on Generative Adversarial Network (GAN): Build a GAN model: including the generator (G) and the discriminator (D). The generator is responsible for generating denoised images, and the discriminator is responsible for distinguishing between generated images and real images.
[0036] Adversarial training: By continuously iteratively training the generator and discriminator, the generator can generate more and more realistic denoised images, while the discriminator finds it increasingly difficult to distinguish between real and fake images.
[0037] Apply GAN denoising: Input the noisy sub-image into the trained generator to obtain the denoised image.
[0038] Among them, the denoising based on statistical models uses low-rank matrix approximation, including: Build Image Matrix: Convert the sub-image into an image matrix.
[0039] Low-rank approximation: Use low-rank matrix approximation methods (such as singular value decomposition (SVD)) to decompose the image matrix, retain the basic features (that is, larger singular values), and remove the smaller singular values corresponding to the noise.
[0040] Reconstruct image: Reconstruct the image matrix using the retained base layer features and convert the reconstructed image matrix into a sub-image.
[0041] Graph-based regularization: Build a graph model: treat pixels or image blocks in a sub-image as nodes of a graph, and construct edges and weights based on the similarity between pixels.
[0042] Regularized optimization: The regularization term of the graph is used to construct an optimization problem, and the noise is removed by solving the problem while maintaining the structural information of the image.
[0043] The denoised sub-images are subjected to multi-scale complementary learning to obtain denoised images fused at different scales, including the following steps: Image preprocessing: Preprocess the input image, such as standardization, cropping, etc., for subsequent processing.
[0044] Multi-scale feature extraction: Use filters or convolution kernels of different scales to extract features from images and generate feature maps of multiple scales. These feature maps capture the detailed information of the image at different scales.
[0045] Detail feature learning: directly learn image detail features from images through specific network structures (such as U-shaped network structure, residual network, etc.) to obtain image-scale image detail features.
[0046] Noise feature learning: Another branch network is used to learn the noise distribution of the image, indirectly learn the image detail features, and obtain the noise scale image detail features.
[0047] Feature fusion: The image detail features learned from the two branches are fused by feature concatenation, weighted summation or other feature fusion methods to obtain the final image detail features. This step aims to separate the noise and useful information in the image.
[0048] Advantages of multi-scale features: Multi-scale feature fusion can describe the complexity of the image more comprehensively and accurately. Feature maps of different scales capture the information of the image at different scales, thereby enhancing the model's ability to understand the details and structure of the image.
[0049] Separation of details and noise: By decomposing the image into a low-frequency part (base layer) and a high-frequency part (detail layer), and processing them separately, the noise and useful information in the image can be separated. The base layer usually contains the main structure and low-frequency information of the image, while the detail layer contains the details and high-frequency information (including noise) of the image.
[0050] The effect of complementary learning: The detail feature learning branch directly learns the image detail features, which helps to retain the useful information in the image; while the noise feature learning branch indirectly learns the image detail features by learning the noise distribution, which helps to remove the noise in the image. The learning results of the two branches are complementary, and the outputs of the two branches are fused to obtain an image that retains the details and removes the noise.
[0051] Improved robustness and generalization capabilities: Multi-scale complementary learning enhances the model's adaptability to complex factors such as scale changes, object size, shape complexity, and spatial position relationships by fusing features of different scales. This makes the model more robust and generalizable when processing different types of images.
[0052] Step S3: Add the fused denoised sub-image detail features to the base layer in the original image to obtain the final denoised image.
[0053] Image reconstruction: Obtain the added image, that is, add the fused denoised sub-image detail features to the base layer (or low-frequency part) of the original image to obtain the denoised image. This step completes the image reconstruction, so that the denoised image retains the detail information of the original image and reduces the impact of noise.
[0054] Convolutional Neural Network (CNN): Use convolutional neural network CNN to extract abstract features of images layer by layer, gradually transitioning from low-level image edge and texture information to high-level semantic information.
[0055] Among them, the method description is: CNN can learn the local features of the image and reconstruct the image through these features. In the reconstruction of the fused denoised image, a CNN model can be designed, which takes the denoised image as input, extracts deeper features through multiple convolutional layers, activation layers and possible pooling layers, and finally restores the original size and details of the image through deconvolution layers or upsampling layers based on the abstract features.
[0056] Data processing: During the data processing, CNN extracts abstract features of the image layer by layer, which gradually transition from low-level edge and texture information to high-level semantic information. In the reconstruction stage, these features are used to guide the image restoration process, gradually restoring the details of the image through deconvolution or upsampling operations.
[0057] Generative Adversarial Network (GAN): Method description: GAN consists of two parts: the generator and the discriminator. The generator is responsible for receiving random noise (or fused denoised images) and generating fake images as close to the real image as possible; the discriminator is responsible for distinguishing whether the input image is real or a fake image generated by the generator. Through adversarial training between the two networks, the generator can learn how to generate more realistic and clear images.
[0058] Data processing: In GAN, the fused denoised image can be used as one of the input conditions of the generator and used together with random noise to generate the reconstructed image. The generator optimizes the quality of the generated image by continuously adjusting its parameters so that it can deceive the discriminator. In this process, the processing of data is highly nonlinear and relies on the adversarial effect between the two networks.
[0059] In the generative adversarial network (GAN), the fused denoised image is used as one of the input conditions of the generator, and the reconstructed image is generated together with random noise. The generator continuously adjusts the parameters to optimize the generated image quality.
[0060] Sparse representation and dictionary learning: Method description: Sparse representation represents the image as a linear combination of a set of sparse coefficients, which are selected from a predefined dictionary. In the reconstruction process, a dictionary learning algorithm can be used to learn a suitable dictionary from the fused denoised image, and the image can be reconstructed through a sparse coding algorithm.
[0061] Data processing: In the sparse representation framework, data processing mainly involves the iterative process of sparse coding and dictionary updating. Sparse coding finds the sparse coefficients that best represent the image by solving optimization problems, while dictionary updating optimizes the atoms (i.e., basis vectors) in the dictionary based on the current sparse coefficients to better adapt to the distribution of image data.
[0062] In the sparse representation framework, sparse coding is used to solve the optimization problem to obtain the sparse coefficients that best represent the image, and dictionary update is used to optimize the atoms in the dictionary with the current sparse coefficients.
[0063] The above-mentioned image reconstruction methods include but are not limited to inverse Laplace pyramid reconstruction, inverse wavelet transform reconstruction, inverse Fourier transform reconstruction and the like.
[0064] The present invention also includes a display module for displaying the denoised image.
[0065] Based on the above method, the present invention provides an image denoising system based on multi-scale complementary learning, including: an acquisition module, a denoising module and a reconstruction module.
[0066] Among them, the acquisition module is used to obtain the original image data to be denoised, and decompose the original image into sub-images at different spatial frequencies; wherein, the sub-image with a spatial frequency lower than a threshold represents the base layer of the original image; the denoising module is used to perform denoising on each sub-image separately, directly learn the image scale detail features of the image from the denoised sub-image, and indirectly learn the noise scale detail features of the image by learning the noise distribution of the denoised sub-image, and fuse the image scale detail features and the noise scale detail features to obtain the fused denoised sub-image detail features; the reconstruction module is used to add the fused denoised sub-image detail features to the base layer in the original image to obtain the final denoised image.
[0067] The present invention also provides a computer device, including a memory and a processor, wherein a program is stored in the memory, and when the program is executed by the processor, the processor executes the steps of an image denoising method based on multi-scale complementary learning.
[0068] According to the disclosed embodiments, a computing device may communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth communications, etc.), or with any device (e.g., routers, modems, etc.) that enables a computing device to communicate with one or more other computing devices.
[0069] The present invention also provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of an image denoising method based on multi-scale complementary learning are implemented.
[0070] According to the disclosed embodiments, the storage medium may be a non-volatile computer-readable storage medium, such as but not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.
[0071] The above content is a further detailed description of the present invention in combination with a specific preferred embodiment. For technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as belonging to the protection scope of the present invention.
Claims
1. An image denoising method based on multi-scale complementary learning, characterized in that: The steps include: Obtaining an original image to be denoised, and decomposing the original image into sub-images at different spatial frequencies; wherein the sub-images with spatial frequencies lower than a threshold represent the base layer of the original image; De-noising is performed on each sub-image respectively, and the image scale detail features of the image are directly learned from the denoised sub-image, and the noise scale detail features of the image are indirectly learned by learning the noise distribution of the denoised sub-image. The image scale detail features and the noise scale detail features are fused to obtain the fused denoised sub-image detail features; The fused denoised sub-image detail features are added to the base layer in the original image to obtain the final denoised image.
2. The image denoising method based on multi-scale complementary learning as claimed in claim 1, characterized in that: The denoising process is performed on each sub-image separately, including: performing denoising process on each sub-image separately using one or more denoising processes selected from filter-based denoising, deep learning-based denoising and statistical model-based denoising.
3. The image denoising method based on multi-scale complementary learning as claimed in claim 2, characterized in that: The denoising based on the statistical model includes: Convert the sub-image to an image matrix; The image matrix is decomposed using the low-rank matrix approximation method, and the basic features are retained to remove the singular values corresponding to the noise; The image matrix is reconstructed using the retained base layer features, and the reconstructed image matrix is converted into a sub-image.
4. The image denoising method based on multi-scale complementary learning as claimed in claim 1, characterized in that: The step of adding the fused denoised sub-image detail features to the base layer in the original image to obtain a final denoised image includes the following steps: Obtain the added image; Inputting the added image into a convolutional neural network (CNN) to extract abstract features of the added image layer by layer; the abstract features include features that gradually transition from image edges and texture information to semantic information; Based on the abstract features, the image is gradually restored through deconvolution or upsampling operations to obtain the final denoised image.
5. The image denoising method based on multi-scale complementary learning as claimed in claim 1, characterized in that: When the original image is decomposed into sub-images at different spatial frequencies, the method of decomposing the original image data includes Laplace pyramid decomposition, wavelet transform decomposition and Fourier transform decomposition methods.
6. An image denoising system based on multi-scale complementary learning, characterized in that: include: An acquisition module is used to obtain the original image data to be denoised and decompose the original image into sub-images at different spatial frequencies; wherein the sub-image with a spatial frequency lower than a threshold represents the base layer of the original image; A denoising module is used to perform denoising on each sub-image, directly learn the image scale detail features of the image from the denoised sub-image, and indirectly learn the noise scale detail features of the image by learning the noise distribution of the denoised sub-image, and fuse the image scale detail features and the noise scale detail features to obtain the fused denoised sub-image detail features; The reconstruction module is used to add the fused denoised sub-image detail features to the base layer in the original image to obtain the final denoised image.
7. A computer device, characterized in that: It comprises a memory and a processor, wherein a program is stored in the memory, and when the program is executed by the processor, the processor executes the steps of an image denoising method based on multi-scale complementary learning as claimed in any one of claims 1 to 5.
8. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of an image denoising method based on multi-scale complementary learning as described in any one of claims 1 to 5 are implemented.