Self-supervised image denoising method and system combining local and global features

Abstract

The invention discloses a self-supervised image denoising method and system combining local and global features, and belongs to the field of image processing. The method comprises the following steps: constructing a Patch-level mask mechanism convolution module PLMC, and processing an input image by adopting PLMCs with convolution kernels of different sizes; constructing a local feature extraction branch used for carrying out pixel shuffling down-sampling operation on a feature map output by the PLMC with the convolution kernel of the first size and then carrying out local feature extraction; constructing a global feature extraction branch used for carrying out pixel shuffling downsampling operation on a feature map output by the PLMC with the convolution kernel of the second size and then carrying out global feature extraction; and carrying out element addition on the local features and the global features, then carrying out pixel shuffling up-sampling operation, and recombining into a de-noised image with the same size as the original image as a final reconstructed de-noised image. The denoising effect of the real image is improved.

Description

Technical Field

[0001] This invention belongs to the field of image processing. Specifically, this invention relates to a self-supervised image denoising method and system that combines local and global features. Background Technology

[0002] Image denoising has been a crucial research area in computer vision technology for decades. Due to various interferences during acquisition and transmission, images often contain different forms of noise. This not only degrades image quality but also affects subsequent image processing tasks. Therefore, accurately removing noise from noisy images using appropriate methods is essential. In recent years, with the development of neural networks, learning-based methods have significantly outperformed traditional algorithms. However, these methods require training with a large number of labeled image pairs. Simply synthesizing Gaussian noise images cannot solve the problem—the complexity of real-world noise makes such synthetic data ineffective in practical scenarios. Although real-world noise datasets exist, their acquisition process requires strict environmental control and is costly: obtaining a clear original image requires long exposures or multiple shots, which is almost impractical in dynamic scenes.

[0003] To overcome the reliance on paired training data (i.e., noisy images and their corresponding clean images), image denoising methods without clean target supervision have gradually become a research hotspot. The pioneering work in this field, Noise2Noise (N2N), first demonstrated the possibility of training using only noisy image pairs, but it still relies on multiple noisy images captured in the same scene. More practically valuable research focuses on self-supervised learning of single noisy images. Against this backdrop, Blind Spot Networks (BSNs) have made significant progress by designing strategies for predicting the center blind pixel from neighboring pixels, providing an effective paradigm for single-image denoising. However, the core mechanism of BSNs is based on a key assumption—that noise between pixels is independent. This simplification significantly deviates from the spatially correlated noise prevalent in real imaging systems, caused by sensor physical characteristics (such as correlated noise and the spatial clustering effect of photon shot noise), thus limiting its performance in complex real-world noisy scenes.

[0004] Therefore, this invention proposes a self-supervised image denoising method and system that combines local and global features. Summary of the Invention

[0005] This invention aims to overcome the shortcomings of existing technologies and proposes a self-supervised image denoising method and system that combines local and global features to achieve the following objectives: to solve the loss of details caused by the coarse consideration of the correlation of real noise, and the lack of global interaction caused by the inherent limitations of the receptive field of previous denoising networks, thereby improving the denoising effect of real images.

[0006] To achieve the above objectives, the technical solution adopted by this invention is as follows: a self-supervised image denoising method combining local and global features. The method is used to construct a self-supervised image denoising model combining local and global features to achieve image denoising, and includes the following steps:

[0007] Step S1: Construct a Patch-level masking mechanism convolutional module (PLMC), and process the input image using PLMCs with convolutional kernels of different sizes.

[0008] Step S2: Construct a local feature extraction branch, which is used to perform pixel shuffling downsampling operation on the feature map output by the PLMC with the first size convolution kernel and then perform local feature extraction.

[0009] Step S3: Construct a global feature extraction branch, which is used to perform pixel shuffling downsampling on the feature map output by the PLMC with the second-size convolutional kernel and then perform global feature extraction; the first size is smaller than the second size;

[0010] Step S4: After adding the local features and global features element by element, perform pixel shuffling and upsampling operation, and recombine them into a denoised image with the same size as the original image, which is the final reconstructed denoised image.

[0011] Preferably, in step S1, the construction of the Patch-level masking mechanism convolutional module (PLMC) includes:

[0012] The dense sampling convolution kernel is constructed as follows:

[0013] ;

[0014] in, This represents a densely sampled convolution kernel. It is a regular convolution kernel. For noise correlation mask;

[0015] The dense sampling convolution kernel is used to extract features from the input image to obtain a high-dimensional feature map;

[0016] After downsampling the high-dimensional feature map, features are further extracted through dilated convolution.

[0017] Preferably, the dilated convolution is expressed by the following formula:

[0018] ;

[0019] In the formula, Indicates the location Convolution output at the point; The convolution kernel is represented by the first... One weight parameter; Indicates the first The position of each convolutional kernel; Represents the pixel values ​​of the input feature map; This represents the fixed distance the convolution kernel is offset towards the center pixel during the testing phase. The offset ratio of the kernel is adjusted accordingly. Enhance the ability to capture local details.

[0020] Preferably, in step S2, dilated convolution incorporating residual modules is used for local feature extraction, including:

[0021] Feature map T is obtained by performing pixel shuffling downsampling on the feature map output by PLMC with a first-size convolution kernel;

[0022] Perform dilated convolution on the feature map T;

[0023] The feature map T is then added to the output of the dilated convolution through skip connections for fusion, and the fused features are used as the final extracted local features.

[0024] Preferably, in step S3, an adaptive channel attention module is used for global feature extraction.

[0025] Preferably, the specific construction process of the adaptive channel attention module includes:

[0026] Feature map F is obtained by performing pixel shuffling downsampling on the feature map output by PLMC with a second-size convolution kernel;

[0027] The feature map F is converted into channel descriptors U using global average pooling. Then, a local channel interaction matrix B is introduced to model local dependencies, and a diagonal matrix is ​​used to extract global channel dependencies. Correspondingly, the local and global channel information are represented as follows:

[0028] , ;

[0029] In the formula, and These represent local and global information of the channel, respectively. The first element of the local interaction matrix represents the... Each weight, Represents the first of the global diagonal matrix Each weight, The dimension of the local interaction matrix. This represents the number of channels that are globally dependent.

[0030] Next, the final channel attention weights are calculated based on the local and global information of the channels. ;

[0031] The weight The final extracted global features are obtained by weighted fusion with feature map F.

[0032] Preferably, the weight The calculation is as follows:

[0033] ;

[0034] In the formula, This represents the final channel attention weight. This represents the Sigmoid activation function. This represents a learnable balancing factor used to adjust the importance of global and local information.

[0035] Preferably, the method further includes: initializing the image denoising model constructed based on steps S1 to S4, including:

[0036] Set an adaptive learning rate for adaptive channel attention and train it; calculate the output image and the real image at each training epoch. The loss function is used to adjust the learning rate, and backpropagation is used to optimize the model parameters. Finally, the image denoising model after initialization is obtained, and it is applied to actual image denoising tasks.

[0037] Preferably, the The loss is defined as follows:

[0038] ;

[0039] In the formula, Indicates the number of training samples. Indicates the first The pixel values ​​of a real image. Indicates the first image of the output image Each predicted pixel value.

[0040] The present invention also provides a self-supervised image denoising system that combines local and global features, the system comprising a computer for executing a computer program constructed according to the self-supervised image denoising method that combines local and global features as described above.

[0041] The technical effects of this invention are as follows:

[0042] This invention achieves denoising effects by directly training on noisy images without requiring labeled ones. Compared to traditional supervised learning methods, this invention can better handle both local and global information. Furthermore, the self-supervised learning method is more adaptable to images of different types and sources. This is crucial for natural image processing, as the model needs to effectively denoise images of various styles.

[0043] This invention combines local and global feature extraction to simultaneously capture detailed information and global contextual information, effectively improving image denoising performance. This allows for the preservation of crucial anatomical details while removing noise, thus enhancing the quality of the denoised image. Global feature extraction employs an adaptive channel attention module, enabling the network to precisely allocate feature weights at different granularities. This achieves adaptive fusion of channel-wide and local information, resulting in more accurate global features and significantly improving the network's denoising performance. Local feature extraction utilizes dilated convolution to enhance local feature extraction capabilities, while a residual module is introduced to further enhance information flow and improve the overall denoising effect. Attached Figure Description

[0044] Figure 1 This is a flowchart of a self-supervised image denoising method that combines local and global features, provided as an embodiment of the present invention. Detailed Implementation

[0045] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. This is to help those skilled in the art to have a more complete, accurate, and in-depth understanding of the inventive concept and technical solutions of the present invention, and to facilitate its implementation. It should be noted that the terms "first," "second," etc., used in this application are only for the convenience of describing the technical solutions and to distinguish components; the corresponding component configurations may be the same or different, and are not intended to limit the scope of this application. To make the technical solutions of the present invention clearer, the present invention will be explained and illustrated through the following embodiments.

[0046] This invention provides a self-supervised image denoising method that combines local and global features, aiming to address the loss of detail caused by the coarse consideration of the correlation of real noise, and the lack of global interaction caused by the inherent limitations of the receptive field in previous denoising networks, thereby improving the denoising effect of realistic images. Previous self-supervised denoising methods largely relied on the assumption that noise is pixel-independent. However, experiments have shown that noise is spatially correlated. Furthermore, some denoising networks do not have a good sampling density for neighboring pixels during training, resulting in the loss of local texture details and significantly reducing denoising performance. This embodiment considers that pixels, like signals, have spatial correlation, better encoding local detailed structures and capturing global representations. It mitigates the loss of detail texture by carefully considering the spatial correlation in real noise, and introduces long-distance interaction for blind spot networks using an adaptive channel attention mechanism. This invention also introduces a Patch-level Masking Mechanism (PLMC) convolutional module. Based on previous statistics of the spatial correlation of real noise, this invention considers more neighboring pixels, forming a denser receptive field, enabling the model to recover more detailed structures.

[0047] This invention also designs two branches sampling at different densities. One branch uses a smaller mask convolution as the local information branch, while the other branch uses a larger mask convolution as the global information branch to obtain a wide range of global contextual information. An adaptive channel attention mechanism is employed to compensate for long-distance correlations that cannot be established using a self-attention mechanism. Finally, the results of the two branches are fused, allowing us to fully utilize both local and global features.

[0048] Specifically, embodiments of the present invention construct a self-supervised image denoising model that combines local and global features, enabling the model to learn from noisy images without relying on previously used synthetic noise as a supervisory signal, ultimately achieving image denoising, such as... Figure 1 As shown, the specific method includes the following steps:

[0049] Step S1: Construct a Patch-level masking mechanism convolutional module (PLMC), and process the input image using PLMCs with convolutional kernels of different sizes.

[0050] Step S2: Construct a local feature extraction branch, which is used to perform pixel shuffling downsampling operation on the feature map output by the PLMC with the first size convolution kernel and then perform local feature extraction.

[0051] Step S3: Construct a global feature extraction branch, which is used to perform pixel shuffling downsampling on the feature map output by the PLMC with the second-size convolutional kernel and then perform global feature extraction; the first size is smaller than the second size;

[0052] Step S4: After adding the local features and global features element by element, perform pixel shuffling and upsampling operation, and recombine them into a denoised image with the same size as the original image, which is the final reconstructed denoised image.

[0053] Referring to step S1, this embodiment constructs a Patch-level mask mechanism convolution (PLMC) module, which enhances the ability to capture local details by sampling only pixels with low correlation to the center pixel through a mask; after generating high-dimensional features, downsampling is performed and features are extracted using dilated convolution, and the detail preservation is improved by adjusting the convolution kernel offset ratio.

[0054] Specifically, the construction of the Patch-level masking mechanism convolutional module (PLMC) includes:

[0055] Construct dense sampling convolution kernels to enhance the sampling density of local features, as expressed by the following formula:

[0056] ;

[0057] in, This represents a densely sampled convolution kernel. It is a regular convolution kernel. This is a noise correlation mask; by filtering out highly correlated pixel positions through the mask, the model only samples neighboring pixels with low correlation to the center pixel, thereby retaining more detailed information.

[0058] The dense sampling convolution kernel is used to extract features from the input image to obtain a high-dimensional feature map;

[0059] Downsampling is performed on the high-dimensional feature map to break down spatial noise correlation in the image, and features are further extracted through dilated convolution. The dilated convolution is expressed by the following formula:

[0060] ;

[0061] In the formula, Indicates the location Convolution output at the point; The convolution kernel is represented by the first... One weight parameter; Indicates the first The position of each convolutional kernel; Represents the pixel values ​​of the input feature map; This represents the fixed distance the convolution kernel is offset towards the center pixel during the testing phase. The offset ratio of the kernel is adjusted accordingly. Enhance the ability to capture local details.

[0062] Referring to step S2, this embodiment constructs a local feature extraction branch, which is used to perform pixel shuffling downsampling on the feature map output by the PLMC with a first-size convolutional kernel (7×7 in this embodiment, but can be flexibly selected according to actual conditions) before performing local feature extraction. Specifically, this embodiment uses dilated convolution combined with a residual module for local feature extraction, including:

[0063] Feature map T is obtained by performing pixel shuffling downsampling on the feature map output by PLMC with a first-size convolution kernel;

[0064] Perform dilated convolution on the feature map T;

[0065] The feature map T is then added to the output of the dilated convolution via skip connections for fusion, and the fused features are used as the final extracted local features. The use of dilated convolution improves the ability to extract local features, while the introduction of the residual module further enhances information flow and improves the overall denoising effect.

[0066] Referring to step S3, this embodiment constructs a global feature extraction branch, which is used to perform pixel shuffling downsampling on the feature map output by the PLMC with a second-sized convolutional kernel (21×21 in this embodiment, but can be flexibly selected according to actual conditions) before performing global feature extraction; the first size is smaller than the second size. In this embodiment, an adaptive channel attention module is used for global feature extraction.

[0067] Specifically, the construction process of the adaptive channel attention module includes:

[0068] After performing pixel shuffling downsampling on the feature map output by the PLMC with the second-size convolution kernel, feature map F is obtained. Feature map F has rich local and global channel information.

[0069] The feature map F is converted into channel descriptors U using global average pooling. Then, a local channel interaction matrix B is introduced to model local dependencies, and a diagonal matrix is ​​used to extract global channel dependencies. Correspondingly, the local and global channel information are represented as follows:

[0070] , ;

[0071] In the formula, and These represent local and global information of the channel, respectively. The first element of the local interaction matrix represents the... Each weight, Represents the first of the global diagonal matrix Each weight, The dimension of the local interaction matrix. This represents the number of channels that are globally dependent.

[0072] Next, the final channel attention weights are calculated based on the local and global information of the channels. It can be expressed by the following formula:

[0073] ;

[0074] In the formula, This represents the final channel attention weight. This represents the Sigmoid activation function. This represents a learnable balancing factor used to adjust the importance of global and local information.

[0075] The weight The final extracted global features are obtained by weighted fusion with feature map F.

[0076] Finally, referring to step S4, this embodiment adds the extracted local features and global features element by element to ensure the effective combination of features at different scales, thereby improving the denoising accuracy and image detail restoration effect of the model; then, a pixel shuffling upsampling operation is performed to reassemble the denoised image with the same size as the original image, which is the final reconstructed denoised image.

[0077] Before the image denoising model constructed based on steps S1 to S4 can be applied in practice, it needs to be initialized, including:

[0078] Set an adaptive learning rate for adaptive channel attention and train it; calculate the output image and the real image at each training epoch. The loss is used to adjust the learning rate through a loss function, and the model parameters are optimized using backpropagation; finally, the initialized image denoising model is obtained and applied to a practical image denoising task. Wherein, the... The loss is defined as follows:

[0079] ;

[0080] In the formula, Indicates the number of training samples. Indicates the first The pixel values ​​of a real image. Indicates the first image of the output image Each predicted pixel value.

[0081] This embodiment also provides a self-supervised image denoising system that combines local and global features. The system includes a computer for executing a computer program constructed according to the above-described self-supervised image denoising method that combines local and global features.

[0082] The method described in this invention can be trained directly on noisy images without the need for labeled images to achieve denoising effects. Compared to traditional supervised learning methods, this invention can better handle both local and global features. Furthermore, the self-supervised learning method is more adaptable to images of different types and sources. This is crucial for natural image processing, as the model needs to effectively denoise images of various styles.

[0083] This invention extracts local features through small-size patch-level mask convolution and dilated convolution combined with residual modules. At the same time, it extracts global features through large-size patch-level mask convolution and introduces an adaptive channel attention module, thereby achieving collaborative modeling of local details and global context. This balances noise suppression and detail preservation within a self-supervised framework, resulting in more ideal performance in image denoising tasks.

[0084] To verify the performance of the model designed in this embodiment of the invention, a comparative experiment was further conducted. The datasets used in the experiment were SIDD and DND. SIDD used approximately 30,000 noisy images from about 10 scenes captured by 5 representative mobile phones under different lighting conditions. The training set contained 24,542 images, and the validation set contained 200 images. Performance was evaluated by calculating the PSNR and SSIM of the denoised images. The method proposed in this invention was compared with the following three methods: CVF-SID, AP-BSN, and PUCA. Table 1 reports the denoising performance of different methods on the SIDD and DND datasets, demonstrating the rationality of the model design and contributing to the improvement of denoising accuracy. Therefore, this method can be extended to other image denoising fields for wider application.

[0085] Table 1

[0086] The present invention has been described above by way of example with reference to the accompanying drawings. Obviously, the specific implementation of the present invention is not limited to the above-described manner. Any non-substantial improvements made using the inventive concept and technical solution; or the direct application of the inventive concept and technical solution to other situations without modification, are all within the protection scope of the present invention.

Claims

1. A self-supervised image denoising method combining local and global features, characterized in that: The method is used to achieve image denoising by constructing a self-supervised image denoising model that combines local and global features, and includes the following steps: Step S1: Construct a Patch-level masking mechanism convolutional module (PLMC), and process the input image using PLMCs with convolutional kernels of different sizes. Step S2: Construct a local feature extraction branch, which is used to perform pixel shuffling downsampling operation on the feature map output by the PLMC with the first size convolution kernel and then perform local feature extraction. Step S3: Construct a global feature extraction branch, which is used to perform pixel shuffling downsampling on the feature map output by the PLMC with the second-size convolutional kernel and then perform global feature extraction; the first size is smaller than the second size; Step S4: After adding the local features and global features element by element, perform pixel shuffling and upsampling operation, and recombine them into a denoised image with the same size as the original image, which is the final reconstructed denoised image.

2. The self-supervised image denoising method combining local and global features according to claim 1, characterized in that: In step S1, the construction of the Patch-level masking mechanism convolutional module (PLMC) includes: The dense sampling convolution kernel is constructed, as expressed by the following formula: ； in, This represents a densely sampled convolution kernel. It is a regular convolution kernel. For noise correlation mask; The dense sampling convolution kernel is used to extract features from the input image to obtain a high-dimensional feature map; After downsampling the high-dimensional feature map, features are further extracted through dilated convolution.

3. The self-supervised image denoising method combining local and global features according to claim 1, characterized in that: The dilated convolution is expressed by the following formula: ； In the formula, Indicates the location The convolution output at the point; The convolution kernel is represented by the first... One weight parameter; Indicates the first The position of each convolutional kernel; Represents the pixel values ​​of the input feature map; This represents the fixed distance the convolution kernel is offset towards the center pixel during the testing phase. The offset ratio of the kernel is adjusted accordingly. Enhance the ability to capture local details.

4. The self-supervised image denoising method combining local and global features according to claim 1, characterized in that: In step S2, dilated convolution incorporating residual modules is used for local feature extraction, including: Feature map T is obtained by performing pixel shuffling downsampling on the feature map output by PLMC with a first-size convolution kernel; Perform dilated convolution on the feature map T; The feature map T is then added to the output of the dilated convolution through skip connections for fusion, and the fused features are used as the final extracted local features.

5. The self-supervised image denoising method combining local and global features according to claim 1, characterized in that: In step S3, an adaptive channel attention module is used for global feature extraction.

6. The self-supervised image denoising method combining local and global features according to claim 5, characterized in that: The specific construction process of the adaptive channel attention module includes: Feature map F is obtained by performing pixel shuffling downsampling on the feature map output by PLMC with a second-size convolution kernel; The feature map F is converted into channel descriptors U using global average pooling. Then, a local channel interaction matrix B is introduced to model local dependencies, and a diagonal matrix is ​​used to extract global channel dependencies. Correspondingly, the local and global channel information are represented as follows: ， ； In the formula, and These represent local and global information of the channel, respectively. The first element of the local interaction matrix represents the... Each weight, Represents the first of the global diagonal matrix Each weight, The dimension of the local interaction matrix. This represents the number of channels that are globally dependent. Next, the final channel attention weights are calculated based on the local and global information of the channels. ; The weight The final extracted global features are obtained by weighted fusion with the feature map F.

7. The self-supervised image denoising method combining local and global features according to claim 6, characterized in that: The weight The calculation is as follows: ； In the formula, This represents the final channel attention weight. This represents the Sigmoid activation function. This represents a learnable balancing factor used to adjust the importance of global and local information.

8. A self-supervised image denoising method combining local and global features according to any one of claims 5-7, characterized in that: The method further includes: initializing the image denoising model constructed based on steps S1 to S4, including: Set an adaptive learning rate for adaptive channel attention and train it; calculate the output image and the real image at each training epoch. The loss function is used to adjust the learning rate, and backpropagation is used to optimize the model parameters. Finally, the image denoising model after initialization is obtained, and it is applied to actual image denoising tasks.

9. A self-supervised image denoising method combining local and global features according to claim 8, characterized in that: The The loss is defined as follows: ； In the formula, Indicates the number of training samples. Indicates the first The pixel values ​​of a real image. Indicates the first image of the output image Each predicted pixel value.

10. A self-supervised image denoising system combining local and global features, using the self-supervised image denoising method combining local and global features according to any one of claims 1-9, characterized in that: The system includes a computer for executing a computer program constructed according to any one of claims 1-9 of a self-supervised image denoising method combining local and global features.

Images