Low-light image denoising method based on local-global interactive Transform

Abstract

The invention discloses a low-light image denoising method based on a local-global interactive Transform, and the method is characterized in that the method comprises the following steps: 1, carrying out the denoising of a low-light image; the method comprises the following steps: step 1, constructing an overall structure of LGIT for recovering a given degraded image; 2, a Top-N Patches selection module is used for selecting N patches most related to the target patch and providing feature input for subsequent modules; step 3, the Patch selects an attention Transform module, and the Transform module is used for carrying out global feature aggregation based on the information of Top-N Patches; and step 4, mixing the double-gate feed-forward network MDGFF for further processing the features output by the PSATB and generating a final de-noised image. According to the method, the self-attention inside the patch and between the patches is effectively calculated on different scales, and accurate removal and detail recovery of low-light image noise are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of image denoising, and in particular to a low-light image denoising method based on a local-global interactive Transformer. Background Art

[0002] Night vision imaging technology under low illumination conditions has been widely used in all aspects of production and life, and occupies an extremely important position. High-quality low-light imaging has become increasingly important in fields such as military night reconnaissance, public safety monitoring, and biomedical imaging. However, due to the limitations of acquisition equipment, images taken under low illumination environments are affected by noise and can only capture limited visual information, which seriously reduces the image quality and accuracy of analysis. Therefore, improving the quality of low-light images, especially removing noise and restoring image details, is of increasing significance for the development of low-light night vision technology.

[0003] Although night vision performance can be improved by directly using expensive night vision cameras, improving image quality by improving hardware performance is also limited by the physical limits of the optical system. Image denoising technology restores high-quality images from noisy images captured by ordinary cameras, providing a more flexible and economical solution to this problem. Image denoising is an important task in low-level computer vision, especially under low-light conditions. Early denoising methods were designed based on the self-similarity and local consistency of images. However, these methods often fail to achieve ideal results due to their limited ability to handle complex noisy scenes and require complex optimization processes. In recent years, denoising technology based on deep learning has made breakthrough progress. In particular, methods based on convolutional neural networks (CNNs) have once become the mainstream in the field of image denoising due to their excellent local feature extraction capabilities. However, CNNs have limitations in capturing long-range dependencies of images, which is particularly evident in complex low-light image denoising tasks.

[0004] In recent years, Transformer-based methods have shown strong performance in image denoising tasks, thanks to the advantages of its self-attention mechanism in capturing global dependencies and detail features in images. However, it still has some obvious shortcomings when dealing with low-light image denoising tasks. First, the standard Transformer model relies on a global self-attention mechanism in the spatial dimension, and its computational complexity increases quadratically with the number of tokens, resulting in a significant increase in the computational cost when processing high-resolution images. Although the local window-based self-attention strategy attempts to reduce the computational complexity, it fails to fully utilize the non-local self-similarity of the image, limiting its denoising performance under complex low-light conditions. Second, the traditional Transformer method does not fully consider the spatial correlation of the image in the channel dimension, ignoring the potential contribution of the internal spatial structure information of the image to the denoising performance. There are also some methods that intervene in the attention map to improve the score between similar tokens, aiming to utilize the self-similarity of the image, but its receptive field is limited.

[0005] Transformer-based methods can effectively capture the global dependencies of images and show excellent results in multiple visual tasks. However, when the existing Transformer is applied to denoising of noisy images taken under low illumination conditions, it faces two major problems:

[0006] (1) The computational complexity of the global self-attention mechanism is too high in the spatial dimension because its computational complexity is proportional to the square of the number of tokens. Although the local window-based self-attention method reduces the computational complexity, it fails to make good use of the non-local self-similarity of the image.

[0007] (2) The self-attention calculation in the channel dimension does not fully utilize the spatial correlation of the image. Summary of the invention

[0008] In order to overcome the above technical problems, the purpose of the present invention is to provide a low-light image denoising method based on a local-global interactive Transformer, which solves the problems of high computational complexity and insufficient utilization of spatial correlation in the existing Transformer in low-light image denoising through a refined global interaction strategy. The self-attention mechanism guided by superpixel segmentation is deeply explored to dynamically select other patches in the image that are most relevant to the target patch. On this basis, Top-N Patches Cross Attention (TPCA) is constructed to achieve comprehensive and detailed feature extraction and noise suppression of the image while maintaining computational efficiency. In addition, the hybrid dual-gated feedforward network (MDGFF) is introduced to further enhance the model's ability to capture multi-scale local correlations of images. The finally constructed LGIT, through its hierarchical encoder-decoder network architecture, effectively calculates the self-attention within and between patches at different scales, and achieves accurate removal of low-light image noise and detail restoration.

[0009] The technical solution adopted by the present invention is:

[0010] A low-light image denoising method based on a local-global interactive Transformer comprises the following steps;

[0011] Step 1: Construct the overall structure of LGIT to restore the given degraded image. First, construct the overall framework of Local-Global Interaction Transformer (LGIT). LGIT adopts the U-shaped network structure of encoder-decoder, which can extract image features layer by layer through multi-layer up and down sampling process. After inputting the degraded image, the image will be embedded into a high-dimensional feature space through a 3x3 convolution and provide basic features for subsequent steps. In this structure, the output of each stage will be used as the input of subsequent modules (TPSM and PSATB) to gradually enhance the feature expression and restoration capabilities.

[0012] Step 2: Top-N Patches Selection Module (TPSM), which is used to select the N patches most relevant to the target patch and provide feature input for subsequent modules. In the LGIT structure, the feature map output from the first step will enter the Top-N Patches Selection Module (TPSM), which divides the image into multiple similar superpixel regions through superpixel segmentation. Based on these regions, TPSM dynamically selects the top N patches most relevant to the target patch and provides the most similar contextual information for each patch. These selected patches will provide relevant global information for the subsequent Patch Selection Attention Module, enabling the latter to interact globally more efficiently.

[0013] Step 3: Patch-SelectiveAttentionTransformer Block (PSATB) is used to perform global feature aggregation based on the information of Top-N Patches. After TPSM outputs the relevant Top-N Patches information, the PSATB module uses these selected patches to perform a cross-attention mechanism to further enhance the global interaction between patches. PSATB ensures that features can be aggregated and transferred globally by performing cross-attention calculations on each patch and its related Top-NPatches. In this process, the Patch-SelectiveAttention Module utilizes the contextual information provided by TPSM, which significantly improves the image restoration effect and lays the foundation for the final feature processing.

[0014] Step 4: Hybrid Dual Gated Feedforward Network The feedforward network MDGFF is used to further process the features output by PSATB and generate the final denoised image. After PSATB aggregates the global features, the features are input into the hybrid dual gated feedforward network (MDGFF). MDGFF processes the features from PSATB through convolution paths of different scales, and selects and fuses multi-scale information through a dual gating mechanism. In this way, MDGFF can further optimize and enhance the previously extracted local and global features, ensuring that the final denoised image has high detail recovery capabilities and stronger noise resistance. The combined effect of PSATB and MDGFF ensures that the model achieves efficient information processing and recovery while maintaining low computational complexity.

[0015] The step 1 is specifically as follows:

[0016] The noisy image with degraded image quality due to the imperfection of the imaging system and equipment is used as the input degraded image. In the given degraded image, the Top-N Patches selection module first divides the degraded image into superpixels based on texture and color, classifies each pixel into a superpixel area with high similarity, and calculates the information of the most relevant Top-N patches of each patch at different scales;

[0017] Subsequently, the output of the Top-NPatches selection module (TPSM) and the degraded image are taken as input and enter a U-Shaped network composed of multiple Patch-Selective Attention (PSA) Transformer blocks at different scales for image noise extraction; each PSA contains a cross attention that can aggregate Top-Npatches information to the target patch, a window-based self-attention and two hybrid dual-gated feedforward networks (MDGFF); finally, a LGIT model composed of a U-shaped network is constructed.

[0018] Before the degraded image is input to the U-shaped network, a 3×3 convolution is used to transform the original noisy image I noise Embedded into the high-dimensional feature space, after 3×3 convolution, the x0 input to the deep feature extraction module is:

[0019] x0=f embed (I noise )(1)

[0020] The LGIT model (local-global interactive Transformer model) adopts a U-shaped hierarchical network architecture that can retain and fuse features of different scales, and enhances the model's feature transfer and fusion capabilities by introducing jump connections between the upsampling and downsampling stages.

[0021] The feature map x0 obtained by the extraction module after processing the input map is first compressed step by step through four downsampling units, so that the number of feature map channels is doubled and the spatial scale is halved. Taking the first stage as an example, the feature map is x0∈R C×H×W becomes After 4 downsampling units, the feature map reaches its minimum spatial scale, and a bottleneck stage is added to the network to capture dependencies across a larger spatial range.

[0022] In the upsampling stage, the feature map x after 4 downsampling is reconstructed through 4 decoders. Before each decoder, the feature map is upsampled to double its spatial scale and half its number of channels. After upsampling, a 1×1 convolution is used to fuse the feature map transmitted by the jump connection with the feature map of the current layer in the channel dimension.

[0023] The encoding and decoding process of LGIT consists of the Patch-Selective Attention (PSA) Transformer block. The TPCA in PSA identifies the top N most relevant patches for each patch based on the superpixel segmentation results, effectively supporting the accurate recovery of local details. In the final stage of the network, 3×3 convolutions are used to further refine the feature map and generate a noisy residual image R∈R H×W×3 Finally, the restored image is obtained by I output =I noise +R is calculated.

[0024] The step 2 is specifically as follows:

[0025] The Top-NPatches selection module adopts the method of superpixel segmentation and Patch interaction mapping. It first uses the superpixel segmentation results with self-similarity information to pre-identify the information of the Top-N patches most relevant to each patch, and then performs a detailed Patch interaction mapping strategy. Finally, based on the output results of TPSM, it calculates the cross-attention between the target patch and its most relevant N patches. In this way, more non-local self-similar patches in the image can be used for the recovery of the target patch.

[0026] Specifically, in the superpixel segmentation stage, the softk-meansbased superpixel segmentation algorithm in SPIN is first applied to the input degraded image (original input image) to obtain a preliminary superpixel segmentation result S0. The superpixel regions in S0 are then merged to form larger and more meaningful superpixel regions. In this step, the color and spatial properties of each superpixel region are evaluated, and small superpixel regions that are adjacent and have similar colors are merged into larger regions. The merging algorithm is described as follows:

[0027] if‖c i -c j ‖<θand dist(s i ,s j )<φthen merge(2)

[0028] In the formula, c i and c j are the color centers of superpixel regions i and j, dist(si ,s j ) is the spatial distance, θ and φ are the color and spatial merging thresholds.

[0029] Afterwards, the feature maps of different scales are finely mapped using the label information of the differentiable SLIC algorithm. The core of this step is to map the patches divided in PSA to the superpixel area with the largest pixel proportion according to the proportion of its internal pixels in each superpixel area, thereby ensuring that each patch is mapped to the superpixel area with the largest area coverage. Finally, the top N related patches of each patch at different scales are calculated, and the related patches are selected using the consistency of the superpixel area to ensure the compatibility of features between patches and the coherence of information.

[0030] In TPSM, the input image is first segmented into superpixels, then hard-association-based merging is performed and patch mapping is performed according to the feature map size and patch division at different scales. Finally, the Top-N related patches of each patch are calculated.

[0031] The step 3 is specifically as follows:

[0032] PSATB consists of a cross-attention network that aggregates the information of Top-N patches to the target patch, a WSA, and two feed-forward networks MDGFF. TPCA is the core component of PSA. It uses the output of TPSM to obtain the information of the target patch and its most relevant Top-N patches, and then applies cross-attention calculation to each target patch, where the query vector (Q) is generated by the target patch, and the key (K) and value (V) are generated by the feature map aggregated in the channel dimension by the first N patches most relevant to the target patch. This method can effectively utilize the global context information of the image and accurately restore image details while maintaining computational efficiency.

[0033] The key task of Top-N Patches CrossAttention (TPCA) is to effectively convert the superpixel segmented feature map into a new representation with enhanced contextual information in the encoding and decoding stage. Taking the first TPCA in the encoding stage as an example, the input is a five-dimensional tensor processed by the superpixel segmentation module with a structure of [256, 7, 8, 8, 32]. Among them, 256 represents the number of patches into which the original 128×128 input feature map is divided, that is, the number of patches M=8. 7 represents the 7 patches most relevant to the target patch, including the target patch itself. The subsequent three dimensions [8, 8, 32] correspond to the spatial size and number of channels of each patch respectively.

[0034] In TPCA, the feature map of the target patch t∈R M×M×C , assuming the embedding dimension is D, t is transformed into the query matrix Q through linear projection t ∈R C×D , and the feature map x∈R from Top-N Patches M×M×NC Then the key matrix K is generated accordingly x ∈R NC×D Sum Matrix (Equation 3). In this way, the context information of Top-NPatches is effectively transferred to the target patch to achieve feature enhancement and enrichment. The linear transformation used in this process can be represented by the weight matrix and To represent, they are used to calculate Q, K and V respectively. Through the calculation of cross attention, the network can focus on processing the patch information most relevant to the target patch without having to pay attention to other irrelevant feature areas, which greatly improves the calculation efficiency and captures the local structure and details in the image more accurately.

[0035]

[0036] Next, the attention score is obtained by calculating the dot product of Q and K and normalizing it. The attention score reflects the similarity between the target patch and each Top-NPatches. The calculated attention score guides the transfer of information from Top-NPatches to the target patch. The specific mathematical expression of this process is:

[0037]

[0038] The attention score is multiplied by the value vector V to obtain the updated feature representation of the target patch. The final output calculation process can be expressed as:

[0039]

[0040] Through such a cross-attention mechanism, TPCA can effectively integrate the information of globally related patches into the target patch, which not only improves the expressiveness of features but also maintains the efficiency of the calculation process.

[0041] The step 4 is specifically as follows:

[0042] MDGFF is a feedforward network in PSA, which is used to further improve the efficiency of information flow and the expressiveness of features. MDGFF first normalizes the input features (LayerNormalization, LN), and then processes the features through two parallel gated convolution paths. These two paths are designed to be of different scales to aggregate multi-scale contextual information.

[0043] Specifically, path P1 uses 1×1 and 3×3 convolutions, while path P2 uses 1×1 and 5×5 convolutions. The purpose is to expand the receptive field of the model while keeping the computational complexity controllable.

[0044] After the convolution operation, the features on the two paths are transformed nonlinearly by the activation function ReLU, and then pass through the gating unit (G) for feature selection. The function of the gating unit is to learn how to select and fuse information from parallel paths to enhance the network's expressive power. Mathematically, the gating operation can be expressed as:

[0045] G = σ(Conv(F))(6)

[0046] Among them, F represents the feature before gating, σ is the sigmoid function, which is used to generate a weight coefficient between 0 and 1, and Conv represents the convolution operation. The weight coefficient generated by the gating is multiplied by the feature to achieve selective enhancement of the feature.

[0047] The outputs of the two paths are then merged and then passed through a 1×1 convolution again to fuse the information. The final output feature is given by:

[0048] F out =Conv 1×1 (G.F. P1 +(1-G)·F P2 )(7)

[0049] In the formula, F out represents the output features of MDGFF, F P1 and F P2 They represent the features processed on the two paths respectively. Finally, this output feature will be combined with the output of the self-attention module and returned to the main process of TransformerBlock through the residual connection to complete the entire feature extraction and enhancement process.

[0050] Beneficial effects of the present invention:

[0051] Step 1LGIT's overall network result consists of Top-N Patches Selection Module (TPSM), Patch-Selective Attention (PSA) Transformerblock, 1×1 and 3×3 convolutional layers, which can efficiently capture a wide range of global dependencies and detailed local features in the image, greatly enhancing its feature utilization level of the image's intrinsic self-similar characteristics and spatial layout correlation. This improvement enables the model to show better accuracy and efficiency when dealing with image denoising tasks.

[0052] The TPSM in step 2 consists of a superpixel segmentation module and a patch interactive mapping strategy. It fully utilizes the non-local self-similarity characteristics of the image by aggregating the information of multiple highly correlated patches into the target patch. This method not only provides a solid foundation for subsequent feature fusion and image processing, but also significantly enhances the image denoising and feature analysis capabilities, thereby bringing significant performance improvements to image restoration and analysis tasks.

[0053] The U-shaped network structure composed of PSA in step 3 is a deep feature extraction module, which consists of a cross attention that aggregates the information of Top-N patches to the target patch, a WSA, and two feed-forward networks MDGFF. TPCA is the core component of PSA. By cleverly integrating superpixel segmentation technology as prior knowledge, it balances the high computational cost problem faced by the global self-attention mechanism in an innovative way, and significantly improves the performance of the channel-based attention method in capturing local spatial details. In addition, the superpixel-guided strategy not only significantly enhances the fine expression of local features, but also greatly expands the in-depth capture of global features, providing a richer and more accurate perspective for image analysis and processing. This method effectively utilizes local information and global context information while maintaining computational efficiency to accurately restore image details.

[0054] The MDGFF module in step 4 is composed of a mixture of deep convolutional layers and gating mechanisms. It can more comprehensively capture and utilize features of different scales, thereby providing stronger support for image denoising and detail restoration. The design of this multi-scale feedforward network not only enhances the feature representation, but also ensures the computational efficiency of the entire network. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a schematic diagram of the LGIT network structure overview.

[0056] Figure 2 Select module diagram for Top-N Patches.

[0057] Figure 3Schematic diagram of the Top-N Patches cross-attention module.

[0058] Figure 4 Schematic diagram of the hybrid dual-gated feedforward network MDGFF.

[0059] Figure 5 Schematic diagram for visual comparison with state-of-the-art methods in the low-light image denoising task.

[0060] Figure 6 Schematic diagram of ablation analysis for different Top-N values.

[0061] Figure 7 Schematic diagram of ablation analysis for different patch size values.

[0062] Figure 8 Schematic diagram of the MDGFF module ablation experiment. DETAILED DESCRIPTION

[0063] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] Figure 1 (a) shows the overall structure of LGIT. After a degraded image is given, it first passes through the Top-N Patches Selective Module (TPSM), which divides the original image into superpixels based on texture and color, classifies each pixel into a superpixel region with high similarity, and calculates the information of the most relevant Top-N patches for each patch at different scales. Subsequently, the output of TPSM and the degraded image are used as input to a U-Shaped network composed of multiple Patch-Selective Attention (PSA) Transformer blocks at different scales for image noise extraction.

[0065] like Figure 1 As shown in (b), each PSA contains a cross attention (Top-NPatches Cross Attention, TPCA) that can aggregate the Top-Npatches information to the target patch, a window-based self-attention (Window-based Self Attention, WSA), and two hybrid dual-gated feedforward networks (MDGFF).

[0066] Before the degraded image is input to the U-shaped network, a 3×3 convolution is used to transform the original noisy image I noise Embedded into high-dimensional feature space. After 3×3 convolution, the x0 input to the deep feature extraction module is:

[0067] x0=f embed (I noise )(1)

[0068] The LGIT proposed in this application adopts a U-shaped hierarchical network architecture that can retain and fuse features of different scales, and enhances the model's feature transfer and fusion capabilities by introducing jump connections between the upsampling and downsampling stages.

[0069] The input feature map x0 is first compressed step by step through k downsampling units, so that the number of feature map channels is doubled and the spatial scale is halved. Taking k = 1 as an example, the feature map is x0∈R C×H×W becomes After k downsampling units, the feature map reaches its minimum spatial scale, and a bottleneck stage is added to the network to capture dependencies across a larger spatial range.

[0070] In the upsampling stage, the feature map is reconstructed through k decoders. Before each decoder, the feature map is upsampled to double its spatial scale and half its number of channels. After upsampling, a 1×1 convolution is used to fuse the feature map transmitted by the jump connection with the feature map of the current layer in the channel dimension.

[0071] The encoding and decoding process of LGIT consists of the Patch-Selective Attention (PSA) Transformer block. The TPCA in PSA identifies the top N most relevant patches for each patch based on the superpixel segmentation results, effectively supporting the accurate recovery of local details. In the final stage of the network, 3×3 convolutions are used to further refine the feature map and generate a noisy residual image R∈R H×W×3 Finally, the restored image is obtained by I output =I noise +R is calculated.

[0072] Top-NPatches Selective Module

[0073] This application proposes a Top-NPatches Selective Module (TPSM) that calculates the information of the top-N patches most relevant to each patch at different scales based on superpixel segmentation results. Different from the traditional method of limiting attention calculation to regular patches to reduce the computational complexity of Transformer, this application calculates the cross attention between the target patch and its most relevant N patches based on the output results of TPSM, so that more non-local self-similar patches in the image can be used for the recovery of the target patch.

[0074] Specifically, in the superpixel segmentation stage, the softk-meansbased superpixel segmentation algorithm in SPIN is first applied to the input image to obtain a preliminary superpixel segmentation result S0. Then, the superpixel regions in S0 are merged using the hard association-based merging strategy proposed in this application to form larger and more meaningful superpixel regions. In this step, the color and spatial properties of each superpixel region are evaluated, and adjacent and similarly colored small superpixel regions are merged into larger regions. The merging algorithm is described as follows:

[0075] if‖c i -c j ‖<θand dist(s i ,s j )<φthen merge(2)

[0076] In the formula, c i and c j are the color centers of superpixel regions i and j, dist(s i ,s j ) is the spatial distance, θ and φ are the color and spatial merging thresholds.

[0077] Afterwards, the feature maps of different scales are finely mapped using the label information of the differentiable SLIC algorithm. The core of this step is to map the patches divided in PSA to the superpixel area with the largest pixel proportion according to the proportion of its internal pixels in each superpixel area, thereby ensuring that each patch is mapped to the superpixel area with the largest area coverage. Finally, the top N related patches of each patch at different scales are calculated, and the related patches are selected using the consistency of the superpixel area to ensure the compatibility of features between patches and the coherence of information.

[0078] In TPSM, the input image is first segmented into superpixels and merged based on hard association, then patch mapping is performed according to the feature map size and patch division at different scales, and finally, the Top-N related patches of each patch are calculated. Figure 2 is a schematic diagram of TPSM. Figure 2 (c) shows the final result. Four patches are selected in the figure, and the arrows indicate the selection results of the Top-N patches of the corresponding patches.

[0079] Patch-SelectiveAttention Transformerblock

[0080] Low-light noisy images often exhibit significant non-local self-similarity, which is useful prior knowledge for image denoising. Among existing image restoration methods, the traditional Transformer structure is able to process global information, but its global self-attention mechanism has too high computational complexity when processing high-resolution images. Although the local window-based method reduces the computational complexity, its adaptability to specific noise patterns and the utilization of non-local self-similarity when processing low-light images still need to be improved. Channel-based Transformer architectures such as Restormer excel in capturing the global dependency of features. However, this method relies on carefully designed deep feature interactions and complex channel attention mechanisms, resulting in insufficient sensitivity to local contextual details, especially when there are subtle textures or structures in the image.

[0081] In order to achieve better denoising effect, this application adopts a deep feature extraction algorithm guided by superpixel segmentation to aggregate highly similar patches in the image to make more full use of the redundant information of these self-similar patches. This application proposes Top-N Patches CrossAttention (TPCA) guided by superpixel segmentation, introduces superpixel segmentation as prior knowledge, and adopts a novel method to overcome the high computational cost of global self-attention and optimize the shortcomings of channel attention-based methods in capturing local details. TPCA uses the results of superpixel segmentation to guide the attention mechanism, performs self-attention calculations between the target patch and its most relevant N patches, and significantly reduces the computational complexity. Specifically, the method first obtains the information of the target patch and its most relevant Top-N patches through the superpixel segmentation module, and then applies cross-attention calculations to each target patch, where the query vector (Q) is generated by the target patch, and the key (K) and value (V) are generated by the feature map aggregated in the channel dimension from the first N patches most relevant to the target patch. This method can effectively utilize the global context information of the image while maintaining computational efficiency and accurately restore image details.

[0082] The key task of Top-NPatchesCrossAttention (TPCA) is to effectively convert the superpixel segmented feature map into a new representation with enhanced contextual information in the encoding and decoding stage. Taking the first TPCA in the encoding stage as an example, the input is a five-dimensional tensor processed by the superpixel segmentation module with a structure of [256, 7, 8, 8, 32]. Among them, 256 represents the number of patches into which the original 128×128 input feature map is divided, that is, the number of patches M=8. 7 represents the 7 patches most relevant to the target patch, including the target patch itself. The subsequent three dimensions [8, 8, 32] correspond to the spatial size and number of channels of each patch respectively.

[0083] like Figure 3 As shown, in TPCA, the feature map of the target patch t∈R M×M×C , assuming the embedding dimension is D, t is transformed into the query matrix Q through linear projection t ∈R C×D , and the feature map x∈R from Top-NPatches M×M×NC Then the key matrix K is generated accordingly x ∈R NC×D Sum Matrix (Equation 3). In this way, the context information of Top-NPatches is effectively transferred to the target patch to achieve feature enhancement and enrichment. The linear transformation used in this process can be represented by the weight matrix and To represent, they are used to calculate Q, K and V respectively. Through the calculation of cross attention, the network can focus on processing the patch information most relevant to the target patch without having to pay attention to other irrelevant feature areas, which greatly improves the calculation efficiency and captures the local structure and details in the image more accurately.

[0084]

[0085] Next, the attention score is obtained by calculating the dot product of Q and K and normalizing it. The attention score reflects the similarity between the target patch and each Top-NPatches. The calculated attention score guides the transfer of information from Top-NPatches to the target patch. The specific mathematical expression of this process is:

[0086]

[0087] The attention score is multiplied by the value vector V to obtain the updated feature representation of the target patch. The final output calculation process can be expressed as:

[0088]

[0089] Through such a cross-attention mechanism, TPCA can effectively integrate the information of globally related patches into the target patch, which not only improves the expressiveness of features but also maintains the efficiency of the calculation process.

[0090] In TPCA, the self-attention (SA) module is followed by a mixed-scale dual-gated feed-forward network (MDGFF). MDGFF aims to further improve the efficiency of information flow and the expressiveness of features by introducing mixed-scale and dual-gating mechanisms.

[0091] like Figure 4 As shown in the figure, MDGFF first normalizes the input features (Layer Normalization, LN), and then processes the features through two parallel gated convolution paths. These two paths are designed to be of different scales to aggregate multi-scale contextual information.

[0092] Specifically, path P1 uses 1×1 and 3×3 convolutions, while path P2 uses 1×1 and 5×5 convolutions. The purpose is to expand the receptive field of the model while keeping the computational complexity controllable.

[0093] After the convolution operation, the features on the two paths are transformed nonlinearly by the activation function ReLU, and then pass through the gating unit (G) for feature selection. The function of the gating unit is to learn how to select and fuse information from parallel paths to enhance the network's expressive power. Mathematically, the gating operation can be expressed as:

[0094] G = σ(Conv(F))(6)

[0095] Among them, F represents the feature before gating, σ is the sigmoid function, which is used to generate a weight coefficient between 0 and 1, and Conv represents the convolution operation. The weight coefficient generated by the gating is multiplied by the feature to achieve selective enhancement of the feature.

[0096] The outputs of the two paths are then merged and then passed through a 1×1 convolution again to fuse the information. The final output feature is given by:

[0097] F out =Conv 1×1 (G.F. P1 +(1-G)·F P2 )(7)

[0098] In the formula, F out represents the output features of MDGFF, F P1 and F P2 They represent the features processed on the two paths respectively. Finally, this output feature will be combined with the output of the self-attention module and returned to the main process of TransformerBlock through the residual connection to complete the entire feature extraction and enhancement process.

[0099] By introducing the MDGFF module, the model can more comprehensively capture and utilize features of different scales, thus providing stronger support for image denoising and detail restoration. This multi-scale feedforward network design not only enhances feature representation, but also ensures the computational efficiency of the entire network.

[0100] The window-based self-attention mechanism (WSA) calculates self-attention separately within each patch, which can more effectively process the local features of the image, which enables the model to parse the details of each area more carefully. This application uses the standard WSA in Uformer to process patches. Ultimately, the complete PSA will be composed of a cross-attention TPCA+MDGFF that aggregates the information of the Top-N patches to the target patch and a WSA+MDGFF in series.

[0101] This application proposes a Local-Global Interaction Transformer (LGIT), a strategy that adaptively selects relevant patches for global interaction to achieve global autocorrelation calculation with low computational complexity. Specifically, a Top-NPatches Cross Attention (TPCA) guided by superpixel segmentation is designed. By dynamically selecting the top N patches most similar to the target image patch, cross attention is applied to aggregate the information of these N patches to the target patch in the channel dimension, thereby achieving the purpose of better utilizing the non-local self-similarity of the image. In addition, in order to better extract multi-scale local correlations, a mixed-scale dual-gated feed-forward network MDGFF (Mixed-scale Dual Gated Feed-Forward Network) is introduced. Finally, a hierarchical encoder-decoder network LGIT is constructed using TPCA and MDGFF to calculate self-attention within and between patches at different scales. Extensive experiments on real-world image denoising datasets show that LGIT outperforms the most advanced CNN-based and Transformer-based methods in qualitative and quantitative results. For example, LGIT achieves a significant 40.4dB PSNR on the SIDD dataset.

[0102] experiment

[0103] First, we discuss the experimental setup. Then, we verify the effectiveness and efficiency of the LGIT method on the SIDD and DND datasets. Finally, we conduct comprehensive ablation experiments to verify the effects of each component in LGIT.

[0104] Experimental setup

[0105] This application uses SIDD as the training set, which is specially used for image denoising research. SIDD uses 5 representative mobile phones and takes a total of 30,000 noisy images in 10 scenes. These images were taken under different lighting conditions, including low light, normal brightness, and high exposure.

[0106] This application uses the SIDD test set and the DND dataset when testing the algorithm performance. The DND (Darmstadt Noise Dataset) dataset is a dataset specifically used for image denoising research. It was released by Darmstadt University of Technology and aims to provide real noisy images and corresponding noise-free image pairs for evaluating image denoising algorithms.

[0107] The experiment in this application uses the AdamW optimizer, setting β1 to 0.9 and β2 to 0.999. The training data is randomly rotated 90°, 180°, 270° and horizontally flipped for data augmentation. The learning rate is decayed from the initial 2e-4 to 1e-6 using the cosine decay strategy. The patchsize in LGIT is set to 8×8. The number of PSABlocks (N1-N4) in each layer of the network is set to {1,2,8,2}. The PyTorch deep learning framework is used and runs on the Nvidia V100 graphics card, reaching convergence after 200 epochs of training.

[0108] Referring to the previous work on image denoising, this application uses peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) as evaluation indicators. The image is converted from RGB color space to YCbCr color space and evaluated on the Y channel (brightness component). This is because the Y channel has a more significant impact on the visual quality of the human eye, thus more effectively reflecting the effect of image denoising.

[0109] Comparative Experiments (Compared to other state-of-the-art methods)

[0110] The method proposed in this application is compared with other advanced image denoising algorithms, including the traditional method BM3D, CNN-based DnCNN, CBDNet, CycleISP, AINNet, DeamNet models, and Transformer-based Uformer, MAXIM, Restormer, and Xformer models.

[0111] Table 1. Quantitative comparison results on SIDD and DND datasets. The bold and underlined numbers represent the best and second-best results, respectively.

[0112]

[0113]

[0114] The quantitative indicators of different methods are shown in Table 1. It can be observed that, overall, the Transformer-based model outperforms the CNN-based method in terms of PSNR and SSIM. This is because the Transformer has a stronger global information modeling capability than CNN. The Transformer model effectively captures the dependencies between pixels through the self-attention mechanism, which enables it to establish a wider range of connections between the various parts of the image, thereby improving the overall understanding and processing of the image content. However, some Transformer-based methods limit attention to regular patches or channels based on feature maps to reduce computational complexity. This approach does not fully utilize the non-local self-similarity of the image.

[0115] In contrast, the method proposed in this application aggregates highly correlated patches in a superpixel-guided manner, achieving more refined local information processing while maintaining sensitivity to global content. In this way, LGIT can retain the use of global image features while maintaining low computational complexity, effectively removing noise, especially in the details and texture parts of the image. The superpixel-guided aggregation strategy optimizes the information integration process, making the information extracted from each patch richer and more correlated, which directly improves the denoising effect.

[0116] Figure 5 The denoising effect comparison of several methods is shown. The results show that when the superpixel area in the image is obvious and the image has strong non-local self-similarity, the proposed LGIT can effectively restore the noise-contaminated area. In contrast, the method that limits attention to the patch cannot utilize the information of other patches similar to the target patch, and thus cannot achieve a good denoising effect. It can be clearly seen from the figure that the denoising effect of the method of the present application is better than other methods, which further illustrates that the use of non-local self-similarity plays an important role in low-light image denoising.

[0117] Ablation experiment

[0118] The effectiveness of LGIT's model design and its key components are further analyzed. To ensure the fairness of the experiments, all ablation experiments are conducted under the same conditions.

[0119] Effectiveness of Top-NPatchesCrossAttention(TPCA).

[0120] The TPCA module is the core component of LGIT. TPCA aggregates the information of patches in the same superpixel area to the target patch through the cross-attention mechanism, thereby effectively performing image denoising. TPCA is added to different layers of the U-Net backbone, and the performance comparison under different settings is shown in Table 2. The results show that the lack of TPCA leads to a decline in network performance because the use of window self-attention (WSA) alone cannot obtain enough long-range information. As TPCA is integrated in more layers, the network performance gradually improves. In addition, an attempt was made to use only TPCA and remove WSA, and it was found that the performance of this method was slightly lower than that of using only WSA. This confirms the necessity of simultaneously utilizing the locality and non-local self-similarity of the image in image denoising. At the same time, an ablation experiment was conducted on the selection of "Top-N", and the PSNR performance of different "Top-N" values ​​is shown as follows. Figure 6 As shown. It can be observed that setting Top-N to a smaller value will lead to a decrease in performance, because there is not enough information about related patches for the recovery of the target patch. On the other hand, too large a "Top-N" value will introduce interference from irrelevant patches, which will also lead to a decrease in performance. In order to better evaluate the model, Table 2 also shows the comparison of model parameters under different settings. The results show that the model parameter volume of LGIT is 60.12M. Compared with Uformer (50.88M), LGIT improves PSNR by 0.51dB (40.40dB vs. 39.89dB) with only an increase of 9.24M parameters.

[0121] Table 2. Module ablation experimental results in LGIT

[0122]

[0123]

[0124] Compare the impact of patchsize on network performance. Figure 7 It can be seen that when the patchsize is 8, the PSNR of the model reaches the best, which is 0.77dB and 1.45dB higher than when the patchsize is 4 and 16 respectively. This shows that medium-sized patches can effectively capture sufficient contextual information while maintaining low computational complexity, thereby optimizing the quality of image restoration. On the contrary, although a smaller patchsize reduces computational complexity, it limits the model from capturing necessary contextual information, thus affecting performance. Although a larger patchsize can cover a wider range of contexts, it increases computational complexity and may introduce too much irrelevant information, affecting the denoising effect of the model.

[0125] Table 3. Ablation analysis of feedforward network

[0126]

[0127] The effectiveness of Mixed-scale Dual-gated Feed-forward Network (MDGFF). The ablation experiment results are shown in Table 3, showing the contribution of MDGFF in enhancing the performance of the Transformer network. The results show that MDGFF improves PSNR by 0.05dB compared with the standard FN, which proves the effectiveness of the dual-gating mechanism and mixed-scale design. In addition, by using only a single-scale feedforward network for experiments, it can be found that the performance of the single-scale model decreases by 0.03~0.04dB compared with the mixed-scale MDGFF, which further verifies the advantage of the mixed-scale method in capturing multi-level feature information. Figure 8 The PSNR comparison results of the MDGFF module applied to the Uformer, Restormer and LGIT methods are shown. It can be seen that after replacing the original FFN, the PSNR of Restormer increased from 40.02dB to 40.05dB, Uformer increased from 39.89dB to 39.93dB, and LGIT increased from 40.35dB to 40.40dB, which shows that the MDGFF module significantly improves the performance of image reconstruction.

[0128] This application proposes a new model called Local-Global Interaction Transformer (LGIT), which combines the Top-NPatches Cross Attention (TPCA) guided by superpixel segmentation and the hybrid dual-gated feedforward network (MDGFF) to solve the problems of high computational complexity and insufficient utilization of spatial correlation in the low-light image denoising process. LGIT demonstrates its ability to effectively capture non-local self-similarity and detail information of low-light noisy images through a refined global interaction strategy. Extensive experiments show that LGIT outperforms existing advanced techniques on multiple low-light image denoising benchmark datasets, verifying its effectiveness and efficiency in practical applications.

Claims

1. A low-light image denoising method based on local-global interactive Transformer, characterized in that: The steps include: Step 1: Construct the overall structure of LGIT to restore the given degraded image; Step 2: Top-N Patches Selection Module (TPSM), which is used to select the N patches most relevant to the target patch and provide feature input for subsequent modules. In the overall structure of LGIT, the feature map output in the first step will enter the Top-N Patches Selection Module (TPSM), which divides the image into multiple similar superpixel regions through superpixel segmentation. TPSM dynamically selects the top N patches most relevant to the target patch and provides the most similar contextual information for each patch. The selected patches will provide relevant global information for the subsequent Patch Selection Attention Module; Step 3: Patch selection attention Transformer module is used to perform global feature aggregation based on the information of Top-NPatches. After TPSM outputs the relevant Top-N Patches information, the PSATB module uses these selected patches to perform a cross-attention mechanism to enhance the global interaction between patches. Step 4: The hybrid dual-gated feedforward network feedforward network MDGFF is used to further process the features output by PSATB and generate the final denoised image.

2. The low-light image denoising method based on local-global interactive Transformer according to claim 1, characterized in that: The step 1 is specifically as follows: The noisy image whose image quality is degraded due to the imperfection of the imaging system and equipment is used as the input degraded image. In the given degraded image, the Top-N Patches selection module first divides the degraded image into superpixels based on texture and color, classifies each pixel into a superpixel area with high similarity, and calculates the information of the most relevant Top-N patches of each patch at different scales; Subsequently, the output of the Top-NPatches Selection Module (TPSM) and the degraded image are taken as input to a U-Shaped network consisting of multiple Patch-Selective Attention (PSA) Transformer blocks at different scales for image noise extraction. Each PSA contains a cross attention that can aggregate the Top-Npatches information to the target patch, a window-based self-attention, and two hybrid dual-gated feedforward networks (MDGFF); finally, a LGIT model composed of a U-shaped network is constructed.

3. The low-light image denoising method based on local-global interactive Transformer according to claim 2, characterized in that: Before the degraded image is input to the U-shaped network, a 3×3 convolution is used to transform the original noisy image I noise Embedded into the high-dimensional feature space, after 3×3 convolution, the x0 input to the deep feature extraction module is: x0=f embed (I noise ) (1) The LGIT model adopts a U-shaped hierarchical network architecture that can retain and fuse features of different scales, and introduces jump connections between the upsampling and downsampling stages to enhance the model's feature transfer and fusion capabilities. The feature map x0 obtained by the extraction module after processing the input map is first compressed step by step through four downsampling units, so that the number of feature map channels is doubled and the spatial scale is halved. Taking the first stage as an example, the feature map is x0∈R C×H×W becomes After 4 downsampling units, the feature map reaches its minimum spatial scale, and a bottleneck stage is added to the network to capture dependencies across a larger spatial range. In the upsampling stage, the feature map x after 4 downsampling is reconstructed through 4 decoders. Before each decoder, the feature map is upsampled to double its spatial scale and half its number of channels. After upsampling, a 1×1 convolution is used to fuse the feature map transmitted by the jump connection with the feature map of the current layer in the channel dimension. The encoding and decoding process of LGIT consists of Patch-Selective Attention (PSA) Transformer block. The TPCA in PSA identifies the top N most relevant patches of each patch based on the superpixel segmentation results, effectively supporting the accurate recovery of local details. In the final stage of the network, 3×3 convolution is used to further refine the feature map and generate a noisy residual image R∈R H×W×3 Finally, the restored image is obtained by I output =I noise +R is calculated.

4. The low-light image denoising method based on local-global interactive Transformer according to claim 1, characterized in that: The step 2 is specifically as follows: The Top-N Patches selection module adopts the method of superpixel segmentation and patch interaction mapping. It first uses the superpixel segmentation results with self-similarity information to pre-identify the information of the Top-N patches most relevant to each patch, then performs a detailed patch interaction mapping strategy, and finally calculates the cross-attention between the target patch and its most relevant N patches based on the output results of TPSM. In this way, more non-local self-similar patches in the image can be used for the recovery of the target patch.

5. The low-light image denoising method based on local-global interactive Transformer according to claim 4, characterized in that: In the superpixel segmentation stage, the soft k-means based superpixel segmentation algorithm in SPIN is first applied to the input degraded image (original input image) to obtain the preliminary superpixel division result S0, and then the superpixel regions in S0 are merged to form a larger and more meaningful superpixel region; In this step, the color and spatial properties of each superpixel region are evaluated, and small superpixel regions that are adjacent and have similar colors are merged into larger regions. The merging algorithm is described as follows: if ‖c i -c j ‖<θ and dist(s i ,s j )<φ then merge (2) In the formula, c i and c j are the color centers of superpixel regions i and j, dist(s i ,s j ) is the spatial distance, θ and φ are the color and spatial merging thresholds; Afterwards, the feature maps of different scales are finely mapped using the label information of the differentiable SLIC algorithm. Finally, the top N relevant patches of each patch at different scales are calculated, and the relevant patches are selected using the consistency of the superpixel region. In TPSM, the input image is first segmented into superpixels, then hard-association-based merging is performed and patch mapping is performed according to the feature map size and patch division at different scales. Finally, the Top-N related patches of each patch are calculated.

6. The low-light image denoising method based on local-global interactive Transformer according to claim 1, characterized in that: The step 3 is specifically as follows: In TPCA, the feature map of the target patch t∈R M×M×C , assuming the embedding dimension is D, t is transformed into the query matrix Q through linear projection t ∈R C×D , and the feature map x∈R from Top-N Patches M×M×NC Then the key matrix K is generated accordingly x ∈R NC×D Sum Matrix (Formula 3); The linear transformation used uses a weight matrix and To represent, they are used to calculate Q, K and V respectively; Next, the attention score is obtained by calculating the dot product of Q and K and normalizing it. The attention score reflects the similarity between the target patch and each Top-N Patches; the calculated attention score guides the transfer of information from Top-N Patches to the target patch. The specific mathematical expression of this process is: The attention score is multiplied by the value vector V to obtain the updated feature representation of the target patch. The final output calculation process is expressed as:

7. The low-light image denoising method based on local-global interactive Transformer according to claim 1, characterized in that: The step 4 is specifically as follows: MDGFF first normalizes the input features (LayerNormalization, LN), and then processes the features through two parallel gated convolution paths. These two paths are designed to be of different scales to aggregate multi-scale contextual information.

8. The low-light image denoising method based on local-global interactive Transformer according to claim 7, characterized in that: Path P1 uses 1×1 and 3×3 convolutions, and path P2 uses 1×1 and 5×5 convolutions; After the convolution operation, the features on the two paths are transformed nonlinearly by the activation function ReLU, and then pass through the gating unit (G) for feature selection. The gating operation is expressed as: G = σ(Conv(F)) (6) Among them, F represents the feature before gating, σ is the sigmoid function, which is used to generate a weight coefficient between 0 and 1, and Conv represents the convolution operation. The weight coefficient generated by the gating is multiplied by the feature to achieve selective enhancement of the feature; The outputs of the two paths are then merged and then passed through a 1×1 convolution again to fuse the information. The final output feature is given by: F out =Conv 1×1 (G·F P1 +(1-G)·F P2 ) (7) In the formula, F out represents the output features of MDGFF, F P1 and F P2 They represent the features processed on the two paths respectively. Finally, this output feature will be combined with the output of the self-attention module and returned to the main process of TransformerBlock through the residual connection to complete the entire feature extraction and enhancement process.

Images