A dual-domain heterogeneous image denoising method

Abstract

The application relates to the technical field of image denoising, and particularly discloses a dual-domain heterogeneous image denoising method, which comprises the following steps: extracting a preliminary feature map based on a deep separable convolution; adopting an encoder of a dual-domain heterogeneous collaborative architecture in a shallow layer of a hierarchical dual-drive coding and decoding architecture to perform dual-domain heterogeneous collaborative processing, performing hierarchical feature adaptation, giving consideration to details and local structures in the shallow layer, focusing on global semantics in a deep layer, and dynamically allocating computing resources to significantly reduce redundant computing burden; splicing processed image frequency domain information and image spatial domain information, and based on a vertical stripe perception fusion attention mechanism module between the encoder and the decoder, fusing features of all layers after hierarchical processing and outputting the features to the decoder to obtain a denoised image, enhancing sensitivity to vertical stripe noise, and specifically suppressing noise areas.

Description

Technical Field

[0001] This invention relates to the field of image denoising technology, and in particular to a method for denoising dual-domain heterogeneous images. Background Technology

[0002] Currently, image denoising remains a core and crucial research direction in the field of image processing. In today's digital age, images are widely used in numerous fields, from everyday photography and videography to professional medical imaging, satellite remote sensing images, and industrial inspection images. However, during image acquisition, noise inevitably gets mixed into image data due to factors such as the physical characteristics of sensors and environmental interference. During transmission, network instability and signal interference can also lead to noise in the image. During storage, the characteristics and potential damage of the storage medium can also introduce noise. Common Gaussian noise, usually caused by electronic circuit noise and sensor noise, exhibits a grayscale variation that follows a Gaussian distribution, making the overall image blurry. Salt and pepper noise appears as randomly occurring black and white pixels in the image, like salt and pepper sprinkled on it, severely damaging the visual effect. These noises greatly reduce image quality, making image details blurry and significantly decreasing the recognizability of information. For subsequent image analysis tasks, such as feature extraction and classification of objects in an image, if the image contains noise interference, the extracted features may be inaccurate, leading to classification errors. In the field of target recognition, whether it's facial recognition in security monitoring or road target recognition in autonomous driving, noise can cause recognition algorithms to misjudge or fail to identify targets, leading to serious consequences. In medical image diagnosis, noise interference can mask lesion features, causing doctors to misdiagnose or miss diagnoses, endangering patients' lives and health. The emergence of the Transformer architecture has brought new opportunities to solve the problem of long-range dependencies. Based on the self-attention mechanism, it can effectively capture long-range dependencies when processing sequential data, achieving great success in the field of natural language processing and gradually being introduced into image processing. However, Transformer is relatively weak in local feature extraction and has high computational complexity and low processing efficiency. Mamba, as an emerging model architecture, has a unique structure and efficient computational performance, providing a new approach to image denoising. Image denoising methods based on the Mamba+Transformer dual-branch approach are expected to combine the advantages of both, leveraging the efficiency of Mamba and the ability of Transformer to handle long-range dependencies, to perform comprehensive feature extraction and denoising of images from different scales and domains, opening up new avenues for improving image denoising effects, and have broad application prospects and important research value in the field of image processing.

[0003] Traditional image denoising methods are mainly based on filtering techniques. For example, mean filtering replaces the center pixel value with the average value of neighboring pixels to smooth noise; median filtering replaces the center pixel value with the median value of neighboring pixels, which has a good suppression effect on salt-and-pepper noise. However, these methods have obvious limitations. While smoothing noise, they excessively blur image edges and details. For example, when processing images containing subtle textures, the texture information disappears after noise removal, leading to image distortion and failing to meet the requirements of applications with high image detail requirements. With the rapid development of deep learning technology, neural network-based image denoising methods have shown great advantages. Convolutional Neural Networks (CNNs), with their convolutional layers, pooling layers, and other structures, can automatically learn the feature representation of images and have achieved remarkable results in image denoising tasks. However, the convolutional operation of CNNs is inherently local, and their receptive field size is limited. They have difficulty processing long-range dependencies between distant pixels in an image and struggle to capture global image information. This can lead to unreasonable global structures in the denoised image, affecting the integrity of the denoising effect.

[0004] Therefore, this invention proposes a method for denoising dual-domain heterogeneous images. Summary of the Invention

[0005] This invention provides a dual-domain heterogeneous image denoising method. It employs a dual-domain parallel M-TBlock in the shallow layers (layers one and two) and switches to a lightweight T-Block in the deeper layers (layers three and four). Through hierarchical feature adaptation (shallow layers consider details and local structure, while deeper layers focus on global semantics) and dynamic allocation of computational resources (high-resolution features in the shallow layers are processed by the MT Block, while low-resolution features in the deep layers are handled by the lightweight T-Block), redundant computational burden is significantly reduced. The frequency domain (FreqFormer) and spatial domain (SpMamba) branches are combined in parallel, capturing global frequency domain features and long-distance spatial dependencies through Transformer and Mamba respectively, overcoming the limitations of traditional single-domain processing. V-CBAM is embedded in the encoder-decoder skip connection to reconstruct the cross-layer feature fusion process. Channel attention filters important feature channels, and vertical spatial attention is enhanced by adding a vertical Sobel convolution kernel to strengthen sensitivity to vertical stripe noise and specifically suppress noise regions.

[0006] This invention provides a method for denoising dual-domain heterogeneous images, comprising:

[0007] S1: Preliminary feature map of the input image extracted based on depthwise separable convolution;

[0008] S2: A hierarchical dual-drive encoding / decoding architecture is used to process the initial features hierarchically, resulting in multi-layered features including:

[0009] In the shallow layer of the hierarchical dual-drive encoding / decoding architecture, a dual-domain heterogeneous collaborative architecture encoder is used for dual-domain heterogeneous collaborative processing, including:

[0010] Frequency domain processing branch: Performs spectral band reconstruction and attention weight allocation on the image frequency domain information to obtain the processed image frequency domain information;

[0011] Spatial domain processing branch: Captures long-range spatial dependencies of the image, combines convolutional enhancement feedforward network to suppress local pixel forgetting, and obtains spatial domain information of the processed image;

[0012] In the deep switching of the hierarchical dual-drive encoding and decoding architecture to T Block, a multi-head transposed attention mechanism and a deep spatial enhancement feedforward network are used for global context modeling and local feature enhancement.

[0013] S3: Concat the processed image frequency domain information and image spatial domain information to obtain the features of each layer after hierarchical processing;

[0014] S4: Based on the vertical stripe perception fusion attention mechanism module with skip connections between the encoder and decoder, the features of each layer after hierarchical processing are fused and output to the decoder to obtain a denoised image.

[0015] Preferably, the dual-domain heterogeneous image denoising method performs spectral band reconstruction and attention weight allocation on the image frequency domain information to obtain the processed image frequency domain information, including:

[0016] The frequency domain self-attention module is used to extract Q, K and V through depthwise separable convolution, and the spatial domain features are transformed to the spectral domain using fast Fourier transform.

[0017] Frequency domain elements are classified into different frequency bands by spectral band reconstruction, and different attention weights are assigned between different frequency bands. Attention is reduced in low frequency bands and enhanced in high frequency bands to obtain frequency domain attention features.

[0018] The frequency domain attention features are transformed back to the spatial domain through inverse fast Fourier transform, and the spatial features are output through a convolutional layer.

[0019] The frequency domain enhancement feedforward module increases the channel dimension through point convolutional layers and uses parallel dilated convolution for processing. The features of the two branches are then transformed to the frequency domain, and the spectral domain information is enhanced by learnable weights and bias variables. Through a gating mechanism, the activation output of one branch with a long receptive field is used as the gating unit of the other branch to dynamically adjust the feature enhancement and perform feature fusion. The processed image frequency domain information is obtained based on the 1×1 convolution output.

[0020] Preferred, a dual-domain heterogeneous image denoising method captures the long-range spatial dependence of the image, combines a convolutional enhanced feedforward network to suppress local pixel forgetting, and obtains the spatial domain information of the processed image, including:

[0021] The visual state space module scans two-dimensional features along the four diagonal directions to generate four 1D sequences. Then, it applies discrete state space equations to the 1D sequences in each direction to obtain the results in the four directions.

[0022] The results from the four directions are merged by summation, and then the two-dimensional structure is restored by reshape operation. Convolutional enhancement feedforward network is then used to enhance the interaction capability of local features and suppress channel redundancy information to obtain the processed image spatial domain information.

[0023] Preferably, the dual-domain heterogeneous image denoising method, which employs a dual-domain heterogeneous collaborative architecture encoder in the shallow layer of the hierarchical dual-drive encoding / decoding architecture for dual-domain heterogeneous collaborative processing, further includes:

[0024] Calculate the adjusted number of channels for each layer based on the feature map size processed at each layer;

[0025] The computational balance error of the hierarchical dual-drive encoding / decoding architecture is calculated based on the adjusted number of channels in all layers and the feature map size processed in all layers.

[0026] When the computational balance error of the hierarchical dual-drive encoding and decoding architecture exceeds the preset error threshold, the hierarchical channel dynamic allocation strategy is triggered.

[0027] Preferred, the dual-domain heterogeneous image denoising method calculates the adjusted number of channels for each layer based on the feature map size processed at each layer, including:

[0028]

[0029] In the formula, C is the number of channels after adjustment in a single layer, C base Here, α is the base channel number, tanh() is the hyperbolic tangent function, log2() is the logarithmic function to the base 2, H is the feature map height of the corresponding layer, W is the feature map width of the corresponding layer, H0 is the height of the input image, and W0 is the width of the input image.

[0030] Preferably, the dual-domain heterogeneous image denoising method calculates the computational balance error of the hierarchical dual-drive encoding / decoding architecture based on the adjusted number of channels in all layers and the feature map size processed in all layers, including:

[0031]

[0032] In the formula, δ represents the computational balance error of the hierarchical dual-drive codec architecture, n represents the total number of layers in the hierarchical dual-drive codec architecture, and γ represents the total number of layers in the hierarchical dual-drive codec architecture.i Let C be the weight of the i-th layer. i Let H be the number of channels after adjustment for the i-th layer, FLOPS0 be the computational cost per channel, H0 be the height of the input image, W0 be the width of the input image, and H... i W represents the height of the feature map processed in the i-th layer. i The width of the feature map processed in the i-th layer.

[0033] Preferably, in the dual-domain heterogeneous image denoising method, when the computational balance error of the hierarchical dual-drive encoding / decoding architecture exceeds a preset error threshold, a hierarchical channel dynamic allocation strategy is triggered, including:

[0034] When the computational balance error of the hierarchical dual-drive encoding and decoding architecture exceeds the preset error threshold, the feature map size of the corresponding layer, the size of the input image, and the computational balance error of the hierarchical dual-drive encoding and decoding architecture are input into the hierarchical channel dynamic allocation model to obtain the final adjusted number of channels for each layer.

[0035] Preferred, the dual-domain heterogeneous image denoising method employs a multi-head transposed attention mechanism and a deep spatial enhancement feedforward network for global context modeling and local feature enhancement, including:

[0036] Multi-head transposed attention is used to extract the Q, K, and V matrices through depthwise separable convolution and to perform global context modeling.

[0037] A deep spatial augmentation feedforward network is employed, and local features are extracted using depthwise separable convolutions. Key regions are then selected using spatial attention gating.

[0038] The preferred method for denoising dual-domain heterogeneous images is that the vertical stripe perception fusion attention mechanism module optimizes cross-level feature fusion through the synergistic effect of channel attention and vertical enhancement spatial attention.

[0039] Preferably, in the dual-domain heterogeneous image denoising method, the vertical stripe perception fusion attention mechanism module optimizes cross-level feature fusion through the synergistic effect of channel attention and vertical enhancement spatial attention, including:

[0040] Global average pooling and max pooling are performed on the input features, and channel weight vectors are generated through two fully connected layers.

[0041] The vertical Sobel operator is used to extract the vertical edge response, and the result is concatenated with the horizontal convolution result to generate a spatial weight matrix;

[0042] Meanwhile, the weights of the channel attention branch and the spatial attention branch are dynamically allocated based on the enhancement mechanism until the output is the element-wise product of the channel weight vector and the spatial weight matrix.

[0043] The beneficial effects of this invention compared to the prior art are as follows:

[0044] This method utilizes depthwise separable convolution to extract preliminary feature maps from the input image. This convolutional approach reduces computational cost while effectively capturing the image's fundamental features, laying the foundation for subsequent denoising. It can quickly perform preliminary image processing without losing too much feature information, improving the efficiency of the entire denoising process and enabling the method to maintain good performance even when handling large-scale image data.

[0045] A hierarchical dual-drive encoding and decoding architecture is adopted to process the initial features hierarchically, giving full play to the advantages of different processing stages and branches.

[0046] The encoder employs a dual-domain heterogeneous collaborative architecture in the shallow layer, processing data in parallel through two branches: the frequency domain and the spatial domain. The frequency domain processing branch performs spectral band reconstruction and attention weight allocation on the image's frequency domain information, enabling targeted adjustment of the image's energy distribution in the frequency domain, highlighting key information, and removing frequency domain noise. The spatial domain processing branch captures the image's long-range spatial dependencies and, combined with a convolutional enhancement feedforward network, suppresses local pixel forgetting, effectively preserving the image's structural and detailed information in the spatial dimension and avoiding the loss of local information during denoising. The two branches work together to optimize the image from different dimensions.

[0047] In the deep layer, switching to T Block, a multi-head transposed attention mechanism and a deep spatial enhancement feedforward network are used for global context modeling and local feature enhancement. The multi-head transposed attention mechanism can capture the dependencies between different regions of the image globally, while the deep spatial enhancement feedforward network further enhances local features, making the denoised image not only consistent overall but also clearer in local details, effectively improving image quality.

[0048] By concatenating the processed image's frequency and spatial domain information, the advantageous features of both domains can be fully integrated, resulting in richer and more comprehensive features at each layer after hierarchical processing. This fusion method preserves the useful information extracted during the frequency and spatial domain processing, providing a more powerful feature representation for generating high-quality denoised images.

[0049] A vertical stripe-aware fusion attention mechanism module is set up between the encoder and decoder based on skip connections. This module can effectively fuse features from different layers after hierarchical processing and output the fused features to the decoder. This fusion method not only utilizes feature information from different layers of the encoder, but also pays more attention to image features in specific directions through the vertical stripe-aware attention mechanism, further improving the quality of the denoised image. This results in a more detailed and textured image that is closer to the original clear image.

[0050] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in this application.

[0051] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0052] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0053] Figure 1 This is a schematic diagram of a dual-domain heterogeneous image denoising method according to an embodiment of the present invention;

[0054] Figure 2 This is a flowchart illustrating the execution logic of steps S2 and S3 in an embodiment of the present invention.

[0055] Figure 3 This is a flowchart illustrating the execution logic of step S4 in an embodiment of the present invention. Detailed Implementation

[0056] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0057] Example 1:

[0058] This invention provides a method for denoising dual-domain heterogeneous images, with reference to... Figures 1 to 3 ,include:

[0059] S1: Preliminary feature map of the input image extracted based on depthwise separable convolution;

[0060] S2: A hierarchical dual-drive encoding / decoding architecture is used to process the initial features hierarchically, resulting in multi-layered features including:

[0061] In the shallow layer of the hierarchical dual-drive encoding / decoding architecture, a dual-domain heterogeneous collaborative architecture encoder is used for dual-domain heterogeneous collaborative processing, including:

[0062] The frequency domain processing branch performs spectral band reconstruction and attention weight allocation on the image's frequency domain information to obtain the processed image's frequency domain information. First, the image is transformed from the spatial domain to the frequency domain (e.g., through Fourier transform) to obtain its frequency domain representation. Then, different spectral bands in the frequency domain information are recombinated to highlight or adjust information within specific frequency ranges. Simultaneously, attention weights are assigned to different spectral bands, allowing the model to focus more on frequency bands important for denoising, based on the image's characteristics. For example, if noise in the image is mainly concentrated in the high-frequency range, higher attention weights are assigned to the high-frequency bands to better handle the noise. Through spectral band reconstruction and attention weight allocation, the image's frequency domain information can be optimized in a targeted manner, removing noise while preserving important image features, making the processed image's frequency domain information more conducive to subsequent processing and denoising.

[0063] The spatial domain processing branch captures long-range spatial dependencies in the image and combines them with a convolutional augmentation feedforward network to suppress local pixel forgetting, thereby obtaining processed spatial domain information. Long-range spatial dependencies refer to the relationships between distant pixels in an image. Techniques (such as variations of self-attention mechanisms) are used to capture these long-range dependencies, allowing for an understanding of the connections between different regions in the image. Simultaneously, a convolutional augmentation feedforward network is used to process local pixels, preventing the loss of local pixel information during processing. For example, in a large image region, capturing long-range spatial dependencies reveals the relationship between a local region and other distant regions. The convolutional augmentation feedforward network further enhances the feature representation of local pixels, avoiding information loss. Comprehensive processing of the image in the spatial domain considers both the overall structural relationships (long-range spatial dependencies) and the integrity of local pixel information, resulting in high-quality processed spatial domain information that provides richer spatial information support for denoising.

[0064] In the deeper layers of the hierarchical dual-drive codec architecture, a T-block is used, employing a multi-head transposed attention mechanism and a deep spatial enhancement feedforward network for global context modeling and local feature enhancement. The multi-head transposed attention mechanism is used for global context modeling, which can compute the attention distribution of multiple different representation subspaces in parallel, thus capturing the global information of the image more comprehensively. The deep spatial enhancement feedforward network focuses on local feature enhancement, strengthening the features of local regions through a series of convolutions and nonlinear transformations. For example, the multi-head transposed attention mechanism can simultaneously focus on different regions of the image, comprehensively considering the relationships between them, while the deep spatial enhancement feedforward network performs in-depth processing on a specific local region, improving the feature representation of that region. At the deeper layers of the architecture, global context modeling and local feature enhancement further optimize the image's feature representation. Global context modeling helps the model understand the image as a whole, while local feature enhancement enables the model to process local details more meticulously; the combination of the two helps to remove noise more accurately and preserve the image's detailed information.

[0065] S3: The processed image frequency domain information and image spatial domain information are concatenated to obtain the features of each layer after hierarchical processing. The processed image frequency domain information obtained from the frequency domain processing branch and the processed image spatial domain information obtained from the spatial domain processing branch are concatenated along the channel dimension. For example, if the processed image frequency domain information has C1 channels and the processed image spatial domain information has C2 channels, the concatenated feature map will have C1+C2 channels. By concatenating, the processing results of the frequency domain and spatial domain are fused, making full use of the information advantages of the two domains, so that the features of each layer after hierarchical processing contain richer and more comprehensive image information, providing a more powerful feature representation for subsequent fusion and denoising.

[0066] S4: Based on the vertical stripe perception fusion attention mechanism module with skip connections between the encoder and decoder, the features of each layer after hierarchical processing are fused and output to the decoder to obtain a denoised image.

[0067] A vertical stripe-aware fusion attention mechanism module is constructed between the encoder and decoder using skip connections. Skip connections allow shallow features from the encoder to be directly passed to the decoder for fusion with deeper features, avoiding information loss during the encoding-decoding process. The vertical stripe-aware fusion attention mechanism module focuses on fusing features from different layers after hierarchical processing. It assigns attention weights to different features based on their characteristics, prioritizing features relevant to denoising. For example, lower weights are assigned to feature regions containing noisy information, while higher weights are assigned to key structural and texture features of the image. The fused features are then input into the decoder, which uses a series of deconvolution or upsampling operations to restore the low-resolution feature map to a denoised image of the same size as the original image. In this way, features from different levels are effectively fused, highlighting important features and suppressing noise-related features, ultimately outputting a high-quality denoised image, improving the effect and quality of image denoising.

[0068] In this embodiment, the model employs a 4-layer encoder-decoder structure. First, preliminary features are extracted using depthwise separable convolution (DS-Conv). Then, the model enters the encoder (MT Block), which consists of two branches: a Transformer-based frequency domain processing branch (FreqFormer) and a Mamba-based spatial domain processing branch (SpMamba). These two branches process the frequency and spatial information of the image, respectively. Since the image size is reduced after downsampling, the T Block (Transformer-based) is chosen as the spatial domain processing module to extract deeper features while reducing computational cost. Skip connections, through the embedding of a vertical stripe-aware fusion attention mechanism (V-CBAM), highlight key regions and suppress irrelevant parts through channel attention and vertically enhanced spatial attention, further improving image processing capabilities.

[0069] Depthwise separable convolution is an efficient convolution method that decomposes standard convolution into depthwise convolution and pointwise convolution. In this step, depthwise separable convolution is applied to the input image to extract preliminary feature maps. Depthwise convolution performs independent convolution operations on each channel, capturing local spatial features, while pointwise convolution fuses and integrates the features of the depthwise convolution output through 1×1 convolutions. For example, for an RGB three-channel image, depthwise convolution performs convolutions on the red, green, and blue channels separately, extracting local features for each channel. Then, pointwise convolution fuses these channel features to obtain preliminary feature maps. Compared to traditional convolution, depthwise separable convolution significantly reduces computation while effectively extracting low-level image features, providing a foundation for subsequent processing. These preliminary feature maps contain basic image information, such as edges and textures, and are crucial for further image analysis and processing.

[0070] The MT Block consists of a parallel FreqFormer frequency domain processing branch and a SpMamba spatial domain processing branch. The frequency and spatial domain information of the image processed by the two branches are then concatenated.

[0071] SBSA first extracts Q, K, and V using depthwise separable convolutions. Then, it transforms the spatial domain features to the spectral domain using FFT. Next, it classifies the frequency domain elements into different frequency bands using spectral band reorganization (SBR) and assigns different attention weights to these bands. Attention is reduced for low-frequency bands and increased for high-frequency bands. Finally, the frequency domain attention features are transformed back to the spatial domain using IFFT, and the spatial features are output through convolutional layers.

[0072] SEFF increases channel dimensions through point convolutional layers, then processes the data through two parallel branches, each using 3×3 and 3×3 dilated depthwise convolutions. The features from both branches are then transformed to the frequency domain, and spectral information is enhanced using learnable weights and bias variables. A gating mechanism is used, where the activation output of one branch with a longer receptive field acts as the gating unit for the other branch, dynamically adjusting the feature enhancement.

[0073] SpMamba consists of a Visual State Space Module (VSSM) and a Convolutional Augmentation Feedforward Network (CAE).

[0074] Discrete State-Space Model (VSSM) is a state-space model based on Mamba used to capture long-range dependencies in images. Specifically, 2D-SSM flattens two-dimensional image features into 1D sequences that Mamba can process. It scans the image features along four different directions (top-left to bottom-right, bottom-right to top-left, bottom-left to top-right, and top-right to bottom-left), generating four 1D sequences. Discrete state-space equations are then applied to each sequence. Ideally, the results of all sequences are combined using a summation operation, and then the two-dimensional structure is restored using a reshape operation.

[0075] CAE is a convolutional augmented feedforward network. Since Mamba processes 1D sequences, local pixels in a 2D image may be far apart after flattening, leading to the problem of local pixel forgetting. CAE not only enhances the interaction capability of local features but also suppresses channel redundancy information.

[0076] T Block employs the Multi-Head Transposed Attention (MDTA) mechanism from Restormer. MDTA uses depthwise separable convolutions to extract Q, K, and V, maintaining computational efficiency while capturing local features, and the attention computation models the global context. The Deep Spatial Augmentation Feedforward Network (DSFN) uses depthwise convolutions to reduce computational complexity while maintaining the effectiveness of feature extraction. Through the spatial attention mechanism, key regions are highlighted, less informative features are suppressed, and only useful information is allowed to propagate further.

[0077] A skip connection embedding vertical stripe-aware fusion attention mechanism (V-CBAM) optimizes cross-level feature fusion through the synergistic effect of channel attention and vertically enhanced spatial attention. Channel attention filters important feature channels, while the spatial attention branch adds a vertical Sobel convolution kernel to enhance the response to vertical stripe noise and suppress redundant information. This design ensures that features in noisy regions are accurately suppressed during decoding while preserving image details.

[0078] By combining the frequency domain (FreqFormer) and spatial domain (SpMamba) branches in parallel, and capturing global frequency domain features and long-distance spatial dependencies through Transformer and Mamba respectively, the limitations of traditional methods in single-domain processing are overcome.

[0079] The shallow layers (layers 1 and 2) employ a dual-domain parallel MT Block, while the deep layers (layers 3 and 4) switch to a lightweight TB Block. Through hierarchical feature adaptation (shallow layers take into account details and local structure, while deep layers focus on global semantics) and dynamic allocation of computing resources (high-resolution features in shallow layers are processed by the MT Block, while low-resolution features in deep layers are handled by the lightweight T Block), redundant computational burden is significantly reduced.

[0080] Targeted feature enhancement mechanisms are introduced in different modules. The Convolutional Enhancement Feedforward Network (CAE) in SpMamba solves the problem of local pixel forgetting in Mamba in 2D image processing, enhances the interaction capability of local features, and suppresses channel redundancy information. The Deep Spatial Enhancement Feedforward Network (DSFN) in T Block solves the problem that the transformer's local feature extraction capability is not as good as that of CNN and the problem of over-processing of irrelevant regions through depthwise separable convolution and spatial attention mechanism. While maintaining computational efficiency, it enhances the model's local feature extraction capability, focuses on key regions, and suppresses features in irrelevant regions.

[0081] V-CBAM is embedded in the encoder-decoder skip connection to reconstruct the cross-level feature fusion process. Channel attention filters important feature channels, and vertical spatial attention adds a vertical Sobel convolution kernel to enhance sensitivity to vertical stripe noise and specifically suppress noise regions.

[0082] Example 2:

[0083] Based on Example 1, spectral band reconstruction and attention weight allocation are performed on the image frequency domain information to obtain the processed image frequency domain information, including:

[0084] The frequency domain self-attention module extracts Q, K, and V through depthwise separable convolution, and transforms the spatial domain features to the spectral domain using Fast Fourier Transform (FFT). This is achieved by leveraging the depthwise separable convolution within the frequency domain self-attention module. Depthwise separable convolution breaks down conventional convolution into depthwise convolution and pointwise convolution, effectively reducing computational cost. In this step, this convolution method extracts the query (Q), key (K), and value (V) from the image's frequency domain information. These three features are crucial in the self-attention mechanism: Q determines the focus of attention, K matches relevant information, and V provides the actual feature representation. For example, when processing an image containing multiple textures, Q might focus on a particular texture, K searches for related parts of the image, and V carries the specific feature information of these textures. The Fast Fourier Transform (FFT) is then used to transform the spatial domain features to the spectral domain. FFT efficiently converts spatial domain signals into frequency domain representations, revealing the signal's components at different frequencies. Features that originally described the pixel distribution of an image in the spatial domain are transformed into representations of the intensity of different frequency components in the frequency domain after FFT. For example, after converting a natural landscape image from the spatial domain to the frequency domain, we can see that the low-frequency components reflect the general outline of the image, while the high-frequency components reveal details and textures. The Q, K, and V values ​​extracted by depthwise separable convolution provide the basic data for subsequent attention calculations, while the conversion to the spectral domain allows subsequent processing to manipulate image information in the frequency dimension, which helps to more effectively analyze and process image frequency domain information and prepares for tasks such as denoising.

[0085] Frequency domain elements are classified into different frequency bands through spectral band reconstruction, and different attention weights are assigned to each band. Attention to low-frequency bands is reduced, while attention to high-frequency bands is increased, thus obtaining frequency domain attention features. Frequency domain elements are reconstructed by frequency to classify them into different frequency bands. Since noise is mostly concentrated in the high-frequency range, attention to low-frequency bands is intentionally reduced, while attention to high-frequency bands is increased. In this way, while preserving the low-frequency information of the main image structure, high-frequency information related to noise and details is captured, making the processed frequency domain features more conducive to noise reduction.

[0086] The frequency domain attention features are transformed back to the spatial domain using inverse fast Fourier transform (IFFT), and then output as spatial features through a convolutional layer. The frequency domain attention features, after spectral band reconstruction and weight allocation, are then transformed back to the spatial domain using IFFT to match the spatial dimension of the original image. Next, these spatial domain features are processed and output using convolutional layers to further optimize the feature representation, preparing for subsequent fusion with other spatial domain features or further processing.

[0087] A frequency-domain enhancement feedforward module increases channel dimension through point convolutional layers, employs parallel dilated convolutions for processing, and then transforms the features of both branches into the frequency domain. Learnable weights and bias variables are then used to enhance spectral information. A gating mechanism is employed, using the activation output of one branch with a longer receptive field as the gating unit for the other branch, dynamically adjusting feature enhancement and performing feature fusion. Finally, the processed image frequency domain information is obtained based on the 1×1 convolution output. The frequency-domain enhancement feedforward module first increases channel dimension through point convolutional layers, enabling the model to learn richer features. Then, parallel dilated convolutions are used to extract features from different scales, expanding the receptive field. Next, the features of both branches are transformed into the frequency domain, and learnable weights and bias variables are used to enhance spectral information. Through a gating mechanism, the feature enhancement level of the other branch is dynamically adjusted based on the perception of a longer range of information by one branch, achieving feature fusion. Finally, the processed image frequency domain information is output through a 1×1 convolution, comprehensively improving the quality of frequency domain information and providing strong support for image denoising.

[0088] Example 3:

[0089] Based on Example 1, long-range spatial dependencies of the image are captured, and a convolutional enhanced feedforward network is used to suppress local pixel forgetting to obtain processed image spatial domain information, including:

[0090] The visual state space module scans two-dimensional features along the four diagonal directions to generate four 1D sequences. Then, it applies discrete state space equations to the 1D sequences in each direction to obtain the results in the four directions.

[0091] The visual state space module processes 2D image features by scanning along the four diagonal directions (from top left to bottom right, from top right to bottom left, from bottom left to top right, and from bottom right to top left), transforming the 2D features into a 1D sequence in four directions. This is done because the discrete state space equations to be applied later are better suited for processing 1D sequences. Then, the discrete state space equations are applied to each 1D sequence. Discrete state space equations can capture long-range dependencies in the sequence. For images, this method captures the long-distance correlation information between pixels at different locations in the image, thus obtaining the processing results in four directions. For example, in an image containing a complex scene, this operation can discover potential connections between distant regions, such as a visual long-range dependency between distant buildings and nearby roads.

[0092] The results from the four directions are combined using a summation operation, and then a reshape operation is used to restore the two-dimensional structure. This summation operation integrates the long-range dependency information captured in the four directions, avoiding focusing on only one direction while ignoring important connections in others. Then, the reshape operation restores the combined one-dimensional data to a two-dimensional structure, matching its dimension to the original image. This step prepares for further processing at the two-dimensional image level, reintegrating the information processed by long-range dependency capture back into the image's two-dimensional space.

[0093] Then, a convolutional enhancement feedforward network is used to enhance the interaction capabilities of local features and suppress channel redundancy, obtaining the processed image spatial domain information. After recovering the feature information of the two-dimensional structure in the previous steps, there are issues such as the forgetting of local pixel information and channel information redundancy during processing. Therefore, a convolutional enhancement feedforward network is used to address these issues. Through a series of convolutional operations, the convolutional enhancement feedforward network can enhance the interaction capabilities between local features, allowing the model to focus more on the connections between local pixels, thereby suppressing the forgetting of local pixels. Simultaneously, it can also filter and integrate channel information, suppressing channel redundancy, making the processed image spatial domain information more refined and effective, providing high-quality spatial domain feature support for image denoising. For example, a local region in an image may contain rich details; the convolutional enhancement feedforward network can better preserve and highlight these details while removing some unnecessary channel information, improving the accuracy of image denoising.

[0094] Example 4:

[0095] Based on Example 1, the dual-domain heterogeneous image denoising method further includes: employing a dual-domain heterogeneous collaborative architecture encoder in the shallow layer of the hierarchical dual-drive encoder-decoder architecture for dual-domain heterogeneous collaborative processing;

[0096] The adjusted number of channels for each layer is calculated based on the feature map size processed at each layer. In the shallow layers of a hierarchical dual-drive encoding / decoding architecture, due to the differences in feature map sizes across different layers, the adjusted number of channels for each layer needs to be determined based on the feature map size processed at each layer to make more efficient use of computational resources. The feature map size includes height and width information. Through a specific calculation method, the number of channels is adjusted by comprehensively considering the relationship between the input image size and the feature map sizes of each layer. This adjustment helps the model to extract and represent features more effectively at different layers, avoiding waste of computational resources or insufficient feature extraction due to an unreasonable number of channels. For example, if the feature map size of a certain layer is small, the number of channels may be appropriately reduced to decrease the computational load; if the feature map size is large, the number of channels can be increased to capture more features.

[0097] The computational balance error of the hierarchical dual-drive codec architecture is calculated based on the adjusted number of channels in all layers and the feature map sizes processed in all layers. Further calculations of the computational balance error of the hierarchical dual-drive codec architecture are then performed based on the previously calculated adjusted number of channels in all layers and the feature map sizes processed in all layers. This error reflects the degree of balance in the allocation of computational resources across the entire architecture. The calculation involves factors such as layer weights and single-channel computational load. These factors are combined with the number of channels and feature map sizes to assess whether the distribution of computational load across layers is reasonable. A large computational balance error indicates an uneven allocation of computational resources across layers, potentially resulting in some layers having excessively high or low computational loads.

[0098] When the computational balance error of the hierarchical dual-drive encoding / decoding architecture exceeds a preset error threshold, a hierarchical channel dynamic allocation strategy is triggered. This indicates that the current channel allocation cannot achieve an ideal balance of computational resources across layers. In this case, the hierarchical channel dynamic allocation strategy needs to be triggered. This strategy readjusts the number of channels in each layer based on information such as the feature map size processed by the corresponding layer, the size of the input image, and the computational balance error, to optimize the allocation of computational resources and improve the overall efficiency and performance of the model. For example, through dynamic allocation, computational resources can be shifted from layers with excessive computational load to layers with insufficient computational load, making the entire architecture more computationally balanced and efficient.

[0099] Example 5:

[0100] Based on Example 4, the dual-domain heterogeneous image denoising method calculates the adjusted number of channels for each layer based on the feature map size processed at each layer, including:

[0101]

[0102] In the formula, C is the number of channels after adjustment in a single layer;

[0103] C base The base number of channels is the starting baseline value for adjusting the number of channels. It is the number of channels initially set for the model and provides a basic framework for feature extraction.

[0104] α is the expansion intensity coefficient, which determines the degree to which the number of channels adjusts according to changes in feature map size. A larger value indicates a greater variation in the number of channels with feature map size, and vice versa.

[0105] `tanh()` is the hyperbolic tangent function, which maps the output of the logarithmic function to a range of -1 to 1. Its purpose is to perform a non-linear transformation on the result of the logarithmic function, making channel number adjustment more flexible and adaptable, avoiding the limitations of simple linear adjustment. Because the complexity of image features is not a linear relationship, the hyperbolic tangent function can better fit this complex variation;

[0106] log2() is the base-2 logarithmic function;

[0107] H is the height of the feature map processed by the corresponding layer, and W is the width of the feature map processed by the corresponding layer.

[0108] H0 is the height of the input image, and W0 is the width of the input image.

[0109] This formula is used in the Mamba+Transformer two-branch image denoising method to accurately calculate the adjusted number of channels for each layer based on the feature map size processed. It allows the model to dynamically adjust the number of channels according to the feature map size, thus allocating computational resources efficiently.

[0110] Example 6:

[0111] Based on Example 4, the dual-domain heterogeneous image denoising method calculates the computational balance error of the hierarchical dual-drive encoding / decoding architecture based on the adjusted number of channels in all layers and the feature map size processed in all layers, including:

[0112]

[0113] In the formula, δ is the computational balance error of the hierarchical dual-drive encoding and decoding architecture, which is a key indicator for measuring whether the computational resources of the entire architecture are balanced.

[0114] n represents the total number of layers in the hierarchical dual-drive encoding and decoding architecture, covering all layers involved in the computation within the architecture;

[0115] γ i The weight of the i-th layer reflects the relative importance of that layer in the entire architecture. The weights of different layers can be set according to their roles in feature extraction, processing, etc.

[0116] C i The number of channels in the i-th layer is adjusted. This is calculated based on the feature map size in Example 5. The number of channels affects the feature processing capability and computational load of this layer.

[0117] FLOPS0 is the computational cost per channel, which is the basic unit for measuring the computational resources required by each channel when processing features.

[0118] H0 is the height of the input image, and W0 is the width of the input image, representing the size information of the input image;

[0119] H i W represents the height of the feature map processed in the i-th layer. i The width of the feature map processed in the i-th layer reflects the size of the feature map in that layer.

[0120] The formula calculates the balance error by comprehensively considering factors related to the computational load of each layer. For each layer, its contribution to the computational load balance error is obtained using its weights, adjusted number of channels, computational load per channel, and the size ratio of the input image to the feature map of that layer. The contributions of all layers are then summed to obtain the computational load balance error of the entire hierarchical dual-drive encoding / decoding architecture. For example, if a layer has a large weight, a large number of channels, and a significant difference between the feature map size and the input image size, then that layer will contribute more to the computational load balance error, potentially indicating an imbalance in computational resource allocation between that layer and other layers, thus increasing the overall computational load balance error.

[0121] Example 7:

[0122] Based on Example 4, the dual-domain heterogeneous image denoising method triggers a hierarchical channel dynamic allocation strategy when the computational balance error of the hierarchical dual-drive encoding / decoding architecture exceeds a preset error threshold, including:

[0123] When the computational balance error of the hierarchical dual-drive encoding and decoding architecture exceeds the preset error threshold, the feature map size of the corresponding layer, the size of the input image, and the computational balance error of the hierarchical dual-drive encoding and decoding architecture are input into the hierarchical channel dynamic allocation model to obtain the final adjusted number of channels for each layer.

[0124] When the calculated computational imbalance error of the hierarchical dual-drive encoding / decoding architecture exceeds a pre-set error threshold, it indicates an imbalance in the allocation of computational resources across the architecture layers, impacting model efficiency. In this case, a hierarchical dynamic channel allocation strategy is triggered. The feature map dimensions (including height and width) for the corresponding layer, the input image dimensions (height and width), and the calculated computational imbalance error are all input into a pre-trained hierarchical dynamic channel allocation model. This model comprehensively analyzes this data, considering the relationship between the feature maps of each layer and the input image dimensions, as well as the degree of current computational imbalance. The model outputs the final adjusted number of channels for each layer. In this way, the number of channels in each layer is rationally adjusted, optimizing the allocation of computational resources across layers, making the model more efficient and improving the overall performance of image denoising.

[0125] Example 8:

[0126] Based on Example 1, the dual-domain heterogeneous image denoising method employs a multi-head transposed attention mechanism and a deep spatial enhancement feedforward network for global context modeling and local feature enhancement, including:

[0127] Multi-head transposed attention is employed to extract Q, K, and V matrices through depthwise separable convolution, and global context modeling is performed. In the dual-domain heterogeneous image denoising method, the multi-head transposed attention mechanism is used to extract Q (query), K (key), and V (value) matrices from image features through depthwise separable convolution. This convolution method effectively reduces computational cost. Then, the multi-head transposed attention mechanism is used to compute attention distributions in parallel across multiple subspaces, allowing the model to comprehensively capture long-distance relationships between different regions of the image, thereby constructing a global context model that reflects the overall layout and relationships of the image. For example, when processing a cityscape image, this can be used to establish a global connection between distant high-rise buildings and nearby streets, enabling the model to understand the entire scene.

[0128] This approach employs a deep spatial augmentation feedforward network and uses depthwise separable convolutions to extract local features, then filters key regions through spatial attention gating. It leverages depthwise separable convolutions to capture local image details, such as textures and small patterns. Subsequently, the spatial attention gating mechanism filters key regions based on their importance to image denoising and feature representation, suppressing unimportant parts. For example, in an image, it highlights key outlines and detailed textures of objects while weakening irrelevant backgrounds, making the model more targeted in its local feature processing, improving the quality of local features, and aiding in image denoising.

[0129] Example 9:

[0130] Based on Example 1, the dual-domain heterogeneous image denoising method optimizes cross-level feature fusion through the synergistic effect of channel attention and vertical enhancement spatial attention in the vertical stripe perception fusion attention mechanism module.

[0131] Channel attention focuses on different channels of a feature. Each channel of an image contains different types of information, such as color and texture. By analyzing the importance of each channel, key channels are given higher weights to highlight important information, suppress redundant channels, and improve denoising and feature representation.

[0132] Vertical enhancement spatial attention focuses on spatial dimensions, especially the vertical direction. By employing specialized methods to enhance sensitivity to vertical stripe features, both noise stripes and important vertical structures can be accurately captured and processed, thus strengthening the role of vertical features in the fusion process.

[0133] These two attention mechanisms work together in the encoder-decoder structure to dynamically adjust the fusion weights for features of different scales and levels of abstraction at different levels. This effectively combines shallow details with deep semantics, optimizes cross-level feature fusion, improves image denoising quality, and makes the processed image better in terms of detail, structure, and overall quality.

[0134] Example 10:

[0135] Based on Example 1, the dual-domain heterogeneous image denoising method, the vertical stripe perception fusion attention mechanism module optimizes cross-level feature fusion through the synergistic effect of channel attention and vertical enhancement spatial attention, including:

[0136] Global average pooling and max pooling are performed on the input features to generate channel weight vectors through two fully connected layers. Global average pooling and max pooling are then applied to the input features to obtain global channel information. These are then processed by two more fully connected layers to generate channel weight vectors, thereby determining the importance of each channel in the feature representation and preparing for channel weighting during cross-level feature fusion.

[0137] A spatial weight matrix is ​​generated by extracting vertical edge responses using the vertical Sobel operator and concatenating them with the horizontal convolution results. This matrix emphasizes vertical features, reflects the spatial location weight distribution of the image, and helps to highlight the importance of specific spatial locations for denoising and feature fusion.

[0138] Simultaneously, the weights of the channel attention branch and the spatial attention branch are dynamically allocated based on the enhancement mechanism until the output is the element-wise product of the channel weight vector and the spatial weight matrix. Based on the enhancement mechanism, the weights of the channel and spatial attention branches are dynamically adjusted, ultimately outputting their element-wise product. In this way, by integrating channel and spatial attention information, the weights during cross-level feature fusion are precisely controlled, optimizing the fusion effect to improve denoising quality.

[0139] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of this invention and its equivalents, this invention also intends to include these modifications and variations.

Claims

1. A dual domain heterogeneous image denoising method, characterized in that, Comprise: S1: extracting a preliminary feature map of an input image based on a depth separable convolution; S2: performing hierarchical processing on the preliminary feature using a hierarchical dual-drive encoding and decoding architecture to obtain multiple layers of features, including: In the shallow layer of the hierarchical dual-drive encoding and decoding architecture, a dual-domain heterogeneous collaborative architecture encoder is used for dual-domain heterogeneous collaborative processing, including: A frequency domain processing branch: using a frequency domain self-attention module to extract Q, K and V through a depth separable convolution, and using a fast Fourier transform to transform spatial domain features into a spectral domain; Classify the frequency domain elements into different frequency bands through spectral band reorganization, and allocate different attention weights between different frequency bands, the attention of low frequency bands is reduced, while the attention of high frequency bands is enhanced, to obtain frequency domain attention features; Convert the frequency domain attention features back to the spatial domain through an inverse fast Fourier transform, and output spatial features through a convolution layer; Using a frequency domain enhancement feedforward module, increase the channel dimension through a point convolution layer, process using parallel hole convolution, then convert the features of the two branches to the frequency domain, and enhance the spectral domain information through learnable weight and bias variables, and through a gating mechanism, use the activation output of one branch with a longer range of receptive field as the gating unit of the other branch, dynamically adjust the feature enhancement and perform feature fusion, and based on the 1x1 convolution output, obtain the processed image frequency domain information; A spatial domain processing branch: the visual state space module scans the two-dimensional features along four diagonal directions to generate four 1D sequences in each direction, and then applies a discrete state space equation to each 1D sequence to obtain four directional results; Merge the four directional results through a summation operation, restore the two-dimensional structure through a reshape operation, and then use a convolution enhancement feedforward network to enhance the interaction ability of local features and suppress channel redundant information to obtain processed image spatial domain information; In the deep layer of the hierarchical dual-drive encoding and decoding architecture, switch to a T Block, and use a multi-head transpose attention mechanism and a deep spatial enhancement feedforward network for global context modeling and local feature enhancement; S3: Concatenate the processed image frequency domain information and image spatial domain information to obtain the hierarchical processed features of each layer; S4: Based on the vertical stripe perception fusion attention mechanism module between the encoder and the decoder, fuse the hierarchical processed features of each layer and output to the decoder to obtain a denoised image.

2. The dual domain heterogeneous image denoising method of claim 1, wherein, In the shallow layer of the hierarchical dual-drive encoding and decoding architecture, a dual-domain heterogeneous collaborative architecture encoder is used for dual-domain heterogeneous collaborative processing, which also includes: Based on the feature map size of each layer of processing, calculate the adjusted channel number of each layer; Based on the adjusted channel number of all layers and the feature map size of all layers, calculate the calculation amount balance error of the hierarchical dual-drive encoding and decoding architecture; When the calculation amount balance error of the hierarchical dual-drive encoding and decoding architecture is greater than a preset error threshold, the hierarchical channel dynamic allocation strategy is triggered.

3. The dual domain heterogeneous image denoising method of claim 2, wherein, Based on the feature map size of each layer of processing, calculate the adjusted channel number of each layer, including: wherein, is the adjusted number of channels for a single layer, is the base number of channels, is the expansion strength coefficient, is the hyperbolic tangent function, is the logarithm function with base 2, is the height of the feature map processed by the corresponding layer, is the width of the feature map processed by the corresponding layer, is the height of the input image, is the width of the input image.

4. The dual domain heterogeneous image denoising method of claim 2, wherein, Based on the adjusted channel number of all layers and the feature map size of all layers, calculate the calculation amount balance error of the hierarchical dual-drive encoding and decoding architecture, including: In the formula, To balance the computational load and error in a hierarchical dual-drive encoding / decoding architecture, This represents the total number of layers in the hierarchical dual-drive encoding / decoding architecture. For the first Layer weights, For the first The number of channels after layer adjustment For single-channel computation, The height of the input image. The width of the input image. For the first The height of the feature map processed by the layer. For the first The width of the feature map processed by the layer.

5. The dual domain heterogeneous image denoising method of claim 2, wherein, When the calculation amount balance error of the hierarchical dual-drive coding architecture is greater than a preset error threshold, a hierarchical channel dynamic allocation strategy is triggered, including: When the calculation amount balance error of the hierarchical dual-drive coding architecture is greater than a preset error threshold, the feature map size corresponding to layer processing, the size of the input image, and the calculation amount balance error of the hierarchical dual-drive coding architecture are input to a hierarchical channel dynamic allocation model, and the final adjusted channel number of each layer is obtained.

6. The dual domain heterogeneous image denoising method of claim 1, wherein, Global context modeling and local feature enhancement are performed using a multi-head transpose attention mechanism and a deep spatial enhancement feedforward network, including: A multi-head transpose attention is used to extract Q, K, and V matrices through a depth separable convolution, and global context modeling is performed; A deep spatial enhancement feedforward network is used to extract local features using a depth separable convolution, and a spatial attention gate is used to filter key regions.

7. The dual domain heterogeneous image denoising method of claim 1, wherein, The vertical stripe perception fusion attention mechanism module optimizes cross-level feature fusion through the synergistic effect of channel attention and vertically enhanced spatial attention.

8. The dual domain heterogeneous image denoising method of claim 7, wherein, The vertical stripe perception fusion attention mechanism module optimizes cross-level feature fusion through the synergistic effect of channel attention and vertically enhanced spatial attention, including: Global average pooling and maximum pooling are performed on the input features, and a channel weight vector is generated through two fully connected layers; A vertical Sobel operator is used to extract vertical edge responses, and a spatial weight matrix is generated after concatenating the horizontal convolution results; At the same time, the weights of the channel attention branch and the spatial attention branch are dynamically allocated based on the enhancement mechanism, until the output is the element-wise product of the channel weight vector and the spatial weight matrix.

Images