Image denoising method based on double-branch feature fusion

Abstract

The invention discloses an image denoising method based on double-branch feature fusion, and the method specifically comprises the steps: 1, carrying out the collection of an image, and carrying out the preprocessing of the collected image; 2, a denoising network with double-branch feature fusion is built, the network comprises a generation network and a discrimination network, and authenticity discrimination is carried out on a denoised image generated by the generation network and a real image through the discrimination network; step 3, adversarial training is carried out on the generative network and the discriminant network to obtain a trained dual-branch feature fusion denoising network; and step 4, performing denoising processing on the image by using the trained generative network. The method is remarkably superior to a traditional denoising method in denoising quality, model generalization ability and operation time, different types of noise can be effectively removed, and more image details and structure information are reserved.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to an image denoising method based on double-branch feature fusion. Background Art

[0002] With the continuous advancement of medical imaging technology and the popularization of deep learning technology, cardiac magnetic resonance imaging (MRI) plays an increasingly significant role in medical diagnosis and treatment. However, cardiac images are often interfered by various noises during the acquisition process, which affects the image quality and the accuracy of diagnosis. At present, cardiac image denoising mainly relies on traditional methods such as filtering and signal processing. Although these methods are effective, it is difficult to take into account both the details and the overall quality of the image when dealing with complex noise patterns, and the results are often unsatisfactory.

[0003] In recent years, with the development of deep learning technology, especially the emergence of generative adversarial networks (GANs), a new solution for cardiac image denoising has been provided. Through adversarial training of the generator and the discriminator, high-quality images that are very similar to real images can be generated while effectively removing noise.

[0004] However, these methods also have some problems, such as image smoothing and loss of details during the denoising process, feature loss when the image is scaled, low model reuse rate, and high computing power required for denoising. These problems have become the key to how to perform higher quality image denoising and how to perform image denoising in real time. Summary of the invention

[0005] Purpose of the invention: In order to solve the problems existing in the above-mentioned prior art, the present invention provides an image denoising method based on dual-branch feature fusion.

[0006] Technical solution: The present invention provides an image denoising method based on dual-branch feature fusion, specifically:

[0007] Step 1: Collect images and preprocess the collected images;

[0008] Step 2: Build a denoising network with dual-branch feature fusion, which includes a generating network and a discriminant network. The generating network includes an initial downsampling module, a dual-branch encoder, a fusion module, a bottleneck layer and a dual-branch decoder. The initial downsampling module downsamples the preprocessed image to obtain an initial image, which is then sent to the dual-branch encoder; the dual-branch encoder includes two encoders, and the dual-branch decoder includes two decoders. The dual-branch encoders encode the initial image respectively and input them into the fusion module to obtain fused features. The fused features are input into the bottleneck layer to reduce the size of the feature map, and the output of the bottleneck layer is input into the dual-branch decoder. The dual-branch decoder is jump-connected with the dual-branch encoder, and the size of the feature map is enlarged layer by layer while restoring the detail information lost in the downsampling process, thereby completing the reconstruction of the denoised image; then the denoised image generated by the generating network and the real image are distinguished by the discriminant network for authenticity;

[0009] Step 3: Perform adversarial training on the denoising network to obtain a trained dual-branch feature fusion denoising network;

[0010] Step 4: Use the trained denoising network to denoise the image.

[0011] Furthermore, the dual-branch encoder includes a feedforward encoder and a standard encoder, the feedforward encoder includes first to third downsampling modules connected in sequence, the downsampling module includes a DWDC module and an FFD module connected to each other, the DWDC module includes a wavelet transform module and a hole convolution connected to the wavelet transform module, and the standard encoder includes first to third DWDC modules connected in sequence.

[0012] Furthermore, the wavelet transform module includes a low-pass filter and a high-pass filter. The wavelet transform module decomposes the input image into four parts, splices the four parts, and then performs a Sequential operation on the spliced ​​result; the wavelet transform module decomposes the input image into four parts, specifically: applying a low-pass filter to the input image in both the horizontal and vertical directions, applying a low-pass filter to the input image in the horizontal direction, and applying a high-pass filter in the vertical direction; applying a high-pass filter to the input image in the horizontal direction, and applying a low-pass filter in the vertical direction; applying a high-pass filter to the input image in the horizontal direction and applying a low-pass filter in the vertical direction; applying a high-pass filter to the input image in both the horizontal and vertical directions.

[0013] Furthermore, the dual-branch decoder includes a feedforward multi-scale decoder and a standard multi-scale decoder; the feedforward multi-scale decoder includes first to fourth upsampling modules connected in sequence, each upsampling module includes a deconvolution module and an FFD module connected to each other, the FFD modules in the first, second and third upsampling modules correspond to the FFD modules in the third, second and first downsampling modules one by one, the two corresponding FFD modules are jump-connected, and the FFD module in the fourth upsampling module is jump-connected to the initial image;

[0014] The standard multi-scale decoder includes first to fourth deconvolution modules, the first, second and third deconvolution modules correspond to the third, second and first DWDC modules in the standard encoder one by one, the corresponding modules are jump-connected, and the fourth deconvolution module is jump-connected to the initial image.

[0015] Furthermore, the fusion module concatenates the output feature x of the feedforward encoder and the output feature y of the standard encoder in the channel dimension to obtain a feature image Feature, calculates the mean and maximum value of the Feature along the channel dimension to obtain two tensors, then merges the two tensors along the channel dimension, and performs convolution processing on the merged tensor to obtain feature S; performs a global average pooling operation on Feature, and then obtains feature C through two convolution layers and a ReLU activation function; then adds feature S and feature C to obtain feature pattn1; Feature and pattn1 are first expanded in dimension, and then connected in the channel dimension, and after convolution and Sigmod activation function processing, a pixel-level attention map pattn2 is generated, and then pattn2 is processed according to the following formula to obtain the result Result output by the fusion module:

[0016] Result=conv2d[Feature+(x*pattn2)+(y*(1-pattn2))]

[0017] Among them, conv2d represents Conv2d convolution.

[0018] Furthermore, the method also includes performing a convolution operation on the denoised image generated by the generative network before sending the denoised image generated by the generative network and the real image into the discriminator, then upsampling the result after the convolution operation, and combining the upsampled result with the real image and sending them into the discriminator.

[0019] Furthermore, the PointNet++ neural network is used to upsample the denoised image after the convolution operation.

[0020] Furthermore, the discriminator includes a CNN module and a PCDNet network; the initial layer of the PCDNet network performs convolution, batch normalization and LeakyReLU activation function operations on the input features in sequence, inputs the outputs of the initial layer into several CNN modules respectively, combines the outputs of the CNN modules, and the PCDNet network performs 3×3 convolution, LeakyReLU activation function, 7×7 convolution, LeakyReLU activation function, 5×5 convolution and 3×3 convolution operations on the combined features in sequence, inputs the obtained feature image into the Dropout layer, randomly discards neurons with a preset probability, inputs the output of the Dropout layer into the Sigmoid activation function layer, splices the output of the Sigmoid activation function layer and the output features of the combined CNN module along the channel direction, then performs a partial convolution operation on the spliced ​​features, and finally performs a convolution operation on the features after the partial convolution operation.

[0021] Furthermore, the partial convolution divides the input image into two parts in the channel dimension: x1 and x2, performs a convolution operation on x1, and connects x2 and the convolved x1 in the channel dimension.

[0022] Furthermore, in the initial downsampling module, a Conv2d convolution layer and a BatchNorm2d pooling layer are first used, and then a LeakyReLU activation function is used to introduce nonlinear characteristics.

[0023] Beneficial effects: The present invention combines the generative adversarial network (GAN) based on dual-branch feature fusion and the attention mechanism. The fusion of these technologies not only significantly improves the denoising effect, but also greatly enhances the ability to capture and identify structures of different scales in the image. In particular, for areas with rich details and dynamic changes in cardiac images, this method can more accurately retain key anatomical and physiological information.

[0024] The present invention effectively improves the feature extraction capability of the model by introducing the attention mechanism in the generator and discriminator networks, while avoiding the loss of edge information. Regularization technology and data enhancement and other optimization measures are adopted to reduce the problem of model overfitting, realize real-time denoising, and improve processing efficiency and consistency. Experimental results show that this method is significantly superior to traditional denoising methods in denoising quality, model generalization ability and operation time, and can effectively remove different types of noise and retain more image details and structural information.

[0025] The present invention can not only effectively remove noise from images, but also retain key structural and physiological information, providing more reliable image data support for the diagnosis and treatment of heart disease. In terms of economic benefits, clear heart images can help doctors detect heart disease earlier, thereby formulating more effective treatment plans and reducing the cost of complex operations and patient health risks caused by delayed treatment. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is an overall block diagram of the present invention.

[0027] Figure 2 This is the structure diagram of the initial downsampling module;

[0028] Figure 3 It is the structural diagram of the feedforward encoder module;

[0029] Figure 4 This is the structural diagram of the fusion module;

[0030] Figure 5 This is the structure diagram of the standard multi-scale decoder;

[0031] Figure 6 It is the structure diagram of the discriminator;

[0032] Figure 7 The figure is a comparison chart of the results obtained by the invention of the present invention and the traditional method. DETAILED DESCRIPTION

[0033] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0034] The main steps of the technical solution of the present invention are as follows:

[0035] Step 1: Dataset collection and image preprocessing. This embodiment obtains MICCAI'17 heart clear images (test set) and noisy data sets (training set), including normal, systolic heart failure with infarction, dilated cardiomyopathy, hypertrophic cardiomyopathy, right ventricular abnormalities and other data, and selects relevant heart images in the data sets.

[0036] Step 2: Build a dual-branch feature fusion image denoising network, which mainly includes two parts: a generation network composed of a feedforward branch and a standard branch, and a PCDNet discriminant network.

[0037] Step 3: Input the cardiac noise image obtained in step 1 into the generative adversarial network in step 2 for training, calculate the loss function and update the network parameters to obtain the optimal parameter model.

[0038] Step 4: Send the noisy data in step 1 to the best trained model and output a noise-free heart image.

[0039] The steps of step 1 data set collection and image preprocessing are:

[0040] Obtain the MICCAI'17 cardiac image dataset and divide it into a test set and a training set according to the clarity and blur of the image. The file format of this dataset is .nii, and the size of each image is about 256×256 pixels. Convert the .nii format data to .jpg image format and name the images in the order of 1, 2, 3, 4, etc.

[0041] The second step is to build a dual-branch feature fusion cardiac image denoising network, including the generator network and the discriminator network. The steps are as follows:

[0042] Step 2.1: First, the heart image affected by noise is preprocessed and then input into the feedforward branch and the standard branch respectively. The feedforward branch strengthens the feature transfer capability and allows the noise and signal to be processed separately at different decomposition levels, expanding the receptive field of feature extraction and making full use of the contextual information of the heart image; the standard branch only uses the DWDC module and the standard convolution block for processing, reducing the problem of detail loss in the feature transfer process and retaining more of the original features of the image.

[0043] Step 2.2: Next, the feature maps generated by the feedforward encoder and the standard encoder are fused through channels. The fusion module includes channel attention mechanism, spatial attention mechanism and pixel attention mechanism. The pixel attention mechanism combines the output of spatial attention and channel attention with the original input feature map and adjusts the importance of each pixel through a convolution operation. This allows the model to dynamically adjust the feature response at the pixel level, enhancing the ability to express features.

[0044] Step 2.3: After the feature fusion, a set of high-level abstract feature maps are obtained. In order to further reduce the spatial size of the feature map while keeping the number of channels unchanged, a bottleneck layer is introduced to halve the size of the feature map.

[0045] Step 2.4: Then, the feature maps in step 2.3 are respectively input into the feedforward multi-scale decoder and the standard multi-scale decoder for reconstruction and restoration. The reconstruction and restoration capability of the damaged heart image is improved by using deconvolution kernels of different sizes. The spatial resolution of the image is gradually restored by enlarging the size of the feature map layer by layer, and the skip connection is used to restore the detail information that may be lost during the downsampling process.

[0046] Step 2.5: Finally, the denoised heart image generated by the generative network and the real heart image are distinguished by the PCDNet network, and the reconstruction loss, adversarial loss and style loss are used to jointly train the generative network and the discriminative network to improve the heart denoising effect.

[0047] The process of calculating the loss function and updating the network parameters in step 3 to obtain the optimal parameter model is as follows:

[0048] Initialize the generator and discriminator models, feed the training data into the training loop, train the generator and discriminator, use the Adam optimizer to optimize the training process, use the loss function to calculate the difference, use the mean square error loss (MSE_LOSS) to measure the difference between the output of the discriminator and the true label, and use the improved VGG loss (VGG_LOSS19) to measure the difference between the image generated by the generator and the target image.

[0049] The feedforward encoder and standard encoder network structures are as follows:

[0050] The feedforward encoder branch is composed of wavelet transform, dilated convolution and feedforward propagation module FFD. Wavelet transform and dilated convolution constitute the DWDC module. The feedforward encoder includes three consecutive DWDC modules + FFD, which uses wavelet transform to provide good time-frequency localization characteristics. At the same time, dilated convolution is used to expand the receptive field of the convolution kernel to improve the perception of contextual semantic features of cardiac images. The feedforward propagation module FFD further processes features through a deep separable convolution. The standard encoder branch includes three DWDC modules, which achieve maximum information transmission through brief and effective processing, improve information utilization, and reduce the loss of cardiac image detail information.

[0051] Then, the feature maps obtained by the feedforward gated encoder branch and the standard encoder branch are channel fused. The dual-branch encoding method is used to retain the original detail feature information of the cardiac image while paying better attention to the important semantic feature information of the image, significantly improving the feature perception ability of the cardiac image.

[0052] The fusion module structure is as follows:

[0053] First, the module receives the input x of the feedforward encoder and the input y of the standard encoder, and the calculation formula is as follows:

[0054] Feature=x+y

[0055] Then, the features are input into the spatial attention and channel attention mechanisms respectively. The detailed processes are as follows:

[0056] SA: Calculate the mean and maximum value of the feature along the channel dimension (dim = 1) to obtain two [1, 1, 64, 64] tensors, and then merge these two tensors along the channel dimension to obtain [1, 2, 64, 64]. Use a convolutional layer to process the merged tensor, and the output is called S, with a size of [1, 1, 64, 64].

[0057] CA: Use global average pooling to reduce the input to [1, 64, 1, 1], pass through a sequence of two convolutional layers and an activation function, first reduce the number of channels from 64 to 8, and then increase it back to 64, the output is called C, and the size is [1, 64, 1, 1].

[0058] Next, S and C are added to generate pattn1. Then, in the pixel attention mechanism (PixelAttention), Feature and pattn1 are first expanded in dimension, and then connected in channel dimension. After convolution and activation function processing, a pixel-level attention map is generated. Use 2 pairs of sums for weighted fusion, add the result to Feature, and then process the fused result through a convolution layer to obtain the final result Result:

[0059] Result=conv2d[Feature+(x*pattn2)+(y*1-pattn2)]

[0060] The final output tensor has the same dimensions as the input sum.

[0061] The structure of the dual-branch multi-scale decoder is as follows:

[0062] After the bottleneck layer, four consecutive multi-scale upsampling are used to achieve the decoding of the feature map. Like the dual-branch encoder, this paper adopts the corresponding dual-branch multi-scale decoder to decode the image after feature fusion respectively, which is divided into a feedforward multi-scale decoder and a standard multi-scale decoder. The feedforward multi-scale decoder contains an FFD module and a deconvolution module, which enables better matching of the feature map from the feedforward encoder branch. The two share the same module, which consists of a deconvolution layer, a batch normalization layer, and an activation function. The deconvolution layer achieves upsampling of the feature map through convolution operations and restores the spatial size of the image. Through gradual upsampling and feature synthesis, these upsampling layers can effectively generate realistic images, thereby completing the task of image reconstruction. The standard multi-scale decoder includes four consecutive deconvolution modules.

[0063] After continuous multi-scale decoding, a set of synthetic feature maps are obtained. In order to obtain the final generated image, the decoder uses the deconvolution layer to upsample, thereby restoring the spatial size of the image. The output of the final upsampling layer passes through the activation function to scale the pixel values ​​to the range of [-1,1] to obtain the generated image.

[0064] After the generated image undergoes a 3*3 convolution operation, it is upsampled using the PointNet++ neural network. The upsampled result and the real image are concatenated and sent to the discriminator.

[0065] The structure of the discriminator is as follows:

[0066] The initial layer of the discriminator receives the input image and includes a convolution layer and an activation function. The convolution layer uses a 4*4 convolution kernel with a stride of 2 to reduce the spatial size of the input image. The activation function helps introduce nonlinearity so that the discriminator can handle different types of images. By performing a 3*3 convolution operation on the input feature map and combining it with the activation function, the noise level in the image can be effectively evaluated. The nonlinear nature of the activation function helps the network learn feature representation better. At the same time, a layer of partial convolution operation is added after the activation function to divide the input feature map into two parts in the channel dimension: x1 and x2, apply convolution to x1, and then concatenate the convolved x1 and the unprocessed x2 in the channel dimension to return the result.

[0067] The partial convolution module of the present invention implements partial convolution through segmentation and connection operations. This partial convolution technology can increase the model's processing ability for different parts of the input features while maintaining the model capacity and computational efficiency. This convolution operation provides an important reference for subsequent processing and helps improve the accuracy and stability of image processing. Therefore, noise assessment is not only a key step in neural networks, but also an important link in optimizing image processing effects.

[0068] Multi-scale denoising is achieved through two convolutional layers, which can reduce the size of image features and eliminate noise. In order to prevent overfitting, a layer is introduced to reduce the model's dependence on specific features. In this way, the image quality and effect can be effectively improved.

[0069] The present embodiment is described in detail with reference to the modules in the accompanying drawings:

[0070] The overall structure of the present invention is as follows Figure 1As shown in the figure, first, the original size of the recorded image is [1,3,256,256]. The cardiac MRI image affected by noise is first initially downsampled, specifically after the Conv2d standard convolution and BatchNorm2d standard pooling, and then after the LeakyReLU activation function, the scale is transformed to [1, 32, 128, 128], and then input into the feedforward encoder and the standard encoder respectively. The feedforward encoder introduces wavelet transform, dilated convolution, and FFD modules, which enhances the feature transfer capability and allows noise and signal to be processed separately at different decomposition levels, expands the receptive field range of feature extraction, and makes full use of the context information of the cardiac image; the standard encoder only uses the DWDC module (wavelet transform + dilated convolution) and the standard convolution block for processing, which reduces the problem of detail loss in the feature transfer process and retains more of the original features of the image. Then, the feature maps generated by the dual-branch encoders are channel fused. In this step, a pixel attention mechanism is specially proposed through the fusion module (TAFusion), which combines the output of spatial attention and channel attention with the original input feature map and adjusts the importance of each pixel through a convolution operation. This method allows the model to dynamically adjust the feature response at the pixel level, further enhancing the ability of feature expression. Then, the fused dual-branch feature map is input into the feedforward multi-scale decoder and the standard multi-scale decoder for reconstruction and restoration. The decoder uses deconvolution kernels of different scales to improve the reconstruction and repair ability of damaged heart images. This dual-branch multi-scale decoder design gradually restores the spatial resolution of the image by enlarging the size of the feature map layer by layer, and uses jump connections to restore the detail information that may be lost during the downsampling process. Finally, the denoised heart image generated by the generative network and the real heart image are distinguished by the PCDNet network, and the reconstruction loss, adversarial loss and style loss are used to jointly train the generative network and the discriminative network to improve the heart denoising effect.

[0071] The initial downsampling structure is shown in FIG. Figure 2 As shown in the figure: In the initial downsampling layer, a Conv2d convolution layer and a BatchNorm2d pooling layer are used, and the LeakyReLU activation function is used to introduce nonlinear characteristics, so that the model can learn complex mapping relationships and better extract image features.

[0072] The structure diagram of the feedforward encoder is as follows Figure 3As shown: The feedforward encoder branch is mainly composed of the DWDC module and the FFD module. The DWDC module uses wavelet transform and dilated convolution to perform three consecutive downsamplings. Unlike the Fourier transform, the wavelet transform not only focuses on frequency information, but also can determine the position of these frequencies in time (or space). First, the input image is subjected to the hear wavelet transform, which involves two filters, namely the low-pass filter (L): used to calculate the approximate coefficients (low-frequency part), the core of which is the average operation; the high-pass filter (H), the core of which is the difference operation. For a two-dimensional image, the wavelet transform first applies these filters in each direction, and then downsamples the image to decompose the image into four parts:

[0073] LL (Low-Low): The result of applying a low-pass filter in both directions.

[0074] LH (Low Frequency-High Frequency): The result of applying a low-pass filter horizontally and a high-pass filter vertically.

[0075] HL (High-Low): The result of applying a high-pass filter horizontally and a low-pass filter vertically.

[0076] HH (High-High): The result of applying a high-pass filter in both directions

[0077] The specific formula is as follows:

[0078] yL=dowmsample(L*x*L T )

[0079] yH[0]=dowmsample(L*x*H T )

[0080] yH[1]=dowmsample(H*x*L T )

[0081] yH[2]=dowmsample(H*x*H T )

[0082] Where * represents the convolution operation, L and H are low-pass and high-pass filters respectively, T represents the transpose operation for vertical convolution, and downsample is the downsampling operation.

[0083] At this point, the processed image size is [1, 64, 64, 64]. It then goes through a layer of Sequential operation, which includes a convolution layer, batch normalization, and ReLU function, to keep the size of [1, 64, 64, 64]. The image after Sequential is then input to the FFD layer through a dilated convolution.

[0084] The FFD layer receives the input feature map X and undergoes three consecutive convolution operations:

[0085] The first layer is an expansion of the feature dimension, with 64 input channels and 256 output channels. After a 1×1 convolution kernel with a step size of 1 and 0 padding, the output Y1 is calculated to have a size of [1, 256, 64, 64]. The formula is as follows:

[0086] Y1=ReLU(W1*X+b1)

[0087] Among them, W1 is the convolution kernel weight and b1 is the bias term.

[0088] The second layer is grouped convolution, with 256 input channels and 256 output channels. The 3×3 convolution kernel is padded with a step size of 1 and the number of groups is 64 (each group processes four channels). After calculation, the output Y2 is obtained. The formula is as follows:

[0089] Y2=ReLU(W2*Y1+b2)

[0090] Among them, W2 is the convolution kernel weight, b2 is the bias term, the convolution operation of each group is performed independently, and the scale is maintained at [1, 256, 64, 64].

[0091] The third layer is feature dimension compression, with 256 input channels and 64 output channels. After a 1×1 convolution kernel with a step size of 1 and 0 padding, the output Y3 is calculated. The formula is as follows:

[0092] Y3=ReLU(W3*Y2+b3)

[0093] Among them, W3 is the convolution kernel weight, b3 is the bias term, and the scale becomes [1, 64, 64, 64].

[0094] The features passed through the FFD module enhance the expressiveness by expanding and compressing the feature dimensions, while reducing the number of parameters and computational complexity through grouped convolution, maintaining the invariance of the spatial size.

[0095] The output feature map is then input as y into the fusion module for feature fusion.

[0096] The standard encoder branch only contains the DWDC module, which maintains the original features of the heart image to a greater extent while performing scale transformation, which is conducive to retaining more details when performing feature fusion. The feature map after the standard encoder branch will be used as input to the CGA module.

[0097] The fusion module structure diagram is as follows Figure 4 As shown: First, the fusion module receives the input x of the feedforward encoder branch and the input y of the standard encoder branch, and calculates the feature:

[0098] Feature = x + y;

[0099] Then, the features are input into the spatial attention and channel attention mechanisms respectively. The detailed processes are as follows:

[0100] SA: Calculate the mean and maximum value of the feature along the channel dimension (dim = 1) to obtain two [1, 1, 64, 64] tensors, and then merge these two tensors along the channel dimension to obtain [1, 2, 64, 64]. Use a convolutional layer to process the merged tensor, and the output is called S, with a size of [1, 1, 64, 64].

[0101] CA: Use global average pooling AdAvgPool to reduce the feature to [1,64,1,1]. Through a sequence of two convolutional layers and a ReLU activation function, the number of channels is first reduced from 64 to 8 and then increased back to 64. The output is called C and its size is [1,64,1,1].

[0102] Next, S and C are added to generate pattn1. Then, in the pixel attention mechanism (PixelAttention), Feature and pattn1 are first expanded in dimension, and then connected in channel dimension. After convolution and Sigmod activation function processing, pixel-level attention map pattn2 is generated. Use pattn2 to perform weighted fusion of x and y, add the result to Feature, and then process the fused result through a convolution layer to obtain the final result Result:

[0103] Result=conv2d[Feature+(x*pattn2)+(y*1-pattn2)]

[0104] The final output tensor Result has the same dimensions as the input x and y.

[0105] In this embodiment, after three consecutive layers of DWDC and FFD in the encoder, a set of high-level abstract feature maps are obtained, and the image size is [1, 256, 16, 16]. In order to further reduce the spatial size of the feature map while keeping the number of channels unchanged, a bottleneck layer is introduced after the fusion module. In the bottleneck layer, a Conv2d convolution layer is used, which is used to halve the size of the feature map to [1, 256, 8, 8]. Through the processing of the bottleneck layer, high-level features of the image can be further extracted to prepare for the subsequent dual-branch multi-scale decoder.

[0106] Like the dual-branch encoder, the corresponding dual-branch multi-scale decoder is used to decode the image after feature fusion respectively, which is divided into a feedforward multi-scale decoder and a standard multi-scale decoder. The feedforward multi-scale decoder contains a feedforward propagation module FFD, which enables better matching of feature maps from the feedforward encoder branch. The two share the same Seq module, which consists of a deconvolution layer, a batch normalization layer, and a ReLU activation function to ensure that the image size output is [1, 256, 16, 16]. The deconvolution layer implements upsampling of the feature map through convolution operations and restores the spatial size of the image. Through gradual upsampling and feature synthesis, these upsampling layers can effectively generate realistic images, thereby completing the task of image reconstruction. After three layers of upsampling, the image size is restored to [1, 3, 256, 256]. The structure of the standard multi-scale decoder is as follows Figure 5 As shown in FIG. 4 , it includes four consecutive deconvolution modules, and the feedforward multi-scale decoder is jump-connected with the feedforward encoder and the initial image, and the standard multi-scale decoder is jump-connected with the standard encoder and the initial image.

[0107] After the generated image undergoes a 3*3 convolution operation, it is upsampled using the PointNet++ neural network. The upsampled result and the real image are concatenated and sent to the discriminator.

[0108] like Figure 6As shown in the figure, the initial layer of the PCDNet discriminator receives the input image and includes a convolution layer and a LeakyReLU activation function. The convolution layer uses a 4x4 convolution kernel with a step size of 2 to reduce the spatial size of the input image to [1, 256, 27, 27]. Then the CNN module is introduced. The main function of the CNN block is to further extract the features of the image, which helps the discriminator to more accurately distinguish the real image from the generated image. The batch normalization layer normalizes the input of each channel, which helps to speed up the convergence speed and improve the stability of the model. The LeakyReLU activation function introduces a certain degree of nonlinearity, which enables the discriminator to better handle complex image data. Through the combination of these CNN blocks, the discriminator can more effectively learn and extract the features of the image, thereby accurately distinguishing the real image from the generated image. The image size changes in the process are [1,64,63,63], [1,128,30,30], [1,256,27,27]. The outputs of all CCN blocks are combined, and then convolution operations are performed using a convolution kernel of size 3×3, a stride of 1, and a padding of 1. At this time, the image size is [1, 256, 27, 27]. The noise level in the input image is evaluated, and then multiple convolutional layers are used for multi-scale denoising operations. The convolution kernel sizes of the convolutional layers are 7×7, 5×5, and 3×3, respectively, to further reduce the image feature size. A Dropout layer is then introduced to randomly drop neurons with a certain probability (here 0.5) during the training process. This helps prevent overfitting and improves the generalization ability of the model on unseen data. The final activation operation is performed through the Sigmoid activation function. The Sigmoid function maps the output of the model to the range of (0,1), indicating the probability that the generated image is a real image. The Sigmoid function fuses the output of the model with the output of the CCN block so that its inputs are spliced ​​together along the channel direction. Then, add another layer of partial convolution operation to divide the input feature map x into two parts in the channel dimension: x1 and x2, apply convolution to x1, and then connect the convolved x1 and the unprocessed x2 in the channel dimension and then connect a 4*4 convolution, and then return the result.

[0109] The present invention implements partial convolution through segmentation and connection operations. This partial convolution technique can increase the model's processing capability for different parts of the input features while maintaining the model capacity and computational efficiency. This convolution operation provides an important reference for subsequent processing and helps improve the accuracy and stability of image processing.

[0110] The results of the method of the present invention (DHGAN) are compared with those of the traditional denoising method. Figure 7 As shown by Figure 7It can be seen that the present invention is significantly superior to traditional denoising methods in terms of denoising quality, model generalization ability and computing time. It can effectively remove different types of noise and retain more image details and structural information.

Claims

1. An image denoising method based on dual-branch feature fusion, characterized in that: The specific steps include: Step 1: Collect images and preprocess the collected images; Step 2: Build a dual-branch feature fusion denoising network, which includes a generation network and a discriminant network. The generation network includes an initial downsampling module, a dual-branch encoder, a fusion module, a bottleneck layer and a dual-branch decoder. The initial downsampling module downsamples the preprocessed image to obtain the initial image, which is then sent to the dual-branch encoder; The dual-branch encoder includes two encoders, and the dual-branch decoder includes two decoders. The dual-branch encoders encode the initial image respectively, and input the encoded image into a fusion module to obtain fused features. The fused features are input into a bottleneck layer to reduce the size of the feature map, and the output of the bottleneck layer is input into a dual-branch decoder. The dual-branch decoder is jump-connected with the dual-branch encoder to enlarge the size of the feature map layer by layer while restoring the detail information lost in the downsampling process, thereby completing the reconstruction of the denoised image; and then the denoised image generated by the generation network and the real image are distinguished by the discriminant network for authenticity. Step 3: Perform adversarial training on the denoising network to obtain a trained dual-branch feature fusion denoising network; Step 4: Use the trained denoising network to denoise the image.

2. The image denoising method based on dual-branch feature fusion according to claim 1, characterized in that: The dual-branch encoder includes a feedforward encoder and a standard encoder, the feedforward encoder includes first to third down-sampling modules connected in sequence, the down-sampling modules include a DWDC module and an FFD module connected to each other, the DWDC module includes a wavelet transform module and a hole convolution connected to the wavelet transform module, and the standard encoder includes first to third DWDC modules connected in sequence.

3. The image denoising method based on dual-branch feature fusion according to claim 2 is characterized in that: The wavelet transform module includes a low-pass filter and a high-pass filter. The wavelet transform module decomposes the input image into four parts, splices the four parts, and then performs a Sequential operation on the spliced ​​result; the wavelet transform module decomposes the input image into four parts, specifically: applying a low-pass filter to the input image in both the horizontal and vertical directions, applying a low-pass filter to the input image in the horizontal direction and a high-pass filter in the vertical direction; applying a high-pass filter to the input image in the horizontal direction and a low-pass filter in the vertical direction; applying a high-pass filter to the input image in the horizontal direction and a low-pass filter in the vertical direction; applying a high-pass filter to the input image in both the horizontal and vertical directions.

4. The image denoising method based on dual-branch feature fusion according to claim 2 is characterized in that: The dual-branch decoder includes a feedforward multi-scale decoder and a standard multi-scale decoder; the feedforward multi-scale decoder includes first to fourth up-sampling modules connected in sequence, each up-sampling module includes a deconvolution module and an FFD module connected to each other, the FFD modules in the first, second and third up-sampling modules correspond to the FFD modules in the third, second and first down-sampling modules one by one, two corresponding FFD modules are jump-connected, and the FFD module in the fourth up-sampling module is jump-connected to the initial image; The standard multi-scale decoder includes first to fourth deconvolution modules, the first, second and third deconvolution modules correspond to the third, second and first DWDC modules in the standard encoder one by one, the corresponding modules are jump-connected, and the fourth deconvolution module is jump-connected to the initial image.

5. The image denoising method based on dual-branch feature fusion according to claim 1, characterized in that: The fusion module concatenates the output feature x of the feedforward encoder and the output feature y of the standard encoder in the channel dimension to obtain a feature image Feature, calculates the mean and maximum value of the Feature along the channel dimension to obtain two tensors, then merges the two tensors along the channel dimension, and performs convolution processing on the merged tensor to obtain a feature S; Perform a global average pooling operation on Feature, and then obtain feature C through two convolution layers and a ReLU activation function; then add feature S and feature C to obtain feature pattn1; Feature and pattn1 are first expanded in dimension, and then connected in channel dimension. After convolution and Sigmod activation function processing, pixel-level attention map pattn2 is generated, and then pattn2 is processed according to the following formula to obtain the result output by the fusion module: Result=conv2d[Feature+(x*pattn2)+(y*(1-pattn2))] Among them, conv2d represents Conv2d convolution.

6. The image denoising method based on dual-branch feature fusion according to claim 1, characterized in that: The method also includes performing a convolution operation on the denoised image generated by the generative network before sending the denoised image generated by the generative network and the real image to the discriminator, then upsampling the result after the convolution operation, and combining the upsampling result with the real image and sending them to the discriminator.

7. The image denoising method based on dual-branch feature fusion according to claim 6, characterized in that: As described above, the PointNet++ neural network is used to upsample the denoised image after the convolution operation.

8. The image denoising method based on dual-branch feature fusion according to claim 1, characterized in that: The discriminator includes a CNN module and a PCDNet network; the initial layer of the PCDNet network sequentially performs convolution, batch normalization and LeakyReLU activation function operations on the input features, and the outputs of the initial layer are respectively input into several CNN modules, and the outputs of the CNN modules are combined. The PCDNet network sequentially performs 3×3 convolution, LeakyReLU activation function, 7×7 convolution, LeakyReLU activation function, 5×5 convolution and 3×3 convolution operations on the combined features, and the obtained feature image is input into the Dropout layer, and neurons are randomly discarded with a preset probability, and the output of the Dropout layer is input into the Sigmoid activation function layer, and the output features of the Sigmoid activation function layer and the combined CNN module are spliced ​​along the channel direction, and then the spliced ​​features are partially convolved, and finally the features after the partial convolution operation are convolved.

9. The image denoising method based on dual-branch feature fusion according to claim 8, characterized in that: The partial convolution divides the input image into two parts in the channel dimension: x1 and x2, performs a convolution operation on x1, and connects x2 and the convolved x1 in the channel dimension.

10. The image denoising method based on dual-branch feature fusion according to claim 1, characterized in that: The initial downsampling module first uses a Conv2d convolution layer and a BatchNorm2d pooling layer, and then uses the LeakyReLU activation function to introduce nonlinear characteristics.

Images