A method and system for blind image denoising based on enhanced Transformer

Abstract

The present invention discloses a method and system for blind image denoising based on an enhanced Transformer. The method combines a dynamic convolutional layer with an enhanced Transformer module to adaptively fuse features from multiple modules in a weighted manner. The dynamic convolutional layer is introduced to adaptively adjust parameters without increasing the network depth and width, greatly improving the model's expressive power. A residual learning operation is added within the Transformer module to address the difficulty of training the Transformer module, more effectively extracting global features and semantic information and improving the denoising effect. A residual learning operation is used to fuse the hierarchical features obtained by the convolutional layer, the dynamic convolutional layer, and the enhanced Transformer module, respectively, transferring the memory capacity of each network layer. A concatenation operation is used to fuse the features of the enhanced convolutional layer, the dynamic convolutional layer, and the enhanced Transformer module, followed by obtaining weights through Softmax. A secondary feature extraction method using attention is used to further obtain significant features. The present invention achieves good results in blind image denoising tasks.

Description

Technical Field

[0001] The present invention belongs to the technical fields of image processing, deep learning and computer vision, and specifically relates to an image blind denoising method and system based on enhanced Transformer. Background Art

[0002] In recent years, with the rapid development of mobile devices, digital images have become increasingly easy to obtain due to their portability. The number of digital images in the physical world has increased dramatically, and the application of image processing has become more and more extensive.

[0003] However, due to their small size, mobile devices forgo large photosensitive elements, a key factor limiting clear imaging. Furthermore, due to the portability of mobile devices, people rarely carry professional photography equipment, such as tripods and stabilizers, when taking photos. These factors all result in a certain amount of noise in the captured images. Through mathematical modeling, a noisy image can be considered as y = x + μ, where y is the noisy image, x is the clean image, and μ is the noise. Image denoising techniques primarily restore noise points in noisy images to their original pixel values, thereby obtaining a clean, clear image. Traditional denoising methods can be categorized as spatial pixel feature denoising and transform domain denoising. Spatial pixel feature denoising focuses on the statistical characteristics of a pixel or pixels within a domain and applies certain spatial transformations to these pixels. Transform domain denoising differs from spatial pixel feature denoising in that it uses mathematical transformations to separate signal and noise in the transform domain, separating noise into high, medium, and low frequency noise. The noise is then suppressed, leaving only the signal. An inverse transform is then performed to convert the signal from the transform domain back to the original spatial domain, ultimately removing image noise. Although these methods have achieved good performance in some scenarios, they still face the following challenges:

[0004] (1) Some methods have poor generalization performance and can only adapt to one or several types of noise and can only be used in certain scenarios.

[0005] (2) Some methods have good performance, but their high complexity limits their application scenarios.

[0006] Deep learning originated in the 1960s, but its algorithms were complex and lacked powerful computing resources. In recent years, the rapid development of computing devices, especially graphics processing units (GPUs), and the increasing sophistication of big data have revitalized deep learning. In the field of digital image processing, convolutional neural networks (CNNs) are currently the most popular technology. CNNs leverage the GPU's ability to quickly process data, combined with the powerful learning capabilities of neural networks, the lack of manual parameter tuning, translation invariance, and shared weights. They significantly reduce the number of parameters while effectively extracting local features, making them widely used in image denoising. For example, "Beyond a Gaussian Denoiser: Residual Learning of Deep CNNs for Image Denoising" pioneered the use of CNNs in image denoising. Specifically, this method improves the performance of the denoising network by stacking convolutional layers, the Relu activation function, and batch normalization (BN) layers. The Fast and Flexible Denoising Network (FFDNet) uses both noise and noise maps as network inputs, effectively performing image denoising. While these methods offer good denoising performance, the following challenges remain unaddressed:

[0007] (1) The above methods rely on increasing the number of network layers to improve performance, but infinitely increasing the number of network layers cannot produce the optimal model. Since the convolution operation has limited ability to extract features, these methods cannot be well applied to blind denoising problems.

[0008] (2) The above methods cannot use one model to handle tasks such as real noise and blind noise.

[0009] In 2017, the Transformer completely abandoned the RNN (Recurrent Neural Network) and CNN architectures, adopting only the self-attention mechanism, achieving excellent results in machine translation tasks. This solved the problem of the poor parallelism of the RNN architecture and rapidly gained popularity in natural language processing over the following years. In 2020, the Transformer was first introduced to the field of computer vision. Dividing an image evenly into several blocks as input, the Transformer also achieved good results due to its excellent ability to extract global features. Subsequently, the Transformer was widely studied in the field of computer vision. Summary of the Invention

[0010] The technical problem to be solved by the present invention is to address the deficiencies in the above-mentioned prior art and provide a method and system for blind image denoising based on an enhanced Transformer. This method replaces some ordinary convolutional layers in a convolutional neural network with dynamic convolutional layers, and adds enhanced Transformer blocks after the dynamic convolutional layers to improve the ability to extract global features. Furthermore, it adaptively fuses features across multiple modules using a weighted approach, improving the effectiveness of blind denoising. This invention is of great significance for real-world applications such as medical diagnosis, remote sensing positioning, and portrait enhancement.

[0011] The present invention adopts the following technical solutions:

[0012] An image blind denoising method based on enhanced Transformer includes the following steps:

[0013] S1, dividing the noisy image into several image blocks;

[0014] S2. Build a denoising network based on the enhanced Transformer, and fuse the features of the convolutional layer, dynamic convolutional layer, and enhanced Transformer module in the denoising network through residual learning operations, connection operations, and attention mechanisms;

[0015] S3. Input the noisy image blocks segmented in S1 into the denoising network constructed in step S2 to obtain the image blocks output by the denoising network, and splice the image blocks of the same image together to obtain a clean image.

[0016] Specifically, the denoising network in step S2 includes an 8-layer structure, specifically:

[0017] The 1st, 2nd and 3rd layers are all composed of convolutional layers and activation functions ReLU; the 4th and 5th layers are all composed of dynamic convolutional layers and activation functions ReLU; the 6th layer is an enhanced Transformer module, and the 7th layer is composed of a convolutional layer and activation functions ReLU; the 8th layer is composed of convolutional layers; among them, the convolution kernel size in the 1st and 8th convolutional layers is 3×3, and the convolution kernel size in the 2nd, 3rd and 7th convolutional layers is 5×5; the outputs of the 3rd, 5th and 6th layers are fused through residual learning operations, connection operations, attention mechanisms and other operations, and then serve as the input of the 7th layer.

[0018] Furthermore, the dynamic convolution layer is specifically:

[0019] First, the input features are compressed from C×M×M to C×1×1 through a pooling operation, and then the C×1×1 features are compressed to 4×1×1 through two 1×1 convolutional layers, where C represents the number of channels of the feature. Then, four weights are calculated through the SoftMax function. Subsequently, the four convolution kernels are multiplied by the corresponding weights and the final convolution kernel is obtained, which is the convolution kernel of the dynamic convolution layer. The convolution operation of the dynamic convolution layer is the same as that of the convolution layer, as follows:

[0020] w1,w2,w3,w4=softmax(con 1×1 (ReLU((conv 1×1 (P(input dynamic )))))

[0021] K dynamic =w1K comm1 +w2K comm2 +w3K comm3 +w4K comm4

[0022] Out dynamic =f dynamic (input dynamic )

[0023] Among them, w1, w2, w3, w4 are the weights corresponding to the four convolution kernels, K dynamic is the dynamic convolution kernel; input dynamic is the input feature, Out dynamic is the output feature; P(*) is the pooling operation; Conv 1×1 (*) is a 1×1 convolution layer, ReLU (*) is a ReLU activation function, softmax (*) is a SoftMax function, K comm1 ,K comm2 ,K comm3 ,K comm4 Represent four convolution kernels respectively; fdynamic Represents a convolution operation.

[0024] Furthermore, the enhanced Transformer module includes an encoder Block1 and a decoder Block2, and both encoder Block1 and decoder Block2 are set to 5 layers; the input of the enhanced Transformer module and the output of the 1st, 2nd, 3rd and 4th layer encoder Block1 are feature fused through residual learning operations, and the fused features are used as the input of the 5th layer encoder; the output of the 5th layer encoder Block1 is used as the input of the 1st layer decoder Block2; in addition, the output of the 5th layer encoder Block1 is also input into all decoder Block2 in a memory manner.

[0025] Furthermore, the enhanced Transformer module is specifically:

[0026]

[0027]

[0028]

[0029] where

[0030] where

[0031]

[0032] in, are the outputs of the i-th layer encoder Block1 and the i-th layer decoder Block2, i∈{1,2,3,4,5}; input transformer To enhance the input of the Transformer module, Out transformer To enhance the output of the Transformer module; Memory is the second input of decoder Block2; is the output of Block1 of the i-th layer encoder, To enhance the Transformer module input, the first, second, third and fourth layer encoder Block1 outputs are fused through residual learning operations; represents the i-th layer encoder Block1, represents the i-th layer decoder Block2, i∈{1,2,3,4,5};

[0033] Furthermore, the encoder Block 1 is as follows:

[0034] Out MSA =f MSA (LN(input En ),LN(input En ),LN(input En ))

[0035] Out En =f FC (LN(input En +Out MSA ))+input En +Out MSA

[0036] Among them, input En The input of encoder Block1, Out MSA is the output of multi-head attention, Out En is the output of encoder Block1, LN(*) is the layer normalization function, f MSA (*) is multi-head attention, f FC (*) is a fully connected feedforward network;

[0037] Decoder Block2 is as follows:

[0038]

[0039]

[0040] where

[0041]

[0042] Among them, input De is the input of decoder Block2, is the output of the first multi-head attention, is the output of the second multi-head attention, Memeory is the output of the 5th layer encoder Block1, Out Decoder is the output of decoder Block2, LN(*) is the layer normalization function, f MSA (*) is multi-head attention, f FC (*) is a fully connected feedforward network;

[0043] Going further, the multi-head attention is as follows:

[0044]

[0045] where Q=W Q *inputQ , K=W K *input K , V=W V *input V

[0046] f MSA (input Q ,input K ,input V )=Cat(head1,...,head 12 )W O

[0047] where head i =f SA (input Q ,input K ,input V )

[0048] Among them, input Q ,input K ,input V They are the three inputs of multi-head attention; f SA (*) is the self-attention layer; f MSA (*) is multi-head attention; softmax(*) is the SoftMax function; Q, K, and V are input Q ,input K ,input V , and the weight matrix W Q 、W K 、W V The intermediate variable obtained by multiplication; W O is the weight matrix; d is the normalization parameter, head i is the output of the i-th self-attention layer, i∈{1,...,12}; Cat(*) is the concatenation function;

[0049] The fully connected feedforward network is as follows:

[0050] f FC (input FC )=W2*ReLU(W1*input FC )

[0051] Among them, input FC is the input of the fully connected feedforward network, ReLU(*) is the ReLU activation function, W1 and W2 are weight matrices respectively;

[0052] The layer normalization function is as follows:

[0053] Y′=LN(Y)

[0054]

[0055] y i ∈R l×1 ,Y,Y′∈R l×s

[0056] Among them, Y, Y′ are the input and result of layer normalization respectively, Y, Y′∈R l×s (Y and Y′ are both matrices with l rows and s columns), y i , y′ i are the i-th components of Y and Y′, i∈{1,...,s}, LN(*) is the layer normalization function, μ i l is the mean of the first dimension of Y, i∈{1,...,s}, σ i l is the standard deviation of the first dimension of Y, i∈{1,...,s}.

[0057] Specifically, in step S2, the feature fusion process is performed through residual learning operations, connection operations, and attention mechanisms, specifically:

[0058] Use the attention mechanism to calculate weights: concatenate the output features of the 3rd, 5th, and 6th layers in the denoising network in the channel dimension, and compress the features to C×1×1 through a pooling operation, where C represents the number of channels of the feature. Then, perform dimensionality reduction in the channel dimension through a 1×1 convolutional layer. Subsequently, calculate the weight corresponding to each part of the feature through the SoftMax function. Finally, multiply each part of the feature by the corresponding weight (attention mechanism), and sum to obtain the fused feature, as follows:

[0059] W1,W2,W3=SoftMax(Conv 1×1 (P(Cat(F3,F5,F6))))

[0060] F=W1*F3+W2*F5+W3*F6

[0061] Among them, W1, W2, W3 are the weights corresponding to each part of the features, Cat(*) is the operation of connecting the three features in the channel dimension, P(*) is the pooling operation, Conv 1×1 (*) is a 1×1 convolutional layer, F3, F5, F6, and F are the features output by the 3rd layer, the 5th layer, the 6th layer, and the weighted fusion features, respectively.

[0062] Specifically, in step S3, the input size of the denoising network is 3×M×M, and the output size is 3×M×M, where 3×M×M represents that the input channel and output channel of the image block are 3, and the height and width of the image block are M.

[0063] Another technical solution of the present invention is an image blind denoising system based on an enhanced Transformer, comprising:

[0064] Segmentation module, which divides the noisy image into several image blocks;

[0065] Building a denoising network based on the enhanced Transformer module, using residual learning operations, connection operations, and attention mechanisms to fuse the features of the convolutional layer, dynamic convolutional layer, and enhanced Transformer module in the denoising network;

[0066] The denoising module uses the segmentation module to segment the noisy image into image blocks, inputs the segmented image blocks into the denoising network constructed by the construction module, obtains the image blocks output by the denoising network, and splices the image blocks belonging to the same image to obtain a complete clean image.

[0067] Compared with the prior art, the present invention has at least the following beneficial effects:

[0068] A blind image denoising method based on an enhanced Transformer combines dynamic convolutional layers with enhanced Transformer blocks, and adaptively fuses features from multiple modules in a weighted manner. The image is then segmented and fed into a denoising network, which not only improves the network's training efficiency but also significantly reduces the storage space required for network operation.

[0069] Furthermore, the denoising network combines the local features extracted by 5 convolutional layers and 2 dynamic convolutional layers with the global features extracted by 1 enhanced Transformer module, effectively improving the denoising effect.

[0070] Furthermore, the dynamic convolution layer adjusts the weight of the convolution kernel according to the input data, so that the dynamic convolution layer is no longer a linear mapping, which improves the expressive power of the model.

[0071] Furthermore, in the enhanced Transformer module, the output features of the first four layers of encoder Block1 are fused through residual learning operations to reduce the impact of gradient vanishing during training and avoid the degradation problem of the enhanced Transformer module.

[0072] Furthermore, in the enhanced Transformer module, the 5-layer serial encoder Block1 converts the input sequence into a fixed-length vector for processing, and then inputs it into each decoder Block2, guiding the decoder Block2 to extract more robust global features.

[0073] Furthermore, encoder Block 1 converts the extracted global features into a fixed-length vector to guide the subsequent decoder Block 2. Decoder Block 2 consists of two multi-head attention cascades, which can better extract global features.

[0074] Furthermore, multi-head attention processes all word vectors in parallel and calculates the weights between them, making it a core module for extracting global features. Cascading a fully connected feedforward network after the multi-head attention further enhances the module's expressiveness. Furthermore, adding a layer normalization function to the Transformer module helps stabilize model training.

[0075] Furthermore, feature fusion through residual learning operations, connection operations, and attention mechanisms not only avoids gradient vanishing and facilitates the training of the denoising network, but also reduces the influence of redundant features by adjusting the weights, making the network more expressive.

[0076] Furthermore, dividing the image into image blocks for training does not require too much running space and can also increase the number of training samples.

[0077] In summary, the combination of convolutional layers, dynamic convolutional layers, and enhanced Transformer modules effectively extracts local and global features from the image. Feature fusion is performed using residual learning operations, connection operations, and attention mechanisms, effectively improving the performance of the denoising network.

[0078] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 Flowchart of the present invention;

[0080] Figure 2 This is the structural diagram of the denoising network;

[0081] Figure 3 This is the structural diagram of the dynamic convolution layer;

[0082] Figure 4 To enhance the structural diagram of the Transformer module;

[0083] Figure 5 This is the structure diagram of encoder Block1;

[0084] Figure 6 This is the structure diagram of decoder Block2;

[0085] Figure 7 Where, (a) is the original noisy image, (b) is the noise extracted by the denoising network based on the enhanced Transformer module, (c) is the two blocks in the original noisy block image, (d) is the two noise blocks extracted by the denoising network based on the enhanced Transformer module, and (e) is the two blocks of the clean image;

[0086] Figure 8 This is the entire image tested by the present invention, where (a) is the original image and (b) is the clean image restored by the method of the present invention. DETAILED DESCRIPTION

[0087] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0088] In the description of the present invention, it is to be understood that the terms “include” and “comprise” indicate the presence of the described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0089] It should also be understood that the terms used in the present specification are only for the purpose of describing particular embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0090] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0091] The accompanying drawings illustrate various schematic diagrams of structures according to embodiments disclosed herein. These figures are not drawn to scale; for clarity, some details are exaggerated and some details may be omitted. The shapes of the various regions and layers shown in the figures, as well as their relative sizes and positional relationships, are merely exemplary and may deviate in practice due to manufacturing tolerances or technical limitations. Those skilled in the art may design regions / layers with different shapes, sizes, and relative positions as needed.

[0092] The present invention provides an image blind denoising method based on an enhanced Transformer, which adopts a method combining a dynamic convolution layer with an enhanced Transformer block, and improves the residual learning operation and the connection operation. Instead of using a simple addition mode to fuse data, an attention mechanism is used to calculate the weights of each module, and then adaptively fuses features of multiple modules in a weighted manner. The denoising network implemented by the present invention is divided into 8 layers, specifically including: the first 3 layers are all convolution layers and activation functions ReLU; the next 2 layers are all dynamic convolution layers and activation functions ReLU; the 6th layer is an enhanced Transformer module, the 7th layer is a convolution layer and activation function ReLU, and the 8th layer is a convolution layer. The convolution layer can effectively extract local features and improve the receptive field by stacking the number of convolution layers. The dynamic convolution layer adaptively adjusts parameters without increasing the additional network depth and width, greatly improving the expressive power of the model. Because the enhanced Transformer block has a very strong ability to extract global features, an enhanced Transformer block is added after the dynamic convolution layer to extract global features. This paper enhances the Transformer block by incorporating residual learning into the Transformer. Furthermore, the paper uses an attention mechanism to calculate the weights of each module, multiplying each module by its corresponding weight and summing the results to perform feature fusion. This paper has achieved promising results in blind image denoising tasks.

[0093] See also Figure 1 The present invention provides an image blind denoising method based on enhanced Transformer, comprising the following steps:

[0094] S1, preprocessing the noisy image and dividing the noisy image into several image blocks;

[0095] Each noisy image is randomly divided into several image blocks for model training and testing;

[0096] S2, build an 8-layer denoising network;

[0097] The features of the convolutional layer, dynamic convolutional layer and enhanced Transformer module in the denoising network are integrated through residual learning operations, connection operations and attention mechanisms;

[0098] See also Figure 2, the 1st to 3rd layers in the denoising network are composed of convolutional layers and activation functions ReLU; the 4th to 5th layers are composed of dynamic convolutional layers and activation functions ReLU, the 6th layer is an enhanced Transformer module, the 7th layer is composed of convolutional layers and activation functions ReLU, and the 8th layer is composed of convolutional layers. The outputs of the 3rd layer, the 5th layer, and the 6th layer are fused through residual learning operations, connection operations, and attention mechanisms, and then used as the input of the 7th layer.

[0099] See also Figure 2 , through residual learning operations, connection operations, and attention mechanisms to fuse the features of multiple modules. First, the attention mechanism is used to calculate the weights: the features output by the third layer, the features output by the fifth layer, and the features output by the sixth layer are spliced ​​in the channel dimension and compressed through the pooling operation; then the dimensionality is reduced in the channel dimension through a 1×1 convolution layer, and the corresponding weights are calculated using the SoftMax function; finally, each part of the feature is multiplied by the corresponding weight, and the sum is obtained to obtain the fused feature; the details are as follows:

[0100] W1,W2,W3=SoftMax(Conv 1×1 (P(Cat(F3,F5,F6))))

[0101] F=W1*F3+W2*F5+W3*F6

[0102] Among them, W1, W2, and W3 are the weights corresponding to each part of the features, Cat(*) represents the operation of connecting the three features in the channel dimension; P(*) represents the pooling operation; Conv 1×1 (*) indicates a 1×1 convolutional layer; F3, F5, F6, and F represent the features output by the third layer, the features output by the fifth layer, the features output by the sixth layer, and the weighted fusion features, respectively.

[0103] The input size of the denoising network is 3×M×M, and the output size is 3×M×M. 3×M×M represents that the input channel and output channel are 3, the height and width are M, the convolution kernel size of the 1st and 8th layers is 3×3, and the convolution kernel size of the 2nd, 3rd and 7th layers is 5×5.

[0104] See also Figure 3 , the dynamic convolution layer (Dynamic Convolution Layer) is specifically:

[0105] First, the input features are compressed from C×M×M to C×1×1 through a pooling operation, and then the C×1×1 features are compressed to 4×1×1 through two 1×1 convolutional layers, where C represents the number of channels of the feature. Then, four weights are calculated through the SoftMax function. Subsequently, the four convolution kernels are multiplied by the corresponding weights and the final convolution kernel is obtained, which is the convolution kernel of the dynamic convolution layer. The convolution operation of the dynamic convolution layer is the same as that of the convolution layer, as follows:

[0106] w1,w2,w3,w4=softmax(con 1×1 (ReLU((conv 1×1 (P(input dynamic )))))

[0107] K dynamic =w1K comm1 +w2K comm2 +w3K comm3 +w4K comm4

[0108] Out dynamic =f dynamic (input dynamic )

[0109] Among them, w1, w2, w3, w4 are the weights corresponding to the four convolution kernels, K dynamic is the dynamic convolution kernel; input dynamic is the input feature, Out dynamic is the output feature; P(*) is the pooling operation; Conv 1×1 (*) is a 1×1 convolution layer, ReLU (*) is a ReLU activation function, softmax (*) is a SoftMax function, K comm1 ,K comm2 ,K comm3 ,K comm4 Represent four convolution kernels respectively; f dynamic Represents a convolution operation.

[0110] See also Figure 4 The enhanced Transformer module includes encoder Block1 and decoder Block2; encoder Block1 and decoder Block2 are both set with 5 layers, and the input of the enhanced Transformer module and the output of the 1st to 4th layer encoder Block1 are feature fused through residual learning operations; the output of the 5th layer encoder Block1 is used as the input of the 1st layer decoder Block2; in addition, the output of the 5th layer encoder Block1 is also input into all decoder Block2 in a memory manner.

[0111] The specific formula is as follows:

[0112]

[0113]

[0114]

[0115] where

[0116] where

[0117]

[0118] in, are the outputs of the i-th layer encoder Block1 and the i-th layer decoder Block2, i∈{1,2,3,4,5}; input transformer To enhance the input of the Transformer module, Out transformer To enhance the output of the Transformer module; Memory is the second input of decoder Block2; is the output of Block1 of the i-th layer encoder, To enhance the Transformer module input, the first, second, third and fourth layer encoder Block1 outputs are fused through residual learning operations; represents the i-th layer encoder Block1, represents the i-th layer decoder Block2, i∈{1,2,3,4,5};

[0119] See also Figure 5 , the formula of encoder Block1 is as follows:

[0120] Out MSA =f MSA (LN(input En ),LN(input En ),LN(input En ))

[0121] Out En =f FC (LN(input En +Out MSA ))+input En +Out MSA

[0122] Among them, inputEn The input of encoder Block1, Out MSA is the output of multi-head attention, Out En is the output of encoder Block1, LN(*) is the layer normalization function, f MSA (*) is multi-head attention, f FC (*) is a fully connected feedforward network;

[0123] See also Figure 6 , the formula of decoder Block2 is as follows:

[0124]

[0125]

[0126] where

[0127]

[0128] Among them, input De is the input of decoder Block2, is the output of the first multi-head attention, is the output of the second multi-head attention, Memeory is the output of the 5th layer encoder Block1, Out Decoder is the output of decoder Block2, LN(*) is the layer normalization function, f MSA (*) is multi-head attention, f FC (*) is a fully connected feed-forward network.

[0129] Figure 5 and Figure 6 The formula for the multi-head attention involved is as follows:

[0130]

[0131] where Q=W Q *input Q , K=W K *input K , V=W V *input V

[0132] f MSA (input Q ,input K ,input V )=Cat(head1,...,head 12 )W O

[0133] where head i =f SA (input Q ,input K ,input V )

[0134] Among them, input Q ,input K ,input V They are the three inputs of multi-head attention; f SA (*) is the self-attention layer; f MSA (*) is multi-head attention; softmax(*) is the SoftMax function; Q, K, and V are input Q ,input K ,input V , and the weight matrix W Q 、W K 、W V The intermediate variable obtained by multiplication; W O is the weight matrix; d is the normalization parameter, head i is the output of the i-th self-attention layer, i∈{1,...,12}; Cat(*) is the concatenation function.

[0135] Figure 5 and Figure 6 The fully connected feedforward network involved is as follows:

[0136] f FC (input FC )=W2*ReLU(W1*input FC )

[0137] Among them, input FC is the input of the fully connected feedforward network, ReLU(*) is the ReLU activation function, W1 and W2 are weight matrices respectively;

[0138] Figure 5 and Figure 6 The layer normalization functions involved are as follows:

[0139] Y′=LN(Y)

[0140]

[0141] y i ∈R l×1 ,Y,Y′∈R l×s

[0142] Among them, Y, Y′ are the input and result of layer normalization respectively, Y, Y′∈Rl×s (Y and Y′ are both matrices with l rows and s columns), y i , y′ i are the i-th components of Y and Y′, i∈{1,...,s}, LN(*) is the layer normalization function, μ i l is the mean of the first dimension of Y, i∈{1,...,s}, σ i l is the standard deviation of the first dimension of Y, i∈{1,...,s}.

[0143] S3. Input the image blocks with side length s after segmentation in step S1 into the denoising network constructed by S2, output the image blocks with side length M, obtain the image blocks output by the denoising network, and splice the image blocks belonging to the same image to obtain a complete clean image.

[0144] When testing or using the denoising network, a whole noisy image is input into the denoising network through segmentation, and several M×M image blocks are output. The M×M image blocks are restored to a complete clean image through splicing.

[0145] In addition to being able to process blind noise, the present invention can also process fixed Gaussian noise and real noise.

[0146] In another embodiment of the present invention, a blind image denoising system based on an enhanced Transformer is provided. The system can be used to implement the above-mentioned blind image denoising method based on an enhanced Transformer. Specifically, the blind image denoising system based on an enhanced Transformer includes a segmentation module, a construction module and a denoising module.

[0147] Among them, the segmentation module divides the noisy image into several image blocks;

[0148] Building a denoising network based on the enhanced Transformer module, integrating the features of the convolutional layer, dynamic convolutional layer, and enhanced Transformer module in the denoising network by using residual learning operations, connection operations, and attention mechanisms;

[0149] The denoising module uses the segmentation module to segment the noisy image into image blocks, inputs the segmented image blocks into the denoising network constructed by the construction module, obtains the image blocks output by the denoising network, and splices the image blocks belonging to the same image to obtain a complete clean image.

[0150] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0151] The present invention proposes an image blind denoising method based on enhanced Transformer. Dynamic convolutional layers and enhanced Transformer modules are added to the model to improve the ability to extract features, and feature fusion is adaptively performed on multiple modules in a weighted manner.

[0152] The present invention takes 40 levels of blind Gaussian noise as an example. When training the building block, the batch size is set to 64; a total of 120 cycles are trained; the initial learning rate is set to 1×10 -3 ; and at the 30th, 60th, and 90th cycles, the learning rate is reduced to one tenth of the original. In the denoising module, Figure 8 (a) is an example of a clean image obtained by the denoising network. Figure 8 (b), its brightness and contrast are consistent with the noisy image, and detailed textures such as ripples on the water surface are restored.

[0153] See also Figure 7 , Figure 7 (a) is the complete noisy image; Figure 7 (b) is the complete noise identified by the denoising network; Figure 7 (c) as an example, Figure 7 (c) Figure 7 The two image blocks segmented from (a) Figure 7 The denoising process of the remaining image blocks segmented in (a) is the same as Figure 7 (c) the same, not shown in the description); Figure 7 (d) is from Figure 7 Two noise blocks identified in (c); Figure 7 (e) Yes Figure 7 (c) The clean image block obtained by the denoising network; Figure 7 (e) and from Figure 7 The remaining image blocks after denoising that are segmented in (a) are recombined to obtain Figure 8 (a).

[0154] See also Figure 8 , Figure 8 (a) is the complete noisy image; Figure 8 (b) Yes Figure 8 (a) Complete clean image obtained after denoising network.

[0155] from Figure 7 and Figure 8 As can be seen from the figure, the brightness and contrast of the clean image obtained by the denoising network are consistent with those of the noisy image, and detailed textures such as ripples on the water surface are restored.

[0156] In summary, the present invention provides a method and system for blind image denoising based on an enhanced Transformer. This method incorporates an enhanced Transformer block into the original CNN model to enhance the model's extraction of global features. This, combined with the local features extracted by the convolutional layer, significantly improves model performance. Furthermore, the present invention replaces some convolutional layers with dynamic convolutional layers, enhancing the expressive power of the denoising network and the generalization of the model. The method also promotes feature fusion across layers by employing residual learning operations, concatenation operations, and attention mechanisms, thereby enhancing network performance.

[0157] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0158] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0159] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0160] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0161] The above content is only for explaining the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. A blind image denoising method based on enhanced Transformer, characterized in that: The following steps are involved: S1, dividing the noisy image into several image blocks; S2. Build a denoising network based on the enhanced Transformer. Through residual learning operations, connection operations, and attention mechanisms, the features of the convolutional layer, dynamic convolutional layer, and enhanced Transformer module in the denoising network are integrated. The denoising network consists of an 8-layer structure, specifically: The first, second, and third layers are composed of convolutional layers and activation functions ReLU; the fourth and fifth layers are composed of dynamic convolutional layers and activation functions ReLU; the sixth layer is an enhanced Transformer module, the seventh layer is composed of convolutional layers and activation functions ReLU; the eighth layer is composed of convolutional layers; the convolution kernel size in the first and eighth convolutional layers is , the convolution kernel size in the 2nd, 3rd and 7th convolutional layers is The outputs of the 3rd, 5th, and 6th layers are fused through residual learning, connection, and attention mechanisms, and then used as the input of the 7th layer. The enhanced Transformer module includes an encoder Block 1 and a decoder Block 2. Both encoder Block 1 and decoder Block 2 have 5 layers. The input of the enhanced Transformer module and the outputs of the 1st, 2nd, 3rd, and 4th layer encoder Block 1 are fused through residual learning, and the fused features are used as the input of the 5th layer encoder. The output of the 5th layer encoder Block 1 is used as the input of the 1st layer decoder Block 2. In addition, the output of the 5th layer encoder Block 1 is also input into all decoder Blocks 2 in a memory manner. S3. Input the noisy image blocks segmented in S1 into the denoising network constructed in step S2 to obtain the image blocks output by the denoising network, and splice the image blocks of the same image together to obtain a clean image.

2. The image blind denoising method based on enhanced Transformer according to claim 1, characterized in that: The dynamic convolution layer is specifically: First, the input features are transformed from Compress to , and then through two The convolutional layer will Features of size are compressed to , The number of channels representing the feature is then used to calculate the four weights using the SoftMax function. Subsequently, the final convolution kernel is obtained by multiplying the four convolution kernels by the corresponding weights and summing them up. This is the convolution kernel of the dynamic convolution layer. The convolution operation of the dynamic convolution layer is the same as that of the convolution layer, as follows: in, are the weights corresponding to the four convolution kernels, is the dynamic convolution kernel; is the input feature, is the output feature; It is a pooling operation; for The convolutional layer, is the ReLU activation function, is the SoftMax function, Represent four convolution kernels respectively; Represents a convolution operation.

3. The image blind denoising method based on enhanced Transformer according to claim 1, characterized in that: The enhanced Transformer module is specifically: ， ， in, 、 Respectively Layer encoder Block1 and The output of layer decoder Block2, ; To enhance the input of the Transformer module, To enhance the output of the Transformer module; Memory is the second input of decoder Block2; For the The output of layer encoder Block1, To enhance the Transformer module input, the first, second, third and fourth layer encoder Block1 outputs are fused through residual learning operations; Indicates the Layer encoder Block1, Indicates the Layer decoder Block2, .

4. The image blind denoising method based on enhanced Transformer according to claim 1, characterized in that: Encoder Block1 is as follows: in, is the input of encoder Block1, is the output of multi-head attention, is the output of encoder Block1, is the layer normalization function, For multiple attentions, It is a fully connected feedforward network; Decoder Block2 is as follows: in, is the input of decoder Block2, is the output of the first multi-head attention, is the output of the second multi-head attention, is the output of the 5th layer encoder Block1, is the output of decoder Block2, is the layer normalization function, For multiple attentions, It is a fully connected feed-forward network.

5. The image blind denoising method based on enhanced Transformer according to claim 4, characterized in that: The multi-head attention is as follows: ， ， in, They are the three inputs of multi-head attention; is the self-attention layer; For the multi-headed attention; is the SoftMax function; They are , and the weight matrix 、 、 The intermediate variable obtained by multiplication; is the weight matrix; is the normalization parameter, For the The output of the self-attention layer, ; is the splicing function; The fully connected feedforward network is as follows: in, is the input of the fully connected feedforward network, for activation function, 、 are weight matrices respectively; The layer normalization function is as follows: 、 in, are the input and result of layer normalization, respectively. ( All are one rows, s columns), They are No. A quantity, , is the layer normalization function, for The mean of dimension 1, , for The standard deviation of dimension 1, .

6. The image blind denoising method based on enhanced Transformer according to claim 1, characterized in that: In step S2, the feature fusion process is performed through residual learning operations, connection operations, and attention mechanisms, specifically: Use the attention mechanism to calculate the weights: concatenate the output features of the 3rd layer, the 5th layer, and the 6th layer in the denoising network in the channel dimension, and compress the features to , The number of channels representing the features; then The convolutional layer performs dimensionality reduction on the channel dimension; then, the weight corresponding to each part of the feature is calculated through the SoftMax function; finally, each part of the feature is multiplied by the corresponding weight, and the sum is obtained to obtain the fused feature, as follows: in, are the weights corresponding to each feature, To connect the three features in the channel dimension, For the pooling operation, for The convolutional layer, They are the features output by the 3rd layer, the features output by the 5th layer, the features output by the 6th layer, and the features after weighted fusion.

7. The image blind denoising method based on enhanced Transformer according to claim 1, characterized in that: In step S3, the input size of the denoising network is , the output size is , It represents that the input channel and output channel of the image block are 3, and the height and width of the image block are M.

8. An image blind denoising system based on enhanced Transformer, characterized by: include: Segmentation module, which divides the noisy image into several image blocks; Construct a denoising network based on the enhanced Transformer module. Use residual learning operations, connection operations, and attention mechanisms to fuse the features of the convolutional layer, dynamic convolutional layer, and enhanced Transformer module in the denoising network. The denoising network consists of an 8-layer structure, specifically: The first, second, and third layers are composed of convolutional layers and activation functions ReLU; the fourth and fifth layers are composed of dynamic convolutional layers and activation functions ReLU; the sixth layer is an enhanced Transformer module, the seventh layer is composed of convolutional layers and activation functions ReLU; the eighth layer is composed of convolutional layers; the convolution kernel size in the first and eighth convolutional layers is , the convolution kernel size in the 2nd, 3rd and 7th convolutional layers is The outputs of the 3rd, 5th, and 6th layers are fused through residual learning, connection, and attention mechanisms, and then used as the input of the 7th layer. The enhanced Transformer module includes an encoder Block 1 and a decoder Block 2. Both encoder Block 1 and decoder Block 2 have 5 layers. The input of the enhanced Transformer module and the outputs of the 1st, 2nd, 3rd, and 4th layer encoder Block 1 are fused through residual learning, and the fused features are used as the input of the 5th layer encoder. The output of the 5th layer encoder Block 1 is used as the input of the 1st layer decoder Block 2. In addition, the output of the 5th layer encoder Block 1 is also input into all decoder Blocks 2 in a memory manner. The denoising module uses the segmentation module to segment the noisy image into image blocks, inputs the segmented image blocks into the denoising network constructed by the construction module, obtains the image blocks output by the denoising network, and splices the image blocks of the same image together to obtain a clean image.

Images