Picture denoising method and device

Abstract

The invention relates to a picture denoising method and device. The method comprises the following steps: acquiring a to-be-denoised picture; performing down-sampling processing on the to-be-denoised picture to obtain a feature map of the to-be-denoised picture; performing feature reconstruction on different features in the feature map by adopting different feature reconstruction modes to obtain a feature reconstruction result; and performing up-sampling processing on the feature reconstruction result to obtain a de-noised picture. By applying the method, a good picture denoising effect can be achieved, and meanwhile, the picture denoising efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a method and device for image denoising. Background Art

[0002] At present, the application of pictures is involved in many application scenarios such as teaching, office, and entertainment. Among them, photography is one of the important sources of pictures.

[0003] Furthermore, in actual applications, there are various factors, such as shooting environment factors, shooting technique factors, etc., which lead to a large amount of noise and noise points in the captured pictures, which is not conducive to picture applications.

[0004] Therefore, there is an urgent need for a solution that can effectively and efficiently denoise images. Summary of the Invention

[0005] The present application provides a method and apparatus for image denoising to implement a solution that can effectively and efficiently denoise images.

[0006] In a first aspect, the present application provides a method for image denoising, the method comprising:

[0007] Performing downsampling processing on the image to be denoised to obtain a feature map of the image to be denoised;

[0008] Reconstructing features using different feature reconstruction methods for different features in the feature map to obtain feature reconstruction results;

[0009] The feature reconstruction result is up-sampled to obtain a denoised image.

[0010] In a possible implementation, downsampling the image to be denoised to obtain a feature map of the image to be denoised includes:

[0011] Downsampling the image to be denoised using the encoding network in the trained image denoising model to obtain a feature map of the image to be denoised;

[0012] The reconstructing different features in the feature map using different feature reconstruction methods to obtain feature reconstruction results includes:

[0013] Using a feature reconstruction network in a trained image denoising model to reconstruct different features in the feature map using different feature reconstruction methods to obtain feature reconstruction results;

[0014] The upsampling process is performed on the feature reconstruction result to obtain the denoised image, including:

[0015] The feature reconstruction result is upsampled using a decoding network in a trained image denoising model to obtain the denoised image.

[0016] In one possible implementation, the feature reconstruction network includes a Transformer module, a first convolutional layer, and a first upsampling module; the feature reconstruction network in the trained image denoising model is used to reconstruct different features in the feature map using different feature reconstruction methods to obtain feature reconstruction results, including:

[0017] Using the Transformer module to perform deep feature learning on different features in the feature map to obtain deep feature learning results;

[0018] Performing a first convolution process on the deep feature learning result using the first convolution layer to obtain a first convolution result;

[0019] The first upsampling module is used to perform a first upsampling process on the first convolution result to obtain the feature reconstruction result.

[0020] In one possible implementation, the Transformer module includes a convolutional self-attention module and a convolutional feedforward module; and using the Transformer module to perform deep feature learning on different features in the feature map to obtain deep feature learning results includes:

[0021] Performing self-attention learning on the feature map using the convolutional self-attention module to obtain a self-attention learning result, wherein the self-attention learning result includes weights of different features in the feature map and sub-feature maps of the different features obtained by processing according to the weights;

[0022] The convolution feedforward module is used to perform convolution feedforward processing on the sub-feature maps of different features in the self-attention learning result to obtain the deep feature learning result.

[0023] In one possible implementation, the convolutional self-attention module includes a first normalization layer, a second convolutional layer, a softmax layer, and a third convolutional layer; and performing self-attention learning on the feature map using the convolutional self-attention module to obtain a self-attention learning result includes:

[0024] Performing a first normalization process on the feature map using the first normalization layer to obtain a first normalization result;

[0025] Performing a second convolution process on the first normalized result using the second convolution layer to obtain a second convolution result;

[0026] Performing first weight learning on the second convolution result using the softmax layer to obtain a first weight learning result, where the first weight learning result includes weights of different features in the feature map;

[0027] Performing a first operation on the second convolution result using the first weight learning result to obtain a first operation result;

[0028] Performing a third convolution process on the first operation result using the third convolution layer to obtain a third convolution result;

[0029] A second operation is performed using the third convolution result and the feature map to obtain the self-attention learning result.

[0030] In one possible implementation, the second convolution result includes a first feature matrix, a second feature matrix, and a third feature matrix; and performing first weight learning on the second convolution result using the softmax layer to obtain a first weight learning result includes:

[0031] performing a third operation on the second characteristic matrix and a transposed matrix of the third characteristic matrix to obtain a third operation result;

[0032] Performing first weight learning on the third operation result using the softmax layer to obtain the first weight learning result;

[0033] The performing a first operation on the second convolution result using the first weight learning result to obtain a first operation result includes:

[0034] A first operation is performed on the first feature matrix using the first weight learning result to obtain a first operation result.

[0035] In one possible implementation, the convolutional feedforward module includes a second normalization layer, a fourth convolutional layer, an activation layer, and a fifth convolutional layer; and using the convolutional feedforward module to perform convolutional feedforward processing on the self-attention learning result to obtain the deep-level feature learning result includes:

[0036] Performing a second normalization process on the self-attention learning result using the second normalization layer to obtain a second normalized result;

[0037] Performing a fourth convolution process on the second normalized result using the fourth convolution layer to obtain a fourth convolution result;

[0038] Performing activation processing on the fourth convolution result using the activation layer to obtain an activation result;

[0039] Performing a fifth convolution process on the activation result using the fifth convolution layer to obtain a fifth convolution result;

[0040] A fourth operation is performed on the fifth convolution result and the self-attention learning result to obtain the deep feature learning result.

[0041] In one possible implementation, the encoding network includes a sixth convolutional layer, a first denoising module, and a seventh convolutional layer; and downsampling the image to be denoised using the encoding network in the trained image denoising model to obtain a feature map of the image to be denoised includes:

[0042] Performing a sixth convolution process on the image to be denoised using the sixth convolution layer to obtain a sixth convolution process result;

[0043] Performing a first denoising process on the sixth convolution processing result using the first denoising module to obtain a first denoising processing result;

[0044] Perform a seventh convolution process on the denoising result using the seventh convolution layer to obtain a feature map of the image to be denoised.

[0045] In one possible implementation, the decoding network includes: an eighth convolutional layer, a second denoising module, a second upsampling module, and a ninth convolutional layer. The upsampling process of the feature reconstruction result using the decoding network in the trained image denoising model to obtain the denoised image includes:

[0046] Performing an eighth convolution process on the feature reconstruction result using the eighth convolution layer to obtain an eighth convolution process result;

[0047] performing a second denoising process on the eighth convolution processing result using the second denoising module to obtain a second denoising processing result;

[0048] performing a second upsampling process on the second denoising result using the second upsampling module to obtain a second upsampling result;

[0049] Performing a ninth convolution process on the second upsampling result using the ninth convolution layer to obtain the denoised image.

[0050] In one possible implementation, the first denoising module includes: a first channel attention module and a first visual channel attention module; and performing a first denoising process on the sixth convolution processing result using the first denoising module to obtain a first denoising processing result includes:

[0051] Using the first channel attention module to perform channel attention learning on the sixth convolution processing result to obtain a first channel attention learning result;

[0052] The first visual channel attention module is used to perform visual channel attention learning on the first channel attention learning result to obtain a first visual channel attention learning result.

[0053] In one possible implementation, the second denoising module includes: a second channel attention module and a second visual channel attention module; and performing a second denoising process on the eighth convolution processing result using the second denoising module to obtain a second denoising processing result includes:

[0054] Using the second channel attention module to perform channel attention learning on the eighth convolution processing result, to obtain a second channel attention learning result;

[0055] The second visual channel attention module is used to perform visual channel attention learning on the second channel attention learning result to obtain a second visual channel attention learning result.

[0056] In one possible implementation, the image denoising model is trained in the following manner:

[0057] Get the original image set;

[0058] Degrading the original pictures in the original picture set according to a preset picture degradation method to obtain a degraded picture set;

[0059] Constructing a training sample set using the original picture set and the degraded picture set, wherein the training samples in the training sample set use the corresponding degraded pictures and original pictures as input values ​​and label values ​​respectively;

[0060] The initial convolutional neural network model is trained using the training sample set to obtain the image denoising model, wherein the initial convolutional neural network model includes an encoding network, a feature reconstruction network, and a decoding network.

[0061] In a second aspect, the present application provides an image denoising device, the device comprising:

[0062] Image acquisition module, used to obtain the image to be denoised;

[0063] A downsampling module, configured to perform downsampling processing on the image to be denoised to obtain a feature map of the image to be denoised;

[0064] A feature reconstruction module, configured to reconstruct different features in the feature map using different feature reconstruction methods to obtain feature reconstruction results;

[0065] The upsampling module is used to perform upsampling processing on the feature reconstruction result to obtain a denoised image.

[0066] In a possible implementation, the downsampling module is specifically configured to:

[0067] Downsampling the image to be denoised using the encoding network in the trained image denoising model to obtain a feature map of the image to be denoised;

[0068] The feature reconstruction module is specifically used to: use the feature reconstruction network in the trained image denoising model to reconstruct different features in the feature map using different feature reconstruction methods to obtain feature reconstruction results;

[0069] The upsampling module is specifically used to: use the decoding network in the trained image denoising model to upsample the feature reconstruction result to obtain the denoised image.

[0070] In one possible implementation, the feature reconstruction network includes a Transformer module, a first convolutional layer, and a first upsampling module; the feature reconstruction module includes:

[0071] A feature learning subunit is used to perform deep feature learning on different features in the feature map using the Transformer module to obtain deep feature learning results;

[0072] A first convolution subunit, configured to perform a first convolution process on the deep feature learning result using the first convolution layer to obtain a first convolution result;

[0073] The first upsampling subunit is configured to perform a first upsampling process on the first convolution result using the first upsampling module to obtain the feature reconstruction result.

[0074] In one possible implementation, the Transformer module includes a convolutional self-attention module and a convolutional feedforward module; the feature learning subunit includes:

[0075] A self-attention subunit is configured to perform self-attention learning on the feature map using the convolutional self-attention module to obtain a self-attention learning result, wherein the self-attention learning result includes weights of different features in the feature map and sub-feature maps of different features obtained by processing according to the weights;

[0076] A convolution feedforward subunit is used to use the convolution feedforward module to perform convolution feedforward processing on the sub-feature maps of different features in the self-attention learning result to obtain the deep-level feature learning result.

[0077] In one possible implementation, the convolutional self-attention module includes a first normalization layer, a second convolutional layer, a softmax layer, and a third convolutional layer; the self-attention subunit includes:

[0078] a first normalization subunit, configured to perform a first normalization process on the feature map using the first normalization layer to obtain a first normalization result;

[0079] a second convolution subunit, configured to perform a second convolution process on the first normalized result using the second convolution layer to obtain a second convolution result;

[0080] a first weight learning subunit, configured to perform first weight learning on the second convolution result using the softmax layer to obtain a first weight learning result, where the first weight learning result includes weights of different features in the feature map;

[0081] A first operator unit, configured to perform a first operation on the second convolution result using the first weight learning result to obtain a first operation result;

[0082] a second convolution subunit, configured to perform a third convolution process on the first operation result using the third convolution layer to obtain a third convolution result;

[0083] The second operator unit is used to perform a second operation using the third convolution result and the feature map to obtain the self-attention learning result.

[0084] In a possible implementation, the second convolution result includes a first characteristic matrix, a second characteristic matrix, and a third characteristic matrix; and the first weight learning subunit is specifically configured to:

[0085] performing a third operation on the second characteristic matrix and a transposed matrix of the third characteristic matrix to obtain a third operation result;

[0086] Performing first weight learning on the third operation result using the softmax layer to obtain the first weight learning result;

[0087] The first operator unit is specifically configured to:

[0088] A first operation is performed on the first feature matrix using the first weight learning result to obtain a first operation result.

[0089] In one possible implementation, the convolutional feedforward module includes a second normalization layer, a fourth convolutional layer, an activation layer, and a fifth convolutional layer; the convolutional feedforward subunit includes:

[0090] a second normalization subunit, configured to perform a second normalization process on the self-attention learning result using the second normalization layer to obtain a second normalized result;

[0091] a fourth convolution subunit, configured to perform a fourth convolution process on the second normalized result using the fourth convolution layer to obtain a fourth convolution result;

[0092] an activation subunit, configured to perform activation processing on the fourth convolution result using the activation layer to obtain an activation result;

[0093] a fifth convolution subunit, configured to perform a fifth convolution process on the activation result using the fifth convolution layer to obtain a fifth convolution result;

[0094] The fourth operator unit is used to perform a fourth operation on the fifth convolution result and the self-attention learning result to obtain the deep feature learning result.

[0095] In one possible implementation, the encoding network includes a sixth convolutional layer, a first denoising module, and a seventh convolutional layer; and the feature extraction unit includes:

[0096] a sixth convolution subunit, configured to perform a sixth convolution process on the image to be denoised using the sixth convolution layer to obtain a sixth convolution process result;

[0097] a first denoising subunit, configured to perform a first denoising process on the sixth convolution processing result using the first denoising module to obtain a first denoising processing result;

[0098] The seventh convolution subunit is configured to perform a seventh convolution on the denoising result using the seventh convolution layer to obtain a feature map of the image to be denoised.

[0099] In one possible implementation, the decoding network includes: an eighth convolutional layer, a second denoising module, a second upsampling module, and a ninth convolutional layer, and the feature denoising unit includes:

[0100] an eighth convolution subunit, configured to perform an eighth convolution process on the feature reconstruction result using the eighth convolution layer to obtain an eighth convolution process result;

[0101] a second denoising subunit, configured to perform a second denoising process on the eighth convolution processing result using the second denoising module to obtain a second denoising processing result;

[0102] a second upsampling subunit, configured to perform a second upsampling process on the second denoising result using the second upsampling module to obtain a second upsampling result;

[0103] A ninth convolution subunit is configured to perform a ninth convolution process on the second upsampling result using the ninth convolution layer to obtain the denoised image.

[0104] In a possible implementation, the first denoising module includes: a first channel attention module and a first visual channel attention module;

[0105] The first denoising subunit is specifically configured to: perform channel attention learning on the sixth convolution processing result using the first channel attention module to obtain a first channel attention learning result;

[0106] The first visual channel attention module is used to perform visual channel attention learning on the first channel attention learning result to obtain a first visual channel attention learning result.

[0107] In a possible implementation, the second denoising module includes: a second channel attention module and a second visual channel attention module;

[0108] The second denoising subunit is specifically configured to: perform channel attention learning on the eighth convolution processing result using the second channel attention module to obtain a second channel attention learning result;

[0109] The second visual channel attention module is used to perform visual channel attention learning on the second channel attention learning result to obtain a second visual channel attention learning result.

[0110] In a possible implementation, the device further includes:

[0111] Sample construction module, used to obtain the original image set;

[0112] Degrading the original pictures in the original picture set according to a preset picture degradation method to obtain a degraded picture set;

[0113] Constructing a training sample set using the original picture set and the degraded picture set, wherein the training samples in the training sample set use the corresponding degraded pictures and original pictures as input values ​​and label values ​​respectively;

[0114] A model training module is used to train the initial convolutional neural network model using the training sample set to obtain the image denoising model, wherein the initial convolutional neural network model includes an encoding network, a feature reconstruction network and a decoding network.

[0115] In a possible implementation, the sample construction module includes:

[0116] The degradation unit is configured to degrade different original pictures in the original picture set according to different picture degradation methods to obtain a degraded picture set.

[0117] In a possible implementation, the sample construction module further includes:

[0118] A sharpening unit is used to perform sharpening processing on the original pictures in the original picture set, so as to use the sharpened original picture set to perform the step of constructing a training sample set using the original picture set and the degraded picture set.

[0119] In a third aspect, the present application provides an electronic device comprising: a processor and a memory, wherein the processor is configured to execute an image denoising program stored in the memory to implement the image denoising method described in any one of the first aspects.

[0120] In a fourth aspect, the present application provides a storage medium storing one or more programs, which can be executed by one or more processors to implement the image denoising method described in any one of the first aspects.

[0121] The above-mentioned technical solution provided by the embodiment of the present application has the following advantages over the prior art: the technical solution provided by the embodiment of the present application, by down-sampling the image to be denoised, obtains a feature map of the image to be denoised, adopts different feature reconstruction methods to reconstruct different features in the feature map to obtain a feature reconstruction result, and up-samples the feature reconstruction result to obtain a denoised image, thereby realizing a solution for denoising images based on deep learning, which has good image denoising efficiency, and in this method, by adopting different feature reconstruction methods to reconstruct different features in the feature map, different feature reconstruction is achieved for different features in the image, so that the features that need to be retained and the noise features that need to be removed in the image can be well distinguished, so that the noise features can be effectively removed, thereby achieving good image denoising effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0122] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0123] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0124] One or more embodiments are exemplarily illustrated by pictures in the corresponding drawings. These exemplifications do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings are represented as similar elements. Unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0125] Figure 1 A flowchart of an embodiment of an image denoising method provided in an embodiment of the present application;

[0126] Figure 2 This is an example of a noisy image;

[0127] Figure 3 This is another example of a noisy image;

[0128] Figure 4 A schematic diagram of the structure of an image denoising model provided by the technical solution of an embodiment of the present application;

[0129] Figure 5 Schematic diagram of the coding network structure;

[0130] Figure 6 It is a structural diagram of the denoising module;

[0131] Figure 7 Schematic diagram of the structure of the feature reconstruction network;

[0132] Figure 8 A flowchart of an embodiment for performing feature reconstruction processing on a feature map using a feature reconstruction network to obtain a feature reconstruction result;

[0133] Figure 9 This is a schematic diagram of the structure of the Transformer module designed in the technical solution of the embodiment of the present application;

[0134] Figure 10 A flowchart of an embodiment for using a convolutional self-attention module to perform self-attention learning on a feature map and obtain self-attention learning results;

[0135] Figure 11 A flowchart of an embodiment for using a convolutional feedforward module to perform convolutional feedforward processing on the self-attention learning results to obtain deep feature learning results;

[0136] Figure 12 A schematic diagram of the overall structure of the Transformer module;

[0137] Figure 13 Schematic diagram of the decoding network structure;

[0138] Figure 14 Schematic diagram of the structure of the second denoising module;

[0139] Figure 15 A schematic diagram of the structure of an image denoising model provided by the technical solution of an embodiment of the present application;

[0140] Figure 16 A flowchart of another method for image denoising provided in an embodiment of the present application;

[0141] Figure 17 A flowchart of an embodiment of a method for training an image denoising model provided in an embodiment of the present application;

[0142] Figure 18 This is a block diagram of an embodiment of an image denoising device provided in an embodiment of the present application;

[0143] Figure 19 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0144] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0145] The disclosure below provides many different embodiments or examples for implementing different structures of the present application. In order to simplify the disclosure of the present application, the components and settings of specific examples are described below. Of course, these are merely examples and are not intended to limit the present application. In addition, the present application may repeat reference numbers and / or letters in different examples. Such repetition is for the purpose of simplicity and clarity and does not in itself indicate the relationship between the various embodiments and / or settings discussed.

[0146] This application provides a method and device for image denoising, which can achieve good image denoising effects and improve image denoising efficiency.

[0147] Figure 1 This is a flow chart of an embodiment of a method for image denoising provided in an embodiment of the present application. Figure 1 As shown, the process includes the following steps:

[0148] Step 101: Obtain a picture to be denoised.

[0149] In an exemplary application scenario, users can take pictures through mobile terminals, and with the continuous development of mobile terminal photography technology, the pictures taken by mobile terminals can reach millions or even tens of millions of pixels. When users want to circulate the pictures they take on the Internet, application service providers usually reduce the resolution of the pictures, or lossily compress the pictures into a format suitable for network transmission to reduce the transmission bandwidth of the pictures. Sometimes they even dynamically adjust the picture compression parameters according to the user's network status and network bandwidth limitations to ensure that the application service can respond normally to the user's picture upload request. However, no matter which of the above methods is used, a large amount of noise and noise will be generated in the picture, resulting in damage to the picture quality. For example, see Figure 2 , is an example of a noisy image.

[0150] In the technical solution of the embodiment of the present application, the noisy image generated in the above application scenario can be obtained, for example Figure 2 The exemplified picture is used as the picture to be denoised, thereby obtaining the picture to be denoised.

[0151] In another exemplary application scenario, when a user photographs a sports scene, such as a speeding car or a sprinter on a track, or when the camera shakes, the captured image may contain a lot of noise and pixels, resulting in a blurry image. Figure 3 , is another example of a noisy image.

[0152] In the technical solution of the embodiment of the present application, it is also possible to obtain noisy images generated in the above application scenarios, for example Figure 3 The exemplified picture is used as the picture to be denoised, thereby obtaining the picture to be denoised.

[0153] It should be noted that the above application scenarios are merely exemplary descriptions of application scenarios that may produce noisy images. In actual applications, there are other scenarios that may also produce noisy images, and the embodiments of the present application are not limited to this.

[0154] Step 102: downsample the image to be denoised to obtain a feature map of the image to be denoised.

[0155] From the perspective of feature types, the above feature map may include but is not limited to the following features: color features, texture features, shape features, and spatial relationship features.

[0156] From the perspective of image content, the above feature map can include object features, noise features, and background features. Among them, objects refer to objects in the image, such as Figure 2 and Figure 3(For example, the object here is text). Noise refers to the noise and noise points in the image. For convenience, we will refer to it as noise here. Background refers to the background of the image, such as the image area other than the foreground object.

[0157] Step 103: Reconstruct different features in the feature map using different feature reconstruction methods to obtain feature reconstruction results.

[0158] In one embodiment, the different features mentioned in step 103 refer to different features from the perspective of image content.

[0159] It can be seen from the description of step 103 that in the technical solution of the embodiment of the present application, different feature reconstruction methods are used to reconstruct different features in the feature map, for example, object features are amplified, noise features are suppressed, and background features are smoothed.

[0160] This shows that by using different feature reconstruction methods for different features in the feature map, useless features in the feature map, such as noise features, can be effectively suppressed, and useful features in the feature map, such as object features, can be amplified. This allows the noise features to be effectively removed and the object features to be accurately restored in subsequent processing steps.

[0161] Step 104: Upsample the feature reconstruction result to obtain a denoised image.

[0162] The technical solution provided in the embodiment of the present application downsamples the denoised image to obtain a feature map of the image to be denoised, adopts different feature reconstruction methods to reconstruct different features in the feature map to obtain a feature reconstruction result, and upsamples the feature reconstruction result to obtain a denoised image, thereby realizing a solution for denoising images based on deep learning, which has good image denoising efficiency. In addition, in this method, different feature reconstruction methods are adopted to reconstruct different features in the feature map to achieve differentiated feature reconstruction of different features in the image, thereby being able to well distinguish between features that need to be retained and noise features that need to be removed in the image, thereby being able to effectively remove noise features, thereby achieving good image denoising effect.

[0163] In order to realize the above technical solution, the embodiment of the present application provides an image denoising model, which can effectively denoise the noise and noise points in the image, improve the image quality, and make the objects in the image (such as Figure 2 and Figure 3 The readability of the example (where the object is text) is greatly improved.

[0164] It should be noted that most image denoising solutions in the existing technology are implemented based on the Transformer architecture. Although the application of the Transformer architecture can achieve good image denoising effects, the image denoising model implemented based on the Transformer architecture often has a slow processing speed due to the large amount of computation, and has strict requirements on the image size. It requires that the size of the input image to be processed be a fixed size, which leads to many limitations in the application of the image denoising solutions in the existing technology.

[0165] In this regard, the technical solution of the embodiment of the present application proposes an image denoising model that combines a convolutional neural network and a Transformer architecture.

[0166] Since convolutional neural networks can process images of dynamic sizes and have a relatively fast model processing speed, and the Transformer architecture can achieve good image denoising effects, the image denoising model in the embodiment of the present application can achieve a good balance between processing speed and denoising effect, and no longer limits the size of the image to be denoised. In addition, in order to improve the denoising effect of the image, the present application also adopts a denoising processing strategy that is different from the existing technology, which is detailed below.

[0167] See also Figure 4 , is a structural diagram of the image denoising model provided by the technical solution of the embodiment of this application. Figure 4 As shown in the figure, the image denoising model includes an encoding network, a feature reconstruction network, and a decoding network. The input of the encoding network is the input of the entire image denoising model, the output of the encoding network is connected to the input of the feature reconstruction network, and the output of the feature reconstruction network is connected to the input of the decoding network. The output of the decoding network is the output of the entire image denoising model.

[0168] based on Figure 4 The model structure of the example, in the technical solution of the embodiment of the present application, the picture to be denoised is input into the trained picture denoising model, and the processing process of obtaining the denoised picture output by the picture denoising model includes: inputting the picture to be denoised into the trained picture denoising model, and using the encoding network in the picture denoising model to downsample the picture to be denoised to obtain a feature map of the picture to be denoised; inputting the feature map of the picture to be denoised obtained by the encoding network into the feature reconstruction network, and using the feature reconstruction network to perform feature reconstruction processing on the input feature map to obtain a feature reconstruction result; inputting the feature reconstruction result obtained by the feature reconstruction network into the decoding network, and using the decoding network to upsample the feature reconstruction result to obtain a denoised picture.

[0169] The technical solution provided in the embodiment of the present application is to obtain an image denoising model by combining a convolutional neural network with a Transformer architecture, and use the image denoising model to process the denoised image to obtain a denoised image. Since the convolutional neural network can process images of dynamic sizes and has a faster model processing speed, and the Transformer architecture can achieve good image denoising effects, the above-mentioned image denoising model can be used to denoise images of any size and has good flexibility. In addition, applying the above-mentioned image denoising model for image denoising can not only achieve good image denoising effects, but also improve image denoising efficiency.

[0170] In order to better understand the technical solution of the embodiment of the present application, the three parts of the image denoising model provided by the technical solution of the embodiment of the present application and their processing procedures are explained below.

[0171] (1) Coding Network

[0172] The encoding network is used to downsample the image to be denoised to obtain the feature map of the image to be denoised.

[0173] See also Figure 5 , is a schematic diagram of the coding network structure. Figure 5 As shown, the encoding network includes a sixth convolutional layer, a first denoising module, and a seventh convolutional layer. The input of the sixth convolutional layer is the input of the entire encoding network, the output of the sixth convolutional layer is connected to the input of the first denoising module, the output of the first denoising module is connected to the input of the seventh convolutional layer, and the output of the seventh convolutional layer is the output of the entire encoding network.

[0174] On this basis, the specific implementation of using the encoding network to downsample the denoised image to obtain the feature map of the image to be denoised includes: using the sixth convolution layer to perform the sixth convolution processing on the denoised image to obtain the sixth convolution processing result; using the first denoising module to perform the first denoising processing on the sixth convolution processing result to obtain the first denoising processing result; using the seventh convolution layer to perform the seventh convolution processing on the first denoising processing result to obtain the feature map of the image to be denoised.

[0175] The sixth convolutional layer can be conv2, and its convolution kernel size is 3*3. For the image to be denoised, the sixth convolution layer is used to perform the sixth convolution processing on it, and the sixth convolution processing result is obtained, which can realize the mapping of the image to be denoised from a low-dimensional channel to a high-dimensional channel. For example, the image to be denoised is recorded as a tensor of (b, 3, 515, 512). The sixth convolution layer is used to perform the sixth convolution processing on it, and the 3-channel image to be denoised is mapped to a 64-channel image to obtain a tensor of (b, 64, 515, 512).

[0176] For the sixth convolution processing result, the first denoising module is further used to perform the first denoising processing to obtain the first denoising processing result, which can achieve learning of the noise characteristics in the image to be denoised.

[0177] In one embodiment, see Figure 6 , is a structural diagram of the first denoising module. Figure 6 As shown, the first denoising module includes a first channel attention module and a first visual channel attention module, wherein the input end of the first channel attention module is the input end of the entire first denoising module, the output end of the first channel attention module is connected to the input end of the first visual channel attention module, and the output end of the first visual channel attention module is the output end of the entire first denoising module.

[0178] On this basis, the processing process of the first denoising module includes: using the first channel attention module to perform channel attention learning on the input sixth convolution processing result to obtain the first channel attention learning result, and using the first visual channel attention module to perform visual channel attention learning on the first channel attention learning result to obtain the first visual channel attention learning result.

[0179] The essence of the channel attention mechanism lies in modeling the importance of each feature. This allows for simple and effective feature allocation based on the input for different tasks. It uses automatic learning, using a new neural network to determine the importance of each channel in the feature map. This importance is then used to assign a weight to each feature, allowing the neural network to prioritize certain feature channels, boosting those useful for the task and suppressing those less useful. Here, the focus is on the feature channels of the image objects that are useful for the task, while the less useful ones are on the feature channels of noise and other artifacts.

[0180] The visual attention mechanism can be understood as dedicating limited visual processing power to areas of interest (task-related), which improves observation efficiency. For machines, focusing on task-related areas instead of focusing on areas irrelevant to the task effectively improves feature extraction efficiency. Here, task-irrelevant areas correspond to noise and pixelation in the image, while task-relevant areas correspond to the objects in the image.

[0181] In one embodiment, the first channel attention module includes a third normalization layer, a tenth convolutional layer, an eleventh convolutional layer, a first channel attention layer, and a twelfth convolutional layer. The input of the third normalization layer is the input of the entire first channel attention module, the output of the third normalization layer is connected to the input of the tenth convolutional layer, the output of the tenth convolutional layer is connected to the input of the eleventh convolutional layer, the output of the eleventh convolutional layer is connected to the input of the first channel attention layer, the output of the first channel attention layer is connected to the input of the twelfth convolutional layer, and the output of the twelfth convolutional layer is the output of the entire first channel attention module.

[0182] The third normalization layer can be a layernorm layer. The tenth convolutional layer can be conv2, with a convolution kernel size of 1*1. The eleventh convolutional layer can be conv2, with a convolution kernel size of 3*3. Here, the tenth and eleventh convolutional layers form a set of depthwise separable convolutions. In depthwise separable convolutions, 1*1 convolutions are used to increase the dimension, while 3*3 convolutions are used for activation. The twelfth convolutional layer can be conv2, with a convolution kernel size of 1*1.

[0183] The first visual channel attention module includes a fourth normalization layer, a thirteenth convolutional layer, a fourteenth convolutional layer, a first visual channel attention layer, and a fifteenth convolutional layer. The input of the fourth normalization layer is the input of the entire first visual channel attention module, the output of the fourth normalization layer is connected to the input of the thirteenth convolutional layer, the output of the thirteenth convolutional layer is connected to the input of the fourteenth convolutional layer, the output of the fourteenth convolutional layer is connected to the input of the first visual channel attention layer, the output of the first visual channel attention layer is connected to the input of the fifteenth convolutional layer, and the output of the fifteenth convolutional layer is the output of the entire first visual channel attention module.

[0184] The fourth normalization layer can be a layernorm layer. The thirteenth convolutional layer can be conv2, with a convolution kernel size of 1*1. The fourteenth convolutional layer can be conv2, with a convolution kernel size of 3*3. Here, the thirteenth and fourteenth convolutional layers form a set of depthwise separable convolutions. The fifteenth convolutional layer can be conv2, with a convolution kernel size of 1*1.

[0185] On this basis, the detailed processing of the first denoising module includes: first, using the layernorm layer (i.e., the third normalization layer) to normalize the input features of the first denoising module, and then convolving the above normalized results through a set of depthwise separable convolutions (i.e., the combination of the tenth and eleventh convolutional layers). The above convolution results are then calculated using the channel attention mechanism. The workflow is to first perform average pooling on the input convolution results to obtain a set of weight values, then perform nonlinear mapping through 1*1 convolution, and finally weight the nonlinear mapping results to the input features.

[0186] Subsequently, 1*1 convolution (i.e., the twelfth convolution layer) is used for convolution processing to reduce the channel dimension of the input features, reduce the amount of calculation and the number of parameters, thereby improving the processing efficiency of the model.

[0187] Subsequently, the convolution processing result at this time is weighted to the input feature and input to the next layernorm layer (i.e., the fourth normalization layer).

[0188] The next layernorm layer is used to normalize the input features, and then the normalized results are convolved with a set of depthwise separable convolutions (i.e., a combination of the 13th and 14th convolutional layers). The convolution results are then subjected to the visual channel attention mechanism. The workflow is to first use average pooling to obtain a set of weights, then use 1*1 convolution to reduce the dimensionality, then use ReLU layers to increase nonlinearity, and then use 1*1 convolution to increase the dimensionality. Finally, a sigmoid layer is used to obtain a set of weights ranging from 0 to 1, which are used to weight the input features.

[0189] Subsequently, a 1x1 convolution (i.e., the fifteenth convolutional layer) is used to reduce the channel dimension of the input features, reducing the amount of computation and the number of parameters, thereby improving the efficiency of the model. The convolution result is then weighted onto the input features to obtain the first visual channel attention learning result.

[0190] As a side note, using the visual channel attention mechanism is more complex than the channel attention mechanism, but it can more fully learn features. The difference between the visual channel attention mechanism and the channel attention mechanism is that in the channel attention mechanism, the global pooling operation compresses the features of each channel to obtain a global feature vector, which is then learned through the convolutional layer; while in the visual attention mechanism, through dimensionality reduction and dimensionality increase operations, the global feature vector can better express the relationship between channels, and then the channel weights are learned through the convolutional layer.

[0191] This completes the explanation of the structure and processing process of the first denoising module.

[0192] The seventh convolutional layer can be conv2, and its convolution kernel size is 2*2. The seventh convolutional layer is used to perform the seventh convolution on the result of the first denoising process to obtain the feature map of the image to be denoised. This can achieve dimensionality increase of the feature map of the noise-removed feature, such as increasing the dimension of the tensor (b, 64, 512, 512) to a tensor (b, 128, 256, 256).

[0193] In addition, in one embodiment, the encoding network further includes two combinations of first denoising modules and convolutional layers, wherein the input of the first combination is connected to the output of the seventh convolutional layer, the output of the first combination is connected to the input of the second combination, and the output of the second combination is connected to the input of the feature reconstruction network. The output of the first denoising module in each combination is connected to the input of the convolutional layer in that combination.

[0194] On this basis, after the seventh convolutional layer is used to perform the seventh convolution processing on the first denoising result, the convolution processing result at this time is continued to be input into the above-mentioned first combination, so that the first denoising module in the first combination continues to perform feature learning on the convolution processing result, and then the convolution layer in the first combination is used to perform convolution processing on the aforementioned learned feature map. The convolution processing at this time can achieve the dimensionality upgrade of the aforementioned learned feature map, such as upgrading the tensor of (b, 128, 256, 256) to the tensor of (b, 256, 128, 128).

[0195] The convolution processing result of the first combination is input into the second combination, so that the first denoising module in the second combination can continue to perform feature learning on the convolution processing result. Then, the convolution layer in the second combination performs convolution processing on the feature map learned above. The convolution processing at this time can achieve the dimensionality increase of the feature map learned above, such as increasing the dimension of the tensor (b, 256, 128, 128) to the tensor (b, 512, 64, 64). At this point, the feature map of the image to be denoised extracted by the encoding network is obtained.

[0196] Here, multiple downsampling steps can gradually reduce the spatial size of the input image, thereby expanding the receptive field. This helps the model better capture global and local features in the input image and extract both noisy and non-noise information. Furthermore, the smaller spatial size reduces computational effort and speeds up model training and inference.

[0197] (2) Feature Reconstruction Network

[0198] The feature reconstruction network is used to reconstruct the intermediate features, that is, the different features in the feature map extracted by the above-mentioned encoding network using different feature reconstruction methods, so that the decoding network can more accurately remove the noise in the image when restoring the image based on the feature map.

[0199] See also Figure 7 , is a structural diagram of the feature reconstruction network. Figure 7 As shown, the feature reconstruction network includes a Transformer module, a first convolutional layer, and a first upsampling module. The input of the Transformer module is connected to the output of the encoding network, the output of the Transformer module is connected to the input of the first convolutional layer, the output of the first convolutional layer is connected to the input of the first upsampling module, and the output of the first upsampling module is connected to the input of the decoding network.

[0200] On this basis, see Figure 8 , a flow chart of an embodiment of performing feature reconstruction processing on a feature map using a feature reconstruction network to obtain a feature reconstruction result, including the following steps:

[0201] Step 801: Use the Transformer module to perform deep feature learning on different features in the feature map to obtain deep feature learning results.

[0202] In the technical solution of the embodiment of the present application, the Transformer module performs deep feature learning on different features in the feature map. For example, the Transformer module is used to learn the noise features that need to be removed and the object features that need to be restored, as well as the background features that need to be smoothed from the feature map. The objects mentioned here refer to the objects in the image to be denoised, such as Figure 1 and Figure 2 The object in the image to be denoised is text. This means that the above deep feature learning results can represent the noise features that need to be removed, the object features that need to be restored, and the background features that need to be smoothed in the feature map.

[0203] In one embodiment, the feature reconstruction network may include multiple layers of Transformer modules, such as 6 layers of Transformer modules. By stacking multiple layers of Transformer modules, the depth of the model is increased, which enables the model to process more complex inputs and learn more abstract feature representations. In addition, the deep model has stronger representation capabilities and can better distinguish features, so as to achieve the effect of more accurately removing noise features.

[0204] In one embodiment, see Figure 9, is a schematic diagram of the structure of the Transformer module designed in the technical solution of the embodiment of this application. Figure 9 As shown in Figure 1, the Transformer module includes a convolutional self-attention module and a convolutional feedforward module. The input of the convolutional self-attention module is the input of the entire Transformer module, the output of the convolutional self-attention module is connected to the input of the convolutional feedforward module, and the output of the convolutional feedforward module is the output of the entire Transformer module.

[0205] On this basis, the Transformer module is used to perform deep feature learning on different features in the feature map. The specific implementation of the deep feature learning results includes: using the convolutional self-attention module to perform self-attention learning on the feature map to obtain the self-attention learning results. The self-attention learning results include the weights of the different features in the feature map and the sub-feature maps of different features obtained by processing according to the weights, such as the sub-feature maps of object features, noise features, and background features. Subsequently, the convolutional feedforward module is used to perform convolutional feedforward processing on the feature maps of different features in the self-attention learning results to obtain the deep feature learning results.

[0206] The following explains the above implementation process in combination with the structures of the convolutional self-attention module and the convolutional feedforward module:

[0207] First, see Figure 9 The convolutional self-attention module includes a first normalization layer, a second convolutional layer, a softmax layer, and a third convolutional layer. The input of the first normalization layer is the input of the entire convolutional self-attention module, the output of the first normalization layer is connected to the input of the second convolutional layer, the output of the second convolutional layer is connected to the input of the softmax layer, the output of the softmax layer is connected to the input of the third convolutional layer, and the output of the third convolutional layer is the output of the entire convolutional self-attention module.

[0208] On this basis, see Figure 10 , a flowchart of an embodiment of using a convolutional self-attention module to perform self-attention learning on a feature map to obtain a self-attention learning result, including the following steps:

[0209] Step 1001: Use a first normalization layer to perform a first normalization process on the feature map to obtain a first normalization result.

[0210] The first normalization layer may be a layernorm layer.

[0211] In the embodiment of the present application, the main effects of performing the first normalization processing on the feature map using the first normalization layer include the following three points:

[0212] 1. Accelerate network convergence. Layer normalization helps accelerate the convergence of convolutional neural networks. By standardizing each sample in a specific dimension, layer normalization can reduce the impact of internal covariate shift, thereby providing a more stable and consistent gradient signal and speeding up the training process.

[0213] 2. Improve the model's generalization ability. Layer normalization helps improve the model's generalization ability, making it more adaptable to different input data. By reducing the correlation between features, layer normalization can reduce the model's dependence on specific samples and improve the model's generalization ability to unseen data.

[0214] 3. Alleviate the vanishing or exploding gradient problem. In deep convolutional neural networks, gradient propagation is difficult to effectively carry out, which may lead to the vanishing or exploding gradient problem. Layer normalization standardizes the input, keeping the activation values ​​of each layer within a relatively small range, thereby alleviating the gradient problem and helping to better propagate the gradient.

[0215] Step 1002: Use a second convolution layer to perform a second convolution process on the first normalized result to obtain a second convolution result.

[0216] The second convolutional layer includes three sets of depthwise separable convolutions. Here, the depthwise separable convolution can be a combination of 1*1 convolution and 3*3 convolution. The purpose of using 1*1 convolution is to increase the dimension, and the purpose of using 3*3 convolution is to activate. Here, using depthwise separable convolution has the same effect as directly using 3*3 convolution, but with fewer parameters than directly using 3*3 convolution. Therefore, it can reduce the amount of calculation and the number of parameters, thereby improving the processing efficiency of the model.

[0217] On this basis, the second convolution layer is used to perform a second convolution processing on the first normalized result. The specific implementation of obtaining the second convolution result includes: using three groups of depth-wise separable convolutions to perform a second convolution processing on the first normalized result respectively, to obtain three groups of feature matrices: the first feature matrix, the second feature matrix and the third feature matrix.

[0218] The first characteristic matrix may be a v (value) matrix, the second characteristic matrix may be a q (query) matrix, and the third characteristic matrix may be a k (key) matrix.

[0219] The v, q, and k matrices above represent the concepts of the self-attention mechanism in the Transformer. Specifically, the q matrix represents the guidance information contained in the input information, the k matrix represents the content information contained in the input information, and the v matrix represents the input information itself.

[0220] Step 1003: Use the softmax layer to perform first weight learning on the second convolution result to obtain a first weight learning result.

[0221] In one embodiment, performing first weight learning on the second convolution result using a softmax layer to obtain the first weight learning result includes: performing a third operation on the transposed matrix of the second feature matrix and the third feature matrix to obtain a third operation result, and performing first weight learning on the third operation result using a softmax layer to obtain the first weight learning result. The first weight learning result includes the weights of different features in the feature map.

[0222] The above-mentioned third operation can be a dot multiplication operation. That is, the transpose of the second feature matrix (q matrix) and the third feature matrix (v matrix) is subjected to a dot multiplication operation to obtain the third operation result. In addition, in order to stabilize the gradient, the third operation result obtained by the above-mentioned dot multiplication operation can be further divided by a temperature coefficient. The role of the temperature coefficient is to adjust the attention difference between different positions. A higher temperature coefficient will make the attention distribution more even, reduce the difference between different positions, and make the model more inclined to pay attention to the entire feature on average. On the contrary, a lower temperature coefficient will enhance the difference between different positions, making the model more inclined to pay attention to the local area in the input feature. By adjusting the temperature coefficient, the degree of attention of the model to global and local information can be controlled.

[0223] Step 1004: Use the first weight learning result to perform a first operation on the second convolution result to obtain a first operation result.

[0224] In one embodiment, performing a first operation on the second convolution result using the first weight learning result to obtain the first operation result includes: performing a first operation on the first feature matrix using the first weight learning result to obtain the first operation result.

[0225] The first operation may be a weighted operation, that is, the first weight learning result obtained by using softmax is weighted to the first feature matrix (v matrix).

[0226] Step 1005: Use the third convolution layer to perform a third convolution process on the first operation result to obtain a third convolution result.

[0227] In the technical solution of the embodiment of the present application, a third convolution layer is used to perform a third convolution processing on the above-mentioned first operation result to obtain a third convolution result.

[0228] The third convolution layer may be conv2, and the convolution kernel size thereof is 1*1. Performing the third convolution process on the first operation result using the third convolution layer can reduce the dimension of the first operation result.

[0229] Step 1006: Perform a second operation using the third convolution result and the feature map to obtain a self-attention learning result.

[0230] The second operation may be an addition operation.

[0231] Since the third convolution layer has been used to reduce the dimension of the first operation result, the third convolution result obtained by dimensionality reduction and the feature map can be added in step 1006 to obtain the self-attention learning result.

[0232] Continue to see Figure 9 The convolutional feedforward module includes a second normalization layer, a fourth convolutional layer, an activation layer, and a fifth convolutional layer. The input of the second normalization layer is the input of the entire convolutional feedforward module, the output of the second normalization layer is connected to the input of the fourth convolutional layer, the output of the fourth convolutional layer is connected to the input of the activation layer, the output of the activation layer is connected to the input of the fifth convolutional layer, and the output of the fifth convolutional layer is the output of the entire convolutional feedforward module.

[0233] On this basis, see Figure 11 , a flow chart of an embodiment of using a convolutional feedforward module to perform convolutional feedforward processing on sub-feature maps of different features in the self-attention learning result to obtain a deep feature learning result, including the following steps:

[0234] Step 1101: Use the second normalization layer to perform a second normalization process on the self-attention learning result to obtain a second normalized result.

[0235] The second normalization layer may be a layernorm layer.

[0236] In the embodiment of the present application, the main functions of using the second normalization layer to perform a second normalization processing on the sub-feature maps of different features in the self-attention learning results also include three points. The detailed explanation of these three points can be found in the main functions of the first normalization processing described in the above step 1001, which will not be repeated here.

[0237] Step 1102: Use a fourth convolution layer to perform a fourth convolution process on the second normalized result to obtain a fourth convolution result.

[0238] Among them, the fourth convolutional layer includes two sets of depth-wise separable convolutions. As above, the depth-wise separable convolution can be a combination of 1*1 convolution and 3*3 convolution. The purpose of using 1*1 convolution is to increase the dimension, and the purpose of using 3*3 convolution is to perform activation.

[0239] On this basis, the fourth convolution layer is used to perform a fourth convolution processing on the second normalized result. The specific implementation of obtaining the fourth convolution result includes: using two sets of depth-wise separable convolutions to perform a fourth convolution processing on the second normalized result respectively, to obtain two sets of feature matrices, which are called the fourth feature matrix and the fifth feature matrix for convenience of description.

[0240] The fourth feature matrix represents the weight value of each point on the feature map.

[0241] Step 1103: Use the activation layer to activate the fourth convolution result to obtain an activation result.

[0242] Here, performing activation processing on the fourth convolution result by using the activation layer means: activating the fourth feature matrix through the activation layer and then weighting it to the fifth feature matrix to obtain an activation result.

[0243] Step 1104: Use the fifth convolution layer to perform a fifth convolution process on the activation result to obtain a fifth convolution result.

[0244] The fifth convolution layer may be conv2, and the convolution kernel size is 1*1. Performing the fifth convolution process on the activation result using the fifth convolution layer can reduce the dimension of the activation result.

[0245] Step 1105: Perform a fourth operation using the fifth convolution result and the self-attention learning result to obtain a deep feature learning result.

[0246] The fourth operation may be an addition operation.

[0247] Since the activation result has been processed for dimensionality reduction using the fifth convolutional layer, the activation result obtained by dimensionality reduction and the self-attention learning result can be added in step 1105 to obtain a deep feature learning result.

[0248] This concludes the description of the structures and processing procedures of the convolutional self-attention module and the convolutional feedforward module.

[0249] For ease of understanding, the following Figure 12 The overall structure of the Transformer module is shown as an example, explaining the overall processing of the Transformer module:

[0250] See also Figure 12For the feature map extracted by the encoding network, the layernorm layer is first used to normalize the feature map. Subsequently, the normalized results are convolved through three sets of depth-wise separable convolutions (a combination of 1*1 convolution and 3*3 convolution) to obtain three feature matrices q, k, and v. Subsequently, the transpose of the q feature matrix and the k feature matrix are vector-wise multiplied. Here, in order to stabilize the gradient, the transpose of the q feature matrix is ​​multiplied by the k feature matrix and divided by a temperature coefficient. Subsequently, the softmax layer is used to obtain the weight, and the weight is weighted to the v feature matrix, thus obtaining the self-attention calculation result. Subsequently, the self-attention calculation result is reduced in dimension using a 1*1 convolution, added to the feature map, and input into the next normalization layer.

[0251] As above, the next normalization layer normalizes the input, and then convolves the normalized results through two sets of depth-wise separable convolutions (a combination of 1*1 convolution and 3*3 convolution) to obtain two sets of feature matrices.

[0252] The gelu function is then used to activate the feature matrix on the left and weight it to the feature matrix on the right, thus obtaining the attention calculation result. The difference between this attention mechanism and the self-attention mechanism mentioned above is that the self-attention mechanism above learns the weights of the feature map, while the attention mechanism here learns the weights of each pixel on the feature map.

[0253] Finally, the above attention calculation results are reduced in dimension through 1*1 convolution and added to the input of the next normalization layer to obtain the deep feature learning results.

[0254] Step 802: Use the first convolution layer to perform a first convolution process on the deep feature learning result to obtain a first convolution result.

[0255] In the technical solution of the embodiment of the present application, the first convolution layer is used to perform a first convolution processing on the deep feature learning result learned by the Transformer module from the feature map to obtain a first convolution result.

[0256] The first convolutional layer may be Conv2, whose convolution kernel size is 1*1. The first convolutional layer is used to perform a first convolution process on the above-mentioned deep feature learning result, and the first convolution result obtained is, for example, to increase the dimension of the deep feature learning result from (b, 512, 64, 64) to (b, 1024, 64, 64).

[0257] Step 803: Use a first upsampling module to perform a first upsampling process on the first convolution result to obtain a feature reconstruction result.

[0258] In the technical solution of the embodiment of the present application, the first convolution result of the first convolution layer is subjected to a first upsampling process using a first upsampling module to obtain a final feature reconstruction result.

[0259] Here, the first upsampling module can be nn.PixelShuffle(2). The first upsampling module is used to perform a first upsampling process on the first convolution result, and the feature reconstruction result obtained is, for example, to increase the dimension of the first convolution result from (b, 1024, 64, 64) to (b, 256, 128, 128). At this point, the feature reconstruction result is obtained by the feature reconstruction network reconstructing the feature map extracted by the encoding network.

[0260] (3) Decoding Network

[0261] See also Figure 13 , is a schematic diagram of the decoding network structure. Figure 13 As shown, the decoding network includes: an eighth convolutional layer, a second denoising module, a second upsampling module, and a ninth convolutional layer. The input of the eighth convolutional layer is the input of the entire decoding network, the output of the eighth convolutional layer is connected to the input of the second denoising module, the output of the second denoising module is connected to the input of the second upsampling module, the output of the second upsampling module is connected to the input of the ninth convolutional layer, and the output of the ninth convolutional layer is the output of the entire decoding network.

[0262] On this basis, the decoding network is used to upsample the feature reconstruction result to obtain the denoised image. The specific implementation includes: using the eighth convolution layer to perform the eighth convolution processing on the feature reconstruction result to obtain the eighth convolution processing result; using the second denoising module to perform the second denoising processing on the eighth convolution processing result to obtain the second denoising processing result; using the second upsampling module to perform the second upsampling processing on the second denoising processing result to obtain the second upsampling result; using the ninth convolution layer to perform the ninth convolution processing on the second upsampling result to obtain the denoised image.

[0263] The eighth convolutional layer can be conv2, whose convolution kernel size is 1*1. The second upsampling module can be nn.PixelShuffle(2). The ninth convolutional layer can be conv2, whose convolution kernel size is 3*3.

[0264] In one embodiment, see Figure 14 , is a schematic diagram of the structure of the second denoising module. Figure 14As shown, the second denoising module includes a second channel attention module and a second visual channel attention module, wherein the input end of the second channel attention module is the input end of the entire second denoising module, the output end of the second channel attention module is connected to the input end of the second visual channel attention module, and the output end of the second visual channel attention module is the output end of the entire second denoising module.

[0265] On this basis, the processing process of the second denoising module includes: using the second channel attention module to perform channel attention learning on the input eighth convolution processing result to obtain the second channel attention learning result, and using the second visual channel attention module to perform visual channel attention learning on the second channel attention learning result to obtain the second visual channel attention learning result.

[0266] In one embodiment, the second channel attention module includes a fifth normalization layer, a sixteenth convolutional layer, a seventeenth convolutional layer, a second channel attention layer, and an eighteenth convolutional layer. The input of the fifth normalization layer is the input of the entire second channel attention module, the output of the fifth normalization layer is connected to the input of the sixteenth convolutional layer, the output of the sixteenth convolutional layer is connected to the input of the seventeenth convolutional layer, the output of the seventeenth convolutional layer is connected to the input of the second channel attention layer, the output of the second channel attention layer is connected to the input of the eighteenth convolutional layer, and the output of the eighteenth convolutional layer is the output of the entire second channel attention module.

[0267] The fifth normalization layer can be a layernorm layer. The sixteenth convolutional layer can be conv2, with a convolution kernel size of 1*1. The seventeenth convolutional layer can be conv2, with a convolution kernel size of 3*3. Here, the sixteenth and seventeenth convolutional layers form a set of depthwise separable convolutions. In depthwise separable convolutions, the purpose of using 1*1 convolutions is to increase the dimension, and the purpose of using 3*3 convolutions is to perform activation. The eighteenth convolutional layer can be conv2, with a convolution kernel size of 1*1.

[0268] The second visual channel attention module includes a sixth normalization layer, a nineteenth convolutional layer, a twentieth convolutional layer, a second visual channel attention layer, and a twenty-first convolutional layer. The input of the sixth normalization layer is the input of the entire second visual channel attention module, the output of the sixth normalization layer is connected to the input of the nineteenth convolutional layer, the output of the nineteenth convolutional layer is connected to the input of the twentieth convolutional layer, the output of the twentieth convolutional layer is connected to the input of the second visual channel attention layer, the output of the second visual channel attention layer is connected to the input of the twenty-first convolutional layer, and the output of the twenty-first convolutional layer is the output of the entire second visual channel attention module.

[0269] The sixth normalization layer can be a layernorm layer. The nineteenth convolutional layer can be conv2, with a convolution kernel size of 1*1. The twentieth convolutional layer can be conv2, with a convolution kernel size of 3*3. Here, the nineteenth and twentieth convolutional layers form a set of depthwise separable convolutions. The twenty-first convolutional layer can be conv2, with a convolution kernel size of 1*1.

[0270] On this basis, the detailed processing process of the second denoising module includes: first, using the layernorm layer (i.e., the fifth normalization layer) to normalize the input features of the second denoising module, and then convolving the above normalized results through a set of depthwise separable convolutions (i.e., the combination of the eleventh convolutional layer and the twelfth convolutional layer). The above convolution results are then calculated using the channel attention mechanism. The workflow is to first perform average pooling on the input convolution results to obtain a set of weight values, then perform nonlinear mapping through 1*1 convolution, and finally weight the nonlinear mapping results to the input features.

[0271] Subsequently, 1*1 convolution (i.e., the thirteenth convolution layer) is used for convolution processing to reduce the channel dimension of the input features, reduce the amount of calculation and the number of parameters, thereby improving the processing efficiency of the model.

[0272] Subsequently, the convolution processing result at this time is weighted to the input feature and input to the next layernorm layer (i.e., the sixth normalization layer).

[0273] The next layernorm layer is used to normalize the input features, and then convolves the normalized results through a set of depthwise separable convolutions (i.e., a combination of the 19th and 24th convolutional layers). The convolution results are then subjected to the visual channel attention mechanism. The workflow is to first use average pooling to obtain a set of weights, then use 1x1 convolution to reduce the dimensionality, then use ReLU layers to increase nonlinearity, and then use 1x1 convolution to increase the dimensionality. Finally, a sigmoid layer is used to obtain a set of weights ranging from 0 to 1, which are used to weight the input features.

[0274] Subsequently, a 1x1 convolution (i.e., the 21st convolutional layer) is used to reduce the channel dimension of the input features, reducing the amount of computation and the number of parameters, thereby improving the efficiency of the model. The convolution result is then weighted onto the input features to obtain the attention learning results of the second visual channel.

[0275] This completes the explanation of the structure and processing process of the second denoising module.

[0276] In addition, in one embodiment, the decoding network includes a combination of two groups of eighth convolutional layers, a second denoising module, and a second upsampling module, the input end of the first combination is connected to the output end of the feature reconstruction network, the output end of the first combination is connected to the input end of the second combination, and the output end of the second combination is connected to the input end of the ninth convolutional layer.

[0277] Based on this, the decoding network's processing flow includes: The eighth convolutional layer in the first combination upscales the feature reconstruction network's input (b, 512, 64, 64) to a (b, 512, 128, 128) tensor. The second denoising module in the first combination then upscales the (b, 512, 128, 128) tensor to a (b, 128, 256, 256) tensor.

[0278] Then, the eighth convolutional layer in the second combination upscales the (b, 128, 256, 256) tensor to a (b, 256, 256, 256) tensor. Then, the second denoising module in the second combination upscales the (b, 256, 256, 256) tensor to a (b, 64, 512, 512) tensor, and then combines it with the (b, 64, 512, 512) tensor on the left.

[0279] Finally, the ninth convolutional layer maps the (b, 64, 512, 512) tensor to the (b, 3, 512, 512) tensor, resulting in the final denoised image.

[0280] It should be noted that for convolution kernels of the same size in different convolutional layers, the parameters in the convolution kernels may be different, and the embodiments of the present application do not limit this.

[0281] At this point, the three parts of the image denoising model and their processing processes provided by the technical solution of the embodiment of the present application are explained respectively.

[0282] On this basis, see Figure 15 , which is a structural diagram of the image denoising model provided by the technical solution of the embodiment of the present application. Figure 15 In the example, DB stands for DenosingBlock, which represents the denoising module, and TB stands for TransformerBlock, which represents the Transformer module. The specific processing of this image denoising model is described above and will not be repeated here.

[0283] See also Figure 16 , is a flow chart of another embodiment of an image denoising method provided in an embodiment of the present application, such as Figure 16 As shown, the process includes the following steps:

[0284] Step 1601: Obtain the image to be denoised.

[0285] Step 1602: Input the image to be denoised into the trained image denoising model to obtain a denoised image output by the image denoising model.

[0286] The specific implementation of step 1601 and step 1602 can be found in the explanation in the above embodiment and will not be repeated here.

[0287] Step 1603: sharpen the denoised image to obtain a sharpened denoised image.

[0288] It can be seen from the description of step 1603 that after the denoised image is output by the image denoising model, the denoised image can be further sharpened, which can make the objects in the image more prominent and conspicuous, thereby improving the readability of the denoised image.

[0289] Figure 16 The process shown in Figure 1 Based on the process shown, a convolutional neural network is combined with the Transformer architecture to design an image denoising model. This image denoising model is used to process the denoised image to obtain a denoised image. Since the convolutional neural network can process images of dynamic sizes and has a relatively fast model processing speed, and the Transformer architecture can achieve good image denoising effects, the above image denoising model can be used to denoise images of any size and has great flexibility. In addition, applying this image denoising model to image denoising can achieve both good image denoising effects and improved image denoising efficiency. Furthermore, by sharpening the denoised image output by the image denoising model, the objects in the image are made more prominent and conspicuous, thereby improving the readability of the denoised image.

[0290] See also Figure 17 , is a flow chart of an embodiment of a method for training an image denoising model provided in an embodiment of the present application. Figure 17 As shown, the process includes the following steps:

[0291] Step 1701: Obtain the original image set.

[0292] Step 1702: Degrade the original pictures in the original picture set according to a preset picture degradation method to obtain a degraded picture set.

[0293] Step 1703: construct a training sample set using the original picture set and the degraded picture set, wherein the training samples in the training sample set use the corresponding degraded pictures and original pictures as input values ​​and label values, respectively.

[0294] For ease of understanding, steps 1701 to 1703 are described below in a unified manner:

[0295] The training of deep learning models relies on large datasets. For the image denoising model used in the embodiments of this application, its training relies on a large number of paired low-quality and high-quality images. Here, low-quality images refer to images containing noise and artifacts, while high-quality images refer to clear images without noise and artifacts, such as HD images. However, in actual applications, it is usually not possible to directly obtain paired low-quality and high-quality images.

[0296] In this regard, an embodiment of the present application proposes to use a method of degrading high-quality images to obtain pairs of low-quality images and high-quality images for model training.

[0297] Based on the above description, the original image set includes multiple original images, and the original images here are clear images without noise and noise, such as high-definition images. In one embodiment, multiple high-definition images can be obtained from an open source high-definition image library as original images to form the original image set.

[0298] The aforementioned preset image degradation methods include, but are not limited to, one or a combination of the following methods: adding noise, filtering, random perturbation, momentum noise, JPEG compression noise, etc. When the image degradation method includes a combination of the aforementioned methods, the image degradation methods may be sequentially applied to an original image, such as sequentially applying noise, random perturbation, and momentum noise to the original image.

[0299] In one embodiment, different original images in the original image set are degraded according to different image degradation methods to obtain a degraded image set. Here, the different image degradation methods can be different single image degradation methods or a combination of different image degradation methods, and the embodiment of the present application does not limit this. In addition, the same original image in the original image set can be degraded multiple times according to different image degradation methods. In this way, multiple degraded images can be obtained for a single original image, thereby forming multiple pairs of low-quality images and high-quality images for the single original image.

[0300] Whether degrading different original images in the original image set according to different image degradation methods, or degrading the same original image in the original image set multiple times according to different image degradation methods, the richness of the training samples can be increased, and the image denoising effect of the trained image denoising model can be further improved.

[0301] In addition, in one embodiment, the original pictures in the original picture set can also be sharpened, so as to use the sharpened original picture set to execute the step of constructing a training sample set using the original picture set and the degraded picture set. By constructing the training sample in this way, it is possible to learn the process from low-quality pictures to high-quality sharpened pictures in the process of training the picture denoising model, and then the denoising picture can be directly denoised and sharpened by the trained picture denoising model, thereby eliminating the above-mentioned Figure 16 Step 1603 in the illustrated embodiment further improves the denoising effect of the image denoising model.

[0302] Step 1704: Use the training sample set to train the initial convolutional neural network model to obtain an image denoising model, wherein the initial convolutional neural network model includes an encoding network, a feature reconstruction network, and a decoding network.

[0303] The encoding network includes a sixth convolutional layer, a denoising module, and a seventh convolutional layer. The input of the sixth convolutional layer is the input of the entire encoding network, the output of the sixth convolutional layer is connected to the input of the denoising module, the output of the first denoising module is connected to the input of the seventh convolutional layer, and the output of the seventh convolutional layer is the output of the entire encoding network.

[0304] Furthermore, the first denoising module includes a first channel attention module and a first visual channel attention module, wherein the input end of the first channel attention module is the input end of the entire first denoising module, the output end of the first channel attention module is connected to the input end of the first visual channel attention module, and the output end of the first visual channel attention module is the output end of the entire first denoising module.

[0305] The first channel attention module includes a third normalization layer, a tenth convolutional layer, an eleventh convolutional layer, a first channel attention layer, and a twelfth convolutional layer. The input of the third normalization layer is the input of the entire first channel attention module, the output of the third normalization layer is connected to the input of the tenth convolutional layer, the output of the tenth convolutional layer is connected to the input of the eleventh convolutional layer, the output of the eleventh convolutional layer is connected to the input of the first channel attention layer, the output of the first channel attention layer is connected to the input of the twelfth convolutional layer, and the output of the twelfth convolutional layer is the output of the entire first channel attention module.

[0306] The third normalization layer can be a layernorm layer. The tenth convolutional layer can be conv2, with a convolution kernel size of 1*1. The eleventh convolutional layer can be conv2, with a convolution kernel size of 3*3. Here, the tenth and eleventh convolutional layers form a set of depthwise separable convolutions. In depthwise separable convolutions, 1*1 convolutions are used to increase the dimension, while 3*3 convolutions are used for activation. The twelfth convolutional layer can be conv2, with a convolution kernel size of 1*1.

[0307] The first visual channel attention module includes a fourth normalization layer, a thirteenth convolutional layer, a fourteenth convolutional layer, a first visual channel attention layer, and a fifteenth convolutional layer. The input of the fourth normalization layer is the input of the entire first visual channel attention module, the output of the fourth normalization layer is connected to the input of the thirteenth convolutional layer, the output of the thirteenth convolutional layer is connected to the input of the fourteenth convolutional layer, the output of the fourteenth convolutional layer is connected to the input of the first visual channel attention layer, the output of the first visual channel attention layer is connected to the input of the fifteenth convolutional layer, and the output of the fifteenth convolutional layer is the output of the entire first visual channel attention module.

[0308] The fourth normalization layer can be a layernorm layer. The thirteenth convolutional layer can be conv2, with a convolution kernel size of 1*1. The fourteenth convolutional layer can be conv2, with a convolution kernel size of 3*3. Here, the thirteenth and fourteenth convolutional layers form a set of depthwise separable convolutions. The fifteenth convolutional layer can be conv2, with a convolution kernel size of 1*1.

[0309] In one embodiment, the encoding network further includes two combinations of first denoising modules and convolutional layers, wherein the input of the first combination is connected to the output of the seventh convolutional layer, the output of the first combination is connected to the input of the second combination, and the output of the second combination is connected to the input of the feature reconstruction network. The output of the first denoising module in each combination is connected to the input of the convolutional layer in that combination.

[0310] The feature reconstruction network includes a Transformer module, a first convolutional layer, and a first upsampling module. The input of the Transformer module is connected to the output of the encoding network, the output of the Transformer module is connected to the input of the first convolutional layer, the output of the first convolutional layer is connected to the input of the first upsampling module, and the output of the first upsampling module is connected to the input of the decoding network.

[0311] Furthermore, the Transformer module includes a convolutional self-attention module and a convolutional feedforward module. The input of the convolutional self-attention module is the input of the entire Transformer module, the output of the convolutional self-attention module is connected to the input of the convolutional feedforward module, and the output of the convolutional feedforward module is the output of the entire Transformer module.

[0312] The convolutional self-attention module includes a first normalization layer, a second convolutional layer, a softmax layer, and a third convolutional layer. The input of the first normalization layer is the input of the entire convolutional self-attention module, the output of the first normalization layer is connected to the input of the second convolutional layer, the output of the second convolutional layer is connected to the input of the softmax layer, the output of the softmax layer is connected to the input of the third convolutional layer, and the output of the third convolutional layer is the output of the entire convolutional self-attention module.

[0313] The convolutional feedforward module includes a second normalization layer, a fourth convolutional layer, an activation layer, and a fifth convolutional layer. The input of the second normalization layer is the input of the entire convolutional feedforward module, the output of the second normalization layer is connected to the input of the fourth convolutional layer, the output of the fourth convolutional layer is connected to the input of the activation layer, the output of the activation layer is connected to the input of the fifth convolutional layer, and the output of the fifth convolutional layer is the output of the entire convolutional feedforward module.

[0314] The decoding network includes an eighth convolutional layer, a second denoising module, a second upsampling module, and a ninth convolutional layer. The input of the eighth convolutional layer serves as the input of the entire decoding network. The output of the eighth convolutional layer is connected to the input of the second denoising module, which in turn is connected to the input of the second upsampling module. The output of the second upsampling module is connected to the input of the ninth convolutional layer, which serves as the output of the entire decoding network.

[0315] The second denoising module includes a second channel attention module and a second visual channel attention module, wherein the input end of the second channel attention module is the input end of the entire second denoising module, the output end of the second channel attention module is connected to the input end of the second visual channel attention module, and the output end of the second visual channel attention module is the output end of the entire second denoising module.

[0316] The second channel attention module includes a fifth normalization layer, a sixteenth convolutional layer, a seventeenth convolutional layer, a second channel attention layer, and an eighteenth convolutional layer. The input of the fifth normalization layer is the input of the entire second channel attention module, the output of the fifth normalization layer is connected to the input of the sixteenth convolutional layer, the output of the sixteenth convolutional layer is connected to the input of the seventeenth convolutional layer, the output of the seventeenth convolutional layer is connected to the input of the second channel attention layer, the output of the second channel attention layer is connected to the input of the eighteenth convolutional layer, and the output of the eighteenth convolutional layer is the output of the entire second channel attention module.

[0317] The fifth normalization layer can be a layernorm layer. The sixteenth convolutional layer can be conv2, with a convolution kernel size of 1*1. The seventeenth convolutional layer can be conv2, with a convolution kernel size of 3*3. Here, the sixteenth and seventeenth convolutional layers form a set of depthwise separable convolutions. In depthwise separable convolutions, the purpose of using 1*1 convolutions is to increase the dimension, and the purpose of using 3*3 convolutions is to perform activation. The eighteenth convolutional layer can be conv2, with a convolution kernel size of 1*1.

[0318] The second visual channel attention module includes a sixth normalization layer, a nineteenth convolutional layer, a twentieth convolutional layer, a second visual channel attention layer, and a twenty-first convolutional layer. The input of the sixth normalization layer is the input of the entire second visual channel attention module, the output of the sixth normalization layer is connected to the input of the nineteenth convolutional layer, the output of the nineteenth convolutional layer is connected to the input of the twentieth convolutional layer, the output of the twentieth convolutional layer is connected to the input of the second visual channel attention layer, the output of the second visual channel attention layer is connected to the input of the twenty-first convolutional layer, and the output of the twenty-first convolutional layer is the output of the entire second visual channel attention module.

[0319] The sixth normalization layer can be a layernorm layer. The nineteenth convolutional layer can be conv2, with a convolution kernel size of 1*1. The twentieth convolutional layer can be conv2, with a convolution kernel size of 3*3. Here, the nineteenth and twentieth convolutional layers form a set of depthwise separable convolutions. The twenty-first convolutional layer can be conv2, with a convolution kernel size of 1*1.

[0320] In addition, in one embodiment, the decoding network includes a combination of two groups of eighth convolutional layers, a second denoising module, and a second upsampling module, the input end of the first combination is connected to the output end of the feature reconstruction network, the output end of the first combination is connected to the input end of the second combination, and the output end of the second combination is connected to the input end of the ninth convolutional layer.

[0321] In one embodiment, the initial convolutional neural network model is iteratively trained using a training sample set. After each round of iterative training, the loss value of the currently trained convolutional neural network model is calculated. When the loss value does not meet the preset iteration termination condition, the next round of iterative training is continued until the loss value of the trained convolutional neural network model meets the preset iteration termination condition. The convolutional neural network model obtained by the last round of training is determined as the trained image denoising model.

[0322] Among them, as an optional implementation method, the CharbonnierLoss loss function is used to calculate the loss value of the convolutional neural network model during the training process. This loss function can handle outliers well and improve the realism of the denoising effect of the image denoising model obtained by the final training.

[0323] The technical solution provided in the embodiment of the present application degrades the original images in the original image set according to a preset image degradation method to obtain a degraded image set, and uses the original image set and the degraded image set to construct a training sample set, thereby solving the problem of difficulty in obtaining paired low-quality images and high-quality images in practical applications, and meeting the prerequisites for training image denoising models.

[0324] Figure 18 This is a block diagram of an embodiment of an image denoising device provided in an embodiment of the present application. Figure 18 As shown, the device includes:

[0325] Image acquisition module 1801, used to acquire the image to be denoised;

[0326] A downsampling module 1802 is configured to perform downsampling processing on the image to be denoised to obtain a feature map of the image to be denoised;

[0327] A feature reconstruction module 1803 is configured to reconstruct different features in the feature map using different feature reconstruction methods to obtain feature reconstruction results;

[0328] The upsampling module 1804 is configured to perform upsampling processing on the feature reconstruction result to obtain a denoised image.

[0329] In a possible implementation, the downsampling module 1802 is specifically configured to:

[0330] Downsampling the image to be denoised using the encoding network in the trained image denoising model to obtain a feature map of the image to be denoised;

[0331] The feature reconstruction module 1803 is specifically configured to: utilize a feature reconstruction network in a trained image denoising model to perform feature reconstruction on different features in the feature map using different feature reconstruction methods to obtain a feature reconstruction result;

[0332] The upsampling module 1804 is specifically configured to perform upsampling processing on the feature reconstruction result using a decoding network in a trained image denoising model to obtain the denoised image.

[0333] In one possible implementation, the feature reconstruction network includes a Transformer module, a first convolutional layer, and a first upsampling module; the feature reconstruction module 1803 includes:

[0334] A feature learning subunit is used to perform deep feature learning on different features in the feature map using the Transformer module to obtain deep feature learning results;

[0335] A first convolution subunit, configured to perform a first convolution process on the deep feature learning result using the first convolution layer to obtain a first convolution result;

[0336] The first upsampling subunit is configured to perform a first upsampling process on the first convolution result using the first upsampling module to obtain the feature reconstruction result.

[0337] In one possible implementation, the Transformer module includes a convolutional self-attention module and a convolutional feedforward module; the feature learning subunit includes:

[0338] A self-attention subunit is configured to perform self-attention learning on the feature map using the convolutional self-attention module to obtain a self-attention learning result, wherein the self-attention learning result includes weights of different features in the feature map and sub-feature maps of different features obtained by processing according to the weights;

[0339] A convolution feedforward subunit is used to use the convolution feedforward module to perform convolution feedforward processing on the sub-feature maps of different features in the self-attention learning result to obtain the deep feature learning result.

[0340] In one possible implementation, the convolutional self-attention module includes a first normalization layer, a second convolutional layer, a softmax layer, and a third convolutional layer; the self-attention subunit includes:

[0341] a first normalization subunit, configured to perform a first normalization process on the feature map using the first normalization layer to obtain a first normalization result;

[0342] a second convolution subunit, configured to perform a second convolution process on the first normalized result using the second convolution layer to obtain a second convolution result;

[0343] a first weight learning subunit, configured to perform first weight learning on the second convolution result using the softmax layer to obtain a first weight learning result, where the first weight learning result includes weights of different features in the feature map;

[0344] A first operator unit, configured to perform a first operation on the second convolution result using the first weight learning result to obtain a first operation result;

[0345] a second convolution subunit, configured to perform a third convolution process on the first operation result using the third convolution layer to obtain a third convolution result;

[0346] The second operator unit is used to perform a second operation using the third convolution result and the feature map to obtain the self-attention learning result.

[0347] In a possible implementation, the second convolution result includes a first characteristic matrix, a second characteristic matrix, and a third characteristic matrix; and the first weight learning subunit is specifically configured to:

[0348] performing a third operation on the second characteristic matrix and a transposed matrix of the third characteristic matrix to obtain a third operation result;

[0349] Performing first weight learning on the third operation result using the softmax layer to obtain the first weight learning result;

[0350] The first operator unit is specifically configured to:

[0351] A first operation is performed on the first feature matrix using the first weight learning result to obtain a first operation result.

[0352] In one possible implementation, the convolutional feedforward module includes a second normalization layer, a fourth convolutional layer, an activation layer, and a fifth convolutional layer; the convolutional feedforward subunit includes:

[0353] a second normalization subunit, configured to perform a second normalization process on the self-attention learning result using the second normalization layer to obtain a second normalized result;

[0354] a fourth convolution subunit, configured to perform a fourth convolution process on the second normalized result using the fourth convolution layer to obtain a fourth convolution result;

[0355] an activation subunit, configured to perform activation processing on the fourth convolution result using the activation layer to obtain an activation result;

[0356] a fifth convolution subunit, configured to perform a fifth convolution process on the activation result using the fifth convolution layer to obtain a fifth convolution result;

[0357] The fourth operator unit is used to perform a fourth operation on the fifth convolution result and the self-attention learning result to obtain the deep feature learning result.

[0358] In one possible implementation, the encoding network includes a sixth convolutional layer, a first denoising module, and a seventh convolutional layer; and the feature extraction unit includes:

[0359] a sixth convolution subunit, configured to perform a sixth convolution process on the image to be denoised using the sixth convolution layer to obtain a sixth convolution process result;

[0360] a first denoising subunit, configured to perform a first denoising process on the sixth convolution processing result using the first denoising module to obtain a first denoising processing result;

[0361] The seventh convolution subunit is configured to perform a seventh convolution on the denoising result using the seventh convolution layer to obtain a feature map of the image to be denoised.

[0362] In one possible implementation, the decoding network includes: an eighth convolutional layer, a second denoising module, a second upsampling module, and a ninth convolutional layer, and the feature denoising unit includes:

[0363] an eighth convolution subunit, configured to perform an eighth convolution process on the feature reconstruction result using the eighth convolution layer to obtain an eighth convolution process result;

[0364] a second denoising subunit, configured to perform a second denoising process on the eighth convolution processing result using the second denoising module to obtain a second denoising processing result;

[0365] a second upsampling subunit, configured to perform a second upsampling process on the second denoising result using the second upsampling module to obtain a second upsampling result;

[0366] A ninth convolution subunit is configured to perform a ninth convolution process on the second upsampling result using the ninth convolution layer to obtain the denoised image.

[0367] In a possible implementation, the first denoising module includes: a first channel attention module and a first visual channel attention module;

[0368] The first denoising subunit is specifically configured to: perform channel attention learning on the sixth convolution processing result using the first channel attention module to obtain a first channel attention learning result;

[0369] The first visual channel attention module is used to perform visual channel attention learning on the first channel attention learning result to obtain a first visual channel attention learning result.

[0370] In a possible implementation, the second denoising module includes: a second channel attention module and a second visual channel attention module;

[0371] The second denoising subunit is specifically configured to: perform channel attention learning on the eighth convolution processing result using the second channel attention module to obtain a second channel attention learning result;

[0372] The second visual channel attention module is used to perform visual channel attention learning on the second channel attention learning result to obtain a second visual channel attention learning result.

[0373] In a possible implementation, the device further includes:

[0374] Sample construction module, used to obtain the original image set;

[0375] Degrading the original pictures in the original picture set according to a preset picture degradation method to obtain a degraded picture set;

[0376] Constructing a training sample set using the original picture set and the degraded picture set, wherein the training samples in the training sample set use the corresponding degraded pictures and original pictures as input values ​​and label values ​​respectively;

[0377] A model training module is used to train the initial convolutional neural network model using the training sample set to obtain the image denoising model, wherein the initial convolutional neural network model includes an encoding network, a feature reconstruction network and a decoding network.

[0378] In a possible implementation, the sample construction module includes:

[0379] The degradation unit is configured to degrade different original pictures in the original picture set according to different picture degradation methods to obtain a degraded picture set.

[0380] In a possible implementation, the sample construction module further includes:

[0381] A sharpening unit is used to perform sharpening processing on the original pictures in the original picture set, so as to use the sharpened original picture set to perform the step of constructing a training sample set using the original picture set and the degraded picture set.

[0382] like Figure 19As shown, an embodiment of the present application provides an electronic device, including a processor 111, a communication interface 112, a memory 113 and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.

[0383] Memory 113, for storing computer programs;

[0384] In one embodiment of the present application, the processor 111 is configured to execute a program stored in the memory 113 to implement the image denoising method provided by any of the aforementioned method embodiments, including:

[0385] Get the image to be denoised;

[0386] Performing downsampling processing on the image to be denoised to obtain a feature map of the image to be denoised;

[0387] Reconstructing features using different feature reconstruction methods for different features in the feature map to obtain feature reconstruction results;

[0388] The feature reconstruction result is up-sampled to obtain a denoised image.

[0389] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the image denoising method provided in any of the aforementioned method embodiments are implemented.

[0390] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0391] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general hardware platform, or of course, by hardware. Based on this understanding, the above technical solution, in essence, or the part that contributes to the relevant technology, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiment.

[0392] It should be understood that the terms used herein are for the purpose of describing specific example embodiments only and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "comprise", "include", "contain" and "have" are inclusive and therefore specify the presence of stated features, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not to be construed as necessarily requiring them to be performed in the specific order described or illustrated, unless the order of execution is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0393] The foregoing is merely a list of specific embodiments of the present application, intended to enable those skilled in the art to understand or implement the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the broadest scope consistent with the principles and novel features of the present application.

Claims

1. A method for image denoising, characterized in that: The method comprises: Get the image to be denoised; Performing downsampling processing on the image to be denoised to obtain a feature map of the image to be denoised; Reconstructing features using different feature reconstruction methods for different features in the feature map to obtain feature reconstruction results; The feature reconstruction result is up-sampled to obtain a denoised image.

2. The method according to claim 1, characterized in that The downsampling process is performed on the image to be denoised to obtain a feature map of the image to be denoised, including: Downsampling the image to be denoised using the encoding network in the trained image denoising model to obtain a feature map of the image to be denoised; The reconstructing different features in the feature map using different feature reconstruction methods to obtain feature reconstruction results includes: Using a feature reconstruction network in a trained image denoising model to reconstruct different features in the feature map using different feature reconstruction methods to obtain feature reconstruction results; The upsampling process is performed on the feature reconstruction result to obtain the denoised image, including: The feature reconstruction result is upsampled using a decoding network in a trained image denoising model to obtain the denoised image.

3. The method according to claim 2, characterized in that The feature reconstruction network includes a Transformer module, a first convolutional layer, and a first upsampling module. The feature reconstruction network in the trained image denoising model is used to reconstruct different features in the feature map using different feature reconstruction methods to obtain feature reconstruction results, including: Using the Transformer module to perform deep feature learning on different features in the feature map to obtain deep feature learning results; Performing a first convolution process on the deep feature learning result using the first convolution layer to obtain a first convolution result; The first upsampling module is used to perform a first upsampling process on the first convolution result to obtain the feature reconstruction result.

4. The method according to claim 3, characterized in that The Transformer module includes a convolutional self-attention module and a convolutional feedforward module; the Transformer module is used to perform deep feature learning on different features in the feature map to obtain deep feature learning results, including: Performing self-attention learning on the feature map using the convolutional self-attention module to obtain a self-attention learning result, wherein the self-attention learning result includes weights of different features in the feature map and sub-feature maps of the different features obtained by processing according to the weights; The convolution feedforward module is used to perform convolution feedforward processing on the sub-feature maps of different features in the self-attention learning result to obtain the deep feature learning result.

5. The method according to claim 4, characterized in that The convolutional self-attention module includes a first normalization layer, a second convolutional layer, a softmax layer, and a third convolutional layer; the self-attention learning is performed on the feature map using the convolutional self-attention module to obtain a self-attention learning result, including: Performing a first normalization process on the feature map using the first normalization layer to obtain a first normalization result; Performing a second convolution process on the first normalized result using the second convolution layer to obtain a second convolution result; Performing first weight learning on the second convolution result using the softmax layer to obtain a first weight learning result, where the first weight learning result includes weights of different features in the feature map; Performing a first operation on the second convolution result using the first weight learning result to obtain a first operation result; Performing a third convolution process on the first operation result using the third convolution layer to obtain a third convolution result; A second operation is performed using the third convolution result and the feature map to obtain the self-attention learning result.

6. The method according to claim 5, characterized in that The second convolution result includes a first feature matrix, a second feature matrix, and a third feature matrix; and performing first weight learning on the second convolution result using the softmax layer to obtain a first weight learning result includes: performing a third operation on the second characteristic matrix and a transposed matrix of the third characteristic matrix to obtain a third operation result; Performing first weight learning on the third operation result using the softmax layer to obtain the first weight learning result; The performing a first operation on the second convolution result using the first weight learning result to obtain a first operation result includes: A first operation is performed on the first feature matrix using the first weight learning result to obtain a first operation result.

7. The method according to claim 4, characterized in that The convolution feedforward module includes a second normalization layer, a fourth convolution layer, an activation layer, and a fifth convolution layer; the convolution feedforward module is used to perform convolution feedforward processing on the self-attention learning result to obtain the deep feature learning result, including: Performing a second normalization process on the self-attention learning result using the second normalization layer to obtain a second normalized result; Performing a fourth convolution process on the second normalized result using the fourth convolution layer to obtain a fourth convolution result; Performing activation processing on the fourth convolution result using the activation layer to obtain an activation result; Performing a fifth convolution process on the activation result using the fifth convolution layer to obtain a fifth convolution result; A fourth operation is performed on the fifth convolution result and the self-attention learning result to obtain the deep feature learning result.

8. The method according to claim 2, characterized in that The encoding network includes a sixth convolutional layer, a first denoising module, and a seventh convolutional layer; the encoding network in the trained image denoising model is used to downsample the image to be denoised to obtain a feature map of the image to be denoised, including: Performing a sixth convolution process on the image to be denoised using the sixth convolution layer to obtain a sixth convolution process result; Performing a first denoising process on the sixth convolution processing result using the first denoising module to obtain a first denoising processing result; The seventh convolution layer is used to perform a seventh convolution on the denoising result to obtain a feature map of the image to be denoised.

9. The method according to claim 2, characterized in that The decoding network includes: an eighth convolutional layer, a second denoising module, a second upsampling module, and a ninth convolutional layer. The decoding network in the trained image denoising model is used to upsample the feature reconstruction result to obtain the denoised image, including: Performing an eighth convolution process on the feature reconstruction result using the eighth convolution layer to obtain an eighth convolution process result; performing a second denoising process on the eighth convolution processing result using the second denoising module to obtain a second denoising processing result; performing a second upsampling process on the second denoising result using the second upsampling module to obtain a second upsampling result; Performing a ninth convolution process on the second upsampling result using the ninth convolution layer to obtain the denoised image.

10. The method according to claim 8, characterized in that The first denoising module includes: a first channel attention module and a first visual channel attention module; the first denoising module is used to perform a first denoising process on the sixth convolution processing result to obtain a first denoising processing result, including: Using the first channel attention module to perform channel attention learning on the sixth convolution processing result to obtain a first channel attention learning result; The first visual channel attention module is used to perform visual channel attention learning on the first channel attention learning result to obtain a first visual channel attention learning result.

11. The method according to claim 9, characterized in that The second denoising module includes: a second channel attention module and a second visual channel attention module; the second denoising module is used to perform a second denoising process on the eighth convolution processing result to obtain a second denoising processing result, including: Using the second channel attention module to perform channel attention learning on the eighth convolution processing result, to obtain a second channel attention learning result; The second visual channel attention module is used to perform visual channel attention learning on the second channel attention learning result to obtain a second visual channel attention learning result.

12. The method according to any one of claims 2 to 10, characterized in that: The image denoising model is trained in the following way: Get the original image set; Degrading the original pictures in the original picture set according to a preset picture degradation method to obtain a degraded picture set; Constructing a training sample set using the original picture set and the degraded picture set, wherein the training samples in the training sample set use the corresponding degraded pictures and original pictures as input values ​​and label values ​​respectively; The initial convolutional neural network model is trained using the training sample set to obtain the image denoising model, wherein the initial convolutional neural network model includes an encoding network, a feature reconstruction network, and a decoding network.

13. An image denoising device, characterized in that: The device comprises: Image acquisition module, used to obtain the image to be denoised; A downsampling module, configured to perform downsampling processing on the image to be denoised to obtain a feature map of the image to be denoised; A feature reconstruction module, configured to reconstruct different features in the feature map using different feature reconstruction methods to obtain feature reconstruction results; The upsampling module is used to perform upsampling processing on the feature reconstruction result to obtain a denoised image.

Images