A denoising model and a method and device for establishing the same, and an image denoising method and device

Abstract

The application provides a denoising model and a method and device for establishing the same, and an image denoising method and device. The method for establishing the denoising model comprises the following steps: constructing a reference clean image according to the non-local similarity of adjacent pixels of a noisy image; and training parameters in the denoising model according to the noisy image and the reference clean image. The input of a first network layer in the denoising model is connected to the input of a second network layer through an edge detection module, the output of the first network layer is connected to the output of the second network layer after being multiplied by a point, and the multiplied result is connected to a first image reconstruction layer. The image reconstruction layer is used for reconstructing the features obtained by the multiplication to obtain a clean image. The application can extract enhanced local features and enhanced edge features, use the extracted edge features to guide the first network layer to obtain more edge detail information in an attention mechanism mode, obtain more accurate clean image detail information, suppress noise, and improve robustness and denoising capability.

Description

Technical Field

[0001] This article relates to the field of image restoration, and in particular to a denoising model and its establishment method and apparatus, as well as an image denoising method and apparatus. Background Technology

[0002] Images are a crucial medium for human-computer interaction. Therefore, they are widely used in aerospace exploration, oceanographic surveys, atmospheric monitoring, medical diagnosis, and public entertainment. However, during image acquisition, cameras are often affected by camera shake, the shooting environment, and moving objects, resulting in noisy images. To address this problem, researchers have proposed image denoising methods. First, based on the mechanisms of image degradation caused by cameras, researchers construct denoising models using prior knowledge to process noisy images. For example, block-matching 3D filter methods convert two-dimensional images into three-dimensional arrays, and then use collaborative filters to enhance the sparsity of these 3D arrays, achieving noise reduction. To further improve denoising efficiency, researchers have proposed decomposing similar image patches using similar sparsity patterns, integrating nonlocal averaging and sparse coding methods into a single framework to address image denoising and mosaic removal problems. To enhance denoising performance, researchers use image nonlocal self-similarity to obtain estimates of the sparse coding of the original image, and then modulate the effect of these estimates by centering the sparse coding coefficients of the observed image. This effectively handles various image restoration problems such as image denoising, image deblurring, and image super-resolution. The weighted kernel normalization minimization method utilizes singular values ​​to obtain different weights as solutions, and then relies on nonlocal self-similarity for image denoising.

[0003] While prior knowledge can lead to better denoising results, these methods require manual parameter setting and complex optimization algorithms to obtain optimal parameters. To compensate for these shortcomings, discriminative learning methods are used to train image prior models. For example, a semi-quadratic splitting optimization method is used to construct a contraction field cascade model to restore eroded images; gradient descent inference of image prior knowledge in the expert domain is used to train the denoising model. Although discriminative learning methods have achieved good performance in image denoising, these methods are all limited by the priors used. Furthermore, these methods require several manual parameter tuning steps to obtain optimal parameters. In addition, these methods train the denoiser using only a fixed noise level, making them ineffective for blind denoising.

[0004] Recently proposed Convolutional Neural Networks (CNNs) have been widely used in image denoising due to their flexible, plug-and-play architecture. For example, a deep convolutional neural denoising network can be constructed using block normalization and residual learning operations. The normalization problem addresses the uneven distribution of training samples caused by convolution operations during CNN training by normalizing samples within blocks. Simultaneously, residual learning operations are applied to a given noisy image and a predicted noisy mapping image to reconstruct a clean image. The synergistic effect of these two techniques enables the denoiser to effectively handle various low-level visual tasks, such as image denoising, image super-resolution, and image decompression. These multi-task capabilities rely on well-designed network structures. If the designed network cannot handle complex low-level visual tasks well, such as real-world noisy images and blindly noisy images, it will fail to recover the details of the clean image, resulting in less effective denoising performance. To address this issue, researchers have proposed multi-feature fusion methods to enhance the robustness of the acquired features. For example, to further address the problem of denoising more severely damaged images, a dual-network structure is constructed by expanding the network width to restore more complex noisy images. Simultaneously, to prevent the loss of depth information, dilated convolution is only used in one sub-network, allowing it to capture more contextual depth information through the dilation factor. This complements the feature extraction width of the dual-network structure, contributing to obtaining more robust features. Furthermore, to address the problem of uneven sample distribution during training due to limited hardware resources, renormalized samples are used in both sub-networks. They normalize the entire sample set to unify the sample distribution and accelerate the convergence speed of the denoising network. Moreover, to further suppress noise, residual learning operations are applied to the ends of both sub-networks to obtain a staged potential clean image. Over-connection operations are then used to fuse the two clean images, enhancing the pixels of the clean image. The synergistic work of these key technologies makes the resulting dual-network denoiser highly effective for restoring complex eroded images, such as those with realistic noise. While these methods effectively solve the image denoising problem, they all rely on training the denoiser with a labeled clean image (Ground Truth). However, the degree of damage to the captured noisy images is not uniform. Furthermore, there are no corresponding clean reference images for the collected noisy images, which limits the denoising networks mentioned above in denoising real-world noisy images without a matching pair. Summary of the Invention

[0005] This paper addresses the technical problems of existing neural network denoisers, which are not applicable to clean images without references during training, and the trained denoisers are not suitable for images with blind noise, resulting in a narrow range of application and poor recognition performance.

[0006] To address the aforementioned technical problems, the first aspect of this paper provides a denoising model, including:

[0007] The network consists of a first network layer, an edge detection module, a second network layer, and an image reconstruction layer.

[0008] The inputs of the first network layer and the second network layer are connected through an edge detection module, and the outputs of the first network layer and the second network layer are multiplied together and then connected to the image reconstruction layer.

[0009] The first network layer is used to extract enhanced local features from the noisy image;

[0010] The edge detection module is used to generate an edge image based on the noise image;

[0011] The second network layer is used to extract enhanced edge features from the edge image;

[0012] The image reconstruction layer is used to reconstruct the features obtained by dot product to obtain a clean image.

[0013] In a further embodiment of this paper, the first network layer includes N layers, each of which consists of interconnected heterogeneous convolutional layers and a first activation function;

[0014] The second network layer comprises N-1 layers, each consisting of a single convolutional layer and a second activation function. The first N-2 layers have identical convolutional kernels, and the N-1th convolutional layer is... ;

[0015] The output of the Nth layer of the first network layer is multiplied by the output of the (N-1)th layer of the second network layer.

[0016] In a further embodiment of this paper, the first activation function of each layer in the first network layer is connected to the heterogeneous convolutional layer of the next layer, and the heterogeneous convolutional layer includes at least three different convolutions, and the first activation function is the Swish function.

[0017] In a further embodiment of this paper, the second activation function of each layer in the second network layer is connected to the single convolutional layer of the next layer, and the second activation function is the ReLU function.

[0018] In a further embodiment of this paper, the single convolutional layers of the odd-numbered layers and the single convolutional layers of the even-numbered layers that are closest to each other in the second network layer are connected respectively.

[0019] In a further embodiment of this paper, the second network layer further includes an Nth layer, which includes a single convolutional layer and a second activation function, wherein the convolutional kernel of the single convolutional layer of the Nth layer is the same as that of the single convolutional layer of the previous N-2 layers.

[0020] The second aspect of this document provides a method for establishing a denoising model, applicable to the denoising model described in any of the foregoing embodiments, including:

[0021] Construct a reference clean image based on the nonlocal similarity of adjacent pixels in a noisy image;

[0022] The parameters in the denoising model are trained based on the noisy image and the reference clean image.

[0023] In a further embodiment of this paper, the parameters for training the denoising model based on the noisy image and the reference clean image include:

[0024] Both the noisy image and the clean reference image are divided into multiple original image blocks;

[0025] The original image patch of the noisy image is input into the denoising model, and the denoising model calculates a clean image patch.

[0026] A first loss function is constructed using the clean image patch calculated from the noisy image and the original image patch of the reference clean image;

[0027] The parameters in the denoising model are optimized using the first loss function.

[0028] In a further embodiment of this paper, if the second network layer further includes an Nth layer, the Nth layer includes a single convolutional layer and a second activation function, and the convolutional kernel of the single convolutional layer of the Nth layer is the same as that of the single convolutional layer of the previous N-2 layers.

[0029] Based on the noisy image and the reference clean image, the parameters for training the denoising model also include:

[0030] A second loss function is constructed using the edge image patches of the reference clean image and the output of the Nth layer of the second network layer;

[0031] Calculate the total loss function based on the first loss function and the second loss function;

[0032] Optimizing the parameters in the denoising model using the first loss function further involves optimizing the parameters in the denoising model using the total loss function.

[0033] The third aspect of this paper provides a denoising model building apparatus, including:

[0034] The preprocessing module is used to construct a reference clean image based on the non-local similarity of adjacent pixels in the noisy image;

[0035] A training module is used to train the parameters in the denoising model based on the noisy image and a reference clean image. A fourth aspect of this paper provides an image denoising method, including:

[0036] The image to be processed is divided into multiple image blocks.

[0037] The image block to be processed is input into the denoising model established in any of the aforementioned embodiments to calculate a clean image block;

[0038] Combine clean image blocks to obtain a clean image.

[0039] The fifth aspect of this paper provides an image denoising apparatus, comprising:

[0040] The segmentation module is used to segment the image to be processed, resulting in multiple image blocks.

[0041] The calculation module is used to input the image block to be processed into the denoising model trained by the denoising model building device, and calculate the clean image block.

[0042] The combination module is used to combine clean image blocks to obtain a clean image.

[0043] A sixth aspect of this document provides a computer device including a memory, a processor, and a computer program stored in the memory, wherein the computer program, when run by the processor, executes instructions for the denoising model establishment method described in any of the foregoing embodiments.

[0044] The seventh aspect of this document provides a computer storage medium having a computer program stored thereon, which, when run by a processor of a computer device, executes instructions for a denoising model establishment method according to any of the foregoing embodiments.

[0045] This paper constructs a reference clean image based on the non-local similarity of adjacent pixels in a noisy image. The parameters of the denoising model are trained using the noisy image and the reference clean image, making the trained denoising model applicable to blindly noisy images. The denoising model is designed to include a first network layer, an edge detection module, a second network layer, and an image reconstruction layer. The inputs of the first and second network layers are connected through the edge detection module, and the outputs of the first and second network layers are multiplied together and then connected to the first image reconstruction layer. The first network layer extracts enhanced local features from the noisy image, the edge detection module generates an edge image based on the noisy image, the second network layer extracts enhanced edge features from the edge image, and the image reconstruction layer reconstructs the features obtained from the multiplication to obtain a clean image. This method can extract enhanced local and edge features, and uses an attention mechanism to guide the first network layer to obtain more edge detail information, resulting in more accurate clean image detail information, thus suppressing noise and improving robustness and denoising ability.

[0046] To make the above and other objects, features and advantages of this document more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments or prior art described herein, the accompanying drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this article. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 The flowchart of the denoising model establishment method in the embodiments of this paper is shown;

[0049] Figure 2 The structural diagram of the denoising model in the embodiment of this paper is shown;

[0050] Figure 3 The diagram shows the structure of the first network layer in the embodiment of this paper;

[0051] Figure 4 This paper shows a first structural diagram of the second network layer in an embodiment of the invention;

[0052] Figure 5 This paper shows a second structural diagram of the second network layer in an embodiment of the invention;

[0053] Figure 6 The third structural diagram of the second network layer in the embodiments of this paper is shown;

[0054] Figure 7 The flowchart illustrating the parameter training process of the denoising model in the embodiments of this paper is shown.

[0055] Figure 8 A flowchart of the image denoising method in the embodiments of this paper is shown;

[0056] Figure 9 A structural diagram of the denoising model building apparatus in the embodiments of this paper is shown;

[0057] Figure 10 A structural diagram of the image denoising apparatus of the embodiments herein is shown;

[0058] Figure 11A A schematic diagram of a noise image from an embodiment of this paper is shown;

[0059] Figure 11B A schematic diagram of a clean image is shown as a reference in the embodiments described herein;

[0060] Figure 12A and Figure 12B The original image blocks of the noisy image and the reference clean image in the embodiments of this paper are shown respectively;

[0061] Figure 13A and Figure 13B Schematic diagrams of edge image patches of the original image patches in the noisy image and the reference clean image of the embodiments in this paper are shown respectively;

[0062] Figures 14A to 14C The following diagrams illustrate the denoising model, the first network layer, and the second network layer in the embodiments described in this paper.

[0063] Figure 15 A structural diagram of the computer device described in this embodiment is shown.

[0064] Explanation of symbols in the attached drawings:

[0065] 210. First network layer;

[0066] 211. Heterogeneous convolutional layers;

[0067] 212. First activation function;

[0068] 220. Second network layer;

[0069] 221. Single convolutional layer;

[0070] 222. Second activation function;

[0071] 230. Image reconstruction layer;

[0072] 240. Edge detection module;

[0073] 910. Preprocessing module;

[0074] 920. Training module;

[0075] 1010. Divide into modules;

[0076] 1020. Calculation module;

[0077] 1030, Combined Module;

[0078] 1510, First network layer;

[0079] 1520, Second Network Layer;

[0080] 1530, a single convolutional layer;

[0081] 1540. Canny operator module;

[0082] 1602. Computer equipment;

[0083] 1604, Processor;

[0084] 1606. Memory;

[0085] 1608. Drive mechanism;

[0086] 1610. Input / Output Module;

[0087] 1612. Input devices;

[0088] 1614. Output devices;

[0089] 1616. Presentation equipment;

[0090] 1618. Graphical User Interface;

[0091] 1620. Network interface;

[0092] 1622. Communication link;

[0093] 1624. Communication bus. Detailed Implementation

[0094] The technical solutions in the embodiments described below will be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments described herein, and not all of the embodiments. Based on the embodiments described herein, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this document.

[0095] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings herein are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0096] This specification provides the operational steps of the methods described in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operational steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only possible execution order. In actual system or device products, the methods shown in the embodiments or drawings can be executed sequentially or in parallel.

[0097] It should be noted that the denoising model establishment method and apparatus, as well as the image denoising method and apparatus described in this paper, can be used for image denoising processing in any field. For example, they can be applied to disaster relief in harsh environments, aerial exploration and medical diagnosis in low-light environments, and seabed exploration in unstable environments. This paper does not limit the application fields of the denoising model establishment method and apparatus, as well as the image denoising method and apparatus.

[0098] Specifically, such as Figure 1 As shown, Figure 1 This paper illustrates a denoising model establishment method, which addresses the technical problems of existing neural network denoisers being unsuitable for training clean images without references, and the trained denoisers being unable to apply to blindly noisy images (i.e., damaged images with unknown noise distribution types), resulting in a narrow applicable range and poor recognition performance. Specifically, as shown... Figure 1 As shown, the methods for establishing the denoising model include:

[0099] Step 110: Construct a reference clean image based on the non-local similarity of adjacent pixels in the noisy image.

[0100] Here, the noisy image refers to an image acquired by an image acquisition device, and the reference clean image refers to an image without noise. Step 110 specifically includes: calculating the non-local similarity of adjacent pixels in the noisy image, and using the average value of the surrounding pixel values ​​to replace the center pixel value to complete the construction of the reference clean image. The specific calculation process of non-local similarity can refer to existing technologies, and this paper does not limit it.

[0101] Step 120: Based on the noisy image and the reference clean image, train the parameters in the denoising model, and substitute the trained parameters back into the denoising model.

[0102] In this step, the noisy image is first input into the denoising model to calculate the clean image; then, the calculated clean image and the reference clean image are used to construct a loss function; next, it is determined whether the loss function meets the preset conditions. If it does, the parameters in the denoising model are determined. If it does not, the parameters in the denoising model are adjusted (for example, by using the gradient descent method; this paper does not limit the specific adjustment algorithm) and the above process is repeated.

[0103] like Figure 2 As shown, the denoising model includes a first network layer 210, a second network layer 220, an image reconstruction layer 230, and an edge detection module 240. The inputs of the first network layer 210 and the second network layer 220 are connected through the edge detection module 240. The first network layer 210, also known as a Fusion Network for Multi-level Features (FNMLF), is used to extract enhanced local features from the noisy image. The edge detection module 240 is used to generate an edge image based on the noisy image. The second network layer 220, also known as an Edge Residual Network (ERN), is used to fuse the features obtained from the network layers and extract enhanced edge features from the edge image. The output of the first network layer 210 and the output of the second network layer 220 are multiplied together and then connected to the first image reconstruction layer 230. Through the multiplication process, the outputs of the first network layer 210 and the second network layer 220 can be fused together. The image reconstruction layer 230 is used to reconstruct the features obtained by the multiplication to obtain a clean image.

[0104] The edge detection module 240 described in this paper is, for example, the Canny operator, which obtains the edge image by searching for the maximum value around a pixel and setting the gray value mapped to non-maximum points to 0. In specific implementations, the edge detection module 240 can also be the Prewitt operator or the Sobel operator. This paper does not limit the specific algorithm of the edge detection module 240.

[0105] Image reconstruction layer 230 is a single convolutional layer. In specific implementation, the kernel size of this single convolutional layer is [missing information]. The input and output channels are divided into 64 and 3 respectively.

[0106] The features extracted by the first network layer 210 and the second network layer 220 are complementary, which can effectively suppress noise. The first network layer 210 includes N layers, and the second network layer 220 includes at least N-1 layers. The first layer of both the first network layer 210 and the second network layer 220 has 3 input channels (determined by the R, G, and B channels of the image) and 64 output channels (a common setting for neural networks; other numbers of channels can be selected in specific implementations). Each subsequent layer has 64 input and 64 output channels.

[0107] This paper, through steps 110 and 120, makes the trained denoising model applicable to blindly noisy images. The denoising model is designed to include a first network layer, an edge detection module, a second network layer, and an image reconstruction layer. The inputs of the first and second network layers are connected through the edge detection module. The first network layer extracts enhanced local features from the noisy image, the edge detection module generates an edge image based on the noisy image, and the second network layer extracts enhanced edge features from the edge image. The outputs of the first and second network layers are multiplied and then connected to the first image reconstruction layer. The image reconstruction layer reconstructs the features obtained from the multiplication to obtain a clean image. It can extract enhanced local and edge features, and uses the extracted edge features to guide the first network layer to obtain more edge detail information through an attention mechanism. This prevents the loss of edge information, obtains more accurate clean image detail information, achieves noise suppression, and improves robustness and denoising ability.

[0108] In one embodiment of this article, such as Figure 3 As shown, the first network layer 210 includes N layers, N for example, 17 layers, each of which consists of interconnected heterogeneous convolutional layers 211 and a first activation function 212, and the output of the Nth layer of the first network layer 210 constitutes the output of the first network layer.

[0109] In the first network layer 210, the first activation function 212 of each layer is connected to the heterogeneous convolutional layer 211 of the next layer. The heterogeneous convolutional layer 211 includes at least three different convolutions. For example, the heterogeneous convolutional layer includes... , and The convolutional layer has a first activation function 212, which is the Swish function. The Swish function can be expressed as... .in, This represents an adjustable parameter. This represents the output value of the heterogeneous convolutional layer. Heterogeneous convolutional layers can enhance the role of local key information, improving the training efficiency and denoising performance of the denoising model. The Swish function can transform linear noise into nonlinear noise.

[0110] In one embodiment of this article, such as Figure 4 As shown, the second network layer 220 includes N-1 layers, each of which consists of a single convolutional layer 221 and a second activation function 222. The output of the N-1th layer of the second network layer 220 constitutes the output of the second network layer 220. The convolutional kernels of the first N-2 layers are identical, for example, ... The (N-1)th convolutional layer is .

[0111] The output of the Nth layer of the first network layer 210 is multiplied by the output of the (N-1)th layer of the second network layer 220.

[0112] In the second network layer 220, the second activation function 222 of each layer is connected to the next single convolutional layer 221. The second activation function 222 is a ReLU function, which can be expressed as: .

[0113] In a further embodiment, to enhance the expressive power of the second network layer 220, residual learning (essentially a "+" operation) is applied to the network to strengthen the effect of the acquired feature layers. Specifically, as shown below... Figure 4 As shown, the single convolutional layers 221 of the odd-numbered layers and the single convolutional layers 221 of the even-numbered layers in the second network layer 220 are connected respectively. This embodiment can enhance the effect of the obtained feature hierarchy and solve a long-standing problem of deep networks.

[0114] Furthermore, such as Figure 5 As shown, step 120 above, based on the noisy image and the reference clean image, trains the parameters in the denoising model, including:

[0115] Step 510: Divide both the noisy image and the clean reference image into multiple original image blocks;

[0116] Step 520: Input the original image patch of the noisy image into the denoising model, and the denoising model calculates the clean image patch;

[0117] Step 530: Construct a first loss function using clean image patches and original image patches of a reference clean image;

[0118] Step 540: Optimize the parameters in the denoising model using the first loss function.

[0119] In some implementations, for example, both the noisy image and the reference clean image are divided into In other embodiments, the noisy image and the clean reference image can be divided into original image blocks of other sizes, which is not limited herein. In specific implementation, the mapping range of the network can be obtained based on the size of the receptive field, thereby determining the image block size.

[0120] The denoising model used in step 520 is the model during the training process, and the parameter values ​​are not the final values.

[0121] In step 530, the first loss function (Loss1) is constructed based on the mean squared error (MSE) of the clean image patch and the original image patch of the reference clean image. The specific calculation formula is as follows:

[0122] ;

[0123] in, Represents the number of original image patches. Representing the A clean image patch is obtained by using a denoising model on an original image patch. Representing the A block of the original image that serves as a reference clean image.

[0124] When performing step 540 above, it is determined whether the value of the first loss function meets the predetermined conditions. If it does not meet the conditions, the parameters are learned according to the optimization algorithm. If it meets the conditions, the parameter learning is stopped.

[0125] In one embodiment of this article, such as Figure 6 As shown, to further prevent the deep network from losing edge information, and to achieve interaction between the first and second network layers to obtain complementary information, improve denoising performance, and further learn optimal parameters to make the denoising model more robust, the second network layer also includes an Nth layer. This Nth layer includes a single convolutional layer and a second activation function. The convolutional kernel of the Nth layer is the same as that of the single convolutional layers in the previous N-2 layers. The output of the Nth layer is a clean edge image. In some specific implementations, the kernel size of the Nth layer is... The input channels and output channels are 64 and 3 respectively.

[0126] Furthermore, such as Figure 7 As shown, in step 120 above, the parameters trained in the denoising model based on the noisy image and the reference clean image include:

[0127] Step 710: Divide both the noisy image and the clean reference image into multiple original image blocks;

[0128] Step 720: Input the original image patch of the noisy image into the denoising model, and the denoising model calculates the clean image patch;

[0129] Step 730: Construct a first loss function using clean image patches and original image patches of a reference clean image;

[0130] Step 740: Construct a second loss function using edge image patches from the reference clean image and the output of the Nth layer of the second network layer;

[0131] Step 750: Calculate the total loss function based on the first loss function and the second loss function;

[0132] Step 760: Optimize the parameters in the denoising model using the total loss function.

[0133] The implementation of steps 710-730 and 760 can refer to steps 510-530 and 540 described above, and will not be detailed here.

[0134] When implementing step 740, the second loss function (Loss2) can be constructed using the MSE method, referring to step 530.

[0135] When implementing step 750, the first loss function and the second loss function are merged using a weighted method. The formula for calculating the total loss function is as follows:

[0136] ,in, and These are the weighting coefficients. In practice, the designers can set weighting coefficients, for example... It is 0.7. The value is 0.3. This paper does not limit the specific value of the weight coefficient, and the weight coefficient can be adjusted by the designer according to the model training effect.

[0137] In specific training, noisy images can be divided into training set and validation set. The denoising model is trained using the training set, and the training set is used to verify the training effect of the denoising model.

[0138] In one embodiment of this paper, an image denoising method is also provided. This method can run on smart terminals, including smartphones, tablets, desktop computers, etc. It can also be a standalone application, a mini-program embedded in other programs, or a webpage. This paper does not limit the specific implementation method.

[0139] Specifically, such as Figure 8 As shown, image denoising methods include:

[0140] Step 810: Divide the image to be processed into multiple image blocks.

[0141] Step 820: Input the image block to be processed into the denoising model established in any of the aforementioned embodiments to calculate a clean image block;

[0142] Step 830: Combine clean image blocks to obtain a clean image.

[0143] In step 810, the image to be processed can be divided according to the image patch size used during the training of the denoising model. The training process of the denoising model in step 820 is the same as in the previous embodiment and will not be detailed here. Step 830 combines the clean image patches, that is, restores the small image patches to a larger image.

[0144] Based on the same inventive concept, this paper also provides a denoising model building apparatus, as described in the following embodiments. Since the principle of the denoising model building apparatus in solving the problem is similar to that of the denoising model building method, the implementation of the denoising model building apparatus can refer to the denoising model building method, and the repeated parts will not be described again.

[0145] Specifically, such as Figure 9 As shown, the denoising model building device includes:

[0146] Preprocessing module 910 is used to construct a reference clean image based on the non-local similarity of adjacent pixels in the noisy image;

[0147] The training module 920 is used to train the parameters in the denoising model based on the noisy image and the reference clean image, and to substitute the trained parameters back into the denoising model.

[0148] The denoising model includes a first network layer, an edge detection module, a second network layer, and an image reconstruction layer. The inputs of the first network layer and the second network layer are connected through the edge detection module. The first network layer is used to extract enhanced local features from the noisy image, the edge detection module is used to generate an edge image based on the noisy image, and the second network layer is used to extract enhanced edge features from the edge image. The outputs of the first network layer and the second network layer are multiplied together and then connected to the first image reconstruction layer. The image reconstruction layer is used to reconstruct the features obtained by the multiplication to obtain a clean image.

[0149] Furthermore, such as Figure 10 As shown, an image denoising apparatus is also provided, comprising:

[0150] The segmentation module 1010 is used to segment the image to be processed, resulting in multiple image blocks to be processed.

[0151] The calculation module 1020 is used to input the image block to be processed into the denoising model trained by the denoising model building device, and calculate the clean image block.

[0152] The combination module 1030 is used to combine clean image blocks to obtain a clean image.

[0153] This embodiment can extract and enhance local features and edge features, guide edge features into local features, obtain more accurate and clean image detail information, achieve noise suppression, and improve robustness and noise reduction capabilities.

[0154] To more clearly illustrate the technical solution presented in this paper, the following specific embodiment describes the denoising model training process:

[0155] (1) Utilizing pixel nonlocal similarity and replacing the center pixel value with the surrounding pixel values, a reference clean image is constructed from the noisy image. For example, a noisy image is shown below. Figure 11A As shown, the paired clean reference image is as follows Figure 11B As shown.

[0156] (2) To improve the efficiency of training the denoising model, the noisy image and the clean reference image are cropped into... The original image patch, the original image patch of the noisy image, such as Figure 12A As shown, the original image patch for reference to the clean image is as follows: Figure 12B As shown.

[0157] (3) Then, the original image patch of the noisy image is fed into the denoising model to be trained to calculate the clean image patch. For example Figures 14A to 14C As shown, the denoising model includes: a first network layer 1510, a second network layer 1520, a single convolutional layer 1530, and a Canny operator module 1540. The inputs of the first network layer 1510 and the second network layer 1520 are connected through the Canny operator module 1540. The first network layer 1510 is used to extract enhanced local features from the noisy image, and the Canny operator module 1540 is used to generate edge images (such as...) based on the noisy image. Figure 13A As shown, the second network layer 1520 is used to extract enhanced edge features from the edge image. The output of the first network layer 1510 is multiplied by the output of the 16th layer of the second network layer 1520 and then connected to a single convolutional layer 1530. Through the multiplication process, the outputs of the first network layer 1510 and the second network layer 1520 can be fused together. The single convolutional layer 1530 is used to reconstruct the features obtained by the multiplication to obtain a clean image. The single convolutional layer 1530 serves as the last layer of the denoising model and is used to construct a clean image.

[0158] The denoising model has an 18-layer structure, in which the first network layer 1510 and the second network layer 1520 are both 17 layers, and the single convolutional layer 1530 is the 18th layer.

[0159] like Figure 14B As shown, each layer in the first network layer 1510 consists of three different convolutions ( , and The network consists of heterogeneous convolutions and the Swish activation function. The first layer of the network has 3 input channels and 64 output channels, while the second to seventh layers each have 64 input and output channels.

[0160] like Figure 14A As shown, each layer of the second network layer 1520 consists of a single convolution and the ReLU activation function. The kernel size of the convolutions in layers 1-15 and layer 17 of the second network layer 1520 is [missing information]. The kernel size of the 16th layer is Layers 16 and 17 constitute the output of the second network layer 1520. The first layer of the second network layer 1520 has 3 input channels and 64 output channels, the second to 16 layers have 64 input channels and 64 output channels, and the 17th layer has 64 input channels and 3 output channels.

[0161] The kernel size of a single convolutional layer with a size of 1530 is The input channels and output channels are 64 and 3 respectively.

[0162] (4) Construct the first loss function using clean image patches and the original image patches of the reference clean image.

[0163] (5) Utilize edge image patches of a reference clean image (e.g.) Figure 13B As shown in the figure, and the output of the Nth layer of the second network layer, construct the second loss function.

[0164] (6) Calculate the total loss function based on the first loss function and the second loss function.

[0165] The two loss functions of the first and second network layers are combined in a weighted manner to further learn the optimal parameters, making the learned denoising model more robust.

[0166] (7) Optimize the parameters in the denoising model using the total loss function.

[0167] The denoising model trained using the above method can effectively handle denoising of damaged images without paired clean images, demonstrating that this technique can effectively handle blind noise, which better meets the needs of digital devices. Therefore, it has important applications in disaster relief in harsh environments, aerial exploration in low-light environments, medical diagnosis, and seabed exploration in unstable environments.

[0168] In one embodiment of this document, a computer device is also provided for implementing the methods described in any of the foregoing embodiments. Figure 15As shown, computer device 1602 may include one or more processors 1604, such as one or more central processing units (CPUs), each of which may implement one or more hardware threads. Computer device 1602 may also include any memory 1606 for storing information of any kind, such as code, settings, data, etc. Non-limitingly, for example, memory 1606 may include any type of RAM, any type of ROM, flash memory, hard disk, optical disk, etc. More generally, any memory can use any technology to store information. Furthermore, any memory may provide volatile or non-volatile retention of information. Furthermore, any memory may represent a fixed or removable component of computer device 1602. In one case, when processor 1604 executes associated instructions stored in any memory or combination of memories, computer device 1602 may perform any operation of the associated instructions. Computer device 1602 also includes one or more drive mechanisms 1608 for interacting with any memory, such as hard disk drive mechanisms, optical disk drive mechanisms, etc.

[0169] Computer device 1602 may also include an input / output module 1610 (I / O) for receiving various inputs (via input device 1612) and providing various outputs (via output device 1614). A specific output mechanism may include a presentation device 1616 and an associated graphical user interface 1618 (GUI). In other embodiments, the input / output module 1610 (I / O), input device 1612, and output device 1614 may be omitted, and the device may function solely as a computer device within a network. Computer device 1602 may also include one or more network interfaces 1620 for exchanging data with other devices via one or more communication links 1622. One or more communication buses 1624 couple the components described above together.

[0170] Communication link 1622 can be implemented in any way, such as via a local area network, a wide area network (e.g., the Internet), a point-to-point connection, or any combination thereof. Communication link 1622 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc., governed by any protocol or combination of protocols.

[0171] Corresponding to Figure 1 , Figure 7 , Figure 8 In addition to the methods described above, this embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the above-described methods.

[0172] This embodiment also provides a computer-readable instruction, wherein when a processor executes the instruction, the program therein causes the processor to perform the following: Figure 1 , Figure 7 , Figure 8 The method shown.

[0173] It should be understood that in the various embodiments of this document, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this document.

[0174] It should also be understood that, in the embodiments herein, the term "and / or" is merely a description of the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following associated objects have an "or" relationship.

[0175] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this document.

[0176] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0177] In the embodiments provided herein, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections through some interfaces, devices, or units, or they may be electrical, mechanical, or other forms of connection.

[0178] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments described herein, depending on actual needs.

[0179] Furthermore, the functional units in the various embodiments of this document can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0180] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this paper, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this paper. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0181] This document uses specific embodiments to illustrate the principles and implementation methods of this document. The descriptions of the embodiments above are only for the purpose of helping to understand the methods and core ideas of this document. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this document. Therefore, the content of this specification should not be construed as a limitation of this document.

Claims

1. A method for establishing a denoising model, characterized in that, include: Construct a reference clean image based on the nonlocal similarity of adjacent pixels in a noisy image; Based on the noisy image and the reference clean image, train the parameters in the denoising model; The denoising model includes: a first network layer, an edge detection module, a second network layer, and an image reconstruction layer; The input of the first network layer is connected to the input of the edge detection module, the output of the edge detection module is connected to the input of the second network layer, and the output of the first network layer and the output of the second network layer are multiplied and then connected to the image reconstruction layer. The first network layer is used to extract enhanced local features from noisy images; The edge detection module is used to generate an edge image based on the noise image; The second network layer is used to extract enhanced edge features from the edge image; The image reconstruction layer is used to reconstruct the features obtained by dot product to obtain a clean image; The first network layer comprises N layers, each of which consists of interconnected heterogeneous convolutional layers and a first activation function; the heterogeneous convolutional layers comprise at least three different convolutions; The second network layer comprises N-1 layers, each consisting of a single convolutional layer and a second activation function. The first N-2 layers have identical convolutional kernels, and the N-1th convolutional layer is... ; The output of the Nth layer of the first network layer is multiplied by the output of the (N-1)th layer of the second network layer.

2. The denoising model establishment method as described in claim 1, characterized in that, In the first network layer, the first activation function of each layer is connected to the heterogeneous convolutional layer of the next layer, and the first activation function is the Swish function.

3. The denoising model establishment method as described in claim 1, characterized in that, In the second network layer, the second activation function of each layer is connected to the single convolutional layer of the next layer, and the second activation function is the ReLU function.

4. The denoising model establishment method as described in claim 1, characterized in that, Connect the single convolutional layers of the odd-numbered and even-numbered layers that are closest to each other in the second network layer.

5. The denoising model establishment method as described in claim 1, characterized in that, The second network layer also includes an Nth layer, which includes a single convolutional layer and a second activation function. The convolutional kernel of the single convolutional layer of the Nth layer is the same as that of the single convolutional layers of the previous N-2 layers.

6. The denoising model establishment method as described in claim 5, characterized in that, The parameters for training the denoising model based on the noisy image and the reference clean image include: Both the noisy image and the clean reference image are divided into multiple original image blocks; The original image patch of the noisy image is input into the denoising model, and the denoising model calculates the clean image patch. A first loss function is constructed using the clean image patch calculated from the noisy image and the original image patch of the reference clean image; The parameters in the denoising model are optimized using the first loss function.

7. The denoising model establishment method as described in claim 6, characterized in that, The Nth layer of the second network layer includes a single convolutional layer and a second activation function. The convolutional kernel of the single convolutional layer of the Nth layer of the second network layer is the same as that of the single convolutional layers of the first N-2 layers of the second network layer. The parameters used in training the denoising model, based on the noisy image and the reference clean image, also include: A second loss function is constructed using the edge image patches of the reference clean image and the output of the Nth layer of the second network layer; Calculate the total loss function based on the first loss function and the second loss function; Optimizing the parameters in the denoising model using the first loss function further involves optimizing the parameters in the denoising model using the total loss function.

8. A denoising model determination device, characterized in that, include: The preprocessing module is used to construct a reference clean image based on the non-local similarity of adjacent pixels in the noisy image; The training module is used to train the parameters in the denoising model based on the noisy image and the reference clean image; The denoising model includes: a first network layer, an edge detection module, a second network layer, and an image reconstruction layer; The input of the first network layer is connected to the input of the edge detection module, the output of the edge detection module is connected to the input of the second network layer, and the output of the first network layer and the output of the second network layer are multiplied and then connected to the image reconstruction layer. The first network layer is used to extract enhanced local features from noisy images; The edge detection module is used to generate an edge image based on the noise image; The second network layer is used to extract enhanced edge features from the edge image; The image reconstruction layer is used to reconstruct the features obtained by dot product to obtain a clean image; The first network layer comprises N layers, each of which consists of interconnected heterogeneous convolutional layers and a first activation function; the heterogeneous convolutional layers comprise at least three different convolutions; The second network layer comprises N-1 layers, each consisting of a single convolutional layer and a second activation function. The first N-2 layers have identical convolutional kernels, and the N-1th convolutional layer is... ; The output of the Nth layer of the first network layer is multiplied by the output of the (N-1)th layer of the second network layer.

9. An image denoising method, characterized in that, include: The image to be processed is divided into multiple image blocks. The image block to be processed is input into the denoising model established by any one of claims 1 to 7, and a clean image block is calculated. Combine clean image blocks to obtain a clean image.

10. An image denoising apparatus, characterized in that, include: The segmentation module is used to segment the image to be processed, resulting in multiple image blocks. The calculation module is used to input the image block to be processed into the denoising model established by any one of claims 1 to 7, and calculate a clean image block; The combination module is used to combine clean image blocks to obtain a clean image.

11. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, When the computer program is run by the processor, it executes the instructions of the method according to any one of claims 1 to 7.

12. A computer storage medium having a computer program stored thereon, characterized in that, When the computer program is run by the processor of the computer device, it executes the instructions of the method according to any one of claims 1 to 7.

Images