Image denoising method, image denoising device, electronic device and storage medium

The image denoising network constructed by the cascade tree module and the binocular enhancement module solves the problems of poor generalization performance and high computational complexity in the existing technology, and realizes efficient image denoising and reducing training in different scenarios cost.

CN116977650BActive Publication Date: 2024-11-12RES & DEV INST OF NORTHWESTERN POLYTECHNICAL UNIV IN SHENZHEN +2
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310958579.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-07-31
Publication Date
2024-11-12
Estimated Expiration
2043-07-31

AI Technical Summary

Technical Problem

The generalization performance of existing image denoising methods is poor and the computational complexity is high. The traditional methods rely on scenes and noise types. The single feature extraction of deep learning methods leads to poor denoising effect and high training cost.

Method used

The cascaded tree module and binocular enhancement module are used to build a lightweight image denoising network. Multiple feature extraction and feature fusion are performed through the tree module, and the binocular enhancement module performs feature refinement and local feature enhancement to reduce the computational complexity. And improve the noise removal effect.

Benefits of technology

It realizes effective removal of arbitrary noise in different scenarios, reduces the complexity of denoising calculation, improves the generalization performance and denoising effect of image denoising network, reduces the dependence on the composition of noise images-clean images, and reduces the Training cost.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116977650B_ABST
    Figure CN116977650B_ABST
Patent Text Reader

Abstract

The present application provides an image denoising method, an image denoising device, an electronic device and a storage medium, which belong to the field of image processing technology. By acquiring a noisy image, the noisy image is input into a preset image denoising network, and the image denoising network includes a tree module and a binocular enhancement module. Multivariate feature extraction is performed on the noisy image through the tree module to obtain multivariate image features. Through the binocular enhancement module, feature purification is performed on the multivariate image features to obtain image refinement features, local feature enhancement is performed on the multivariate image features to obtain image saliency features, and feature fusion is performed on the image refinement features and the image saliency features, thereby improving the image denoising effect and obtaining a clean image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image denoising method, an image denoising device, an electronic device and a storage medium. Background Art

[0002] In the related art, traditional denoising methods or deep learning methods are used to remove noise from noisy images to improve image quality. However, some traditional denoising methods depend on the application scenario and the type of noise, and can only remove one or several types of noise in a certain scenario, and the generalization performance is poor. Some traditional denoising methods are relatively complex and have certain requirements for computing resources, which limits their application scenarios. Deep learning methods mainly perform image denoising through convolutional neural networks (CNN), such as deep convolutional denoising neural networks (DnCNN), fast and flexible denoising networks (FFDNet), and dense residual networks (RDN). Since these networks are single networks, the image features extracted by the network are relatively single, resulting in poor image denoising effects. In addition, these networks extract image features by stacking convolutional layers. As the depth of the network increases, the complexity and amount of denoising also increase, which limits their use scenarios. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose an image denoising method, an image denoising device, an electronic device and a storage medium, which constructs a lightweight image denoising network by cascading a tree module and a binocular enhancement module, aiming to improve the image denoising effect while reducing the complexity of denoising calculations.

[0004] To achieve the above objective, a first aspect of an embodiment of the present application provides an image denoising method, the method comprising:

[0005] Get a noisy image;

[0006] Inputting the noisy image into a preset image denoising network, wherein the image denoising network includes a tree module and a binocular enhancement module;

[0007] Performing multivariate feature extraction on the noise image through the tree module to obtain multivariate image features;

[0008] Through the binocular enhancement module, the multi-image features are subjected to feature purification to obtain image refinement features, and the local features of the multi-image features are enhanced to obtain image saliency features;

[0009] The image refinement features and the image saliency features are fused to obtain a denoised image.

[0010] In some embodiments, before inputting the noisy image into a preset image denoising network, the image denoising method further includes:

[0011] Extracting the red channel, green channel and blue channel of the noise image;

[0012] Masking the pixels of the red channel by using a first preset mask to obtain a first mask image;

[0013] Masking the pixel points of the green channel by a second preset mask to obtain a second mask image;

[0014] Using the first preset mask to mask the pixels of the blue channel, to obtain a third mask image;

[0015] The first mask image, the second mask image and the third mask image are stitched to obtain the blocked noise image.

[0016] In some embodiments, the tree module includes multiple convolutional layers, and a tree structure is formed between the multiple convolutional layers. The multivariate feature extraction of the noise image through the tree module to obtain the multivariate image feature includes:

[0017] Extracting features of the noise image layer by layer through multiple convolutional layers to obtain hierarchical image features;

[0018] The hierarchical image features are fused layer by layer to obtain the multivariate image features.

[0019] In some embodiments, the multiple convolutional layers include a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a sixth convolutional layer, and a seventh convolutional layer, and the layer-by-layer feature extraction of the noise image through the multiple convolutional layers to obtain the hierarchical image features includes:

[0020] Performing a first feature extraction on the noise image through the first convolutional layer to obtain a first image feature;

[0021] Performing second feature extraction on the first image feature through the second convolutional layer to obtain a second image feature;

[0022] Performing a third feature extraction on the first image feature through the third convolutional layer to obtain a third image feature;

[0023] Performing a fourth feature extraction on the second image feature through the fourth convolutional layer to obtain a fourth image feature;

[0024] Performing a fifth feature extraction on the second image feature through the fifth convolutional layer to obtain a fifth image feature;

[0025] Performing a sixth feature extraction on the third image feature through the sixth convolutional layer to obtain a sixth image feature;

[0026] Performing a seventh feature extraction on the third image feature through the seventh convolutional layer to obtain a seventh image feature;

[0027] The second image feature, the third image feature, the fourth image feature, the fifth image feature, the sixth image feature and the seventh image feature are used as the hierarchical image features.

[0028] In some embodiments, the step of fusing the hierarchical image features layer by layer to obtain the multivariate image features includes:

[0029] Performing a first feature fusion on the fifth image feature and the sixth image feature to obtain a first fused feature;

[0030] Performing a second feature fusion on the first fusion feature, the second image feature and the third image feature to obtain a second fusion feature;

[0031] A third feature fusion is performed on the second fusion feature, the fourth image feature and the seventh image feature to obtain the multivariate image feature.

[0032] In some embodiments, the binocular enhancement module includes a feature refinement module, the feature refinement module includes a pooling layer, a feature refinement convolution layer and an upsampling layer, and the feature purification of the multivariate image features to obtain image refinement features includes:

[0033] Performing pooling processing on the multivariate image features through the pooling layer to obtain image pooling features;

[0034] Performing convolution processing on the image pooling feature through the feature refinement convolution layer to obtain a first image convolution feature;

[0035] Upsampling the first image convolution feature through the upsampling layer to obtain an image upsampling feature;

[0036] The multivariate image feature and the image upsampling feature are fused according to a first preset factor to obtain the image refinement feature.

[0037] In some embodiments, the binocular enhancement module includes a local enhancement module, the local enhancement module includes a local enhancement convolution layer and a gating layer, and the local feature enhancement of the multivariate image features to obtain image saliency features includes:

[0038] Performing convolution processing on the multivariate image feature through the local enhancement convolution layer to obtain a second image convolution feature;

[0039] Performing activation processing on the second image convolution feature through the gating layer to obtain an image activation feature;

[0040] The multi-image feature and the image activation feature are fused according to a second preset factor to obtain the image saliency feature.

[0041] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides an image denoising device, the device comprising:

[0042] An acquisition module, used for acquiring a noise image;

[0043] An input module, used for inputting the noisy image into a preset image denoising network, wherein the image denoising network includes a tree module and a binocular enhancement module;

[0044] A first feature extraction module, used for performing multivariate feature extraction on the noise image through the tree module to obtain multivariate image features;

[0045] A second feature extraction module is used to perform feature purification on the multi-image features through the binocular enhancement module to obtain image refinement features, and perform local feature enhancement on the multi-image features to obtain image saliency features;

[0046] The feature fusion module is used to fuse the image refinement features and the image saliency features to obtain a denoised image.

[0047] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the image denoising method described in the first aspect when executing the computer program.

[0048] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the image denoising method described in the first aspect is implemented.

[0049] The image denoising method, image denoising device, electronic device and computer-readable storage medium proposed in the present application obtain a noisy image and input the noisy image into a preset image denoising network. The image denoising network includes a tree module and a binocular enhancement module. A lightweight image denoising network is constructed by cascading the tree module and the binocular enhancement module, which reduces the complexity of denoising calculation and improves the effect of image denoising. Since a single image feature cannot fully express the image, a multi-dimensional feature extraction of the noisy image is performed by a tree module, and a multi-dimensional image feature, i.e., a multi-dimensional image feature, can be obtained to enhance the feature expression ability of the image denoising network, so that the image denoising network does not depend on the application scenario and the type of noise, and realizes the removal of any noise in any scenario, thereby improving the generalization performance of the image denoising network. In addition, compared with the method of extracting features by stacking convolutional layers, the tree module extracts features in parallel with a tree structure, which can improve the speed of feature extraction. The binocular enhancement module is used to enhance the multi-dimensional image features from different angles, so as to further improve the accuracy of feature extraction by enhancing the features. The multi-dimensional image features are purified and redundant information is filtered to obtain image refinement features. The local features of the multi-image features are enhanced to suppress the noise and obtain the image salient features. The image refinement features and image salient features are fused to obtain better feature representation and improve the image denoising effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a flow chart of an image denoising method provided by an embodiment of the present application;

[0051] Figure 2 is another flow chart of the image denoising method provided by an embodiment of the present application;

[0052] Figure 3 is a schematic diagram of a self-supervisory mask provided in an embodiment of the present application;

[0053] Figure 4 is a network structure diagram of an image denoising network provided in an embodiment of the present application;

[0054] Figure 5 yes Figure 1 Flow chart of step S130 in FIG.

[0055] Figure 6 yes Figure 5 Flow chart of step S510 in FIG.

[0056] Figure 7 yes Figure 5 Flow chart of step S520 in FIG.

[0057] Figure 8 yes Figure 1 Flow chart of step S140 in FIG.

[0058] Fig. 9 yes Figure 1 Another flow chart of step S140 in FIG.

[0059] Fig.10a is a noise image provided by an embodiment of the present application;

[0060] Fig.10b yes Fig.10a Red channel of the noisy image;

[0061] Fig.10c yes Fig.10a Green channel of the noisy image;

[0062] Fig.10d yes Fig.10a Blue channel of the noisy image;

[0063] Fig.10e Yes Fig.10b The red channel is masked to obtain the red channel of the noise image after occlusion;

[0064] Fig.10f Yes Fig.10c The green channel is masked to obtain the green channel of the noisy image;

[0065] Figure 10g Yes Fig.10d The blue channel is the blue channel of the occluded noise image obtained by masking the blue channel;

[0066] Fig.10h is the red channel of the noise image after 10e occlusion, Fig.10f Green channel of the noise image after occlusion, Figure 10g The occluded noise image is obtained by splicing and combining the blue channels of the occluded noise image;

[0067] Fig.10j Yes Fig.10h The occluded noisy image is input into the image denoising network for denoising to obtain a clean image;

[0068] Figure 10k yes Fig.10a Image patches of noisy images;

[0069] Figure 10l Yes Figure 10k The occluded image block is obtained by masking the image block;

[0070] Figure 10m Yes Figure 10l The occluded image block is input into the image denoising network for denoising to obtain a clean image block;

[0071] Figure 10n yes Fig.10a Another image patch of the noisy image;

[0072] Fig.10o Yes Figure 10n The occluded image block is obtained by masking the image block;

[0073] Figure 10p Yes Fig.10o The occluded image block is input into the image denoising network for denoising to obtain a clean image block;

[0074] Fig.11 is a structural schematic diagram of an image denoising device provided in an embodiment of the present application;

[0075] Fig.12 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0076] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0077] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0078] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0079] Image denoising technology is one of the important issues in the field of computer vision. Due to the influence of environment, weather, camera and human factors, the collected images are usually noisy, so it is necessary to denoise the collected images through image denoising technology to improve the image quality. Through mathematical modeling, the noisy image can be expressed as y=x+μ, where y is the noisy image, x is the clean image, and μ is the noise. Image denoising technology mainly restores the damaged pixel values ​​in the noisy image to the original pixel values ​​to obtain a high-quality image.

[0080] In the related art, traditional denoising methods or deep learning methods are used to remove noise from noisy images. Traditional denoising methods can be divided into spatial pixel feature denoising methods and transform domain denoising methods. The spatial pixel feature denoising method is a method for obtaining a new central pixel value by analyzing the direct connection between the central pixel and other adjacent pixels in the grayscale space within a window of a certain size. The transform domain denoising method is different from the spatial pixel feature denoising method. It converts the noisy image from the spatial domain to the transform domain through transformation methods such as Fourier transform, and separates the image signal and noise in the transform domain. For example, the noise is divided into high-frequency noise, medium-frequency noise and low-frequency noise in terms of frequency, and the noise is suppressed. What remains is the image signal. Then, the separated image signal is converted from the transform domain back to the original spatial domain through inverse transformation to achieve the purpose of removing image noise. Although these methods have achieved good performance in some scenarios, these methods still face the following challenges: some traditional denoising methods depend on the application scenario and the type of noise, and can only remove one or several types of noise in a certain scenario, and the generalization performance is poor. Some traditional denoising methods are highly complex and have certain requirements on computing resources, which limits their application scenarios.

[0081] The complexity of deep learning algorithms is high, and computing resources that match the complexity of the algorithms are required. CNN relies on the advantage of the Graphic Processing Unit (GPU) to quickly process data, absorbs the powerful learning and fitting capabilities of neural networks, and has excellent model performance without manual parameter adjustment. In addition, CNN has the characteristics of translation invariance and weight sharing, which greatly reduces the number of network parameters while being able to extract local features well. Therefore, CNN is a very effective image restoration method. CNN is also widely used in image denoising. For example, DnCNN uses CNN for image denoising, and improves the denoising performance of the network by stacking convolutional layers, activation functions ReLU and batch normalization layers (Batch Normalization, BN) layers. FFDNet uses noisy images and noise as network inputs, and is a flexible denoising model. A large number of residual connection operations are used in RDN to achieve feature fusion of deep features and shallow features. However, the following challenges remain unresolved: the above neural networks extract features through a single network, which results in a single extracted feature, limiting the performance of the denoising network; the above neural networks rely on stacking convolutional layers to improve the performance of the denoising network, but as the depth of the network increases, the high complexity and large amount of computation limit the use scenarios of the denoising network; the training of the above neural networks is highly dependent on image pairs consisting of noisy images and clean images, which increases the cost of training a robust image denoising network.

[0082] Based on this, the embodiments of the present application provide an image denoising method, an image denoising device, an electronic device and a computer-readable storage medium, which aim to reduce the computational complexity of denoising, reduce the dependence on image pairs consisting of noisy images and clean images, improve the generalization performance, and improve the effect of image denoising.

[0083] The image denoising method, image denoising device, electronic device and computer-readable storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the image denoising method in the embodiments of the present application is described.

[0084] The image denoising method provided in the embodiments of the present application relates to the fields of image processing technology, deep learning and computer vision. The image denoising method provided in the embodiments of the present application can be applied to a terminal, can be applied to a server, or can be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the image denoising method, etc., but is not limited to the above forms.

[0085] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0086] Figure 1 is an optional flowchart of the image denoising method provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S110 to S150.

[0087] Step S110, obtaining a noise image;

[0088] Step S120, inputting the noisy image into a preset image denoising network, where the image denoising network includes a tree module and a binocular enhancement module;

[0089] Step S130, extracting multivariate features from the noise image through a tree module to obtain multivariate image features;

[0090] Step S140, using a binocular enhancement module, purifying the multi-image features to obtain image refinement features, and enhancing the local features of the multi-image features to obtain image saliency features;

[0091] Step S150: performing feature fusion on the image refinement features and the image saliency features to obtain a denoised image.

[0092] Steps S110 to S150 shown in the embodiment of the present application can reduce the complexity of denoising and improve the generalization performance of the image denoising network by cascading the tree module and the binocular enhancement module to construct a lightweight image denoising network. The tree module can extract multi-dimensional image features to avoid a single image feature being unable to fully express the image, thereby affecting the image denoising effect. The binocular enhancement module is used to enhance multi-dimensional image features from different angles, so as to further improve the accuracy of feature extraction by enhancing features, thereby improving the image denoising effect.

[0093] In step S110 of some embodiments, the noise image is an image containing noise, and the noise may be Gaussian noise, blind noise, or real noise. The noise image may be obtained by photographing a preset object through an image acquisition device, or may be directly obtained from an image database.

[0094] See also Figure 2 In some embodiments, before step S120, the image denoising method may further include but is not limited to steps S210 to S250:

[0095] Step S210, extracting the red channel, green channel and blue channel of the noise image;

[0096] Step S220, masking the pixels of the red channel by using a first preset mask to obtain a first mask image;

[0097] Step S230, masking the pixels of the green channel by a second preset mask to obtain a second mask image;

[0098] Step S240, masking the pixels of the blue channel by using the first preset mask to obtain a third mask image;

[0099] Step S250: stitching the first mask image, the second mask image and the third mask image to obtain a blocked noise image.

[0100] In step S210 of some embodiments, the training of the traditional image denoising network is overly dependent on the image pairs consisting of the noisy image and the clean image, but the image pairs are difficult to obtain, resulting in high training costs for training the robust image denoising network. At the same time, due to the difficulty in obtaining the image pairs, the amount of image pairs used to train the image denoising network is insufficient, resulting in the network parameters of the image denoising network being unable to reach the optimum, affecting the effect of image denoising. The disclosed embodiment uses a self-supervised algorithm to mask the noisy image to solve the problems of high training costs and insufficient training data. The training of the traditional image denoising network is highly dependent on paired noisy images and clean images. However, the acquisition of paired noisy images and clean images requires high production costs, which limits the training of the robust image denoising network. Through the self-supervised mask, the image denoising network is forced to learn the connection between noise and clean pixels, reducing the dependence of the image denoising network training on clean images and reducing the training cost of the image denoising network.

[0101] like Figure 3 As shown, in some embodiments, when the noise image is a color image, the red (Red, R) channel, the green (Green, G) channel and the blue (Blue, B) channel of the noise image are extracted. When the noise image is a grayscale image, the grayscale image can be converted into a color image by pseudo color mapping.

[0102] In some embodiments, in order to increase the number of training samples and speed up the training efficiency, the noisy image is randomly divided into a plurality of image blocks of preset sizes, and the image blocks are masked using a self-supervisory algorithm, wherein the preset size may be 48 × 48. It should be noted that in the image denoising network training stage, the training efficiency of the two methods of first dividing the image blocks and then masking and first masking and then dividing the image blocks is the same, and the output results of the network are the same.

[0103] In step S220 of some embodiments, the first preset mask is a black and white binary image that blocks the red channel. The size of the first preset mask may be the same as the size of the red channel or the preset size, such as Figure 3The mask 1 shown may also be smaller than the size of the red channel or the preset size. When the size of the first preset mask is smaller than the size of the red channel or the preset size, the first preset mask slides on the red channel or the image block with a specific step size until the traversal of the red channel or the image block is completed, and the traversal does not include repeated traversal of pixels. The specific step size is related to the size of the first preset mask. For example, if the size of the first preset mask is 2×2, the specific step size is 2, and each non-overlapping 2×2 patch in the red channel or the image block can be blocked using the first preset mask.

[0104] The i-th pixel P in the first preset mask i The coordinates of (x i ,y i ), when x i and i When it is an odd number or an even number at the same time, the pixel P i The pixel value is 0, when x i is an odd number, y i is an even number, or x i is an even number, y i When it is an odd number, the pixel P i That is, in the first preset mask, when the horizontal coordinate and the vertical coordinate of the pixel position are both odd numbers or both even numbers, the pixel value of the position is 0, and the pixel values ​​of the other positions are 1. Figure 3 As shown, the pixel values ​​of the pixels on the diagonal line of the first preset mask are all 0. The calculation method of the pixel value of the first preset mask is shown in formula (1).

[0105]

[0106] Wherein, i represents the i-th pixel of the first preset mask; represents the pixel value of the i-th pixel of the first preset mask; x i y is the horizontal coordinate of the position of the i-th pixel in the first preset mask; i is the ordinate of the position of the i-th pixel in the first preset mask; % represents the modulo operation.

[0107] The pixel values ​​of the red channel pixels are multiplied by the pixel values ​​of the first preset mask pixels to mask each pixel of the red channel to obtain a first mask image, which is the masked red channel. From the perspective of the spatial domain, the first preset mask shields the pixel values ​​of the adjacent spatial domains of the red channel, so that the image denoising network can learn the relationship between the red channel pixels from the spatial domain.

[0108] In step S230 of some embodiments, the second preset mask is a black and white binary image that blocks the green channel. The size of the second preset mask may be the same as the size of the green channel or the preset size, such as Figure 3 The mask 2 shown. The description related to the size of the second preset mask refers to step S220, which will not be repeated here.

[0109] Different from the first preset mask, the i-th pixel P in the second preset mask i The coordinates of (x i ,y i ), when x i and i When it is an odd number or an even number at the same time, the pixel P i The pixel value is 1, when x i is an odd number, y i is an even number, or x i is an even number, y i When it is an odd number, the pixel P i The pixel value of is 0. That is, in the second preset mask, when the horizontal coordinate and the vertical coordinate of the pixel position are both odd numbers or both even numbers, the pixel value of the position is 1, and the pixel values ​​of the other positions are 0. The calculation method of the pixel value of the second preset mask is shown in formula (2).

[0110]

[0111] Wherein, i represents the i-th pixel of the second preset mask; represents the pixel value of the i-th pixel of the second preset mask; i is the horizontal coordinate of the position of the i-th pixel in the second preset mask; i is the ordinate of the position of the i-th pixel in the second preset mask; % represents the modulo operation.

[0112] The pixel values ​​of the green channel pixels are multiplied by the pixel values ​​of the second preset mask pixels to block each pixel of the green channel to obtain a second mask image, which is the blocked green channel. From the perspective of the spatial domain, the second preset mask shields the pixel values ​​of the adjacent spatial domains of the green channel, so that the image denoising network can learn the relationship between the green channel pixels from the spatial domain. From the perspective of the channel, the first preset mask and the second preset mask shield the pixel values ​​of the two adjacent channels of the red channel and the green channel, so that the image denoising network can learn the relationship between the channels of the noisy image.

[0113] In step S240 of some embodiments, the first preset mask has been described in detail in step S220 and will not be repeated here.

[0114] The pixel values ​​of the blue channel pixels are multiplied by the pixel values ​​of the first preset mask pixels to block each pixel of the blue channel to obtain a third mask image, which is the blocked blue channel. From the perspective of the spatial domain, the first preset mask shields the pixel values ​​of the blue channel in the adjacent spatial domain, so that the image denoising network can learn the relationship between the blue channel pixels from the spatial domain. From the perspective of the channel, the second preset mask and the first preset mask shield the pixel values ​​of the two adjacent channels of the green channel and the blue channel, so that the image denoising network can learn the relationship between the channels of the noisy image.

[0115] In step S250 of some embodiments, the first mask image, the second mask image and the third mask image are stitched to form a blocked noise image, and the blocked noise image is a color image.

[0116] In the above steps S210 to S250, a masking strategy is adopted in which a first preset mask is used to block the red channel and the blue channel and a second preset mask is used to block the green channel. From the perspective of the spatial domain, the masking strategy can shield the pixel values ​​of adjacent spatial domains, forcing the image denoising network to learn the relationship between pixels from a larger spatial domain. From the perspective of the channel, the masking strategy can shield the pixel values ​​of adjacent channels, forcing the image denoising network to learn the relationship between the channels of the noisy image, thereby solving the problems of poor denoising effect and high training cost due to small number of training samples.

[0117] In step S120 of some embodiments, during the training phase of the image denoising network, the occluded noisy image is input into the image denoising network, such as Figure 4 As shown. In the testing phase of the image denoising network, in order to ensure the integrity of the noisy image, the unobstructed noisy image can be directly input into the image denoising network for denoising. The network structure of the image denoising network is shown in Figure 4 As shown in the figure, the image denoising network has two modules, the first module is a tree module, and the second module is a binocular enhancement module. The tree module is used to extract multivariate image features of noisy images; the binocular enhancement module enhances multivariate image features from different angles to further improve the accuracy of feature extraction by enhancing features.

[0118] See also Figure 5 In some embodiments, the tree module includes multiple convolutional layers, and a tree structure is formed between the multiple convolutional layers. Step S130 may include but is not limited to steps S510 to S520:

[0119] Step S510, extracting features of the noise image layer by layer through multiple convolutional layers to obtain hierarchical image features;

[0120] Step S520, performing layer-by-layer feature fusion on the hierarchical image features to obtain multi-dimensional image features.

[0121] In step S510 of some embodiments, if the tree module is represented by f t , the noise image is represented as I N , the multivariate image feature output by the tree module is represented as F de =f t (I N ).

[0122] The tree module borrows the idea of ​​tree-like data structure and arranges multiple convolutional layers into a tree structure. It then extracts features of the noise image layer by layer through the multiple convolutional layers of the tree structure to obtain the image features of the noise image at each layer of the tree structure. The image features of each layer are used as hierarchical image features to obtain diversified features.

[0123] In step S520 of some embodiments, feature fusion refers to splicing multiple features in the feature channel dimension. For example, the number of channels of feature A and feature B are both 16, and feature A and feature B are fused to obtain a new feature, and the number of channels of the new feature is 32. Layer-by-layer feature fusion refers to the successive feature fusion of image features of each layer of the tree structure. For example, the image features output by the second layer of the tree structure are features A and B, and the image features output by the third layer are features A1, A2, B1 and B2, then A2 and B1 of the third layer are fused to obtain C1, C1 is fused with A and B of the second layer to obtain C2, and C2 is fused with A1 and B2 of the third layer to obtain multi-image features.

[0124] In the above steps S510 to S520, the tree module can extract multi-dimensional image features to improve the denoising performance of the image denoising network. In addition, the tree structure of the tree module can extract image features in parallel at each layer, which improves the efficiency of feature extraction compared to the method of extracting features by stacking convolutional layers.

[0125] See also Figure 6 In some embodiments, the plurality of convolutional layers include a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a sixth convolutional layer, and a seventh convolutional layer, and step S510 may include but is not limited to steps S610 to S680:

[0126] Step S610, performing first feature extraction on the noise image through a first convolutional layer to obtain a first image feature;

[0127] Step S620, extracting a second feature from the first image feature through a second convolutional layer to obtain a second image feature;

[0128] Step S630, performing a third feature extraction on the first image feature through a third convolutional layer to obtain a third image feature;

[0129] Step S640, performing fourth feature extraction on the second image feature through a fourth convolutional layer to obtain a fourth image feature;

[0130] Step S650, performing fifth feature extraction on the second image feature through a fifth convolutional layer to obtain a fifth image feature;

[0131] Step S660, performing a sixth feature extraction on the third image feature through a sixth convolutional layer to obtain a sixth image feature;

[0132] Step S670, performing seventh feature extraction on the third image feature through a seventh convolutional layer to obtain a seventh image feature;

[0133] Step S680: taking the second image feature, the third image feature, the fourth image feature, the fifth image feature, the sixth image feature and the seventh image feature as hierarchical image features.

[0134] In step S610 of some embodiments, Figure 4 As shown in the figure, the tree module has 6 layers, all of which are composed of convolutional layers and activation functions ReLU. The first three layers form a tree structure, which includes 7 convolutional layers, namely the first convolutional layer, the second convolutional layer, the third convolutional layer, the fourth convolutional layer, the fifth convolutional layer, the sixth convolutional layer and the seventh convolutional layer. The first convolutional layer is located in the first layer of the tree module, the second convolutional layer and the third convolutional layer are located in the second layer of the tree module, and the fourth convolutional layer, the fifth convolutional layer, the sixth convolutional layer and the seventh convolutional layer are located in the third layer of the tree module. Six branches are formed through these three layers in the tree module, and different features are recursively extracted through these six branches. While keeping the amount of calculation unchanged, multivariate complementary features are extracted. In the tree structure, the number of channels of the convolutional kernel of the convolutional layer decreases layer by layer by multiples of 2 as the number of layers increases.

[0135] The convolution kernel size of the first convolution layer is 3×3, and the number of input channels and output channels of the convolution kernel are 3 and 64 respectively. When extracting the first feature, the noise image is first convolved with a convolution kernel of size 3×3, and the number of input channels and output channels are 3 and 64 respectively to obtain a feature map of the noise image, and the feature map is activated with the Relu activation function to obtain the first image feature. The number of feature channels of the feature map and the first image feature is 64.

[0136] In step S620 of some embodiments, the convolution kernel size of the second convolution layer is 3×3, and the number of input channels and the number of output channels of the convolution kernel are both 32. The second image feature is directly extracted by the second convolution layer to obtain the second image feature, and the number of feature channels of the second image feature is 32. Alternatively, the first image feature is divided into the first feature and the second feature according to the number of feature channels. That is, "s32" means that the dimension of the feature channel after the first image feature is divided into two is 32, such as Figure 4 As shown. For example, if the number of feature channels of the first image feature is 64, the features represented by the first 32 feature channels of the first image feature are divided into the first feature, and the features represented by the last 32 feature channels are divided into the second feature. By bisection, the amount of calculation of each convolution layer can be reduced, and the computational complexity of a single convolution layer can be reduced. The second feature is extracted from the first feature through the second convolution layer to obtain the second image feature, and the number of feature channels of the first feature, the second feature, and the second image feature are all 32. The method for extracting the second feature is the same as that for extracting the first feature, and may refer to step S610, which will not be repeated here.

[0137] In step S630 of some embodiments, the convolution kernel size of the third convolution layer is 3×3, and the number of input channels and the number of output channels of the convolution kernel are both 32. The third feature is directly extracted from the first image feature through the third convolution layer to obtain the third image feature. Alternatively, on the premise that the second feature is extracted from the first feature through the second convolution layer, the third feature is extracted from the second feature through the third convolution layer to obtain the third image feature. The number of feature channels of the third image feature is 32. The method of extracting the third feature is the same as that of extracting the first feature, and can refer to step S610, which will not be repeated here.

[0138] In step S640 of some embodiments, the fourth feature is directly extracted from the second image feature through the fourth convolution layer to obtain the fourth image feature. Alternatively, the second image feature is equally divided into the third feature and the fourth feature according to the number of feature channels, and the fourth feature is extracted from the third feature through the fourth convolution layer to obtain the fourth image feature. That is, "s16" means that the dimension of the feature channel after the image feature is equally divided is 16, such as Figure 4 As shown. The size of the convolution kernel of the fourth convolution layer is 1×1, the number of input channels and the number of output channels of the convolution kernel are both 16, and the number of feature channels of the fourth image feature, the third feature, and the fourth feature are all 16. The number of parameters of the 1×1 convolution is 1 / 9 of that of the 3×3 convolution. By setting the size of the convolution kernel to 1×1, the network parameters can be effectively reduced to avoid overfitting. The fourth feature extraction method is the same as the first feature extraction method, and can refer to step S610, which will not be repeated here.

[0139] It is understandable that 3x3 convolution considers both cross-channel and local information integration, and can extract not only channel features but also spatial features. 1×1 convolution does not need to consider the relationship between pixels and surrounding pixels, but only focuses on the information integration of channels within a single pixel, overcoming the difficulty of effectively extracting channel features due to the small number of third feature channels. By using a combination of 3x3 convolution and 1x1 convolution, both spatial features and channel features can be extracted effectively, and the 1x1 convolution has a small number of parameters, which can reduce network parameters.

[0140] In step S650 of some embodiments, the fifth feature is directly extracted from the second image feature through the fifth convolution layer to obtain the fifth image feature. Alternatively, on the premise that the fourth feature is extracted from the third feature through the fourth convolution layer, the fifth feature is extracted from the fourth feature through the fifth convolution layer to obtain the fifth image feature. The convolution kernel size of the fifth convolution layer is 1×1, the number of convolution kernel input channels and output channels are both 16, and the number of feature channels of the fifth image feature is 16. The method of extracting the fifth feature is the same as that of extracting the first feature, and can refer to step S610, which will not be repeated here.

[0141] In step S660 of some embodiments, the sixth feature is directly extracted from the third image feature through the sixth convolution layer to obtain the sixth image feature. Alternatively, the third image feature is equally divided into the fifth feature and the sixth feature according to the number of feature channels, and the sixth feature is extracted from the fifth feature through the sixth convolution layer to obtain the sixth image feature. The convolution kernel size of the sixth convolution layer is 1×1, the number of convolution kernel input channels and output channels are both 16, and the number of feature channels of the fifth feature, the sixth feature, and the sixth image feature are all 16. The sixth feature extraction method is the same as the first feature extraction method, and can refer to step S610, which will not be repeated here.

[0142] In step S670 of some embodiments, the seventh feature is directly extracted from the third image feature through the seventh convolution layer to obtain the seventh image feature. Alternatively, on the premise that the sixth feature is extracted from the fifth feature through the sixth convolution layer, the seventh feature is extracted from the sixth feature through the seventh convolution layer to obtain the seventh image feature. The convolution kernel size of the seventh convolution layer is 1×1, the number of convolution kernel input channels and output channels are both 16, and the number of feature channels of the seventh image feature is 16.

[0143] In step S680 of some embodiments, the second image features and the third image features output by the second layer, and the fourth image features, the fifth image features, the sixth image features and the seventh image features output by the third layer are used as hierarchical image features.

[0144] It should be noted that the single-branch network uses stacked convolutional layers to extract features. If the number of convolutional kernel channels in each layer is 64, the number of convolutional kernel channels in the first layer of the tree module is 64, the number of convolutional kernel channels in the second and third convolutional layers of the second layer is 32, and the number of convolutional kernel channels in the second layer is 64 in total. The number of convolutional kernel channels in the fourth, fifth, sixth and seventh convolutional layers of the third layer is 16, and the number of convolutional kernel channels in the third layer is also 64 in total. The number of convolutional kernel channels in each layer is 64, so the number of parameters remains unchanged, and the feature resolution and the amount of calculation remain unchanged. For a single convolutional layer, the number of feature channels of the image features input to a single convolutional layer is not all 64, which reduces the amount of calculation and parameters of some single convolutional layers.

[0145] In the above steps S610 to S680, the convolution layer is formed into 6 branches through a tree structure, which can extract different image features from each branch while ensuring that the amount of calculation remains unchanged, thereby obtaining multi-component complementary features.

[0146] See also Figure 7 In some embodiments, step S520 may include but is not limited to steps S710 to S730:

[0147] Step S710, performing a first feature fusion on the fifth image feature and the sixth image feature to obtain a first fused feature;

[0148] Step S720, performing a second feature fusion on the first fusion feature, the second image feature and the third image feature to obtain a second fusion feature;

[0149] Step S730, performing a third feature fusion on the second fusion feature, the fourth image feature and the seventh image feature to obtain a multivariate image feature.

[0150] In step S710 of some embodiments, features are gathered layer by layer through the last three layers of the tree module, and the output of each branch contributes to the image denoising network. The complementary features of the six branches can be smoothly merged to improve the denoising performance of the image denoising network. Figure 4As shown in the figure, the fourth layer of the tree module includes three convolutional layers, which are the eighth convolutional layer, the ninth convolutional layer and the tenth convolutional layer from top to bottom. The fifth layer includes one convolutional layer, which is the eleventh convolutional layer. The sixth layer includes one convolutional layer, which is the twelfth convolutional layer. The convolution kernel size of each convolutional layer in the fourth layer is 3×3, and the convolution kernel size of the fifth and sixth layers is 1×1. The number of input channels and output channels of the convolution kernel of the eighth convolutional layer is 16, the number of input channels and output channels of the convolution kernel of the ninth convolutional layer is 32, the number of input channels and output channels of the convolution kernel of the tenth convolutional layer is 16, the number of input channels and output channels of the convolution kernel of the eleventh convolutional layer is 64 and 32 respectively, and the number of input channels and output channels of the convolution kernel of the twelfth convolutional layer is 64.

[0151] In step S710 of some embodiments, the fifth image feature and the sixth image feature are concatenated by Concat. Figure 4 As shown, "c16" indicates that the fifth image feature and the sixth image feature are copied, both of which have 16 channels, and feature splicing is performed. The image features after feature splicing are extracted through the ninth convolution layer of the fourth layer to obtain the first fusion feature.

[0152] In step S720 of some embodiments, the first fusion feature and the sum of the second image feature and the third image feature are concatenated by Concat. Figure 4 As shown in the figure, "c32" indicates that the second image feature and the third image feature are copied, and their feature channels are both 32; and the features are added, and the sum is concatenated with the first fusion feature. The feature fusion of the four branches of the second and third layers is realized, and the concatenated features are convoluted through the eleventh convolution layer of the fifth layer to obtain the second fusion feature.

[0153] In step S730 of some embodiments, the fourth image feature is extracted by the eighth convolution layer of the fourth layer, the seventh image feature is extracted by the tenth convolution layer of the fourth layer, and the second fusion feature, the fourth image feature after feature extraction, and the seventh image feature after feature extraction are concatenated by Concat. Figure 4As shown, "c16" indicates that the fourth image feature after feature extraction and the seventh image feature after feature extraction are copied, and their feature channels are both 16. The feature fusion of the six branches of the second and third layers is realized, and the features after feature fusion are extracted through the twelfth convolution layer of the sixth layer to obtain multivariate image features. It can be understood that the total number of convolution kernel channels in each of the three layers after the tree module is 64. Compared with the method of stacking convolution layers with a single branch, the parameter quantity of the entire tree module remains unchanged, the feature resolution remains unchanged, and the amount of calculation remains unchanged. In the above steps S710 to S730, the image features of the six branches are gradually fused three times, so as to achieve the smooth fusion of the six branch features, thereby improving the denoising effect of the noisy image.

[0154] See also Figure 8 In some embodiments, the binocular enhancement module includes two branches, the upper branch is a feature refinement module, the feature refinement module includes a pooling layer, a feature refinement convolution layer and an upsampling layer, and step S140 may include but is not limited to steps S810 to S840:

[0155] Step S810, performing pooling processing on the multi-image features through a pooling layer to obtain image pooling features;

[0156] Step S820, performing convolution processing on the image pooling feature through a feature refinement convolution layer to obtain a first image convolution feature;

[0157] Step S830, upsampling the first image convolution feature through an upsampling layer to obtain an image upsampling feature;

[0158] Step S840: performing feature fusion on the multivariate image features and the image upsampling features according to the first preset factor to obtain image refinement features.

[0159] In step S810 of some embodiments, the feature refinement module includes five layers, the first layer is a pooling layer, and the pooling process of the pooling layer can be at least one of maximum pooling, average pooling, random pooling, and global average pooling. Reducing the resolution of the feature by pooling can not only suppress noise, but also reduce the amount of calculation of subsequent modules.

[0160] In step S820 of some embodiments, Figure 4As shown in the figure, the feature refinement convolution layer includes two cascaded convolution layers, which are the first convolution layer (the first Conv-3+ReLU) and the second convolution layer (the second Conv-3+ReLU) from front to back. Conv-3 represents a 3×3 convolution kernel, and ReLU represents the use of the ReLU activation function. The first convolution layer and the second convolution layer are located in the second and third layers of the feature refinement module, respectively, and are both composed of a 3×3 convolution kernel and a ReLU activation function. The image pooling features are convolved through the first convolution layer and the second convolution layer in turn to obtain the first image convolution feature.

[0161] In step S830 of some embodiments, the fourth layer of the feature refinement module is an upsampling layer, which is used to predict high-quality features so that the feature resolution of the upsampled features of the image is greater than the feature resolution of the pooled features of the image.

[0162] In step S840 of some embodiments, there is dynamic residual learning between the input of the pooling layer and the output of the upsampling layer, and feature fusion is achieved through dynamic residual learning. Dynamic residual learning introduces a learnable factor λ1, which fuses the input of the first layer (multiple image features) and the output of the fourth layer (image upsampling features), which not only alleviates the gradient vanishing problem, but also promotes the dynamic fusion of deep features and shallow features. Fusion refers to the in-place addition of multi-image features and image upsampling features. The fifth layer of the feature refinement module is a convolution layer, which consists of a 3×3 convolution kernel and a ReLU activation function. The fused features are convolved through the convolution layer to obtain image refinement features.

[0163] The method for extracting image refinement features is shown in formula (3).

[0164] O s =f c3 (f up (f c3 (f c3 (f P (F de ))))+λ1×F de )Formula (3)

[0165] Among them, O s is the image refinement feature output by the feature refinement module; f P is the pooling layer; F de is the multivariate image feature input to the feature refinement module; f c3 It is a convolution layer consisting of a convolution kernel of size 3×3 and a ReLU activation function; f up is the upsampling layer; λ1 is the first preset factor.

[0166] In the above steps S810 to S840, the feature refinement module adopts a pooling-feature extraction-upsampling strategy, which reduces the amount of calculation and suppresses the noise in the spatial dimension.

[0167] See also Fig. 9 In some embodiments, the lower branch of the binocular enhancement module is a local enhancement module, and the local enhancement module includes a local enhancement convolution layer and a gating layer. Step S140 may include but is not limited to steps S910 to S930:

[0168] Step S910, performing convolution processing on the multi-image feature through a local enhancement convolution layer to obtain a second image convolution feature;

[0169] Step S920, performing activation processing on the second image convolution feature through the gating layer to obtain an image activation feature;

[0170] Step S930: performing feature fusion on the multivariate image features and the image activation features according to the second preset factor to obtain image saliency features.

[0171] In step S910 of some embodiments, Figure 4 As shown in the figure, the local enhancement module has five layers. The local enhancement convolutional layer includes the first convolutional layer (Conv-1+ReLU), the second convolutional layer (Conv-1) and the third convolutional layer (Conv-1+ReLU), which are located in the first, second and third layers respectively. The convolution kernel size of the first convolutional layer, the second convolutional layer and the third convolutional layer is 1×1. The difference is that the first convolutional layer and the third convolutional layer are composed of 1×1 convolution and ReLU, and the second convolutional layer is only composed of 1×1 convolution. The input channel of the first convolutional layer is 64, and the output channel is 16 (i.e. Figure 4 "Ch:64→16" in the second convolutional layer); the input and output channels of the second convolutional layer are both 16 (i.e. Figure 4 "Ch:16→16" in the figure); the input channel of the third convolutional layer is 16, and the output channel is 64 (i.e. Figure 4 The first convolution layer reduces the number of feature channels, the second convolution layer keeps the number of feature channels of the output features of the first convolution layer unchanged, and the third convolution layer increases the number of feature channels of the output features of the second convolution layer. The multi-image features are sequentially input into the first convolution layer, the second convolution layer, and the third convolution layer for convolution processing to obtain the second image convolution features.

[0172] In step S920 of some embodiments, the gating layer is located at the fourth layer of the local enhancement module, and the gating mechanism of the gating layer is composed of a sigmoid function. The second image convolution feature is mapped to a value between 0 and 1 through the sigmoid function, and the value indicates the ratio of the multivariate image feature that can pass through the gating layer. The second image convolution feature that passes the sigmoid function is used as the image activation feature.

[0173] In step S930 of some embodiments, there is dynamic residual learning between the input of the first convolutional layer and the output of the gated layer. Dynamic residual learning introduces a learnable factor, namely a second preset factor, and scales the multivariate image features according to the second preset factor to make the multivariate image features more accurate. The scaled multivariate image features and image activation features are bitwise multiplied to filter redundant information and control the flow of effective information to the convolutional layer of the fourth layer. The filtered features are convolved by the convolutional layer (Conv-1+ReLU) located in the fifth layer of the local enhancement module to obtain image saliency features. The convolutional layer consists of a convolution with a convolution kernel size of 1×1 and ReLU, and the input channel and output channel are both 64. The calculation method of the image saliency feature is shown in formula (4).

[0174] O e =R(f c1 (Sigmoid(R(f c1 (f c1 (R(f c1 (F de )))))))×λ2×F de )) Formula (4)

[0175] Among them, O e is the image saliency feature output by the local enhancement module; c1 is a convolution with a kernel size of 1×1, R is the ReLU activation function, and λ2 is the second preset factor.

[0176] The above steps S910 to S930 can not only effectively reduce the amount of calculation but also suppress the redundant features of the channel dimension through the bottleneck structure of the channel dimension.

[0177] In step S150 of some embodiments, the denoised image is a clean image obtained by removing noise from the noisy image through the image denoising network, and the image channel of the denoised image is 3. If the image input batch size is 64 and the image block size is 48×48, the input size of the image denoising network is 64×3×48×48, and the output size is 64×3×48×48. The calculation method of the denoised image is shown in formula (5).

[0178] O C =f c3(Concat(f s (F de ),f e (F de )))Formula (5)

[0179] Among them, O C is the denoised image; f c3 is a convolution with a kernel size of 3×3; Concat is a connection function; F de is the input of the binocular enhancement module; f s is the feature refinement module; f e It is a local enhancement module.

[0180] The image refinement features output by the feature refinement module and the image saliency features output by the local enhancement module are concatenated through Concat, and the concatenated features are convolved with a convolution kernel size of 3×3 (Conv-3) to obtain a denoised image.

[0181] In the training phase, the loss function is used to calculate the pixel-by-pixel loss of the denoised image and the noisy image output by the image denoising network to obtain the target loss value. The loss function uses the mean squared error loss function (MSE). When the target loss value reaches the preset loss threshold or the number of network iterations reaches the preset threshold, the network training is stopped. The noisy image is input into the trained image denoising network for denoising.

[0182] It can be understood that the noisy image generated by the self-supervised mask and the corresponding noisy image are composed of image pairs to increase the amount of samples used to train the image denoising network.

[0183] Taking 50 levels of noise as an example, the image denoising method based on self-supervised enhanced tree structure in an embodiment of the present application is described in detail.

[0184] Get a noisy image. The noisy image is as follows: Fig.10a As shown. Extract the R channel, G channel and B channel of the noise image. The R channel is as follows Fig.10b As shown, the G channel is Fig.10c As shown, channel B is Fig.10d As shown. The R channel is masked using the mask 1 designed by the self-supervised algorithm to obtain the masked R channel. The masked R channel is shown in Fig.10e As shown. The G channel is masked by using the mask 2 designed by the self-supervised algorithm to obtain the occluded G channel. The occluded G channel is shown in Fig.10f As shown. Use mask 1 to mask the B channel to obtain the blocked B channel. The blocked B channel is as follows Figure 10gThe R channel after occlusion, the G channel after occlusion, and the B channel after occlusion are merged to obtain the noise image after occlusion. The noise image after occlusion is as shown in Fig.10h The masked noise image is divided into multiple image blocks of the same size, as shown in Figures 10I and Fig.10o As shown. The image block shown in FIG10I is used as the first occluded image block. Fig.10o The image block shown is used as the second occluded image block, and the first occluded image block corresponds to an image block of the noise image, such as Figure 10k As shown, the second occluded image block corresponds to another image block of the noise image, such as Figure 10n As shown. Multiple image blocks are input into the initial image denoising network to train the initial denoising network and obtain the target image denoising network, which outputs multiple clean image blocks. The clean image block corresponding to the first occluded image block is shown as Figure 10m As shown, the clean image block corresponding to the second occluded image block is as follows Figure 10p As shown in the figure. During training, the batch size of the noise image is 64, the size of the image block is 48×48, the image channels of the noise image and the image block are both 3, the input size of the image denoising network is 64×3×48×48, and the output size is 64×3×48×48. By analyzing the distribution of noise, using the spatial domain of a single noise image, the correlation of the neighborhood and the independence of noise, a specific mask is designed to block the noise image. This self-supervised training strategy solves the problem of insufficient training data and helps to train a more robust denoising network.

[0185] Get Fig.10aThe noisy image shown in the figure is input into the trained target image denoising network. The target image denoising network includes a tree module and a binocular enhancement module. The tree module has 6 layers, all of which are composed of convolutional layers and ReLU. The convolution kernel size of the first, second and fourth convolutional layers is 3×3, and the convolution kernel size of the third, fifth and sixth convolutional layers is 1×1. The first three layers are divided into 6 branches layer by layer, and multi-component complementary features are extracted through 6 branches. The multi-component complementary features are fused layer by layer through the last three layers. The binocular enhancement module includes a feature refinement module, a local enhancement module and a convolutional layer. The feature refinement module has 5 layers, which are composed of a pooling layer, a convolutional layer, an upsampling layer and a dynamic residual. The convolution kernel size of the convolutional layer is 3×3. The local enhancement module has 5 layers. The first, third and fifth layers are composed of convolutional layers and ReLU. The second layer is composed of a convolutional layer. The fourth layer is a gated layer. The gated layer adopts a gated mechanism, which is implemented by a sigmoid activation function. The feature refinement module is used to purify the features after layer-by-layer fusion to obtain image refinement features. The local enhancement module is used to enhance the local features after layer-by-layer fusion to obtain image salient features. The image refinement features and image salient features are fused and input into the convolution layer for convolution processing to obtain a clean image. The clean image is as follows: Fig.10j shown.

[0186] Different from the traditional image denoising method based on deep convolutional neural network, the embodiment of the present application adopts a method combining a tree module and a binocular enhancement module to construct an enhanced tree-like lightweight image denoising neural network. The tree module draws on the idea of ​​the data structure of the tree relationship to derive six branches, extract multivariate features, and keep the computational complexity unchanged. The features are fused layer by layer so that the features of the six branches can be smoothly unified. The embodiment of the present application achieves good results in the image denoising task. In addition to being able to process Gaussian noise, it can also process blind noise and real noise. The image denoising method of the embodiment of the present application has a low computational cost and has a wide range of application scenarios. It is of great significance for applications such as medical diagnosis, remote sensing positioning and autonomous driving.

[0187] See also Fig.11 The present application also provides an image denoising device, which can implement the above-mentioned image denoising method, and the device includes:

[0188] An acquisition module 1110, configured to acquire a noise image;

[0189] An input module 1120, used to input the noisy image into a preset image denoising network, where the image denoising network includes a tree module and a binocular enhancement module;

[0190] A first feature extraction module 1130 is used to extract multivariate features from the noise image through a tree module to obtain multivariate image features;

[0191] The second feature extraction module 1140 is used to perform feature extraction on the multi-image features through the binocular enhancement module to obtain image refinement features, and perform local feature enhancement on the multi-image features to obtain image saliency features;

[0192] The feature fusion module 1150 is used to fuse the image refinement features and the image saliency features to obtain a denoised image.

[0193] The specific implementation of the image denoising device is substantially the same as the specific implementation of the above-mentioned image denoising method, and will not be described in detail herein.

[0194] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned image denoising method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0195] See also Fig.12 , Fig.12 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0196] The processor 1210 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0197] The memory 1220 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1220 can store an operating system and other applications. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1220, and the processor 1210 calls and executes the image denoising method of the embodiment of the present application;

[0198] Input / output interface 1230, used to implement information input and output;

[0199] The communication interface 1240 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0200] The bus 1250 transmits information between the various components of the device (e.g., the processor 1210, the memory 1220, the input / output interface 1230, and the communication interface 1240);

[0201] The processor 1210 , the memory 1220 , the input / output interface 1230 , and the communication interface 1240 are connected to each other in communication within the device via a bus 1250 .

[0202] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned image denoising method is implemented.

[0203] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0204] The image denoising method, image denoising device, electronic device and computer-readable storage medium provided by the embodiments of the present application can reduce the complexity of denoising and improve the generalization performance of the image denoising network by cascading a tree module and a binocular enhancement module. The tree module can extract multi-dimensional image features to avoid a single image feature being unable to fully express the image, thereby affecting the image denoising effect. The binocular enhancement module is used to enhance multi-dimensional image features from different angles, so as to further improve the accuracy of feature extraction by enhancing features, thereby improving the effect of image denoising.

[0205] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0206] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0207] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0208] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0209] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0210] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0211] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0212] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0213] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0214] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0215] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. An image denoising method, characterized in that: The method comprises: Get a noisy image; Inputting the noisy image into a preset image denoising network, wherein the image denoising network includes a tree module and a binocular enhancement module; Performing multivariate feature extraction on the noise image through the tree module to obtain multivariate image features; Through the binocular enhancement module, the multi-image features are subjected to feature purification to obtain image refinement features, and the local features of the multi-image features are enhanced to obtain image saliency features; Performing feature fusion on the image refinement features and the image saliency features to obtain a denoised image; The binocular enhancement module includes a feature refinement module, and the feature refinement module includes a pooling layer, a feature refinement convolution layer and an upsampling layer. The feature refinement of the multivariate image features is performed to obtain image refinement features, including: Performing pooling processing on the multivariate image features through the pooling layer to obtain image pooling features; Performing convolution processing on the image pooling feature through the feature refinement convolution layer to obtain a first image convolution feature; Upsampling the first image convolution feature through the upsampling layer to obtain an image upsampling feature; The multivariate image feature and the image upsampling feature are fused according to a first preset factor to obtain the image refinement feature.

2. The image denoising method according to claim 1, characterized in that: Before inputting the noisy image into a preset image denoising network, the image denoising method further includes: Extracting the red channel, green channel and blue channel of the noise image; Masking the pixels of the red channel by using a first preset mask to obtain a first mask image; Masking the pixel points of the green channel by a second preset mask to obtain a second mask image; Using the first preset mask to mask the pixels of the blue channel, to obtain a third mask image; The first mask image, the second mask image and the third mask image are stitched to obtain the blocked noise image.

3. The image denoising method according to claim 1, characterized in that: The tree-shaped module includes a plurality of convolutional layers, and a tree-shaped structure is formed between the plurality of convolutional layers. The multivariate feature extraction of the noise image is performed by the tree-shaped module to obtain multivariate image features, including: Extracting features of the noise image layer by layer through multiple convolutional layers to obtain hierarchical image features; The hierarchical image features are fused layer by layer to obtain the multivariate image features.

4. The image denoising method according to claim 3, characterized in that: The multiple convolutional layers include a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a sixth convolutional layer and a seventh convolutional layer. The multiple convolutional layers are used to extract features of the noise image layer by layer to obtain hierarchical image features, including: Performing a first feature extraction on the noise image through the first convolutional layer to obtain a first image feature; Performing second feature extraction on the first image feature through the second convolutional layer to obtain a second image feature; Performing a third feature extraction on the first image feature through the third convolutional layer to obtain a third image feature; Performing a fourth feature extraction on the second image feature through the fourth convolutional layer to obtain a fourth image feature; Performing a fifth feature extraction on the second image feature through the fifth convolutional layer to obtain a fifth image feature; Performing a sixth feature extraction on the third image feature through the sixth convolutional layer to obtain a sixth image feature; Performing a seventh feature extraction on the third image feature through the seventh convolutional layer to obtain a seventh image feature; The second image feature, the third image feature, the fourth image feature, the fifth image feature, the sixth image feature and the seventh image feature are used as the hierarchical image features.

5. The image denoising method according to claim 4, characterized in that: The step of fusing the hierarchical image features layer by layer to obtain the multivariate image features includes: Performing a first feature fusion on the fifth image feature and the sixth image feature to obtain a first fused feature; Performing a second feature fusion on the first fusion feature, the second image feature and the third image feature to obtain a second fusion feature; A third feature fusion is performed on the second fusion feature, the fourth image feature and the seventh image feature to obtain the multivariate image feature.

6. The image denoising method according to any one of claims 1 to 5, characterized in that: The binocular enhancement module includes a local enhancement module, and the local enhancement module includes a local enhancement convolution layer and a gating layer. The local feature enhancement of the multivariate image features to obtain image saliency features includes: Performing convolution processing on the multi-image feature through the local enhanced convolution layer to obtain a second image convolution feature; Performing activation processing on the second image convolution feature through the gating layer to obtain an image activation feature; The multivariate image feature and the image activation feature are fused according to a second preset factor to obtain the image saliency feature.

7. An image denoising device, characterized in that: The device comprises: An acquisition module, used for acquiring a noise image; An input module, used for inputting the noisy image into a preset image denoising network, wherein the image denoising network includes a tree module and a binocular enhancement module; A first feature extraction module, used for performing multivariate feature extraction on the noise image through the tree module to obtain multivariate image features; A second feature extraction module is used to perform feature purification on the multi-image features through the binocular enhancement module to obtain image refinement features, and perform local feature enhancement on the multi-image features to obtain image saliency features; A feature fusion module, used for fusing the image refinement features and the image saliency features to obtain a denoised image; The binocular enhancement module includes a feature refinement module, and the feature refinement module includes a pooling layer, a feature refinement convolution layer and an upsampling layer. The feature refinement of the multivariate image features is performed to obtain image refinement features, including: Performing pooling processing on the multivariate image features through the pooling layer to obtain image pooling features; Performing convolution processing on the image pooling feature through the feature refinement convolution layer to obtain a first image convolution feature; Upsampling the first image convolution feature through the upsampling layer to obtain an image upsampling feature; The multivariate image feature and the image upsampling feature are fused according to a first preset factor to obtain the image refinement feature.

8. An electronic device, characterized in that The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the image denoising method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the image denoising method according to any one of claims 1 to 6 is implemented.