Improved BM3D image denoising method based on image depth feature map

Abstract

The invention relates to an improved BM3D image denoising method based on an image depth feature map, and belongs to the technical field of image processing. Different from a traditional BM3D algorithm which directly carries out block matching on a noise image, the method provided by the invention selects a classic test image for image denoising, adopts a trained UNet denoising network to obtain feature maps of different scales of the noise image, and then carries out block matching operation of the BM3D algorithm on the feature maps. And selecting all non-repeated similar blocks at the same position on the noise image according to the matched similar block position on each feature image, thereby obtaining a block matching group considering the depth feature of the noise image, and carrying out the subsequent operation of the traditional BM3D algorithm on this basis. According to the method, the block matching process of the traditional BM3D algorithm is improved by considering the features of different scales of the noise image, so that the matching result is more accurate, and the denoising effect of the traditional BM3D algorithm is enhanced while the detail features of the image are well reserved.

Description

Technical Field

[0001] This invention relates to an improved BM3D image denoising method based on image depth feature maps, specifically an improved BM3D algorithm that uses a UNet denoising network to obtain feature maps of noisy images and performs block matching operations on the feature maps, belonging to the field of image processing technology. Background Technology

[0002] Noise is inevitably generated during image generation and transformation, which not only degrades the visual quality of the image itself but also affects subsequent advanced visual processing tasks. Therefore, it is necessary to take measures to eliminate noise while preserving image details.

[0003] Currently, image processing algorithms can be broadly categorized into three types: spatial domain denoising, frequency domain denoising, and deep learning denoising. Spatial domain denoising directly averages pixels in the noisy image, such as Gaussian filtering, median filtering, and anisotropic filtering. These methods can preserve edge information well but also easily blur the image. Frequency domain denoising transforms the image from the spatial domain to the transform domain. In the transform domain, the image is divided into low-frequency and high-frequency components, thus separating the real signal from the noise. Noise belongs to the low-frequency component, so thresholding methods can be used to filter out the low-frequency component to achieve denoising. In 2005, based on the nonlocal self-similarity of images, Buades, Coll, and Morel proposed a spatial domain denoising algorithm called NLM, which significantly improved the performance of spatial domain denoising algorithms. To further improve the performance of denoising algorithms, in 2007, Dabov, Foi, and others combined the characteristics of nonlocal self-similarity and frequency domain denoising to propose the BLOCK-MATCHING and 3D filtering (BM3D) algorithm, which has a significant denoising effect and can effectively preserve image details. The BM3D algorithm can be simply summarized into three steps: First, block matching and grouping, that is, dividing the noisy image into blocks, searching for similar blocks of each block according to a certain similarity metric and a set threshold, and combining them to form a 3D matrix cluster; Second, using collaborative Wiener filtering to filter each set cluster; Third, aggregation, that is, re-aggregating the filtered clusters to obtain the output image. With the development of artificial intelligence technology, denoising methods based on deep learning have attracted more and more attention. In 2012, Burger et al. used a multi-layer neural network to solve the image denoising task with excellent performance. Reference [1] proposed a texture-aware neural network TANet, which can recover small details from noisy images well. Reference [2] took advantage of the fact that the UNet network can learn multi-scale features and can be used for image conversion, proposed a two-stage denoising algorithm and achieved competitive denoising performance.

[0004] According to the BM3D steps, block matching is a key step in the algorithm. Currently, the academic community has done a lot of research on block matching. Reference [3] proposed a new block similarity metric, which alleviated the interference of noise and ensured the accuracy of block matching when there is high intensity noise in the image. Reference [4] proposed an improved BM3D technique based on adaptive threshold, which has excellent denoising performance for different types of noise in different images. In order to achieve more accurate block matching, Reference [5] used the regional energy of the AC coefficients of the Discrete Cosine Transform (DCT) to divide each image into three different regions and adaptively select the block size according to the region. It can be seen that the current improved block matching algorithm has achieved certain results from the perspective of improving the similarity metric criteria and improving the accuracy of block matching. However, the block matching operation involved in the above research is often directly processing traditional noisy images, lacking consideration of deep features of the image, such as structure, brightness, contrast, etc. This makes some important features of the image ignored, resulting in inaccurate block matching, and thus the visual quality of the denoised image is not necessarily good.

[0005] In recent years, the rapid development of deep learning has provided ideas for the extraction of deep features of images. Deep learning has the advantages of autonomous learning of target features and high recognition rate. The RGB-D salient object detection model established by Reference [6] uses ResNet-50 [7] and VGG-16 [8] to process RGB images and their corresponding depth maps. It can effectively consider the complementarity of multi-level features and generate high-quality prediction saliency maps. Reference [9] proposed a new method for extracting spatiotemporal local image motion features based on depth images. This method is not only simple to calculate, but can also characterize the local temporal changes of human motion and the local spatial structure of actions. Reference

[10] proposed a UNet (MA-UNet)

[11] based on multi-attention to improve the ability to extract fine-grained features. It was used in the semantic segmentation of remote sensing images and achieved good results. Reference

[12] and others used feature fusion technology to construct a pyramid structure feature map with equally rich location information and semantic information, and performed target detection on multi-scale feature layers to enhance the network's adaptability to scale images. It can be seen that deep learning can learn deep features at different scales of traditional images, and using these features to process other advanced vision tasks can achieve more accurate and effective results. However, there is currently very little research on block matching using deep learning feature maps. Summary of the Invention

[0006] This invention provides an improved BM3D image denoising method based on image depth feature maps, which enhances the denoising effect of the traditional BM3D algorithm while better preserving image detail features.

[0007] The technical solution of this invention is: an improved BM3D image denoising method based on image depth feature maps, the specific steps of which are as follows:

[0008] Step 1: Input the noisy image into a trained UNet denoising network to obtain feature maps of the noisy image at different scales;

[0009] Step 2: Perform improved block matching on the feature map to obtain block matching groups that take into account the depth features of the noisy image, and integrate them into a three-dimensional similar block matrix;

[0010] The specific steps for improving block matching in Step 2 are as follows:

[0011] Construct a two-dimensional sliding window of size N1×N1, and extract image blocks on the noisy image I according to a set step size p. Let the block currently being operated on be Z. R Then, using reference block Z... R Searching for reference block Z within an N2×N2 search region centered on the target. R Similar blocks, denoted as Z Q Define distance d(Z) R Z Q Z is measured by (inversely proportional to similarity). R and Z Q The similarity between two blocks is expressed as shown in equation (1).

[0012]

[0013] In the formula, τ 2D It is a 2D unitary transform operator (e.g., DCT, DFT, etc.). γ is the hard thresholding operator, λ thr2D It is a fixed threshold parameter, σ represents the noise standard deviation, ||·||2 represents the L2 norm, and γ is defined as follows.

[0014]

[0015] Finding reference block Z R The idea behind similar blocks is to select a suitable threshold τ1, if d(Z) R Z Q If ) < τ1, then Z is considered to be Q With Z R They are similar, so add them to set B. R In the middle. Finally, refer to block Z. R and its similar block set B R Integrate into a three-dimensional matrix T R .

[0016] Next, following the method for finding similar blocks described above, a similar process will be performed on the feature maps. Assume three feature maps are selected, namely feature...Figure 1 ,feature Figure 2 and characteristics Figure 3 Search for similar blocks to the reference block R on each of the three feature maps. Among them, the feature maps... Figure 1 The similar blocks to the reference block R are block Q1 and block Q2, with the following features. Figure 2 The similar blocks to the reference block R are block Q1 and block Q3, with the following features. Figure 3 The similar blocks to the reference block R are blocks Q4 and Q5. Then, based on the positions of the similar blocks matched on each feature map, all non-repeating similar blocks are selected at the same positions on the noisy image, and these are used as the set of similar blocks for the reference block R to form a three-dimensional matrix T. R .

[0017] Step 3: Perform a 3D transformation on the similarity block matrix, reduce image noise using a hard thresholding filter, and obtain estimates of the 2D similarity blocks using a 3D inverse transformation method. A weighted average of multiple estimates for each similarity block is then used to obtain the base estimate of the image.

[0018] The specific steps of Step 3 are as follows:

[0019] Step 3.1: For the three-dimensional matrix T R conduct (Two-dimensional Bior wavelet transform and one-dimensional Haar wavelet transform) Three-dimensional unitary transform. Three-dimensional transform domain denoising can separate noise from real information in an image without energy loss. Most of the real information in an image is concentrated at the vertices of the three-dimensional matrix energy, while noise is often concentrated at the bottom of the three-dimensional matrix energy. Therefore, through hard thresholding filtering in the transform domain, most of the image information can be preserved while effectively removing noise. B after hard threshold filtering R The set of estimated values ​​for the middle block, λ thr3D It is a fixed threshold parameter.

[0020] Step 3.2: After hard thresholding shrinkage, each block will obtain a basic estimated weight, N. R express The number of non-zero coefficients in the matrix is ​​determined by referring to block Z. R The basic estimated weights are as follows:

[0021]

[0022] For a given pixel i to appear in multiple blocks, a weighted average of the estimates from these overlapping blocks is needed to obtain the base estimate of i, as shown in the following formula:

[0023]

[0024] In the formula, Z Q Representing set B R For any image patch containing pixel i, there may also be multiple sets B. R Since each set contains an image patch with pixel i, a weighted average of the patch estimates for all sets containing that pixel is needed. For any set B R Block Z containing cell i Q The estimated value, x Q It is a 0-1 variable. and x Q The expression is as follows:

[0025]

[0026]

[0027] Step 4: Use the UNet denoising network to obtain feature maps of different scales of the basic estimated image;

[0028] Step 5: Perform block matching on the feature map of the base estimated image to obtain the block matching groups of the base estimated image, and integrate them into a three-dimensional similar block matrix;

[0029] The specific steps in Step 4 and Step 5 include:

[0030] The basic estimation image Y obtained in the basic estimation stage basic The input is fed into the UNet denoising network to obtain the feature map of the last layer in the last process. The subsequent process is similar to Step 2, forming a new three-dimensional matrix T. R2 At this point, there are two three-dimensional matrices: one is the three-dimensional matrix T obtained in the basic estimation stage, which consists of similar blocks in the noisy image. R The other is a three-dimensional matrix T composed of similar patches in the image generated by the basic estimation. R2 .

[0031] Step 6: Perform 3D transformation processing on the similar block matrices of the noisy image and the basic estimated image respectively. Treat the similar block matrix in the basic estimate as the energy spectrum of the real signal. Use this energy spectrum to perform collaborative Wiener filtering processing on the noisy image. Perform inverse transformation on the processed data and return to the original position of the similar block to obtain the final estimated value. Perform weighted averaging processing on pixels with multiple estimated values ​​to obtain the final image estimate.

[0032] The specific steps of Step 6 are as follows:

[0033] Step 6.1: Unlike the hard threshold filtering used in the basic estimation stage, the final estimation uses Wiener filtering. The formula for calculating the attenuation coefficient of the Wiener filter is as follows:

[0034]

[0035] Then, through the 3D transform coefficients of the noisy image The product of the Wiener contraction coefficient is used to achieve the T R Wiener filtering is then applied. Finally, the estimated values ​​of the block are generated through inverse transform. The Wiener filtering expression is as follows:

[0036]

[0037] The weight of each reference block in the final estimation stage can be defined as:

[0038]

[0039] Step 6.2: Similarly, a weighted average of these block estimates is needed to obtain the final estimate of pixel i:

[0040]

[0041] In the formula, For any set B R Block Z containing cell i Q The estimated value is expressed similarly to formula (5).

[0042] This invention further optimizes the block matching process of the traditional BM3D algorithm. Specifically, unlike the traditional BM3D algorithm which directly performs block matching on the noisy image, this invention first inputs the noisy image into a trained UNet denoising network to obtain feature maps of different scales of the noisy image. Assuming three feature maps are selected, they are: feature... Figure 1 ,feature Figure 2 and characteristics Figure 3 Search for similar blocks to the reference block R on each of the three feature maps. Among them, the feature maps... Figure 1 The similar blocks to the reference block R are block Q1 and block Q2, with the following features. Figure 2 The similar blocks to the reference block R are block Q1 and block Q3, with the following features. Figure 3 The similar blocks to the reference block R are blocks Q4 and Q5. Then, based on the positions of the similar blocks matched on each feature map, all non-repeating similar blocks are selected at the same positions on the noisy image, and these are used as the set of similar blocks for the reference block R to form a three-dimensional matrix T. R .

[0043] The beneficial effects of this invention are:

[0044] A pre-trained UNet denoising network is used to acquire feature maps of noisy images at different scales. Then, block matching operations of the BM3D algorithm are performed on the feature maps. Based on the positions of similar blocks matched on each feature map, all non-repeating similar blocks are selected at the same positions in the noisy image. This results in block matching groups that take into account the depth features of the noisy image, thereby improving the accuracy of block matching and enhancing the denoising effect. Attached Figure Description

[0045] Figure 1 The flowchart shows the improved BM3D image denoising algorithm based on image depth feature maps.

[0046] Figure 2 This figure shows a simplified example of improved block matching, using block matching with three feature maps.

[0047] Figure 3 and Figure 4 The denoising results and local denoising effects of Baboon with σ=10 under the method of this invention, BM3D, NCSR and NLM are respectively.

[0048] Figure 5 and Figure 6 The figures show the denoising results and local denoising effects for Man with σ=10 under four denoising algorithms: the method of this invention, BM3D, NCSR, and NLM. Compared with the three algorithms BM3D, NCSR, and NLM,

[0049] Figure 7 Visualize the PSNR values ​​of various denoising methods under different noise levels. "□": Improved BM3D; "*": NCSR; "○": BM3D; "◇": NLM. Detailed Implementation

[0050] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0051] Example: Figure 1 and Figure 2 As shown, an improved BM3D image denoising method based on image depth feature maps is implemented in the following steps:

[0052] Step 1: Input the noisy image into a trained UNet denoising network to obtain feature maps of the noisy image at different scales;

[0053] The specific steps for obtaining the feature map in Step 1 are as follows:

[0054] The UNet network comprises four processes: Process 1 (head) has 9 layers with 48 channels per layer; Process 2 (down) has 21 layers with 48 channels per layer; Process 3 (up) has 46 layers with 96 channels per layer; and Process 4 (last) has 10 layers with 3 channels per layer. As can be seen, the UNet network has a large number of layers, and the number of channels varies significantly between different process layers. The head and down processes of the UNet network resemble an encoder structure, compressing the image. The up and last processes resemble a decoder structure, restoring the image to a denoised version, making the image features more apparent. The last layer of the last process yields the denoised image of the noisy image. Therefore, to fully consider the deep features of the noisy image, improve block matching accuracy, and thus enhance the denoising effect of the BM3D algorithm, this invention selects the feature map of the last layer of the last process for improved block matching.

[0055] Step 2: Perform improved block matching on the feature map to obtain block matching groups that take into account the depth features of the noisy image, and integrate them into a three-dimensional similar block matrix;

[0056] The specific steps for improving block matching in Step 2 are as follows:

[0057] Construct a two-dimensional sliding window of size N1×N1, and extract image blocks on the noisy image I according to a set step size p. Let the block currently being operated on be Z. R Then, using reference block Z... R Searching for reference block Z within an N2×N2 search region centered on the target. R Similar blocks, denoted as Z Q Define distance d(Z) R Z Q Z is measured by (inversely proportional to similarity). R and Z Q The similarity between two blocks is expressed as shown in equation (1).

[0058]

[0059] In the formula, τ 2D It is a 2D unitary transform operator (e.g., DCT, DFT, etc.). γ is the hard thresholding operator, λ thr2D It is a fixed threshold parameter, σ represents the noise standard deviation, ||·||2 represents the L2 norm, and γ is defined as follows.

[0060]

[0061] Finding reference block Z R The idea behind similar blocks is to select a suitable threshold τ1, if d(Z) R ZQ If ) < τ1, then Z is considered to be Q With Z R They are similar, so add them to set B. R In the middle. Finally, refer to block Z. R and its similar block set B R Integrate into a three-dimensional matrix T R .

[0062] Next, following the method for finding similar blocks described above, a similar process will be performed on the feature maps. Assume three feature maps are selected, namely feature... Figure 1 ,feature Figure 2 and characteristics Figure 3 Search for similar blocks to the reference block R on each of the three feature maps. Among them, the feature maps... Figure 1 The similar blocks to the reference block R are block Q1 and block Q2, with the following features. Figure 2 The similar blocks to the reference block R are block Q1 and block Q3, with the following features. Figure 3 The similar blocks to the reference block R are blocks Q4 and Q5. Then, based on the positions of the similar blocks matched on each feature map, all non-repeating similar blocks are selected at the same positions on the noisy image, and these are used as the set of similar blocks for the reference block R to form a three-dimensional matrix T. R .

[0063] Step 3: Perform a 3D transformation on the similarity block matrix, reduce image noise using a hard thresholding filter, and obtain estimates of the 2D similarity blocks using a 3D inverse transformation method. A weighted average of multiple estimates for each similarity block is then used to obtain the base estimate of the image.

[0064] The specific steps of Step 3 are as follows:

[0065] Step 3.1: For the three-dimensional matrix T R conduct (Two-dimensional Bior wavelet transform and one-dimensional Haar wavelet transform) Three-dimensional unitary transform. Three-dimensional transform domain denoising can separate noise from real information in an image without energy loss. Most of the real information in an image is concentrated at the vertices of the three-dimensional matrix energy, while noise is often concentrated at the bottom of the three-dimensional matrix energy. Therefore, through hard thresholding filtering in the transform domain, most of the image information can be preserved while effectively removing noise. B after hard threshold filtering R The set of estimated values ​​for the middle block, λ thr3D It is a fixed threshold parameter.

[0066] Step 3.2: After hard thresholding shrinkage, each block will obtain a basic estimated weight, N. R express The number of non-zero coefficients in the matrix is ​​determined by referring to block Z.R The basic estimated weights are as follows:

[0067]

[0068] For a given pixel i to appear in multiple blocks, a weighted average of the estimates from these overlapping blocks is needed to obtain the base estimate of i, as shown in the following formula:

[0069]

[0070] In the formula, Z Q Representing set B R For any image patch containing pixel i, there may also be multiple sets B. R Since each set contains an image patch with pixel i, a weighted average of the patch estimates for all sets containing that pixel is needed. For any set B R Block Z containing cell i Q The estimated value, x Q It is a 0-1 variable. and x Q The expression is as follows:

[0071]

[0072]

[0073] Step 4: Use the UNet denoising network to obtain feature maps of different scales of the basic estimated image;

[0074] Step 5: Perform block matching on the feature map of the base estimated image to obtain the block matching groups of the base estimated image, and integrate them into a three-dimensional similar block matrix;

[0075] The specific steps in Step 4 and Step 5 include:

[0076] The basic estimation image Y obtained in the basic estimation stage basic The input is fed into the UNet denoising network to obtain the feature map of the last layer in the last process. The subsequent process is similar to Step 2, forming a new three-dimensional matrix T. R2 At this point, there are two three-dimensional matrices: one is the three-dimensional matrix T obtained in the basic estimation stage, which consists of similar blocks in the noisy image. R The other is a three-dimensional matrix T composed of similar patches in the image generated by the basic estimation. R2 .

[0077] Step 6: Perform 3D transformation processing on the similar block matrices of the noisy image and the basic estimated image respectively. Treat the similar block matrix in the basic estimate as the energy spectrum of the real signal. Use this energy spectrum to perform collaborative Wiener filtering processing on the noisy image. Perform inverse transformation on the processed data and return to the original position of the similar block to obtain the final estimated value. Perform weighted averaging processing on pixels with multiple estimated values ​​to obtain the final image estimate.

[0078] The specific steps of Step 6 are as follows:

[0079] Step 6.1: Unlike the hard threshold filtering used in the basic estimation stage, the final estimation uses Wiener filtering. The formula for calculating the attenuation coefficient of the Wiener filter is as follows:

[0080]

[0081] Then, through the 3D transform coefficients of the noisy image The product of the Wiener contraction coefficient is used to achieve the T R Wiener filtering is then applied. Finally, the estimated values ​​of the block are generated through inverse transform. The Wiener filtering expression is as follows:

[0082]

[0083] The weight of each reference block in the final estimation stage can be defined as:

[0084]

[0085] Step 6.2: Similarly, a weighted average of these block estimates is needed to obtain the final estimate of pixel i:

[0086]

[0087] In the formula, For any set B R Block Z containing cell i Q The estimated value is expressed similarly to formula (5).

[0088] The improvement objective of this invention is to enhance the denoising effect of the traditional BM3D algorithm by considering block matching on the feature map of the noisy image to improve the accuracy of block matching.

[0089] This invention uses classic test images for image denoising, such as Cameraman, boat, and Baboon. Each image is a grayscale image, 256×256 pixels in size, and includes both noise-free images and images with various noise levels. This invention employs pre-trained UNet network models on datasets with different noise levels and utilizes these UNet models to extract feature maps from images with different noise levels for experimental purposes. Furthermore, this invention uses Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) as evaluation metrics for image denoising performance. A higher PSNR indicates better denoising performance, and a higher SSIM indicates higher visual quality of the denoised image. The parameter settings for the traditional BM3D and improved BM3D algorithms are the same, as shown in Table 1.

[0090] Table 1. BM3D Algorithm Parameter Settings

[0091]

[0092] This invention compares two classic nonlocal means denoising algorithms, BM3D and NLM. In addition, it also selects typical denoising algorithms in deep learning, namely UNet denoising network and NCSR denoising algorithm based on sparse coding, for comparison. The experimental results are shown in Table 3.

[0093] Table 2. Comparison of denoising performance of different denoising algorithms based on PSNR and SSIM (σ=10)

[0094]

[0095]

[0096] Note: In the table, the numerical results A / B represent PSNR and SSIM. The larger the values ​​of A and B, the better the denoising performance of the algorithm. The last row "mean" represents the mean of each column, and the bolded values ​​represent the maximum values ​​in that row.

[0097] The results in Table 2 show that the PSNR and SSIM values ​​of the method proposed in this invention are better than the other four methods. Specifically, compared with UNet, the PSNR of the method proposed in this invention is improved by an average of 3.24 dB, and compared with NLM, the SSIM value is improved by an average of 0.2181. Compared with BM3D and NCSR, the PSNR value of the method proposed in this invention is improved by an average of 0.26-0.39 dB, and the SSIM value is improved by an average of 0.0076-0.011.

[0098] Figure 3 The demonstration shows the denoising effects of four denoising algorithms on Baboon. Figure 4The demonstration showcases the local denoising effects of four denoising algorithms on Baboon. It can be seen that after denoising with BM3D, NCSR, and NLM, areas with complex textures such as Baboon's nose, facial hair, and beard become smoother, exhibiting a certain degree of distortion. In contrast, the method described in this invention retains better texture in the denoised image, demonstrating higher visual quality. Similarly, from... Figure 5 and Figure 6 It can also be seen that, compared with other methods, the method in this invention provides clearer details such as the hand contour in the image after denoising. The above experiments demonstrate that, because the proposed method performs block matching on the feature map, it fully considers the deep features at different scales of the image, thus improving the accuracy of block matching. Furthermore, the structural similarity index SSIM used in the proposed method improves the similarity measurement criterion for block matching, further enhancing the visual quality of the denoised image.

[0099] Figure 7 The presentation visualizes the PSNR values ​​of the proposed method, BM3D, NLM, and NCSR methods for denoising under four noise levels: σ=10, σ=20, σ=30, and σ=40.

[0100] from Figure 7 As can be seen, the denoising performance of various denoising methods decreases with increasing noise levels. This is because the more noise there is, the less detailed information the image contains, making it impossible for the algorithm to utilize more of the image's features, thus leading to a decline in denoising performance. However, overall, the method proposed in this invention exhibits the best performance across various noise levels and is consistently superior to other methods.

[0101] References

[0102] 1.B.Fu, L.Wang and Z.Luo, "Robust Image Denoising with Texture-AwareNeuralNetwork," 2021 IEEE International Conference on Multimedia and Expo (ICME), Shenzhen, China, 2021, pp.1-6.

[0103] 2.Q.Tang et al.,"A Two-stage Raman Imaging Denoising Algorithm Basedon DeepLearning,"2022Asia Communications and Photonics Conference(ACP),Shenzhen,China,2022,pp.2096-2099.

[0104] 3.Qinping Feng,Shuping Tao,Chao Xu,Guang Jin.BM3D-GT&AD:animprovedBM3D denoising algorithm based on Gaussian threshold and angulardistance[J].

[0105] IET Image Processing,2020,14(3).

[0106] 4.Mbarki Zouhair,Ben Jabeur Seddik Chiraz,Seddik Hassene.Building amodifiedblock matching kernel based on Wave Atom transform for efficientimagedenoising[J].The Egyptian Journal of Remote Sensing and SpaceSciences,2021,24(3P2).

[0107] 5.Yingkun Hou,Dinggang Shen.Image denoising with morphology-and size-adaptive block-matching transform domain filtering[J].EURASIP Journal onImage and Video Processing,2018,2018(1).

[0108] 6.C.Liu,W.Zhou,Y.Chen and J.Lei,"Asymmetric Deeply Fused NetworkforDetecting Salient Objects in RGB-D Images,"in IEEE Signal ProcessingLetters,

[0109] vol.27,pp.1620-1624,2020.

[0110] 7.E.K.Kumar,P.V.V.Kishore,M.T.K.Kumar,D.A.Kumar and A.S.C.S.Sastry,

[0111] "Three-Dimensional Sign Language Recognition With Angular VelocityMaps andConnived Feature ResNet,"in IEEE Signal Processing Letters,vol.25,no.12,pp.

[0112] 1860-1864,Dec.2018.

[0113] 8.Sarfaraz Masood,Abhinav Rai,Aakash Aggarwal,M.N.Doja,Musheer Ahmad.

[0114] Detecting distraction of drivers using Convolutional Neural Network[J].PatternRecognition Letters,2017(prepublish).

[0115] 9.C.Liang,E.Chen,L.Qi and L.Guan,"Improving Action RecognitionUsingCollaborative Representation of Local Depth Map Feature,"in IEEESignalProcessing Letters,vol.23,no.9,pp.1241-1245,Sept.2016.

[0116] 10.Sun Yu,Bi Fukun,Gao Yangte,Chen Liang,Feng Suting.A Multi-Attention UNetfor Semantic Segmentation in Remote Sensing Images[J].Symmetry,2022,14(5).11.He Xingkun,Wang Can,Zheng Rongyao,Sun Zhibin,Li Xiwen.GPRimagedenoising with NSST-UNET and an improved BM3D[J].DigitalSignalProcessing,2022,123.

[0117] 12.Wenxin Dai,Yuqing Mao,Rongao Yuan,Yijing Liu,Xuemei Pu,Chuan Li.ANovelDetector Based on Convolution Neural Networks for Multiscale SARShipDetection in Complex Background[J].Sensors,2020,20(9)。

Claims

1. An improved BM3D image denoising method based on image depth feature maps, characterized in that, The method specifically includes the following steps: Step 1: Input the noisy image into a trained UNet denoising network to obtain feature maps of the noisy image at different scales; Step 2: Perform improved block matching on the feature map to obtain block matching groups that take into account the depth features of the noisy image, and integrate them into a three-dimensional similar block matrix; Step 3: Perform a 3D transformation on the similarity block matrix, reduce image noise using a hard thresholding filter, and obtain estimates of the 2D similarity blocks using a 3D inverse transformation method. A weighted average of multiple estimates for each similarity block is then used to obtain the base estimate of the image. Step 4: Use the UNet denoising network to obtain feature maps of different scales of the basic estimated image; Step 5: Perform block matching on the feature map of the base estimated image to obtain the block matching groups of the base estimated image, and integrate them into a three-dimensional similar block matrix; Step 6: Perform 3D transformation processing on the similar block matrices of the noisy image and the basic estimated image respectively. Treat the similar block matrix in the basic estimate as the energy spectrum of the real signal. Use this energy spectrum to perform collaborative Wiener filtering processing on the noisy image. Perform inverse transformation on the processed data and return to the original position of the similar block to obtain the final estimated value. Perform weighted averaging processing on pixels with multiple estimated values ​​to obtain the final image estimate.

2. The improved BM3D image denoising method based on image depth feature maps according to claim 1, characterized in that: In Step 1, the approach to obtaining the feature map of the noisy image is as follows: The UNet network comprises four processes: Process 1 (head) has 9 layers with 48 channels per layer; Process 2 (down) has 21 layers with 48 channels per layer; Process 3 (up) has 46 layers with 96 channels per layer; and Process 4 (last) has 10 layers with 3 channels per layer. As can be seen, the UNet network has a large number of layers, and the number of channels varies significantly between different process layers. The head and down processes of the UNet network resemble an encoder structure, compressing the image. The up and last processes resemble a decoder structure, restoring the image to a denoised version, making the image features more apparent. The last layer of the last process yields the denoised image of the noisy image. Therefore, to fully consider the deep features of the noisy image, improve block matching accuracy, and thus enhance the denoising effect of the BM3D algorithm, this invention selects the feature map of the last layer of the last process for improved block matching.

3. The improved BM3D image denoising method based on image depth feature maps according to claim 1, characterized in that: The specific steps for improving block matching in Step 2 are as follows: Construct a two-dimensional sliding window of size N1×N1, and extract image blocks on the noisy image I according to a set step size p. Let the block currently being operated on be Z. R Then, using reference block Z... R Searching for reference block Z within an N2×N2 search region centered on the target. R Similar blocks, denoted as Z Q Define distance d(Z) R Z Q Z is measured by (inversely proportional to similarity). R and Z Q The similarity between two blocks is expressed as shown in equation (1). In the formula, τ 2D It is a 2D unitary transform operator (e.g., DCT, DFT, etc.). γ is the hard thresholding operator, λ thr2D It is a fixed threshold parameter, σ represents the noise standard deviation, ||·||2 represents the L2 norm, and γ is defined as follows. Finding reference block Z R The idea behind similar blocks is to select a suitable threshold τ1, if d(Z) R Z Q If ) < τ1, then Z is considered to be Q With Z R They are similar, so add them to set B. R In the middle. Finally, refer to block Z. R and its similar block set B R Integrate into a three-dimensional matrix T R . Next, following the method for finding similar blocks described above, a similar process is performed on the feature maps. Assume three feature maps are selected: Feature Map 1, Feature Map 2, and Feature Map 3. Search for similar blocks of the reference block R on each of the three feature maps. Specifically, the similar blocks of the reference block R in Feature Map 1 are blocks Q1 and Q2; in Feature Map 2, they are blocks Q1 and Q3; and in Feature Map 3, they are blocks Q4 and Q5. Then, based on the positions of the matched similar blocks on each feature map, select all non-repeating similar blocks at the same positions on the noisy image, and use these as the set of similar blocks for the reference block R, forming a three-dimensional matrix T. R .

4. The improved BM3D image denoising method based on image depth feature maps according to claim 1, characterized in that: The specific steps of Step 3 are as follows: Step 3.1: For the three-dimensional matrix T R conduct (Two-dimensional Bior wavelet transform and one-dimensional Haar wavelet transform) Three-dimensional unitary transform. Three-dimensional transform domain denoising can separate noise from real information in an image without energy loss. Most of the real information in an image is concentrated at the vertices of the three-dimensional matrix energy, while noise is often concentrated at the bottom of the three-dimensional matrix energy. Therefore, through hard thresholding filtering in the transform domain, most of the image information can be preserved while effectively removing noise. B after hard threshold filtering R The set of estimated values ​​for the middle block, λ thr3D It is a fixed threshold parameter. Step 3.2: After hard thresholding shrinkage, each block will obtain a basic estimated weight, N. R express The number of non-zero coefficients in the matrix is ​​determined by referring to block Z. R The basic estimated weights are as follows: For a given pixel i to appear in multiple blocks, a weighted average of the estimates from these overlapping blocks is needed to obtain the base estimate of i, as shown in the following formula: In the formula, Z Q Representing set B R For any image patch containing pixel i, there may also be multiple sets B. R Since each set contains an image patch with cell i, a weighted average is needed to calculate the estimated values ​​of all patches containing that cell. For any set B R Block Z containing cell i Q The estimated value, x Q It is a 0-1 variable. and x Q The expression is as follows:

5. The improved BM3D image denoising method based on image depth feature maps according to claim 1, characterized in that: Steps 4 and 5 include: The basic estimation image Y obtained in the basic estimation stage basic The input is fed into the UNet denoising network to obtain the feature map of the last layer in the last process. The subsequent process is similar to Step 2, forming a new three-dimensional matrix T. R2 At this point, there are two three-dimensional matrices: one is the three-dimensional matrix T obtained in the basic estimation stage, which consists of similar blocks in the noisy image. R The other is a three-dimensional matrix T composed of similar patches in the image generated by the basic estimation. R2 .

6. The improved BM3D image denoising method based on image depth feature maps according to claim 1, characterized in that: The specific steps of Step 6 are as follows: Step 6.1: Unlike the hard threshold filtering used in the basic estimation stage, the final estimation uses Wiener filtering. The formula for calculating the attenuation coefficient of the Wiener filter is as follows: Then, through the 3D transform coefficients of the noisy image The product of the Wiener contraction coefficient is used to achieve the T R Wiener filtering is then applied. Finally, the estimated values ​​of the block are generated through inverse transform. The Wiener filtering expression is as follows: The weight of each reference block in the final estimation stage can be defined as: Step 6.2: Similarly, a weighted average of these block estimates is needed to obtain the final estimate of pixel i: In the formula, For any set B R Block Z containing cell i Q The estimated value is expressed similarly to formula (5).

Images