Software and hardware comprehensive optimization BM3D image denoising algorithm and hardware circuit design method

Through the combination of the improved Canny operator edge detection and SFCM algorithm and combined with hardware design optimization, the BM3D algorithm's resource occupation and time-consuming problems have been solved, and faster processing speed and better noise removal effect have been achieved, which has improved the Retention of image details and textures.

CN119067873APending Publication Date: 2024-12-03PEKING UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310667087.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-31
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The BM3D algorithm has problems in resource occupation and time consumption, especially in block matching and three-dimensional collaborative filtering modules, and the image details and textures are not preserved sufficiently.

Method used

The improved Canny operator edge detection and SFCM algorithm are used for pre-processing, combined with hardware design optimization block matching and three-dimensional collaborative filtering modules, and edge detection is performed using morphological filtering and a single threshold determined by the minimum intra-class variance and the maximum inter-class variance. Class division with flat areas, and parallel computing and pipeline optimization are performed at the hardware level.

Benefits of technology

It significantly improves processing speed and noise removal effect, reduces resource consumption, and improves the retention of image details and textures.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119067873A_ABST
    Figure CN119067873A_ABST
Patent Text Reader

Abstract

The invention discloses a software and hardware comprehensive optimization BM3D image denoising algorithm and a hardware design method. An improved Canny operator edge detection method is provided in software, morphological filtering is adopted to make brightness and darkness more obvious, and a single threshold is determined by using a minimum intra-class variance and a maximum between-class variance in two-value filtering to confirm edge information; performing a class division acceleration algorithm on the flat region of the picture by adopting an SFCM algorithm; in hardware design, optimization design is carried out on delay and marginal regions, a parallel computing hardware structure is proposed to accelerate the speed, an assembly line is inserted at three-dimensional transformation and hard threshold filtering, resources are reused, and a low-power-consumption multiplier design based on a booth algorithm and a Wallace tree structure is proposed. And algorithm optimization and hardware design are carried out on the original BM3D algorithm, the output image effect is optimized, the operation time is shortened, and the consumed resources are reduced. The whole process is efficient and high in transportability, and the bottleneck that a traditional BM3D algorithm cannot be applied due to resources and time consumption is broken through.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention takes the BM3D algorithm in the field of image denoising as the entry point, realizes the dual optimization of BM3D at the software and hardware levels, and achieves the best effect and shortest time consumption of the algorithm. First of all, attention is paid to the shortcomings of the BM3D algorithm, namely the time consumption and accuracy of block matching, and the preservation of image detail texture. For the preservation of image texture details, the present invention proposes an improved Canny operator edge detection as the first step of implementing the input image. On the premise of the original Canny operator edge detection, its first step filtering technique is improved, which is changed to morphological filtering here, so that the brightness and darkness are more obvious and the edges are highlighted. At the same time, the fourth step binary value filtering part is changed to a single threshold determined by the minimum intra-class variance and the maximum inter-class variance, so as to more accurately determine the edge information. At the same time, after edge detection, the present invention adopts the SFCM algorithm to classify the flat areas of the image, which is convenient for the subsequent block matching to search the effective area, while reducing the number of search blocks, effectively reducing the time consumption and improving the accuracy. After the algorithm is improved, the whole is hardware-based, and the block matching and collaborative filtering modules that are particularly time-consuming are hardened. The delay is strictly controlled from the hardware perspective. At the same time, due to the huge amount of calculation, the present invention proposes a parallel computing hardware structure to speed up the speed as much as possible. At the same time, the pipeline is reasonably inserted at the final three-dimensional forward and inverse transformation and hard threshold filtering, and resources are reused to achieve the best situation of area and speed. The basic hardware operation units, such as multipliers, adders, dividers, square roots, etc., have been optimized to varying degrees. Background Art

[0002] Images are something we come into contact with in all aspects of our lives. In terms of visual sense, a picture that has not been polluted by noise can make people feel comfortable. In scientific experiments, information extraction and reprocessing of unpolluted pictures are more accurate, so in-depth research is conducted in the field of image denoising. Now, except for AI denoising, the most recognized image denoising algorithm is the BM3D algorithm. On some devices that do not have NPU support or the NPU is not dedicated to image processing, the application prospects of the BM3D algorithm are very bright.

[0003] The basic idea of ​​BM3D comes from the observation that there are many similar repetitive structures in natural images. These similar structures are collected and aggregated by image block matching, and then orthogonally transformed to obtain a sparse representation of them. The sparsity and structural similarity are fully utilized for filtering. BM3D denoising can fully retain the structure and details of the image and obtain a good signal-to-noise ratio.

[0004] The BM3D algorithm can be divided into two steps, namely the basic estimation and the final estimation. The first step is the basic estimation, which is mainly divided into three parts: 1. Block matching: Taking full advantage of the NLM algorithm, a search window of a certain size is set in the image, a reference image block is set, and matching similar blocks are found in the search window according to similarity conditions, and then all matching similar blocks are stacked into a three-dimensional matrix. 2. Collaborative hard threshold filtering: The three-dimensional matrix formed in the previous step is transformed in the transform domain to obtain a sparse representation of the image group, and then the transform coefficients are hard threshold filtered, and finally the filtered data is inversely transformed in the transform domain, that is, returned to the image domain. 3. Aggregation: Because the previous step is to process a three-dimensional matrix, the returned matrix is ​​also a three-dimensional matrix, that is, many similar two-dimensional image blocks are stacked, and then the initial denoised image is obtained by weighted averaging the entire group according to the weight given by the block matching. The second step is the final estimation, which is also mainly divided into three parts: 1. Block matching: According to the denoised image obtained by the initial estimation in the first step, block matching is performed again, stacked into a three-dimensional matrix, and this three-dimensional matrix is ​​used as a reference. According to its position, the blocks are stacked at the same position of the image that was initially input. 2. Collaborative Wiener filtering: This step mainly performs a 3D domain transformation on the two sets of 3D matrices, uses the initial estimated image value as the empirical value of the Wiener filter, and performs Wiener filtering on the transformation coefficients formed after the domain transformation of the initial noise image, and then returns to the image domain through the inverse domain transformation as in the first step. 3. Aggregation: Same as the initial estimation, weighted average is performed according to the weights to return the final denoised image.

[0005] This paper fully studies the advantages and disadvantages of the BM3D algorithm, innovatively proposes the SFCM algorithm and the Canny modified version of edge detection as improvements to the BM3D algorithm block matching pre-algorithm, and proposes the idea of ​​using hardware design to reduce the time consumption of block matching and three-dimensional collaborative filtering. The above improvements are achieved, and finally a good breakthrough is made in speed and effect compared with the original algorithm. Summary of the invention

[0006] The technical objectives of the present invention are:

[0007] In order to solve the resource occupation and time consumption of the BM3D algorithm, the present invention proposes a method for hardware design of the time-consuming part thereof. In order to solve the speed and search volume problems of the BM3D algorithm, the present invention proposes a pre-class separation processing mode of the SFCM algorithm. In order to solve the problem that the image detail texture finally processed by the BM3D algorithm is not preserved much, it is proposed to use a modified version of the Canny edge detection operator to divide the image into the flat domain and the edge domain. Finally, the overall optimization of the BM3D algorithm in software and hardware is achieved.

[0008] The present invention realizes the following contents:

[0009] The overall operation flow of the present invention is as follows: Figure 1First, we use the improved Canny operator to perform edge detection, which mainly consists of five steps: Gaussian filtering, pixel gradient calculation, non-maximum suppression, hysteresis threshold processing, and isolated weak edge suppression. The overall process is as follows Figure 2 As shown. The present invention partially optimizes the first and fourth steps. In the first step, the Gaussian filter is replaced by a morphological filter, including a top-hat transform and a black-hat transform. The top-hat transform is to open the original image, and then subtract the result of the opening operation from the original image. The opening operation will delete the bright spots whose grayscale values ​​are higher than those of the surrounding images. After subtraction, the bright noise under the dark background can be effectively eliminated. The black-hat transform is opposite to the top-hat transform. After the image is closed, the bright spots whose grayscale values ​​are higher than those of the surrounding images can be deleted. After subtracting the original image, the dark area can be highlighted while reducing the dark noise under the bright background. The result after filtering is the difference between the original image plus the top-hat transform and the difference between the black-hat transform, which makes the bright and dark more obvious, thereby highlighting the edge.

[0010] The second step is pixel gradient suppression. Since the first-order finite difference is basically equal to the image gradient value, edge detection is performed through different templates, that is, partial derivatives are calculated in the x and y directions respectively to obtain the amplitude and direction at each pixel point, which are then combined into a gradient map.

[0011] The third step is non-maximum suppression, which aims to eliminate the stray responses caused by edge detection and play a role in "slimming down" the edge. The basic method is to compare the gradient strength of the current pixel with the gradient strength of the adjacent pixels along the positive and negative gradient directions. If it is the maximum (that is, the extreme value), the pixel is retained as an edge point. If it is not the maximum, it is suppressed and not regarded as an edge point. For more accurate calculation, linear interpolation is usually used between two adjacent pixels across the gradient direction to obtain the pixel gradient to be compared.

[0012] The fourth step is to optimize the double thresholds. The present invention only uses a single threshold. According to the minimum intra-class variance criterion and the maximum inter-class variance method, the image is first divided into two categories, edge and non-edge, and iterative testing is performed. When the total variance of the two categories is the smallest, the threshold at that time is defined as the standard for distinguishing edges and non-edges. The intra-class variance formula is as follows:

[0013]

[0014] The left term is the intra-class variance, t is the intra-class threshold, ω 1 (t) represents the number of pixels whose gradient magnitude is less than t, u 1 (t) represents the average gradient amplitude of these pixels, x i That is, the pixel amplitude is less than t. The first sum is the intra-class variance of non-edge points, and the second is the intra-class variance of edge points. Therefore, the t value with the smallest left term is the intra-class threshold.

[0015] According to the maximum inter-class variance method, the formula is as follows:

[0016] g=w 0 (w 0 -u) 2 +w 1 (u 1 -u) 2

[0017] In the formula, g represents the inter-class variance, u represents the grayscale average of the entire image, and u 0 Represents the average gray value of the edge class, w 0 Represents the proportion of the edge class occupying the entire image, u 1 Represents the average gray value of non-edge classes, w 1 It represents the proportion of the non-edge class in the entire image. The threshold between edge and non-edge classes is the most obvious when g is maximized.

[0018] Finally, the final threshold is obtained by averaging the intra-class threshold and the inter-class threshold.

[0019] The fifth step is to suppress isolated weak edges. Generally speaking, weak edge pixels caused by real edges will be connected to strong edge pixels. When strong edge points are connected, sometimes non-closed edges are formed. In order to form a closed contour, the weak edge pixel and its 8 neighboring pixels can be checked according to their connection with the strong edge. Generally, it can be defined that as long as one of the neighboring pixels is a strong edge pixel, the weak edge can be retained as a strong edge, assigned a value of 1 or 255, that is, a real edge point, and connected, and finally the edge is completely closed.

[0020] Then, according to the image separated from the flat domain and edge domain obtained by the Canny improved edge detection algorithm, the flat domain is divided into classes using the SFCM clustering algorithm. In the image, adjacent pixels are highly correlated or even consistent, so there are many ways of spatial filtering. Therefore, it can be known that the pixels around each pixel are likely to have the same membership, so these points can be classified into one class. Therefore, according to this characteristic, the membership of a pixel is defined as the weighted average of the sample membership and the spatial membership. The main implementation process of the SFCM algorithm is as follows: Figure 3 As shown, that is, 1. Input sample set {x1, x2, x3...xn}; 2. Determine the number of classes to be classified, that is, the value of C, determine the sample membership factor p and the spatial membership q and the termination condition of the iteration (here it can be the number of times or the end threshold): 3. Randomly initialize the membership matrix, and judge whether it meets the iteration termination condition according to the second step; 4. Fix the class center, divide the input samples, and assign them to the nearest class center; 5. Fix the entire membership matrix and recalculate the class center position; 6. Repeatedly iterate the class center and the membership matrix until the algorithm meets the termination condition.

[0021] The data obtained in this way is input into the block matching module, where the hardware design of the block matching module and the subsequent three-dimensional collaborative filtering module and aggregation module is carried out, such as Figure 4 As shown in the figure, Display_timing mainly performs parameter configuration and image data input, outputs data to Ref_gen, and sends control signals to the Blk_extract module. The Ref_gen module mainly performs frame judgment on the input data and sends control signals to the subsequent block matching module. The Block_extract module mainly stores the input data and delays the output through control logic to form a Kernel, while considering the boundary conditions. Finally, it outputs the two-dimensional reference image block and the search image block in the search window, and sends the positions of the current two-dimensional reference image block and the search image block in the search window to the aggregation module. The PE module mainly performs The similarity between the reference image block and the search image block is calculated in parallel, and they are sorted and scored at the same time. They are then stored in Fifo and the score is sent to the aggregation module as a reference for the final aggregation weight. The What_top module mainly performs three-dimensional forward and inverse transforms and hard threshold processing on the input three-dimensional image group, including forward What2-dimensional transform, forward What1-dimensional transform, hard threshold processing module, reverse What2-dimensional transform, reverse What2-dimensional transform, and finally outputs the data and the weights counted after the hard threshold module to the aggregation module. The aggregation module mainly performs weighted average normalization on the input data and weights, and then re-arranges them according to the image block position provided by block matching. The overall architecture is basically as described above.

[0022] Except for the basic storage RAM and necessary FIFO, the present invention performs pipeline design in the wht module, such as Figure 5 As shown. That is, registers are added after the two-dimensional forward wht transformation, the one-dimensional forward wht transformation, the hard threshold filtering, the two-dimensional backward wht transformation, and the one-dimensional backward wht transformation. This is mainly because this is the critical path, and the data flow is unidirectional and executed sequentially. The best effect is achieved while ensuring that the processing time of each level of the pipeline is basically the same. The overall transformation process is divided into several steps to increase the final clock maximum frequency. At the same time, considering that the area of ​​the registers added here actually accounts for a very small proportion in the overall design, it is adopted. In the implementation process, because the two-dimensional wht transformation and the one-dimensional wht transformation actually take different time, some counters are added to the two-dimensional wht transformation to extend the delay, so as to achieve the final basic time consistency and realize the correct pipeline division.

[0023] At the same time, the present invention proposes a low-power multiplier based on the booth algorithm and the Wallace tree structure and a 32-bit carry-lookahead adder with a carry-lookahead architecture with a 4-bit carry-lookahead adder as the core, such as Figure 6As shown, in the square root structure, the hardware design is performed according to the successive approximation algorithm, such as Figure 7 As shown, the overall operation module is optimized to different degrees.

[0024] Compared with the existing BM3D algorithm, the present invention has the following advantages:

[0025] 1. Faster processing speed and less resource consumption

[0026] The SFCM clustering algorithm used in the present invention can effectively narrow the search window and reduce the number of searches, effectively saving time. At the same time, the overall hardware design for block matching and three-dimensional collaborative filtering solves the time-consuming and resource-consuming problems of these two parts of the BM3D algorithm in software.

[0027] 2. The image denoising effect is better

[0028] First, we perform a modified Canny edge detection on the image to effectively divide the flat domain and the edge domain, which is similar to the principle of bilateral filtering. Later, we perform block matching on the edge domain according to the edge contour, and use the SFCM algorithm to perform precise class division and block matching on the flat area. The final result is better, and the degree of detail and texture retention is higher than the original algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 BM3D software and hardware optimization structure calculation process

[0030] Figure 2 Canny operator improved version of edge detection calculation process

[0031] Figure 3 SFCM clustering algorithm classification process

[0032] Figure 4 BM3D algorithm hardware design architecture

[0033] Figure 5 3D collaborative filtering module pipeline division

[0034] Figure 6 16-bit carry-lookahead adder with a carry-lookahead architecture based on a 4-bit carry-lookahead adder

[0035] Figure 7 Calculation process of square root operation module DETAILED DESCRIPTION

[0036] The following describes in detail the BM3D software and hardware comprehensive design implementation method provided by the present invention in conjunction with the accompanying drawings, but does not constitute a limitation to the present invention.

[0037] The present invention is a BM3D image denoising algorithm and a hardware circuit design method with comprehensive software and hardware optimization. The entire implementation process is as follows: Figure 1 The specific hardware design architecture is shown in Figure 4 As shown, the overall operation process is as follows:

[0038] Step 1: The input image is processed by the improved version of the Canny operator algorithm. This step mainly includes morphological transformation. First, the image is enhanced to highlight the edges, and then the pixel gradient is suppressed to obtain the partial derivative of each pixel in the two-dimensional direction, that is, the amplitude and direction, to form a gradient map, and then non-maximum suppression is performed to determine the gradient of the pixel and compare it with the gradient intensity of the intersection of the direction and the pixel matrix. If it is the largest, the pixel is retained, otherwise it is removed, and then the threshold is judged. The minimum contrast method within the class and the maximum variance method between classes are used to comprehensively determine the threshold. The point above the threshold is the edge point, and finally the edge contour is connected, and the strong pixel points around the flat domain are related to the edge gap.

[0039] Step 2: Process the edge processing results with the SFCM clustering algorithm. After edge processing, the edge part is sent to the hardware module for block matching according to the edge contour, and the flat area is divided into classes according to the SFCM algorithm. The main thing is to first determine the number of classes, the membership iteration termination condition, and then initialize the membership matrix. Iterate repeatedly to determine the class center and the membership matrix, considering the sample membership and spatial membership values. In this way, the data processed by the SFCM algorithm is sent to the hardware design module for implementation.

[0040] Step 3: The hardware design module receives data for block matching, 3D collaborative filtering, and aggregation. The hardware module receives data from edge detection and data after SFCM clustering, controls the delay according to the VP port protocol, correctly assembles the kernel, and uses parallel machine selection when calculating similar blocks. Then, the similarity is sorted and sent to the cache, and then the cache is read and sent to the 3D collaborative filtering module for 2D forward Walsh-Hadamard transform, 1D forward Walsh-Hadamard transform, hard threshold filtering, 1D forward Walsh-Hadamard inverse transform, and 2D forward Walsh-Hadamard inverse transform. The data is sent to the aggregation module, and the image is reassembled based on the weights of the 3D collaborative filtering and the weighted average of the position provided by the block matching.

[0041] The present invention proposes a BM3D image denoising algorithm and hardware circuit design method with comprehensive optimization of software and hardware based on the BM3D algorithm, including algorithm optimization, time consumption analysis, hardware design and comprehensive simulation, in pursuit of better software and hardware synergy effect. The BM3D image denoising algorithm module with comprehensive optimization of software and hardware designed by the present invention can optimize the image effect of the original BM3D algorithm while minimizing the running time and resource consumption. The entire design process is effective and highly portable, breaking the bottleneck of the traditional BM3D algorithm that cannot be applied due to resources and time consumption.

[0042] The above embodiments are only used to illustrate the method of the present invention rather than to limit it. A person skilled in the art may modify or make equivalent substitutions to the method of the present invention without departing from the spirit and scope of the present invention. The scope of protection of the present invention shall be subject to the claims.

Claims

1. A BM3D image denoising algorithm and hardware circuit design method with comprehensive software and hardware optimization, including cluster analysis and edge detection at the algorithm level, parallel computing of hardware circuits and 3D forward and inverse transformation. Features: In terms of algorithm optimization, we first use the Canny operator to detect the edge of the image and divide the image into a flat area and an edge area. Then, we perform block matching on the image in the edge area according to the edge contour. In the flat area, we use the SFCM algorithm to perform cluster analysis on the image in advance, so as to improve the accuracy and speed of the block matching process as much as possible while retaining the image details and texture. At the hardware circuit level, the block matching module is delayed and outputted to ensure the correctness of the output kernel. At the same time, the block matching results are calculated in parallel. Finally, the similar blocks obtained are subjected to three-dimensional forward and inverse transformation and hard threshold filtering.

2. The SFCM algorithm selected according to claim 1 optimizes the original algorithm, Features: The BM3D algorithm can only achieve better denoising effect when a large number of similar blocks are searched based on the reference blocks. If you want to search for a large number of high-quality similar blocks, you need to expand the search area, reduce the size of the reference blocks and similar blocks, or increase the similar block judgment threshold, but these will affect the PSNR value and algorithm running time. The SFCM clustering algorithm can determine the number of classes to be divided, then continuously iterate the membership matrix, determine the final class center, and finally obtain the classification result of the entire image. It can minimize the search range and shorten the algorithm running time while ensuring the number and quality of the search blocks.

3. The original algorithm for edge detection optimization by selecting the Canny operator as described in claim 1, Features: Because at the edge of the image, a lot of edge information will be lost after clustering through the SFCM algorithm. If you want to retain the image edge and texture and other detailed information as much as possible, you need to perform edge detection on this part of information in advance for further processing; ideally, the edge of the image should be a step function, but in actual scenes, the sensor device will have a certain impact on the image, and the step edge can easily become a slope edge, and the amplitude change changes from the original instantaneous change to a gradual change. This requires edge detection to be filtered first, followed by enhancement and detection. Filtering can generally reduce some noise interference. Enhancement is to display the edge information of the image, and the final detection is to remove pseudo-points of the edge information; there are also many operators for silver-inlaid edge detection. The present invention mainly uses the canny operator. The canny operator has five main steps for edge detection, namely Gaussian filtering, pixel gradient calculation, non-maximum suppression, hysteresis threshold processing and isolated weak edge suppression.

4. Partial optimization is performed by the Canny operator edge detection described in claim 3, Features: The first step is to replace the Gaussian filter with morphological filtering, including top hat transformation and black hat transformation. The final filtered result is the original image plus the difference of top hat transformation minus the difference of black hat transformation, making the light and dark more obvious, thereby highlighting the edge; the fourth step is to optimize the double threshold, using only one threshold, and averaging the threshold according to the minimum intra-class variance criterion and the maximum inter-class variance method.

5. The BM3D hardware circuit according to claim 1, Features: The whole system complies with the VP port protocol and takes the VP port output delay into consideration. In the block matching module, delay control and edge replication are performed simultaneously, including mirroring and replication modes, to ensure the correct output of the kernel. In the PE module, parallel calculation of similar blocks is implemented. Finally, in the three-dimensional module, Walsh-Hadamard forward and inverse transform is performed, and hard threshold filtering is added. Reasonable pipeline division is performed, and finally weighted average is used to reassemble the image output according to the block matching address.

6. The optimization of the basic operation module in the BM3D hardware circuit according to claim 5, Features: The booth algorithm, Wallace tree structure and 4-2 compressor structure are used to optimize the area and speed of the multiplier. At the same time, a 32-bit adder with a carry-lookahead structure composed of 4-bit carry-lookahead adders is used to optimize the unsigned divider and improve it to adapt to the signed number case. At the same time, a square root operation module is provided according to the successive approximation algorithm.