Image enhancement hardware accelerator of CLAHE algorithm

Abstract

The invention discloses an image enhancement hardware accelerator of a CLAHE (contrast-limited adaptive histogram equalization) algorithm, and aims to provide an efficient and resource-optimized solution to realize the enhancement of image quality. The system is composed of a histogram and gray value mapping generator, an interpolator, a gray value mapping DMA, a gray value mapping memory, a gating circuit of the gray value mapping memory, a gray value mapping MEM group virtual FIFO, an RGB-to-YUV module and the like. Wherein the histogram and gray value mapping generator is responsible for generating gray value mapping of each image block, and the interpolator is used for performing bilinear interpolation to smooth a transition effect between adjacent image blocks. Under the resolution of 1920 * 1080, the highest operation frequency is 135.8 MHz, and the processing speed of about 31.8 frames per second (fps) can be achieved. Meanwhile, compared with the prior art, only 2701 LUTs, 1998 FFs and 9 18K BLOCK RAMs are used while the similar frame rate is kept.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image enhancement hardware accelerator of a CLAHE algorithm. Background Art

[0002] In the field of digital image processing, image enhancement is a key technology that aims to improve the quality of images by adjusting image properties such as color, contrast, and brightness to make them more suitable for human observation or computer vision analysis. Traditional image enhancement methods can be divided into two categories: spatial domain methods and frequency domain methods. Spatial domain methods operate directly on image pixels, while frequency domain methods operate on the spectrum of the image after Fourier transform.

[0003] The CALHE (Contrast Adaptive Logarithmic Histogram Equalization) algorithm is an advanced spatial domain image enhancement technology. It is an improved method based on histogram equalization (HE). Traditional histogram equalization can effectively improve the overall contrast of the image, but it may cause the loss of local details and the problem of over-brightness or over-darkness in local areas. In comparison, CLAHE can better preserve the details of the image and reduce the problem of over-enhancement.

[0004] CLAHE is widely used in medical imaging, satellite remote sensing, biological sciences and other fields, and is particularly suitable for processing images with a large dynamic range and that need to retain detailed information. For example, in medical images, CLAHE can help doctors see tissue structures more clearly and improve diagnostic accuracy.

[0005] The core idea of ​​the CLAHE algorithm is to divide the garden image into several image blocks, and then perform histogram equalization on the pixel values ​​in each image block. Compared with global histogram equalization, the CLAHE algorithm handles the details of the image better. The specific steps are as follows:

[0006] Statistical local histogram: For each image block, the grayscale histogram glv_hist[i][j] of its pixel values ​​is counted (i is the image block index, j is the grayscale value index). The histogram shows the frequency of occurrence of each grayscale value.

[0007] Limit contrast clipping histogram: In order to avoid excessive amplification of noise, a clipping limit value clip_limit[i] is applied to each local histogram. If the number of pixels of a certain gray value exceeds this limit value, the gray value is divided into the excess group, and the excess value excess_value[i][j] is defined as glv_hist[i][j]-clip_limit[i];

[0008] Calculate the excess statistics in the histogram and excess_sum[i]=∑excess_value[i][j];

[0009] Calculate the average value of the exceedance statistics (N(glv) is the total number of grayscale values);

[0010] Calculate the histogram clipping subthreshold excess_upper[i]=clip_limit[i]-excess_avg[i]. For grayscale values ​​outside the excess limit group, grayscale values ​​above the clipping subthreshold are divided into the high count group, and grayscale values ​​below the clipping subthreshold are divided into the low count group;

[0011] For the grayscale value of the exceeding limit group, the new grayscale value is glv_hist[i][j]=clip_limit[i];

[0012] For the grayscale value of the high count group, the new grayscale value is glv_hist[i][j]=clip_limit[i];

[0013] For the grayscale value of the low count group, the new grayscale value is glv_hist[i][j]=glv_hist[i][j]+excess_avg[i].

[0014] Generate grayscale value mapping: Calculate the cumulative distribution function (CDF) based on the pruned histogram, that is, the proportion of all pixels less than or equal to a certain grayscale value to the total number of pixels. Based on the cumulative distribution function, create a new grayscale mapping table. For each original grayscale value, the new grayscale value is usually calculated according to the following formula: k =(L-1)CDF(r k )(s k After rounding, it represents the new gray value, r k Represents the original grayscale value, L-1 is the maximum grayscale value (L = 2 n , n is the bit width of the pixel gray value), CDF(r k ) is the cumulative distribution function value corresponding to the original grayscale value).

[0015] Bilinear interpolation: To avoid abrupt changes in contrast between different image blocks, CLAHE uses bilinear interpolation to smooth the pixel values ​​of adjacent image blocks. Figure 1 As shown, assume that the center points of the four aforementioned image blocks are a, b, c and d respectively, and the connecting line of each center point and its extension line divide the image into three sub-images of red, blue and green.

[0016] For each sub-image: define the initial grayscale value glv=img(x, y) in the sub-image (x is the column coordinate in the sub-image, y is the row coordinate in the sub-image, and the coordinate value range is 0, 1...), and interpolate to generate a new grayscale value glv′. The specific steps are as follows:

[0017] If the pixel is located in the corner area of ​​the image ( Figure 1 For example, point E does not have an adjacent mapping block, so glv′(E)=mapping_a (mapping_a is the mapping value of the grayscale value of point E in mapping block a);

[0018] If the pixel is located in the boundary area of ​​the image ( Figure 1 The blue area in the middle), such as point F, has adjacent mapping blocks in the horizontal direction, so (W is the sub-image width, H is the sub-image height, mapping_a is the mapping value of the grayscale value of point F in mapping block a, and mapping_b is the mapping value of the grayscale value of point F in mapping block b);

[0019] If the pixel is in the internal area of ​​the image ( Figure 1 The green area in the middle), such as point G, has adjacent mapping blocks in both the horizontal and vertical directions. (mapping_a is the mapping value of the grayscale value of point G in mapping block a, mapping_b is the mapping value of the grayscale value of point G in mapping block b, mapping_c is the mapping value of the grayscale value of point G in mapping block c, and mapping_d is the mapping value of the grayscale value of point G in mapping block d). Summary of the invention

[0020] The present invention realizes an image enhancement hardware accelerator of a CLAHE algorithm.

[0021] Figure 2 The architecture diagram of the image enhancement hardware accelerator of the present invention is given.

[0022] like Figure 2 As shown, the hardware accelerator of the present invention is mainly composed of a histogram and grayscale value mapping generator, an interpolator, a grayscale value mapping DMA, a grayscale value mapping memory and its gating circuit, a grayscale value mapping MEM group virtual FIFO, an RGB to YUV conversion module, etc. Among them, the histogram and grayscale value mapping generator and the interpolator are the two most core functional modules.

[0023] The histogram and gray value map generator is used to generate the gray value map of each image block row, mainly completing three sub-steps: generating histogram, pruning histogram and generating gray value map.

[0024] Interpolators are used to generate bilinear interpolation, handle boundary effects, and reconstruct images.

[0025] Gray value mapping DMA is used to move gray value mapping block row data into / out of external memory.

[0026] The gray value mapping memory and its gating circuit are used to cache gray value mapping block row data. The gating circuit gates it to the histogram and gray value mapping generator, interpolator or gray value mapping DMA.

[0027] The gray value mapping MEM group virtual FIFO implements FIFO control logic in units of gray value mapping block rows, which is used to manage and indicate the empty and full conditions of gray value mapping block rows.

[0028] RGB to YUV and YUV to RGB realize the conversion of RGB data stream and YUV data stream. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is the sub-graph classification diagram in the bilinear interpolation stage of the CLAHE algorithm;

[0030] Figure 2 It is a structural diagram of the CLAHE image enhancement hardware accelerator of the present invention;

[0031] Figure 3 It is the data flow path diagram of the first stage of the image enhancement hardware accelerator processing flow;

[0032] Figure 4 It is the data flow path diagram of the second stage of the image enhancement hardware accelerator processing flow. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not constitute a limitation of the present invention.

[0034] The image enhancement hardware accelerator processes each frame of image in two stages, namely the histogram and grayscale value mapping generation stage and the bilinear interpolation generation stage of new grayscale values.

[0035] Meanwhile, the present invention requires that the RGB data stream of each frame of image be input twice, one time to be passed to the histogram and gray value mapping generator, and one time to be passed to the interpolator.

[0036] Figure 3 The data flow path of the first stage is given.

[0037] In the first stage, the RGB data stream enters the RGB to YUV module and is converted into a YUV data stream;

[0038] At this time, the Y component (grayscale) is input into the histogram and grayscale value map generator after passing through the gate, and the grayscale value map memory is gated for use by the histogram and grayscale value map generator;

[0039] The histogram and grayscale value map generator generates a grayscale value map for each image block row, which is divided into three sub-steps: generating histograms, pruning histograms, and generating grayscale value maps:

[0040] The step of generating histograms statistically generates the grayscale histogram of each image block, and the generated histogram is cached by the grayscale value mapping memory;

[0041] In the step of pruning the histogram, the over-limit value, the over-limit statistical sum, the over-limit statistical average, and the pruning sub-threshold are calculated respectively, and the histogram is further pruned. The pruned histogram is cached by the gray value mapping memory;

[0042] The step of generating gray value mapping calculates the cumulative distribution function and finally generates gray value mapping block row data, and the gray value mapping block row data is cached by the gray value mapping memory;

[0043] The grayscale value mapping block row data cached in the grayscale value mapping memory is saved from the grayscale value mapping memory to the external memory through the grayscale value mapping DMA. The above steps are repeated until the grayscale value mapping of all image blocks of the entire frame image is obtained.

[0044] Figure 4 The data flow path of the second stage is given.

[0045] In the second stage, the gray value mapping block row data stored in the external memory is restored to the gray value mapping memory through the gray value mapping DMA for subsequent use by the interpolator;

[0046] At this time, the Y component (grayscale) of the YUV data stream is input into the interpolator after passing through the gate, and the grayscale value mapping memory is gated for use by the interpolator;

[0047] The interpolator uses the grayscale value mapping block row data, the grayscale value of the pixel and its coordinates to calculate the mapping value and interpolation weight of the pixel in the corresponding mapping block, and finally calculates the new grayscale value;

[0048] Finally, the YUV data stream processed by the interpolator is converted into an RGB data stream and output to the accelerator.

[0049] The present invention optimizes resources by module reuse and table lookup, which is described below:

[0050] As described in

[0037] to

[0048] , the gray value mapping memory is resource reused in the histogram and gray value mapping generation stage and the bilinear interpolation generation stage of new gray values, thereby improving the RAM usage efficiency through reuse.

[0051] When generating new grayscale values ​​through bilinear interpolation, the interpolation weight is calculated by table lookup. The point G in

[0019] is used as an example for explanation (the processing methods for other points are similar):

[0052] For example, calculate the interpolation weight of point G When (other weight processing methods are similar), since the pixel flow order is from left to right, for each row of pixels, the row coordinate y in the sub-image in the weight is fixed, and the column coordinate x in the sub-image is increasing, so the initial weights are pre-stored respectively. and weight difference The interpolation weight of each pixel in the row can be calculated only by table lookup and accumulation, thereby avoiding the use of multipliers and dividers and saving hardware resources.

[0053] The performance of the present invention is described below:

[0054] When the image resolution is 1920×1080 and the clock frequency is 100MHz, the running time of simulating 2 pictures is measured to be 85.316ms, so the frame rate can reach Meet real-time requirements.

[0055] For the xc7z020clg400-2 FPGA, logic synthesis is performed, the clock period is set to 10ns (i.e., the clock frequency is 100MHz), and the clock uncertainty is set to 10% of the clock period. Ignoring the delay caused by the output buffer, the setup time margin is 2.641ns, so this design meets the target clock frequency of 100MHz.

[0056] According to comprehensive statistics, the present invention uses 2701 lookup tables (LUTs), 1998 registers (FFs), and 9 18K block memories (BLOCK RAMs), consuming less hardware resources.

[0057] To further illustrate the advantages of the present invention, the present invention is now compared with three different hardware implementation methods, namely, “Real-Time Processing of Contrast Limited Adaptive Histogram Equalization on FPGA” published by K. Kokufuta et al. at the FPL 2010 conference, “Resource efficient real-time processing of Contrast Limited Adaptive Histogram Equalization” published by B. Unal et al. at the FPL conference, and “High Throughput and Low Latency Hardware of Contrast Limited Adaptive Histogram Equalization Algorithm” published by C. Zhang et al. in the ICSICT journal.

[0058] Table 1 shows the resource consumption of the present invention and other three hardware implementations Table 1 Resource consumption

[0059] Table 2 shows the performance of the present invention and three other hardware implementations at a resolution of 1920x1080. Table 2 Performance at 1920x1080 resolution

[0060] The hardware resources used in the present invention in FPGA mainly include LUT, FF and BLOCK RAM, among which FF occupies a small area in FPGA, so the main focus is on the usage of the other two hardware resources. From the comparison of the above two tables, it can be seen that the hardware resource consumption of the present invention is less than that of the existing hardware accelerator when the frame rate is not much different, which has obvious advantages.

[0061] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. An image enhancement hardware accelerator based on CLAHE (Contrast Limited Adaptive Histogram Equalization) algorithm, characterized by: The hardware accelerator includes a histogram and gray value mapping generator, an interpolator, a gray value mapping DMA, a gray value mapping memory and its gating circuit, a gray value mapping MEM group virtual FIFO, an RGB to YUV conversion module and the like. Compared with the prior art, the hardware accelerator not only has good real-time performance and enhancement effect, but also uses less hardware resources.

2. The histogram and grayscale value map generator according to claim 1, characterized in that: It can realize three functions: generating histograms, trimming histograms, and generating grayscale value mappings. In the histogram generation stage, the grayscale histogram of each image block is counted; in the histogram trimming stage, the grayscale histogram counted in the previous stage is adjusted according to the trimming limit value; in the grayscale value mapping generation stage, the cumulative distribution function is calculated according to the adjusted histogram to create a new grayscale mapping table.

3. The interpolator according to claim 1, characterized in that: It consists of an interpolation part and a YUV to RGB part; the interpolation part calculates the mapping value and interpolation weight of the pixel in the corresponding block according to the gray value mapping table, pixel gray value and coordinates to calculate the new gray value; the YUV to RGB part converts the YUV data stream into an RGB data stream.

4. The grayscale value mapping memory and the gating circuit according to claim 1, characterized in that: It consists of a grayscale value mapping memory and a gating control circuit; the grayscale value mapping memory is connected to the histogram and grayscale value mapping generator, the interpolator and the grayscale value DMA, and is used to cache intermediate data; the gating control circuit selects the grayscale value mapping memory for use by each module according to the order of operations to achieve multiplexing.

5. The grayscale value mapping DMA according to claim 1, characterized in that: The grayscale value mapping memory and the external memory are connected to move the grayscale value mapping block row data into / out of the external memory, and the data moved out of the external memory is cached by the grayscale value mapping memory.

6. The gray value mapping MEM group virtual FIFO according to claim 1, characterized in that: The FIFO control logic is implemented in units of gray value mapping block rows to manage and indicate the empty and full conditions of the gray value mapping block rows.

7. The YUV to RGB part according to claim 1, characterized in that: The RGB data stream is converted into a YUV data stream, and the Y component grayscale value is input into the histogram and grayscale value mapping generator or interpolator through the selector.

Images