A method for improving the detail preservation performance in global tone mapping processing of HDR images

Abstract

The present invention discloses a method for improving the detail retention performance in the global tone mapping processing of high dynamic range (HDR) images. First, the input HDR image is converted from the RGB representation to the IC T C P representation, so as to extract its lightness channel. Then, the lightness is segmented into several intervals by solving the K-means clustering algorithm. On this basis, a quantizer from linear to equalization is used to generate a mapping lookup table for each lightness interval, so as to adaptively adjust its distribution. Finally, while applying a highlight limit, the mapping of the lightness channel is performed, the chromaticity channel remains unchanged and it is converted back to the RGB representation, and the global tone mapping method is applied to obtain the mapping result. The present invention processes using the lightness channel that is more in line with human eye perception instead of the luminance channel. The lightness segmentation enables the lightness distribution of each interval to be adaptively adjusted, which can improve the detail retention performance of the global tone mapping processing. At the same time, the highlight limit ensures the naturalness of the mapping result.

Description

Technical Field

[0001] The present invention belongs to the field of high dynamic range (HDR) image tone mapping methods, and particularly relates to a method for improving the detail retention performance in the global tone mapping processing of HDR images. This method segments the image based on lightness, thereby adaptively adjusting the histogram distribution of each lightness interval, and then applying the global tone mapping method can improve its detail retention performance. Background Art

[0002] HDR images can bring users a more delicate, real and more appealing visual experience. In order to reproduce the perception of HDR images or real scenes by the human eye on low dynamic range (LDR) devices, while compressing the dynamic range by tone mapping methods, based on the characteristics of the human visual system, as much as possible, the structural details, local contrast and color perception of the original content are retained.

[0003] The current tone mapping processing methods are mainly divided into two categories: global tone mapping and local tone mapping. The global tone mapping method uses a specific monotonic curve to map the entire image. Its operation is relatively fast, but it will lead to a decrease in local contrast and the loss of details. Summary of the Invention

[0004] In order to make the HDR images before and after the global tone mapping method processing have similar visual perception and detail richness, the present invention provides a tone mapping method based on lightness segmentation to improve the detail retention performance in the global tone mapping processing.

[0005] The specific technical solution adopted by the present invention is as follows:

[0006] A method for improving the detail retention performance in the global tone mapping processing of HDR images, which includes the following steps:

[0007] S1: Convert a high dynamic range (HDR) image from RGB representation to IC T C P representation, so as to extract its lightness channel;

[0008] S2: Use the K-means clustering method to segment the lightness channel into several lightness intervals;

[0009] S3: Use the HALEQ method (Histogram Adjustment based Linear to Equalized Quantizer) to calculate the mapping lookup table for each interval, and linearly adjust the target lightness range of the lookup table;

[0010] S4: Apply a high - light limit to the mapping curve corresponding to the adjusted look - up table and perform brightness mapping;

[0011] S5: Keep the chromaticity channel unchanged, convert the image after brightness mapping back to the RGB representation, and process it using a specific global tone - mapping method.

[0012] Preferably, the specific steps of step S1 are as follows:

[0013] S101: For the input HDR image, convert the RGB response values normalized between 0 and 1 into XYZ tristimulus values:

[0014]

[0015] S102: Convert the XYZ tristimulus values to the LMS space of the human eye cone cells sensitive to long - wave, medium - wavelength, and short - wave visible light respectively:

[0016]

[0017] S103: Apply the inverse transformation of the PQ curve to each channel of LMS respectively:

[0018]

[0019] In the formula, L’, M’, and S’ respectively represent the calculation results of each channel of L, M, and S. is the inverse transformation of the PQ electro - optical conversion curve, and its inverse transformation result for any input Y ∈ (L, M, S) is:

[0020]

[0021] In the formula, m1 = 2610 / 16384, m2 = (2523 / 4096)×128, C1 = 3424 / 4096, C2 = (2413 / 4096)×32, C3 = (2392 / 4096)×32;

[0022] S104: Calculate IC from L’M’S’ T C P Color representation:

[0023]

[0024] In the formula, I is the brightness channel, C T and C P are the chromaticity channels.

[0025] Preferably, the specific steps of step S2 are as follows:

[0026] S201: Statistically analyze the histogram of the brightness channel extracted in S1, and use the K-means clustering algorithm to update the centroid position of the clustering until it stabilizes. The centroid position is the brightness value. Among them, the initial positions of the centroids of the clustering are evenly distributed within the brightness range, and the interval value is τ.

[0027] S202: After the centroids of the clustering are stable, merge adjacent clusters with a distance less than τ:

[0028]

[0029] In the formula, i and j are the serial numbers of adjacent clusters, c i 、c j and c i,j are the brightness values of the centroids before and after merging, n i and n j are the number of pixels in the cluster.

[0030] S203: Use the dynamic programming solution to obtain the global optimal solution of the K-means clustering problem. Each obtained class corresponds to a brightness interval.

[0031] Preferably, the interval value τ = 0.05.

[0032] Preferably, the specific steps of S3 are as follows:

[0033] S301: Based on the brightness intervals obtained in S2, use the HALEQ method to generate a mapping lookup table for each brightness interval. The first column in the table is the brightness of the segmentation point obtained by the HALEQ method, and the second column is the target brightness evenly distributed between 0 and 1.

[0034] S302: Linearly scale the target brightness of the mapping lookup table for each brightness interval to the original brightness range of that brightness interval, and then cascade them in sequence to obtain the first lookup table applicable to the entire image.

[0035] Preferably, the specific steps of S4 are as follows:

[0036] S401: For the mapping curve corresponding to the first lookup table, add an additional segmentation boundary b in its highlight interval. Recalculate the lookup table of the brightness interval closest to b on the side less than b using the HALEQ method, while keeping the brightness values on the side greater than b unchanged, thereby obtaining the second lookup table. Among them, the segmentation boundary b is:

[0037] b = α(l max -l min )+l min

[0038] In the formula: l max and l minrespectively represent the maximum and minimum brightness values of the original HDR image, and α is the boundary adjustment constant;

[0039] S402: Map the brightness values of the original HDR image to the target brightness in a linear interpolation manner based on the second look-up table.

[0040] Preferably, in step S401, if there are already boundary points greater than b, there is no need to add the above-mentioned segmentation boundary b, and only the brightness values in the maximum brightness interval need to be set to remain unchanged during the mapping process.

[0041] Preferably, the boundary adjustment constant α is 0.8.

[0042] Preferably, step S5 is specifically as follows:

[0043] For the HDR image after brightness mapping in S4, use the mapped brightness channel, while keeping the chromaticity channel of the image unchanged, convert it back to the RGB representation, and apply the global tone mapping method to output the LDR image.

[0044] The present invention has the following beneficial effects compared with the prior art:

[0045] The present invention uses ICC T C P to represent and extract the brightness channel, which has better perceptual consistency, brightness and hue constancy, and adaptively adjusts the distribution of each brightness interval through reasonable brightness segmentation, thereby improving the detail retention performance of the global mapping method and making the mapping result have better local contrast. In addition, the highlight limitation preserves the appearance of the highlight area and improves the naturalness of the result of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a result comparison of whether the global tone mapping method applies this method or not. The global tone mapping methods shown in the figure are Tumblin et al [1] , Drago et al [2] , Reinhard et al [3] , Oskarsson [4] and Khan et al (ATT) [5] proposed from top to bottom, and the left and right are the results of not applying and applying this method respectively. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0047] The technical effects of the present invention will be further elaborated and explained below with reference to the drawings and tables.

[0048] In a preferred embodiment of the present invention, a method for improving the detail retention performance in the global tone mapping process of HDR images is provided, which includes the following steps:

[0049] S1: Convert the high dynamic range (HDR) image from RGB representation to IC T C P representation, and extract its lightness channel.

[0050] In this embodiment, step S1 is specifically as follows:

[0051] S101: For the input HDR image, convert the RGB response values normalized between 0 and 1 into XYZ tristimulus values:

[0052]

[0053] S102: Convert the XYZ tristimulus values to the LMS space of the human eye cone cells sensitive to long - wave, medium - wavelength, and short - wave visible light respectively:

[0054]

[0055] S103: Apply the inverse transformation of the PQ curve to each channel of LMS:

[0056]

[0057] In the formula, L’, M’, and S’ represent the operation results of the L, M, and S channels respectively, is the inverse transformation of the PQ electro - optical conversion curve, and its inverse transformation result for any input Y ∈ (L, M, S) is:

[0058]

[0059] In the formula, m1 = 2610 / 16384, m2 = (2523 / 4096)×128, C1 = 3424 / 4096, C2 = (2413 / 4096)×32, C3 = (2392 / 4096)×32;

[0060] S104: Calculate IC from L’M’S’ T C P color representation:

[0061]

[0062] In the formula, I is the lightness channel, C T and C P are the chromaticity channels.

[0063] S2: Use the K - means clustering method to divide the lightness channel into several lightness intervals.

[0064] In this embodiment, step S2 is specifically as follows:

[0065] S201: Statistically analyze the histogram of the brightness channel extracted in S1, and use the classical K-means clustering algorithm to update the centroid position (i.e., brightness value) of the clustering until it stabilizes. Among them, the initial positions of the centroids of the clustering are evenly distributed within the brightness range, and the interval value is τ. In this embodiment, the interval value τ = 0.05.

[0066] S202: After the centroids of the clustering are stable, merge adjacent clusters with a distance less than τ:

[0067]

[0068] In the formula, i and j are the serial numbers of adjacent clusters, c i 、c j and c i,j are the brightness values of the centroids before and after merging, n i and n j are the number of pixels in the clustering.

[0069] S203: Use the dynamic programming solution to obtain the global optimal solution of the K-means clustering problem, and each obtained class corresponds to a brightness interval.

[0070] In this embodiment, the dynamic programming solution can adopt the dynamic programming solution proposed by Oskarsson [4] , and the specific implementation method can refer to the reference.

[0071] S3: Use the HALEQ method (Histogram Adjustment based Linear to Equalized Quantizer) to calculate the mapping lookup table for each interval, and linearly adjust the target brightness range of the lookup table.

[0072] It should be noted that HALEQ is a linear-to-equalized quantizer in the prior art, and the specific implementation can refer to the corresponding reference [6] .

[0073] In this embodiment, step S3 is specifically as follows:

[0074] S301: Based on the brightness intervals obtained in S2, use the HALEQ method to generate the mapping lookup table for each brightness interval. The first column in the table is the brightness of the segmentation point obtained by the HALEQ method, and the second column is the target brightness evenly distributed between 0 and 1;

[0075] S302: Linearly scale the target brightness of the mapping lookup table for each brightness interval to the original brightness range of the brightness interval, and then cascade them in sequence to form the first lookup table applicable to the entire image.

[0076] S4: Apply a high - light limit to the mapping curve corresponding to the adjusted look - up table and perform brightness mapping.

[0077] In this embodiment, step S4 is specifically as follows:

[0078] S401: For the mapping curve corresponding to the first look - up table, add an additional segmentation boundary b in its high - light interval. Recalculate the look - up table of the brightness interval closest to b on the side less than b using the HALEQ method, while keeping the brightness values on the side greater than b unchanged, thus obtaining a second look - up table; where the segmentation boundary b is:

[0079] b = α(l max -l min ) + l min

[0080] In the formula: l max and l min respectively represent the maximum and minimum brightness values of the original HDR image, and α is the boundary adjustment constant; in this embodiment, the boundary adjustment constant α is 0.8.

[0081] However, it should be noted that in step S401, if there are already boundary points greater than b, there is no need to add the above - mentioned segmentation boundary b, and only the brightness values in the maximum brightness interval need to be set as unchanged during the mapping process.

[0082] S402: Map the brightness values of the original HDR image to the target brightness based on the second look - up table by linear interpolation.

[0083] S5: Keep the chrominance channel unchanged, convert the image after brightness mapping back to the RGB representation, and process it using a specific global tone - mapping method.

[0084] In this embodiment, step S5 is specifically as follows:

[0085] For the HDR image after brightness mapping in S4, use the mapped brightness channel, while keeping the chrominance channel of the image unchanged, convert it back to the RGB representation, and apply the global tone - mapping method to output an LDR image.

[0086] Apply the method for improving the detail - retention performance in the global tone - mapping processing of HDR images shown in S1 - S5 above to specific example data to demonstrate its technical effects.

[0087] Taking 20 HDR images in the LVZ - HDR database [7] as an example, using TMQI [8] and TMQI - FSITM [9] as objective evaluation indicators of method performance, the global tone - mapping method includes Tumblin, etc. [1], Drago et al. [2] , Ferwerda et al.

[10] , Kim et al.

[11] , Reinhard et al. [3] , Schlick

[12] , Ward

[13] , Oskarsson [4] and Khan et al. (the two methods are abbreviated as ATT [5] and TMPQ

[14] ). The average TMQI and TMQI - FSITM scores are shown in Table 1. These two metrics range from 0 to 1, and the larger the value, the better the result. The better scores of each global tone mapping method are marked in bold. As can be seen from the table, the application of this method improves the TMQI and TMQI - FSITM scores of the global tone mapping method, with better structure preservation and naturalness, indicating the improvement of this method in the detail retention performance of the global tone mapping method.

[0088] Table 1 Performance comparison of the global tone mapping method with or without the application of this method

[0089]

[0090] Figure 1 Compared the results of the global tone mapping method with or without the application of this method. The method of Tumblin et al. [1] shows good dynamic range compression performance but the overall appearance is on the dark side. After applying this method, it shows a suitable global contrast. The results of Drago et al. [2] are bright in appearance but lack some details, while the results adjusted by this method have better local contrast. In the mapping results of Reinhard et al. [3] , buildings and roads are difficult to distinguish, which is improved in the results of this method. The method of Oskarsson [4] shows good global lightness but insufficient local contrast. When this method is applied to this method, it shows better details in areas such as leaves. For the ATT algorithm [5] , the results with or without the application of this method show similar global appearances, but the mapped images with the application of this method present more abundant details in outdoor high - light areas.

[0091] References:

[0092] [1] Tumblin J, Rushmeier H. Tone reproduction for realistic images[J]. IEEE Computer Graphics and Applications, 1993, 13(6):42 - 48.

[0093] [2] Drago F, Myszkowski K, Annen T, et al. Adaptive logarithmic mapping for displaying high contrast scenes[J]. Computer Graphics Forum, 2003, 22(3): 419-426.

[0094] [3] Reinhard E, Stark M, Shirley P, et al. Photographic tone reproduction for digital images[C]. Proceedings of the 29th annual conference on Computer graphics and interactive techniques, 2002: 267-276.

[0095] [4] Oskarsson M. Temporally consistent tone mapping of images and video using optimal k-means clustering[J]. Journal of Mathematical Imaging Vision, 2017, 57(2): 225-238.

[0096] [5] Khan I R, Rahardja S, Khan M M, et al. A tone-mapping technique based on histogram using a sensitivity model of the human visual system[J]. IEEE Transactions on Industrial Electronics, 2017, 65(4): 3469-3479.

[0097] [6] Duan J, Bressan M, Dance C, et al. Tone-mapping high dynamic range images by novel histogram adjustment[J]. Pattern Recognition, 2010, 43(5): 1847-1862.

[0098] [7]Panetta K, Kezebou L, Oludare V, et al. TMO-Net: A Parameter-Free Tone Mapping Operator Using Generative Adversarial Network, and Performance Benchmarking on Large Scale HDR Dataset[J]. IEEE Access, 2021, 9: 39500-39517.

[0099] [8]Yeganeh H, Wang Z. Objective quality assessment of tone-mapped images[J]. IEEE Transactions on Image Processing, 2012, 22(2): 657-667.

[0100] [9]Nafchi H Z, Shahkolaei A, Moghaddam R F, et al. FSITM: A feature similarity index for tone-mapped images[J]. IEEE Signal Processing Letters, 2014, 22(8): 1026-1029.

[0101]

[10] Ferwerda J A, Pattanaik S N, Shirley P, et al. A model of visual adaptation for realistic image synthesis[C]. Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, 1996: 249-258.

[0102]

[11] Kim M H, Kautz J. Consistent tone reproduction[C]. Proceedings of the Tenth IASTED International Conference on Computer Graphics and Imaging, 2008: 152-159.

[0103]

[12] Schlick C:Quantization techniques for visualization of highdynamic range pictures,Photorealistic rendering techniques:Springer,1995:7-20.

[0104]

[13] Ward G.A contrast-based scalefactor for luminance display[J].Graphics Gems,1994,4:415-21.

[0105]

[14] Khan I R,Aziz W,Shim S-O.Tone-mapping using perceptual-quantizerand image histogram[J].IEEE Access,2020,8:31350-31358.

[0106] In summary, the present invention processes using the lightness channel that is more in line with human eye perception rather than the luminance channel. The lightness segmentation enables the lightness distribution in each interval to be adaptively adjusted, which can improve the detail retention performance of the global tone mapping process. At the same time, the highlight limitation ensures the naturalness of the mapping result.

[0107] The above-described embodiments are only a preferred solution of the present invention, but they are not intended to limit the present invention. Those of ordinary skill in the relevant technical field can still make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, all technical solutions obtained by means of equivalent replacement or equivalent transformation fall within the protection scope of the present invention.

Claims

1. A method for improving the detail retention performance in global tone mapping processing of HDR images, characterized in that, It includes the following steps: S1: Convert a high dynamic range (HDR) image from RGB representation to IC T C P representation, and then extract its lightness channel; S2: Use the K-means clustering method to divide the lightness channel into several lightness intervals; S3: Use the HALEQ method (Histogram Adjustment based Linear to Equalized Quantizer) to calculate the mapping lookup table for each interval, and linearly adjust the target lightness range of the lookup table; S4: Apply a highlight limit to the mapping curve corresponding to the adjusted lookup table and perform lightness mapping; S5: Keep the chrominance channel unchanged, convert the image after lightness mapping back to the RGB representation, and process it using a specific global tone mapping method; The specific steps of step S3 are as follows: S301: Based on the lightness intervals obtained in S2, use the HALEQ method to generate the mapping lookup table for each lightness interval. The first column in the table is the lightness of the segmentation points obtained by the HALEQ method, and the second column is the target lightness evenly distributed between 0 and 1; S302: Linearly scale the target lightness of the mapping lookup table for each lightness interval to the original lightness range of that interval, and then cascade them in sequence to form the first lookup table applicable to the entire image; The specific steps of step S4 are as follows: S401: For the mapping curve corresponding to the first lookup table, add an additional segmentation boundary b in its highlight interval. Recalculate the lookup table of the lightness interval closest to b on the side less than b using the HALEQ method, while keeping the lightness values on the side greater than b unchanged, thereby obtaining the second lookup table; where the segmentation boundary b is: b＝α(l max -l min )+l min where: l max and l min represent the maximum and minimum lightness values of the original HDR image respectively, and α is the boundary adjustment constant; S402: Map the lightness values of the original HDR image to the target lightness based on the second lookup table by linear interpolation.

2. The method for improving the detail retention performance in the global tone mapping process of HDR images according to claim 1, wherein The specific steps of step S1 are as follows: S101: For the input HDR image, convert the RGB response values normalized between 0 and 1 into XYZ tristimulus values: S102: Convert the XYZ tristimulus values to the LMS space of the human eye cone cells sensitive to long-wave, medium-wave, and short-wave visible light respectively: S103: Apply the inverse transformation of the PQ curve to each channel of LMS: where L’, M’, and S’ respectively represent the operation results of the L, M, and S channels. is the inverse transformation of the PQ electro-optical conversion curve, and its inverse transformation result for any input Y ∈ (L, M, S) is: In the formula, m1 = 2610 / 16384, m2 = (2523 / 4096)×128, C1 = 3424 / 4096, C2 = (2413 / 4096)×32, C3 = (2392 / 4096)×32; S104: Calculate IC from L’M’S’ T C P Color representation: Wherein, I is the lightness channel, and C T and C P is the chroma channel.

3. The method for improving the detail preservation performance in the global tone mapping process of HDR images according to claim 1, wherein The specific steps of step S2 are as follows: S201: Statistically analyze the histogram of the lightness channel extracted in S1, and use the K-means clustering algorithm to update the centroid position of the clustering until it is stable. The centroid position is the lightness value; where the initial positions of the centroids of the clustering are evenly distributed within the lightness range, and the interval value is τ; S202: After the centroids of the clustering are stable, merge the adjacent clusters with a distance less than τ: where i and j are the sequence numbers of adjacent clusters, c i , c j and c i,j are the brightness values of the centroids before and after merging, n i and n j are the number of pixels in the cluster; S203: Use the dynamic programming solution method to obtain the global optimal solution of the K-means clustering problem, and each obtained category corresponds to a lightness interval.

4. The method for improving the detail retention performance in the global tone mapping process of HDR images as claimed in claim 3, wherein, The interval value τ = 0.

05.

5. The method for improving the detail preservation performance in the global tone mapping process of HDR images as claimed in claim 1, wherein In step S401, if there are already boundary points greater than b, there is no need to add the above-mentioned segmentation boundary b, and only the lightness value of the largest lightness interval needs to be set to remain unchanged during the mapping process.

6. The method for improving the detail preservation performance in the global tone mapping process of HDR images according to claim 1, characterized in that The boundary adjustment constant α is 0.

8.

7. The method for improving the detail retention performance in the global tone mapping process of HDR images as described in claim 1, characterized in that, The specific steps of step S5 are as follows: For the HDR image after brightness mapping in S4, the mapped brightness channel is used, while keeping the chromaticity channel of the image unchanged, converting it back to the RGB representation, and applying the global tone mapping method to output the LDR image.

Images