Image Processing Method, Apparatus, Electronic Device, and Storage Medium

Abstract

This application relates to an image processing method, apparatus, electronic device, and storage medium. The image processing method includes: converting an original image to the optical domain, extracting a luminance channel, and obtaining a first luminance value, where the original image is an HDR image; performing a PQ transform on the first luminance value to obtain a second luminance value in the PQ domain; the calculation formula of the PQ transform used is #imgabs0#L HDR,PQ is the second luminance value, L light1 is the first luminance value, and γ1 is a PQ transform constant; according to the equation of the global mapping curve, converting the second luminance value to a third luminance value to obtain a first intermediate image; converting the first intermediate image from the PQ domain to the optical domain to obtain a second intermediate image; according to the color coordinates of the second intermediate image and the color conversion matrix, converting from the original color gamut to the target color gamut to obtain a target image; the target image is an SDR image. The technical solution of this application can provide SDR image output with high visual quality for HDR images under different brightness conditions, is hardware-friendly, and has a small amount of calculation.

Description

[0001] This application is a divisional application of the Chinese patent application with the application number 202311377991.0 and the invention title "Image Processing Method, Device, Electronic Device and Storage Medium", which was filed with the Chinese Patent Office on October 23, 2023. Technical Field

[0002] The present invention relates to the technical field of image processing, and specifically relates to an image processing method, device, electronic device and storage medium. Background Art

[0003] High Dynamic Range (HDR) technology has emerged as a new revolution in digital media and has recently been adopted by the industry as a new standard for capturing, transmitting, and displaying video content. However, since most existing commercial displays cannot display true HDR image content, the backward compatibility of HDR image content with these traditional displays is a very important topic. Over the years, several Tone Mapping Operators (TMOs) have been proposed in the industry to convert HDR image formats to Standard Dynamic Range (SDR) image formats. With the latest development of SDR displays, the demand for video TMOs has become crucial.

[0004] In the related art, most advanced TMOs are applicable to specific types of input content, either considering the normal (average brightness) content in HDR format images or focusing on dark and / or bright content, which has certain limitations and cannot provide high-quality visual quality output for different brightness levels and cannot maintain the artistic impression of the original HDR image content. In addition, the current algorithms are not hardware-friendly and have a large amount of computation. Therefore, how to overcome the above technical defects in converting HDR format images to SDR format images is a technical problem that needs to be solved. Summary of the Invention

[0005] The purpose of this application is to provide an image processing method, device, electronic device and storage medium, which can convert the entire HDR image into an SDR image, provide high-quality visual quality output for HDR images under different brightness conditions, be hardware-friendly, and have a small amount of computation.

[0006] According to the first aspect of the embodiments of this application, an image processing method is provided, including:

[0007] Convert the original image to the light domain, and extract the luminance channel for each pixel to obtain a first luminance value, where the original image is a High Dynamic Range (HDR) image;

[0008] Perform a Perceptual Quantization (PQ) transform on the first luminance value to obtain a corresponding second luminance value in the PQ domain; where the calculation formula of the PQ transform used is Where LHDR,PQ is the second luminance value, L light1 is the first luminance value, γ 1 is the PQ transform constant;

[0009] According to the equation of the specified global mapping curve, convert the second luminance value to a third luminance value to obtain a first intermediate image; the third luminance value is the luminance value of the luminance channel of the first intermediate image;

[0010] Convert the first intermediate image from the PQ domain to the optical domain to obtain a second intermediate image;

[0011] According to the color coordinates of the second intermediate image and the color conversion matrix, convert from the original color gamut to the target color gamut to obtain a target image; wherein, the original color gamut is the color gamut of the original image, the target color gamut is the color gamut of the target image, and the target image is a standard dynamic range SDR image.

[0012] In one embodiment, the γ 1 is 4.

[0013] In one embodiment, the converting the first intermediate image from the PQ domain to the optical domain to obtain a second intermediate image includes:

[0014] Use the calculation formula of the inverse PQ transform to convert the first intermediate image from the PQ domain to the optical domain to obtain the second intermediate image, wherein the calculation formula of the inverse PQ transform is L SDR,PQ is the third luminance value, L light2 is the fourth luminance value of the luminance channel of the pixel of the second intermediate image, γ 2 is the inverse PQ transform constant.

[0015] In one embodiment, the γ 2 is 0.25.

[0016] In one embodiment, after converting the second luminance value to the third luminance value according to the equation of the specified global mapping curve to obtain the first intermediate image and before converting the first intermediate image from the PQ domain to the optical domain to obtain the second intermediate image, it further includes:

[0017] For each color channel, perform color adjustment on the first intermediate image according to the specified color adjustment equation, keeping the hue of the color unchanged and the luminance basically unchanged, to obtain a third intermediate image;

[0018] After obtaining the third intermediate image, convert the third intermediate image from the PQ domain to the optical domain to obtain the second intermediate image.

[0019] In one embodiment, the color adjustment equation is

[0020] where C SDR,PQ is the color saturation of any color channel of the third intermediate image in the PQ domain, C HDR,PQ is the color saturation of any color channel of the first intermediate image in the PQ domain, L HDR,PQ is the second luminance value, L SDR,PQ is the third luminance value, and α and β are two constants responsible for controlling the hue and color saturation of the color.

[0021] In one embodiment, the global mapping curve is a piecewise linear curve, including two endpoints, two breakpoints, and three linear curves. The three linear curves form a continuous curve. The two endpoints and the two breakpoints divide the original image into a dark region, a normal region, and a bright region; the three linear curves correspond to the dark region, the normal region, and the bright region one by one;

[0022] Converting the second luminance value to the third luminance value according to the equation of the specified global mapping curve to obtain the first intermediate image includes:

[0023] Determining the two endpoints and the two breakpoints according to the equation of the global mapping curve;

[0024] Dividing the original image into a dark region, a normal region, and a bright region according to the two endpoints and the two breakpoints;

[0025] For each region of the original image, converting the second luminance value to the corresponding third luminance value according to the equation of the global mapping curve to obtain the first intermediate image.

[0026] In one embodiment, the equation of the global mapping curve is:

[0027]

[0028] where L HDR,PQ is the second luminance value, L SDR,PQ is the third luminance value, s 1 , s 2 and s 3 are the slopes of the three linear curves respectively, a 1 , a 2 and a 3 are the intercepts of the three linear curves respectively, x 1 and x 2 are the abscissas of the two breakpoints, HDR min,PQ is the minimum value of the second luminance value, HDR max,PQis the maximum value of the second luminance value.

[0029] In one embodiment, x 1 is calculated by the following calculation formula:

[0030] x 1 = P 1 × (HDR max,PQ - HDR min,PQ ) + HDR min,PQ ;

[0031] x 2 is calculated by the following calculation formula:

[0032] x 2 = HDR max,PQ - P 2 × (HDR max,PQ - HDR min,PQ );

[0033] where x 1 is the abscissa of the boundary point between the dark area and the normal area, x 2 is the abscissa of the boundary point between the normal area and the bright area, and P 1 and P 2 are two constants.

[0034] In one embodiment, P 1 is 0.15 and P 2 is 0.4.

[0035] In one embodiment, the calculation methods of s 1 , s 2 , s 3 , a 1 , a 2 and a 3 are as follows:

[0036] Determine the values of y 1 and y 2 corresponding to x 1 and x 2 in the SDR domain;

[0037] Obtain the coordinates of the two endpoints;

[0038] According to x 1 , x 2 , y 1 , y 2 and the coordinates of the two endpoints, calculate s 1 , s 2 , s 3 , a 1 , a 2 and a3 value

[0039] In one embodiment, y 1 and y 2 The value is calculated by the following formula:

[0040] y 1 = P 3 ×(SDR max,PQ - SDR min,PQ ) + SDR min,PQ ;

[0041] y 2 = P 4 ×(SDR max,PQ - SDR min,PQ ) + SDR min,PQ ;

[0042] Wherein, P 3 and P 4 are two constants, SDR min,PQ is the minimum brightness value of the target SDR display in the PQ domain, and SDR max,PQ is the maximum brightness value of the target SDR display in the PQ domain.

[0043] In one embodiment, P 3 is 0.1369, and P 4 is 0.7518.

[0044] In one embodiment, the global mapping curve is a gamma curve, and the equation of the gamma curve is as follows:

[0045]

[0046] Wherein, L SDR,PQ is the third brightness value, L HDR,PQ is the second brightness value, and c 1 and c 2 are constants respectively;

[0047] The calculation formula of c 2 is as follows:

[0048]

[0049] The calculation formula of c 1 is as follows:

[0050]

[0051] Wherein, HDR min,PQ is the minimum value of the second brightness value, HDR max,PQ is the maximum value of the second brightness value, and SDRmin,PQ is the minimum brightness value of the target SDR display in the PQ domain, SDR max,PQ is the maximum brightness value of the target SDR display in the PQ domain.

[0052] According to a second aspect of the embodiments of the present application, there is provided an image processing apparatus, including:

[0053] A first acquisition module, configured to convert an original image into the optical domain, and extract a luminance channel for each pixel to obtain a first luminance value, where the original image is a high dynamic range HDR image;

[0054] A second acquisition module, configured to perform a perceptual quantization PQ transform on the first luminance value to obtain a corresponding second luminance value in the PQ domain; where the calculation formula of the PQ transform used is where, L HDR,PQ is the second luminance value, L light1 is the first luminance value, γ 1 is the PQ transform constant;

[0055] A first conversion module, configured to convert the second luminance value into a third luminance value according to the equation of a specified global mapping curve, to obtain a first intermediate image; the third luminance value is the luminance value of the luminance channel of the first intermediate image;

[0056] A second conversion module, configured to convert the first intermediate image from the PQ domain into the optical domain to obtain a second intermediate image;

[0057] A third conversion module, configured to convert from an original color gamut to a target color gamut according to the color coordinates and color conversion matrix of the second intermediate image to obtain a target image; where the original color gamut is the color gamut of the original image, the target color gamut is the color gamut of the target image, and the target image is a standard dynamic range SDR image.

[0058] According to a third aspect of the embodiments of the present application, there is provided an electronic device, including a memory and a processor, where the memory is used to store a computer program executable by the processor; the processor is used to execute the computer program in the memory to implement the above method.

[0059] According to a fourth aspect of the embodiments of the present application, there is provided a computer-readable storage medium, on which a computer program is stored, characterized in that when the executable computer program in the storage medium is executed by a processor, the above method can be implemented.

[0060] Compared with the prior art, the beneficial effects of the present application are as follows: Since the original image (HDR image) is converted to the optical domain, and the luminance channel is extracted for each pixel to obtain the first luminance value, and then the first luminance value is subjected to perceptual quantization (PQ) conversion to obtain the corresponding second luminance value in the PQ domain, that is, the HDR image is converted to the perceptual domain, it helps to maintain the overall visual impression of the frame. Moreover, the calculation formula of the PQ transformation adopted is Hardware-friendly, it can be easily implemented with limited resources, has a small amount of calculation, and can reduce the calculation cost.

[0061] Then, according to the equation of the specified global mapping curve, the second luminance value of each pixel is converted into the corresponding third luminance value to obtain the first intermediate image, where the third luminance value is the luminance value of the luminance channel of the first intermediate image. On the one hand, since the global mapping curve is used instead of the local mapping curve, the luminance of each pixel of the entire image of the original image can be converted, that is, all regions with different luminance levels in the HDR image are subjected to luminance conversion, rather than only local luminance conversion for a certain type of region with a specific luminance level. Therefore, it is possible to convert the entire HDR image into an SDR image and provide high-quality visual quality output for HDR images under different luminance conditions. On the other hand, since the global mapping curve is used to adjust the luminance of the entire image of the original image as a whole, rapid changes in luminance that may cause flicker can be avoided, and thus flicker artifacts can be avoided.

[0062] Then, the first intermediate image is converted from the PQ domain to the optical domain to obtain the second intermediate image, and according to the color coordinates and color conversion matrix of the second intermediate image, it is converted from the original color gamut to the target color gamut to obtain the target image (SDR image).

[0063] In summary, the technical solution provided by the present application can convert the entire HDR image into an SDR image, provide high-quality visual quality output for HDR images under different luminance conditions, and is hardware-friendly, can be easily implemented with limited resources, has a small amount of calculation, and can reduce the calculation cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 is a flowchart of an image processing method shown according to an exemplary embodiment.

[0065] Figure 2 is a schematic diagram of a gamma curve shown according to an exemplary embodiment.

[0066] Figure 3 is a flowchart of an image processing method shown according to another exemplary embodiment.

[0067] Figure 4It is a schematic diagram of a piecewise linear curve shown according to an exemplary embodiment.

[0068] Figure 5 It is a flowchart of an image processing method shown according to another exemplary embodiment.

[0069] Figure 6 It is a block diagram of an image processing apparatus shown according to an exemplary embodiment.

[0070] Figure 7 It is a block diagram of an image processing apparatus shown according to another exemplary embodiment.

[0071] Figure 8 It is a block diagram of an electronic device shown according to an exemplary embodiment. Detailed implementation manners

[0072] Unless otherwise defined, the technical terms or scientific terms used in this specification and the claims shall have the ordinary meanings as understood by those of ordinary skill in the technical field to which the present invention belongs. The following will describe the specific implementation manners of the present invention in conjunction with the accompanying drawings. It should be noted that in the specific description of these implementation manners, for the sake of concise description, this specification may not describe all features of the actual implementation manners in detail. Without departing from the spirit and scope of the present invention, those skilled in the art can modify and replace the implementation manners of the present invention, and the obtained implementation manners are also within the protection scope of the present invention.

[0073] In the related art, directly applying traditional TMO to HDR video is not an effective solution because it will generate visual artifacts, such as flickering, inconsistent brightness and colors. On the other hand, the TMO for generating high-quality SDR content is complex and requires manual intervention to set various parameters within the implementation method, so it is not suitable for real-time and hardware-friendly implementation. Another challenge when using existing methods is the inability to maintain the artistic impression of the original HDR content. When converting to the SDR format, it is crucial to maintain the overall visual impression of the input HDR content.

[0074] To solve the above technical problems, the present application proposes an image processing method, apparatus, electronic device and storage medium, which can realize converting the entire HDR image into an SDR image, provide high-quality visual quality output for HDR images under different brightness conditions, and can also avoid problems such as flickering artifacts, inconsistent brightness and colors generated during the image format conversion process. Moreover, it is hardware-friendly, can be easily implemented with limited resources, has a small amount of calculation, and can reduce the calculation cost.

[0075] Figure 1It is a flowchart of an image processing method shown according to an exemplary embodiment. This image processing method is performed in the RGB color space and can be applied to electronic devices with image processing functions such as image processors, display chips, and displays. Please refer to Figure 1 , this image processing method may include the following steps:

[0076] Step 101: Convert the original image to the optical domain and extract the luminance channel for each pixel to obtain the first luminance value, where the original image is a high dynamic range (HDR) image.

[0077] In this step, the input original image (HDR image) is converted from the HDR domain to the optical domain, that is, the encoded values in the HDR format of the original image are converted to grayscale values. Then, the luminance channel is extracted for each pixel to obtain the first luminance value.

[0078] In this embodiment, the color gamut of the original image, that is, the original color gamut is the BT.2020 color gamut. According to the BT.2020 standard, the formula for extracting the luminance channel from the three color channels of red, green, and blue is as follows:

[0079] L light1 = 0.2627R + 0.6780G + 0.0593B;

[0080] where R is the red channel, G is the green channel, B is the blue channel, and L light1 is the first luminance value.

[0081] In the related art, the HDR format includes two formats: HLG10 or HDR10. The difference between these two formats lies in their transfer functions. The HDR10 format uses the PQ function, while the HLG10 format uses the Hybrid Log-Gamma (HLG) function.

[0082] In this embodiment, when the format of the original image is HDR10, the inverse PQ transform function is used to convert the original image to the optical domain, and when the format of the original image is HLG10, the inverse HLG transform function is used to convert the original image to the optical domain.

[0083] Step 102: Perform a perceptual quantization (PQ) transform on the first luminance value to obtain the corresponding second luminance value in the PQ domain; where the calculation formula of the PQ transform used is where L HDR,PQ is the second luminance value, L light1 is the first luminance value, and γ 1 is the PQ transform constant.

[0084] Among them, the PQ (Perceptual Quantizer) domain is a way of video signal encoding and compression. The PQ domain belongs to the perceptual domain and is a perceptually uniform color space, aiming to more closely match the way the human visual system perceives colors. The PQ domain uses a logarithmic transfer function and can make more efficient use of the available dynamic range. This makes it very suitable for displaying high dynamic range (HDR) video content.

[0085] Perceptual quantization (PQ) aims to optimize the light intensity distribution related to HVS attributes and convert physical linear values into perceptually linear values. Perceptual linearity means that the human eye sees any intensity change at any brightness level in the same way (i.e., two consecutive brightness levels are just below the threshold that the HVS has just noticed). Perceptual linearity is the basis for implementation because in this way, what the human eye can see can be recognized, and this information can be mapped to a higher dynamic range in the best way. Therefore, first, the first brightness value of the luminance channel of the input HDR image is transmitted to the PQ domain, so that the luminance values of the HDR image and the SDR image are in the same domain during the mapping process. This enables us to work in the perceptual domain, where adjacent luminance values correspond to differences perceptible to the human eye.

[0086] In this embodiment, the first brightness value is subjected to PQ transformation to obtain the corresponding second brightness value in the PQ domain.

[0087] In this embodiment, a perceptual quantizer can be used to perform PQ transformation on the first brightness value. The perceptual quantizer is a transfer function that can map the linear light intensity of a display device into a non-linear signal, thus better matching the characteristics of the human visual system (HVS). The function of the perceptual quantizer is to optimize the distribution of light intensity so as to make the most effective use of the available dynamic range and provide the best reproduction of the original image.

[0088] In the related art, the calculation formula for PQ transformation is:

[0089]

[0090] where L light1 is the first brightness value, and L HDR,PQ is the second brightness value. c 1 , c 2 , c 3 , m 1 and m 2 are constants, which can be set to 0.8359, 18.8515, 18.6875, 0.1593, and 78.8437 respectively.

[0091] However, the above calculation formula for PQ transformation has a very high calculation cost in hardware implementation and requires a large amount of resources to implement.

[0092] To overcome the above challenges, in this embodiment, the calculation formula of the PQ transform is as follows:

[0093]

[0094] where L HDR,PQ is the second luminance value, L light1 is the first luminance value, and γ 1 is the PQ transform constant.

[0095] In this embodiment, the calculation formula of the PQ transform is simple in form, has few parameters, does not require manual intervention to set various parameters, has a small amount of calculation, has a relatively low calculation cost when implemented in hardware, is hardware-friendly, can be easily implemented with limited resources, and can be extended to real-time applications.

[0096] In this embodiment, γ 1 is 4. According to extensive simulation experiments, setting γ 1 to 4 can achieve high visual quality results in both subjective and objective evaluations.

[0097] Step 103: Convert the second luminance value to a third luminance value according to the equation of the specified global mapping curve to obtain a first intermediate image; the third luminance value is the luminance value of the luminance channel of the first intermediate image.

[0098] Flicker is a common problem that may occur during tone mapping, especially when using local tone mapping algorithms. This is because local algorithms independently adjust the luminance of individual pixels or a small group of pixels, which can cause rapid changes in the luminance of the entire image. These luminance changes may be particularly obvious in high-contrast regions and may be distracting and cause visual impact. To avoid this problem, the global mapping algorithm (the equation of the global mapping curve) in this application uses a different method. Instead of adjusting the luminance of individual pixels, it takes the average value of the luminance values of the entire image. Since the luminance of the entire image is adjusted as a whole, this method avoids the rapid changes in luminance that may cause flicker.

[0099] In this embodiment, the global mapping curve can be determined according to the curve identifier input by the user. For example, when the curve identifier is 00, the specified global mapping curve is a gamma curve, and when the curve identifier is 01, the specified global mapping curve is a piecewise linear curve.

[0100] In this embodiment, the global mapping curve is a gamma curve, and the equation of the gamma curve is as follows:

[0101]

[0102] where L SDR,PQ is the third luminance value, L HDR,PQis the second luminance value, c 1 and c 2 are constants respectively.

[0103] In this embodiment, c can be calculated first using the following formula 2 :

[0104]

[0105] Then, c is calculated using the following formula 1 :

[0106]

[0107] Among them, HDR min,PQ is the minimum value of the second luminance value, HDR max,PQ is the maximum value of the second luminance value, SDR min,PQ is the minimum luminance value of the target SDR display in the PQ domain, SDR max,PQ is the maximum luminance value of the target SDR display in the PQ domain.

[0108] In this embodiment, the gamma curve C1 adopted is as Figure 2 shown.

[0109] In this embodiment, the global mapping curve adopts a gamma curve, which has a simple mathematical form, low computational complexity, and can generate high-quality SDR images. Moreover, the generated SDR images have more details in brighter areas than in darker areas. If we need the generated SDR images to have more details in brighter areas than in darker areas, it is recommended to use a gamma curve.

[0110] In this embodiment, since the maximum and minimum luminance values of the target SDR display are considered to construct the mapping curve. By doing so, the image processing method can generate content that looks pleasant on all types of SDR displays. Therefore, the algorithm provided in this application is display adaptive and can generate SDR image content that matches the functions of any target SDR display. Therefore, the TMO proposed in this paper is display adaptive and can generate SDR content that matches the functions of any target display.

[0111] Step 104, convert the first intermediate image from the PQ domain to the optical domain to obtain a second intermediate image.

[0112] In this embodiment, it is necessary to convert the first intermediate image from the PQ domain to the optical domain to display the image content.

[0113] In the related art, the following inverse PQ transformation formula is used to convert the first intermediate image from the PQ domain to the optical domain:

[0114]

[0115] Among them, L light2 is the fourth luminance value of the luminance channel of the pixels of the second intermediate image.

[0116] However, the above calculation formula of the inverse PQ transform has a high calculation cost when implemented in hardware and requires a large amount of resources to implement. To overcome the above challenges, in this embodiment, the calculation formula of the inverse PQ transform is:

[0117]

[0118] Among them, L SDR,PQ is the third luminance value, L light2 is the fourth luminance value of the luminance channel of the pixels of the second intermediate image, and γ 2 is the inverse PQ transform constant.

[0119] In this embodiment, the calculation formula of the inverse PQ transform has a simple form, few parameters, and a small amount of calculation. The calculation cost when implemented in hardware is relatively low, which is friendly to hardware and can be easily implemented with limited resources.

[0120] In this embodiment, γ 2 is 0.25. According to extensive simulation experiments, setting γ 2 to 0.25 can achieve high visual quality results in both subjective and objective evaluations.

[0121] Step 105: Convert from the original color gamut to the target color gamut according to the color coordinates of the second intermediate image and the color conversion matrix to obtain the target image; among them, the original color gamut is the color gamut of the original image, the target color gamut is the color gamut of the target image, and the target image is a standard dynamic range SDR image.

[0122] In this embodiment, the color gamut of the target image, that is, the target color gamut is BT.709. Since BT.709 is specifically designed for SDR images, therefore, converting the image to BT.709 can ensure that the generated content displays the correct colors.

[0123] Among them, converting the image from the BT.2020 color gamut to the BT.709 color gamut involves converting the color coordinates of a given image or video from one color space to another color space.

[0124] In one embodiment, a 3×3 color conversion matrix can be used to perform gamut conversion, which maps colors in the source color space (BT.2020) to colors in the target color space (BT.709). This process typically starts with converting an image or video from the RGB color space to the native color representation of the color space. Then, the color conversion matrix is applied to the color coordinates of the image to transform them to the new color space. The matrix for converting from the BT.2020 gamut to the BT.709 gamut can be expressed as:

[0125] R709 = T11 × R2020 + T12 × G2020 + T13 × B2020

[0126] G709 = T21 × R2020 + T22 × G2020 + T23 × B2020

[0127] B709 = T31 × R2020 + T32 × G2020 + T33 × B2020

[0128] Wherein, R, G, and B respectively represent the red, green, and blue channels. R709, G709, and B709 are the color coordinates of the target image in the BT.709 gamut, and R2020, G2020, and B2020 are the color coordinates of the second intermediate image in the BT.2020 gamut. T11, T12, T13, T21, T22, T23, T31, T32, and T33 are conversion coefficients.

[0129] It should be noted that this step is quite complex and requires high precision. This color conversion matrix is only an approximation. If precise conversion is required, a three-dimensional color lookup table (3D-CLUT) needs to be used.

[0130] In this embodiment, since the original image (HDR image) is converted to the optical domain, and the luminance channel is extracted for each pixel to obtain the first luminance value. Then, the first luminance value is subjected to perceptual quantization (PQ) conversion to obtain the corresponding second luminance value in the PQ domain, that is, the HDR image is converted to the perceptual domain. Therefore, it helps to maintain the overall visual impression of the frame. Moreover, the calculation formula of the adopted PQ transformation is Friendly to hardware, it can be easily implemented with limited resources, has a small amount of calculation, and can reduce the calculation cost.

[0131] Then, according to the equation of the specified global mapping curve, the second luminance value of each pixel is converted into the corresponding third luminance value, obtaining a first intermediate image, where the third luminance value is the luminance value of the luminance channel of the first intermediate image. On the one hand, since the global mapping curve is used instead of the local mapping curve, the luminance of each pixel in the entire image of the original image can be converted, that is, the luminance conversion is performed on all regions with different luminance levels in the HDR image, rather than only on a specific type of region with a specific luminance level. Therefore, the entire HDR image can be converted into an SDR image, providing a high-quality visual quality output for HDR images under different luminance conditions. On the other hand, since the global mapping curve is used to adjust the luminance of the entire image of the original image as a whole, the rapid change of luminance that may cause flicker can be avoided, and thus the flicker artifact can be avoided.

[0132] Then, the first intermediate image is converted from the PQ domain to the optical domain, obtaining a second intermediate image. According to the color coordinates and color conversion matrix of the second intermediate image, it is converted from the original color gamut to the target color gamut, obtaining the target image (SDR image).

[0133] In summary, the technical solution provided by this application can convert the entire HDR image into an SDR image, providing a high-quality visual quality output for HDR images under different luminance conditions. Moreover, it is hardware-friendly, can be easily implemented with limited resources, has a small amount of calculation, and can reduce the calculation cost.

[0134] Figure 3 It is a flowchart of an image processing method shown according to another exemplary embodiment. Different from the above embodiment, in this embodiment, after converting the second luminance value into the third luminance value according to the equation of the specified global mapping curve to obtain the first intermediate image and before converting the first intermediate image from the PQ domain to the optical domain to obtain the second intermediate image, for each color channel, the first intermediate image is color-adjusted according to the specified color adjustment equation to keep the hue of the color unchanged and the luminance basically unchanged. Please refer to Figure 3 , the image processing method may include the following steps:

[0135] Step 301, convert the original image to the optical domain, and extract the luminance channel for each pixel, obtaining the first luminance value, where the original image is a high dynamic range (HDR) image.

[0136] This step is similar to step 101 above and will not be elaborated here.

[0137] Step 302, perform a perceptual quantization (PQ) transform on the first luminance value to obtain the corresponding second luminance value in the PQ domain; where the calculation formula of the PQ transform adopted is where LHDR,PQ is the second luminance value, L light1 is the first luminance value, γ 1 is the PQ transformation constant.

[0138] This step is similar to step 102 described above and will not be elaborated here.

[0139] Step 303: According to the equation of the specified global mapping curve, convert the second luminance value into a third luminance value to obtain a first intermediate image; the third luminance value is the luminance value of the luminance channel of the first intermediate image.

[0140] In this embodiment, the global mapping curve is a piecewise linear curve. As Figure 4 shown, the global mapping curve C2 includes two endpoints D1, D2, two demarcation points D3, D4, and three linear curves C21, C22, C23. The three linear curves C21, C22, C23 form a continuous curve. The two endpoints D1, D2 and the two demarcation points D3, D4 divide the original image into a dark area, a normal area, and a bright area; the three linear curves correspond one-to-one with the dark area, the normal area, and the bright area. Specifically, the linear curve C21 is the curve corresponding to the dark area, the linear curve C22 is the curve corresponding to the normal area, and the linear curve C23 is the curve corresponding to the bright area. x 1 and x 2 are the abscissas of the two demarcation points D3, D4, y 1 and y 2 are the ordinates of the two demarcation points D3, D4, s 1 , s 2 and s 3 are the slopes of the three linear curves C21, C22, C23 respectively. In other embodiments, the number of segments in the global mapping curve can be extended to more than three.

[0141] In this embodiment, the equation of the global mapping curve is as follows:

[0142]

[0143] where L HDR,PQ is the second luminance value, L SDR,PQ is the third luminance value, s 1 , s 2 and s 3 are the slopes of the three linear curves respectively, a 1 , a 2 and a 3 are the intercepts of the three linear curves respectively, x 1 and x 2 are the abscissas of the two demarcation points, HDR min,PQ is the minimum value of the second luminance value, HDR max,PQis the maximum value of the second luminance value.

[0144] In this embodiment, the above piecewise linear curve is used as the equation of the specified global mapping curve to convert the second luminance value into the third luminance value, obtaining the first intermediate image. Among them, the specific method of global mapping is as follows: First, according to the equation C2 of the global mapping curve, two endpoints D1, D2 and two demarcation points D3, D4 are determined. Then, according to the two endpoints D1, D2 and the two demarcation points D3, D4, the original image is divided into a dark area, a normal area, and a bright area. Next, for each area of the original image, the second luminance value is converted into the corresponding third luminance value according to the equation of the global mapping curve. Among them, for the dark area of the original image, the second luminance value is converted into the corresponding third luminance value according to the linear curve C21. For the normal area of the original image, the second luminance value is converted into the corresponding third luminance value according to the linear curve C22. For the bright area of the original image, the second luminance value is converted into the corresponding third luminance value according to the linear curve C23. This helps to convert each area of the HDR image into their corresponding areas in the SDR domain, and at the same time helps to maintain the overall artistic impression. Using a piecewise linear curve can make the output image more balanced, thus achieving the best balance between the overall brightness and contrast of the image.

[0145] In this embodiment, the generation method of the piecewise linear curve is as follows:

[0146] (1) Determine the segmentation method.

[0147] In this step, first, calculate x 1 and x 2 values to segment the image, dividing the original image into a dark area, a normal area, and a bright area.

[0148] x 1 is calculated through the following calculation formula:

[0149] x 1 = P 1 ×(HDR max,PQ - HDR min,PQ ) + HDR min,PQ ;

[0150] x 2 is calculated through the following calculation formula:

[0151] x 2 = HDR max,PQ - P 2 ×(HDR max,PQ - HDR min,PQ );

[0152] Among them, x 1The abscissa of the boundary point between the dark area and the normal area, x 2 The abscissa of the boundary point between the normal area and the bright area, P 1 And P 2 Are two constants.

[0153] In this embodiment, P 1 Is 0.15, and P 2 Is 0.4. According to extensive research, setting P 1 And P 2 To 0.15 and 0.4 respectively can achieve good results in terms of dividing the HDR image into three brightness regions.

[0154] After calculating the values of x 1 And x 2 , determine the values y 1 And y 2 In the SDR domain that each corresponds to. 1 And y 2 . y 1 And y 2 The values of can be calculated by the following calculation formula:

[0155] y 1 = P 3 ×(SDR max,PQ - SDR min,PQ ) + SDR min,PQ ;

[0156] y 2 = P 4 ×(SDR max,PQ - SDR min,PQ ) + SDR min,PQ ;

[0157] Among them, P 3 And P 4 Are two constants, SDR min,PQ Is the minimum brightness value of the target SDR display in the PQ domain, and SDR max,PQ Is the maximum brightness value of the target SDR display in the PQ domain.

[0158] In this embodiment, P 3 Is 0.1369, and P 4 Is 0.7518. According to extensive simulations, setting P 3 And P 4 To 0.1369 and 0.7518 respectively will produce an SDR image with good visual quality.

[0159] Next, obtain the coordinates of the two endpoints D1 and D2.

[0160] Next, according to x1 , x 2 , y 1 , y 2 and the coordinates of two endpoints D1 and D2, and calculate s 1 , s 2 , s 3 , a 1 , a 2 and a 3 values.

[0161] (2) Generate a piecewise linear curve

[0162] In this embodiment, according to s 1 , s 2 , s 3 , a 1 , a 2 , and a 3 values, generate the above-mentioned piecewise linear curve, that is, obtain the equation of the global mapping curve C2.

[0163] In this embodiment, the proposed TMO works in the perceptual domain to consider the sensitivity of the human eye to brightness changes in different regions of the scene (i.e., dark, normal, and bright). This helps the TMO to process each region differently according to the way the eye perceives them.

[0164] To retain the overall impression of the original HDR image content, the TMO proposed in this paper uses a novel segmentation method to divide the HDR image into dark, normal, and bright regions. This not only enables the overall brightness of each region to be retained, but also enables the TMO to process all types of HDR images, which is a problem that most existing TMOs cannot solve.

[0165] Step 304, for each color channel, perform color adjustment on the first intermediate image according to the specified color adjustment equation, keeping the hue of the color unchanged and the brightness basically unchanged, to obtain the third intermediate image.

[0166] An important consideration in designing an efficient TMO is to maintain color accuracy between the input HDR image and the generated SDR image. This is a challenge because when trying to compress the brightness level from HDR to SDR, color shift may occur. Therefore, an effective color adjustment scheme is very important when designing a TMO. The goal of the color adjustment method is to maintain the hue of the color during the transformation process while making the change in color brightness negligible.

[0167] In this embodiment, to ensure the color accuracy of the generated image, the following color adjustment equation is adopted:

[0168]

[0169] Among them, C SDR,PQ is the color saturation of any color channel (red, green, or blue) of the third intermediate image in the PQ domain, C HDR,PQ is the color saturation of any color channel of the first intermediate image in the PQ domain, L HDR,PQ is the second luminance value, L SDR,PQ is the third luminance value, and α and β are two constants responsible for controlling the hue and color saturation of the color.

[0170] In this embodiment, since color adjustment is performed according to the specified color adjustment equation, the hue of the color is kept unchanged and the luminance is basically unchanged. In this way, the color accuracy between the input HDR image and the output SDR image can be ensured.

[0171] In addition, according to extensive simulations, the above color adjustment equation preserves the hue of the color, and its influence on the luminance when mapping the HDR image to the SDR image can be ignored.

[0172] In this embodiment, by performing color adjustment in the perceptual domain, the image processing method preserves the hue of the color of the HDR image and generates a color of the SDR image that is very close to the color of the HDR image.

[0173] Step 305: Convert the third intermediate image from the PQ domain to the optical domain to obtain a second intermediate image.

[0174] This step is similar to step 104 above and will not be elaborated here.

[0175] Step 306: Convert from the original color gamut to the target color gamut according to the color coordinates and color conversion matrix of the second intermediate image to obtain a target image; where the original color gamut is the color gamut of the original image, the target color gamut is the color gamut of the target image, and the target image is a standard dynamic range SDR image.

[0176] This step is similar to step 105 above and will not be elaborated here.

[0177] In this embodiment, after converting the second luminance value to the third luminance value according to the equation of the specified global mapping curve to obtain the first intermediate image, and before converting the first intermediate image from the PQ domain to the optical domain to obtain the second intermediate image, for each color channel, color adjustment is performed on the first intermediate image according to the specified color adjustment equation to keep the hue of the color unchanged and the luminance basically unchanged. By performing color adjustment in the perceptual domain, the image processing method preserves the hue of the color of the HDR image and generates a color of the SDR image that is very close to the color of the HDR image, and at the same time, the change in color luminance during the color adjustment process can be ignored.

[0178] Figure 5 It is a flowchart of an image processing method shown according to another exemplary embodiment and is also a principle block diagram of the TMO proposed herein. The main part of this method is the part enclosed by the dashed box 51. Please refer to Figure 5 This image processing method may include the following steps:

[0179] (1) Extract the luminance channel

[0180] This step is the same as step 101 and will not be elaborated here.

[0181] (2) PQ transformation

[0182] This step is the same as step 102 and will not be elaborated here.

[0183] (3) Select a mapping curve

[0184] In this embodiment, the mapping curve is a global mapping curve, and the global mapping curve includes a gamma curve and a piecewise linear curve. The electronic device can select the mapping curve according to the curve identifier input by the user. For example, if the curve identifier input by the user is 00, the specified global mapping curve is the gamma curve; if the curve identifier input by the user is 01, the specified global mapping curve is the piecewise linear curve.

[0185] (4) When the curve identifier of the gamma curve is input by the user, generate the gamma curve and perform global mapping

[0186] In this embodiment, when the curve identifier input by the user is the curve identifier of the gamma curve, generate the gamma curve and perform global mapping.

[0187] (5) When the curve identifier of the piecewise linear curve is input by the user, determine the curve segmentation method

[0188] In this embodiment, when the curve identifier input by the user is the curve identifier of the piecewise linear curve, determine the curve segmentation method.

[0189] This step is the same as the content of determining the curve segmentation method in step 303 above and will not be elaborated here.

[0190] (6) Generate the piecewise linear curve and perform global mapping

[0191] In this embodiment, after determining the curve segmentation method, generate the piecewise linear curve and perform global mapping.

[0192] This step is the same as the content of the global mapping method in step 303 above and will not be elaborated here.

[0193] (7) Color adjustment

[0194] This step is the same as step 304 and will not be elaborated here.

[0195] (8), Inverse PQ transformation

[0196] This step is the same as step 305 and will not be elaborated here.

[0197] (9), Inverse color gamut transformation

[0198] This step is the same as step 306 and will not be elaborated here.

[0199] In this embodiment, the HDR image can be converted into a standard dynamic range (SDR) image to achieve high visual quality at all brightness levels. Moreover, it is hardware-friendly and can be implemented in hardware with limited resources.

[0200] Figure 6 is a block diagram of an image processing apparatus shown according to an exemplary embodiment. As Figure 6 shown, in this embodiment, the image processing apparatus includes:

[0201] A first acquisition module 61, configured to convert the original image into the optical domain and extract the luminance channel for each pixel to obtain a first luminance value, where the original image is a high dynamic range (HDR) image;

[0202] A second acquisition module 62, configured to perform a perceptual quantization PQ transform on the first luminance value to obtain a corresponding second luminance value in the PQ domain; where the calculation formula of the PQ transform adopted is where, L HDR,PQ is the second luminance value, L light1 is the first luminance value, γ 1 is the PQ transform constant;

[0203] A first conversion module 63, configured to convert the second luminance value into a third luminance value according to the equation of the specified global mapping curve to obtain a first intermediate image; the third luminance value is the luminance value of the luminance channel of the first intermediate image;

[0204] A second conversion module 64, configured to convert the first intermediate image from the PQ domain into the optical domain to obtain a second intermediate image;

[0205] A third conversion module 65, configured to convert from the original color gamut to the target color gamut according to the color coordinates and color conversion matrix of the second intermediate image to obtain a target image; where the original color gamut is the color gamut of the original image, the target color gamut is the color gamut of the target image, and the target image is a standard dynamic range (SDR) image.

[0206] In one embodiment, as Figure 7 shown, the image processing apparatus further includes:

[0207] A color adjustment module 66, configured to perform color adjustment on the first intermediate image for each color channel according to a specified color adjustment equation, while keeping the hue of the color unchanged and the brightness basically unchanged, to obtain a third intermediate image;

[0208] A second conversion module 64, further configured to convert the third intermediate image from the PQ domain to the optical domain to obtain a second intermediate image.

[0209] An embodiment of the present application further provides an electronic device, including a processor and a memory; the memory is used to store a computer program executable by the processor; the processor is used to execute the computer program in the memory to implement the image processing method of any of the above embodiments.

[0210] An embodiment of the present application further provides a computer-readable storage medium, which can implement the image processing method of any of the above embodiments when the executable computer program in the storage medium is executed by the processor.

[0211] Regarding the device in the above embodiments, the specific manner in which the processor performs operations has been described in detail in the embodiments related to the method, and will not be elaborated here.

[0212] Figure 8 is a block diagram of an electronic device shown according to an exemplary embodiment. For example, the electronic device 800 may be provided as a server. Referring to Figure 8 , the device 800 includes a processing component 822, which further includes one or more processors, and memory resources represented by a memory 832 for storing instructions executable by the processing component 822, such as application programs. The application programs stored in the memory 832 may include one or more modules each corresponding to a set of instructions. In addition, the processing component 822 is configured to execute instructions to perform the above image processing method.

[0213] The device 800 may further include a power supply component 826 configured to perform power management of the device 800, a wired or wireless network interface 850 configured to connect the device 800 to a network, and an input / output (I / O) interface 858. The device 800 may operate based on an operating system stored in the memory 832, such as Windows ServerTM, MacOS XTM, UnixTM, LinuxTM, FreeBSDTM or the like.

[0214] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 832 including instructions, and the above instructions can be executed by a processing component 822 of the device 800 to complete the above method. For example, the non-transitory computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0215] In the present invention, the terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance. The term "plurality" means two or more, unless otherwise clearly defined.

[0216] The above description of the embodiments is for the convenience of those of ordinary skill in the art to understand and apply the present application. Those skilled in the art can obviously make various modifications to these embodiments easily and apply the general principles described herein to other embodiments without creative efforts. Therefore, the present application is not limited to the embodiments herein, and all improvements and modifications made by those skilled in the art within the scope and spirit of the present application based on the disclosure of the present application are within the scope of the present application.

Claims

1. An image processing method, characterized in that, comprising: Converting the original image to the optical domain, and extracting the luminance channel for each pixel to obtain a first luminance value, wherein the original image is a high dynamic range (HDR) image; Perform perceptual quantization PQ transformation on the first luminance value to obtain a corresponding second luminance value in the PQ domain; wherein, the calculation formula of the PQ transformation adopted is wherein, L HDR,PQ is the second luminance value, L light1 is the first luminance value, γ 1 is the PQ transformation constant; Converting the second luminance value to a third luminance value according to the equation of a specified global mapping curve to obtain a first intermediate image; the third luminance value is the luminance value of the luminance channel of the first intermediate image; Converting the first intermediate image from the PQ domain to the optical domain to obtain a second intermediate image; Converting from the original color gamut to the target color gamut according to the color coordinates of the second intermediate image and a color conversion matrix to obtain a target image; wherein the original color gamut is the color gamut of the original image, the target color gamut is the color gamut of the target image, and the target image is a standard dynamic range (SDR) image; The global mapping curve is a gamma curve, and the equation of the gamma curve is as follows: Among them, L SDR,PQ is the third brightness value, and L HDR,PQ is the second brightness value. c 1 and c 2 are constants respectively; c 2 The calculation formula is as follows: c 1 The calculation formula is as follows: Among them, HDR min,PQ is the minimum value of the second luminance value, HDR max,PQ is the maximum value of the second luminance value, SDR min,PQ is the minimum luminance value of the target SDR display in the PQ domain, SDR max,PQ is the maximum luminance value of the target SDR display in the PQ domain.

2. An image processing apparatus, characterized in that, comprising: A first acquisition module configured to convert the original image to the optical domain, and extract the luminance channel for each pixel to obtain a first luminance value, wherein the original image is a high dynamic range (HDR) image; A second acquisition module, configured to perform a perceptual quantization PQ transform on the first luminance value to obtain a corresponding second luminance value in the PQ domain; wherein, the calculation formula of the PQ transform adopted is wherein, L HDR,PQ is the second luminance value, L light1 is the first luminance value, and γ 1 is the PQ transform constant; A first conversion module configured to convert the second luminance value to a third luminance value according to the equation of a specified global mapping curve to obtain a first intermediate image; the third luminance value is the luminance value of the luminance channel of the first intermediate image; A second conversion module configured to convert the first intermediate image from the PQ domain to the optical domain to obtain a second intermediate image; A third conversion module configured to convert from the original color gamut to the target color gamut according to the color coordinates of the second intermediate image and a color conversion matrix to obtain a target image; wherein the original color gamut is the color gamut of the original image, the target color gamut is the color gamut of the target image, and the target image is a standard dynamic range (SDR) image; The global mapping curve is a gamma curve, and the equation of the gamma curve is as follows: Among them, L SDR,PQ is the third brightness value, L HDR,PQ is the second brightness value, c 1 and c 2 are constants respectively; c 2 The calculation formula is as follows: c 1 The calculation formula is as follows: Among them, HDR min,PQ is the minimum value of the second luminance value, HDR max,PQ is the maximum value of the second luminance value, SDR min,PQ is the minimum luminance value of the target SDR display in the PQ domain, SDR max,PQ is the maximum luminance value of the target SDR display in the PQ domain.

3. An electronic device, characterized in that, comprising a memory and a processor, the memory is used to store a computer program executable by the processor; the processor is used to execute the computer program in the memory to implement the method according to claim 1.

4. A computer-readable storage medium, on which a computer program is stored, characterized in that, when the executable computer program in the storage medium is executed by a processor, it can implement the method according to claim 1.

Images