Image processing method, device, electronic device and storage medium

Abstract

The present application relates to an image processing method, device, electronic device and storage medium. The image processing method includes: converting the original image to the light domain and extracting the brightness channel to obtain a first brightness value, wherein the original image is an HDR image; performing a PQ transformation on the first brightness value to obtain a second brightness value in the PQ domain; the calculation formula of the PQ transformation used is #imgabs0#L HDR,PQ is the second brightness value, L light1 is the first brightness value, γ 1 is a PQ transformation constant; according to the equation of the global mapping curve, the second brightness value is converted to a third brightness value to obtain a first intermediate image; the first intermediate image is converted from the PQ domain to the light domain to obtain a second intermediate image; according to the color coordinates of the second intermediate image and the color conversion matrix, the original color domain is converted to the target color domain to obtain a target image; the target image is an SDR image. The technical solution of the present application can provide SDR image output with high visual quality for HDR images of different brightness conditions, and is hardware-friendly and has a small amount of calculation.

Description

[0001] This application is a divisional application of the Chinese patent application submitted to the China Patent Office on October 23, 2023, with application number 202311377991.0 and invention name "Image processing method, device, electronic device and storage medium". Technical Field

[0002] The present invention relates to the technical field of image processing, and in particular to an image processing method, device, electronic equipment and storage medium. Background Art

[0003] High Dynamic Range (HDR) technology has emerged as a new revolution in digital media and has recently been adopted by the industry as a new standard for acquiring, transmitting, and displaying video content. However, since most existing commercial displays cannot display true HDR image content, the backward compatibility of HDR image content with these legacy displays is a very important topic. Over the years, the industry has proposed several Tone Mapping Operators (TMOs) to convert HDR image formats to Standard Dynamic Range (SDR) image formats. With the recent development of SDR displays, the need for video TMOs has become critical.

[0004] In the related art, most advanced TMOs are applicable to specific types of input content, either considering normal (average brightness) content in HDR format images, or focusing on dark and / or bright content, which has certain limitations and cannot provide high-quality visual quality output for different brightness levels, and cannot maintain the artistic impression of the original HDR image content. In addition, the current algorithms are not hardware-friendly and have a large amount of calculation. Therefore, how to overcome the above-mentioned technical defects of converting HDR format images to SDR format images is a technical problem that needs to be solved. Summary of the invention

[0005] The purpose of this application is to provide an image processing method, device, electronic device and storage medium, which can convert the entire HDR image into an SDR image, provide high-quality visual quality output for HDR images with different brightness conditions, and be hardware-friendly and require little calculation.

[0006] According to a first aspect of an embodiment of the present application, there is provided an image processing method, comprising:

[0007] Converting the original image into a light domain, and extracting a brightness channel for each pixel to obtain a first brightness value, wherein the original image is a high dynamic range (HDR) image;

[0008] The first brightness value is subjected to a perceptual quantization PQ transformation to obtain a corresponding second brightness value in the PQ domain; wherein the calculation formula of the PQ transformation used is: Among them, LHDR,PQ is the second brightness value, L light1 is the first brightness value, γ1 is the PQ transformation constant;

[0009] According to the specified global mapping curve equation, the second brightness value is converted into a third brightness value to obtain a first intermediate image; the third brightness value is a brightness value of a brightness channel of the first intermediate image;

[0010] Converting the first intermediate image from the PQ domain to the optical domain to obtain a second intermediate image;

[0011] According to the color coordinates of the second intermediate image and the color conversion matrix, the original color gamut is converted to the target color gamut to obtain a target image; wherein the original color gamut is the color gamut of the original image, the target color gamut is the color gamut of the target image, and the target image is a standard dynamic range SDR image.

[0012] In one embodiment, the γ1 is 4.

[0013] In one embodiment, converting the first intermediate image from the PQ domain to the optical domain to obtain a second intermediate image includes:

[0014] The first intermediate image is converted from the PQ domain to the optical domain using an inverse PQ transformation formula to obtain the second intermediate image, wherein the inverse PQ transformation formula is: L SDR,PQ is the third brightness value, L light2 is the fourth brightness value of the brightness channel of the pixel of the second intermediate image, and γ2 is the inverse PQ transformation constant.

[0015] In one embodiment, the γ2 is 0.25.

[0016] In one embodiment, after converting the second brightness value into a third brightness value according to the equation of the specified global mapping curve to obtain a first intermediate image and before converting the first intermediate image from the PQ domain to the light domain to obtain a second intermediate image, the method further includes:

[0017] For each color channel, adjusting the color of the first intermediate image according to a specified color adjustment equation, keeping the color hue unchanged and the brightness substantially unchanged, to obtain a third intermediate image;

[0018] After obtaining the third intermediate image, the third intermediate image is converted from the PQ domain to the optical domain to obtain the second intermediate image.

[0019] In one embodiment, the color adjustment equation is

[0020] Among them, C SDR,PQ is the color saturation of any color channel of the third intermediate image in the PQ domain, C HDR,PQ is the color saturation of any color channel of the first intermediate image in the PQ domain, L HDR,PQ is the second brightness value, L SDR,PQ is the third brightness value, α and β are two constants responsible for controlling the hue and color saturation of the color.

[0021] In one embodiment, the global mapping curve is a piecewise linear curve, including two endpoints, two dividing points and three linear curves, the three linear curves constitute a continuous curve, the two endpoints and the two dividing points divide the original image into a dark area, a normal area and a bright area; the three linear curves correspond to the dark area, the normal area and the bright area one by one;

[0022] The step of converting the second brightness value into a third brightness value according to the specified global mapping curve equation to obtain a first intermediate image comprises:

[0023] Determining the two endpoints and the two dividing points according to the equation of the global mapping curve;

[0024] According to the two endpoints and the two dividing points, the original image is divided into a dark area, a normal area and a bright area;

[0025] For each region of the original image, the second brightness value is converted into a corresponding third brightness value according to an equation of a global mapping curve to obtain the first intermediate image.

[0026] In one embodiment, the equation of the global mapping curve is:

[0027]

[0028] Among them, L HDR,PQ is the second brightness value, L SDR,PQ is the third brightness value, s1, s2 and s3 are the slopes of the three linear curves, a1, a2 and a3 are the intercepts of the three linear curves, x1 and x2 are the horizontal coordinates of the two dividing points, HDR min,PQ is the minimum value of the second brightness value, HDR max,PQ is the maximum value of the second brightness value.

[0029] In one embodiment, x1 is calculated by the following formula:

[0030] x1=P1×(HDR max,PQ -HDR min,PQ )+HDR min,PQ ;

[0031] x2 is calculated by the following formula:

[0032] x2=HDR max,PQ -P2×(HDR max,PQ -HDR min,PQ );

[0033] Among them, x1 is the horizontal coordinate of the boundary point between the dark area and the normal area, x2 is the horizontal coordinate of the boundary point between the normal area and the bright area, and P1 and P2 are two constants.

[0034] In one embodiment, P1 is 0.15 and P2 is 0.4.

[0035] In one embodiment, s1, s2, s3, a1, a2, and a3 are calculated as follows:

[0036] Determine the values ​​of y1 and y2 corresponding to x1 and x2 in the SDR domain;

[0037] Obtaining the coordinates of the two endpoints;

[0038] According to the coordinates of x1, x2, y1, y2 and the two endpoints, the values ​​of s1, s2, s3, a1, a2 and a3 are calculated.

[0039] In one embodiment, the values ​​of y1 and y2 are calculated by the following formula:

[0040] y1=P3×(SDR max,PQ -SDR min,PQ )+SDR min,PQ ;

[0041] y2=P4×(SDR max,PQ -SDR min,PQ )+SDR min,PQ ;

[0042] Among them, P3 and P4 are two constants, SDR min,PQ is the minimum luminance value of the target SDR display in the PQ domain, SDR max,PQ is the maximum luminance value of the target SDR display in the PQ domain.

[0043] In one embodiment, P3 is 0.1369 and P4 is 0.7518.

[0044] In one implementation, the global mapping curve is a gamma curve, and the equation of the gamma curve is as follows:

[0045]

[0046] Among them, LSDR,PQ is the third brightness value, L HDR,PQ is the second brightness value, c1 and c2 are constants;

[0047] The calculation formula of c2 is as follows:

[0048]

[0049] The calculation formula of c1 is as follows:

[0050]

[0051] Among them, HDR min,PQ is the minimum value of the second brightness value, HDR max,PQ is the maximum value of the second brightness value, SDR min,PQ is the minimum luminance value of the target SDR display in the PQ domain, SDR max,PQ is the maximum luminance value of the target SDR display in the PQ domain.

[0052] According to a second aspect of an embodiment of the present application, there is provided an image processing apparatus, including:

[0053] A first acquisition module is configured to convert an original image into a light domain and extract a brightness channel for each pixel to obtain a first brightness value, wherein the original image is a high dynamic range (HDR) image;

[0054] The second acquisition module is configured to perform a perceptual quantization PQ transformation on the first brightness value to obtain a corresponding second brightness value in the PQ domain; wherein the calculation formula of the PQ transformation used is: Among them, L HDR,PQ is the second brightness value, L light1 is the first brightness value, γ1 is the PQ transformation constant;

[0055] A first conversion module is configured to convert the second brightness value into a third brightness value according to a specified global mapping curve equation to obtain a first intermediate image; the third brightness value is a brightness value of a brightness channel of the first intermediate image;

[0056] A second conversion module is configured to convert the first intermediate image from the PQ domain to the optical domain to obtain a second intermediate image;

[0057] The third conversion module is configured to convert from the original color gamut to the target color gamut according to the color coordinates and the color conversion matrix of the second intermediate image to obtain a target image; wherein the original color gamut is the color gamut of the original image, the target color gamut is the color gamut of the target image, and the target image is a standard dynamic range SDR image.

[0058] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising a memory and a processor, wherein the memory is used to store a computer program executable by the processor; and the processor is used to execute the computer program in the memory to implement the above method.

[0059] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored, characterized in that when the executable computer program in the storage medium is executed by a processor, the above method can be implemented.

[0060] Compared with the prior art, the present application has the following beneficial effects: since the original image (HDR image) is converted to the light domain, and the brightness channel is extracted for each pixel to obtain a first brightness value, and then the first brightness value is subjected to perceptual quantization (PQ) conversion to obtain a corresponding second brightness value in the PQ domain, that is, the HDR image is converted to the perceptual domain, thus helping to maintain the overall visual impression of the frame. Moreover, the calculation formula of the adopted PQ conversion is It is hardware-friendly and can be easily implemented with limited resources. It has small computational complexity and can reduce computational costs.

[0061] Then, according to the equation of the specified global mapping curve, the second brightness value of each pixel is converted into a corresponding third brightness value to obtain a first intermediate image, wherein the third brightness value is the brightness value of the brightness channel of the first intermediate image. On the one hand, since a global mapping curve is used instead of a local mapping curve, the brightness of each pixel of the entire image of the original image can be converted, that is, all areas of different brightness levels in the HDR image are converted in brightness, rather than only performing local brightness conversion on areas of a certain type of specific brightness level. Therefore, the entire HDR image can be converted into an SDR image, providing high-quality visual quality output for HDR images of different brightness conditions. On the other hand, since the brightness of the entire image of the original image is adjusted as a whole using a global mapping curve, rapid changes in brightness that may cause flickering can be avoided, thereby avoiding flickering artifacts.

[0062] Then, the first intermediate image is converted from the PQ domain to the optical domain to obtain the second intermediate image, and according to the color coordinates and the color conversion matrix of the second intermediate image, it is converted from the original color domain to the target color domain to obtain the target image (SDR image).

[0063] To sum up, the technical solution provided by this application can realize the conversion of the entire HDR image into an SDR image, and provide high-quality visual quality output for HDR images with different brightness conditions. Moreover, it is hardware-friendly and can be easily implemented with limited resources, with small amount of calculation, which can reduce the computing cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 The figure is a flowchart of an image processing method according to an exemplary embodiment.

[0065] Figure 2 is a schematic diagram of a gamma curve according to an exemplary embodiment.

[0066] Figure 3 is a flowchart of an image processing method according to another exemplary embodiment.

[0067] Figure 4 is a schematic diagram of a piecewise linear curve according to an exemplary embodiment.

[0068] Figure 5 is a flowchart of an image processing method according to another exemplary embodiment.

[0069] Figure 6 is a block diagram of an image processing apparatus according to an exemplary embodiment.

[0070] Figure 7 is a block diagram of an image processing apparatus according to another exemplary embodiment.

[0071] Figure 8 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0072] Unless otherwise defined, the technical terms or scientific terms used in this specification and claims shall have the usual meanings understood by persons with ordinary skills in the technical field to which the invention belongs. The specific embodiments of the present invention will be described below in conjunction with the accompanying drawings. It should be noted that in the specific description of these embodiments, in order to provide a concise description, it is impossible for this specification to provide a detailed description of all the features of the actual embodiments. Without departing from the spirit and scope of the present invention, those skilled in the art may modify and replace the embodiments of the present invention, and the resulting embodiments are also within the scope of protection of the present invention.

[0073] In the related art, applying traditional TMO directly to HDR video is not an effective solution because it will produce visual artifacts such as flickering, brightness and color inconsistencies. On the other hand, TMO for generating high-quality SDR content is complex and requires manual intervention to set various parameters within the implementation method, so it is not suitable for real-time, hardware-friendly implementation. Another challenge when using existing methods is the inability to maintain the artistic impression of the original HDR content. It is crucial to maintain the overall visual impression of the input HDR content when converting to SDR format.

[0074] In order to solve the above technical problems, this application proposes an image processing method, device, electronic device and storage medium, which can realize the conversion of the entire HDR image into an SDR image, provide high-quality visual quality output for HDR images of different brightness conditions, and avoid flickering artifacts, brightness and color inconsistency and other problems during the image format conversion process. Moreover, it is hardware-friendly, can be easily implemented under limited resources, has a small amount of calculation, and can reduce the calculation cost.

[0075] Figure 1 is a flowchart of an image processing method according to an exemplary embodiment. The image processing method is performed in the RGB color space and can be applied to electronic devices with image processing functions such as image processors, display chips, and displays. Figure 1 , the image processing method may include the following steps:

[0076] Step 101: convert the original image into a light domain, and extract a brightness channel for each pixel to obtain a first brightness value, wherein the original image is a high dynamic range HDR image.

[0077] In this step, the input original image (HDR image) is converted from the HDR domain to the light domain, that is, the HDR format encoding value of the original image is converted into a grayscale value. Then, the brightness channel is extracted for each pixel to obtain a first brightness value.

[0078] In this embodiment, the color gamut of the original image, that is, the original color gamut is the BT.2020 color gamut. According to the BT.2020 standard, the formula for extracting the brightness channel from the three color channels of red, green, and blue is as follows:

[0079] L light1 =0.2627R+0.6780G+0.0593B;

[0080] Among them, R is the red channel, G is the green channel, B is the blue channel, and L is the light1 is the first brightness value.

[0081] In the related art, HDR formats include two formats: HLG10 or HDR10. The difference between these two formats is that their transfer functions are different. The HDR10 format uses the PQ function, while the HLG10 format uses the Hybrid Log-Gamma (HLG) function.

[0082] In this embodiment, when the format of the original image is HDR10, an inverse PQ transformation function is used to convert the original image into the optical domain, and when the format of the original image is HLG10, an inverse HLG transformation function is used to convert the original image into the optical domain.

[0083] Step 102, performing a perceptual quantization PQ transformation on the first brightness value to obtain a corresponding second brightness value in the PQ domain; wherein the calculation formula of the PQ transformation used is: Among them, L HDR,PQ is the second brightness value, L light1 is the first brightness value, and γ1 is the PQ transformation constant.

[0084] Among them, the PQ (Perceptual Quantizer) domain is a way of encoding and compressing video signals. The PQ domain belongs to the perceptual domain and is a perceptually uniform color space designed to more closely match the way the human visual system perceives color. The PQ domain uses a logarithmic transfer function, which can use the available dynamic range more efficiently. This makes it very suitable for displaying high dynamic range (HDR) video content.

[0085] Perceptual Quantization (PQ) aims to optimize the light intensity distribution associated with the HVS properties, converting physically linear values ​​into perceptually linear values. Perceptual linearity means that the human eye sees any intensity change at any brightness level in the same way (i.e., two consecutive brightness levels are just below the threshold that the HVS just notices). Perceptual linearity is fundamental to the implementation because this allows to identify what the human eye can see and map that information to a higher dynamic range in the best way. Therefore, the first luminance value of the luminance channel of the input HDR image is first transferred to the PQ domain, which allows the luminance values ​​of the HDR image and the luminance values ​​of the SDR image to be in the same domain during the mapping process. This allows us to work in the perceptual domain, where adjacent luminance values ​​correspond to differences that are perceptible to the human eye.

[0086] In this embodiment, the first brightness value is subjected to PQ transformation to obtain a corresponding second brightness value in the PQ domain.

[0087] In this embodiment, a perceptual quantizer may be used to perform a PQ transform on the first brightness value. A perceptual quantizer is a transfer function that maps the linear light intensity of a display device to a nonlinear signal to better match the characteristics of a human visual system (HVS). The function of the perceptual quantizer is to optimize the distribution of light intensity so as to most efficiently utilize the available dynamic range and provide the best reproduction of the original image.

[0088] In the related art, the calculation formula of PQ transformation is:

[0089]

[0090] Among them, L light1 is the first brightness value, L HDR,PQ is the second brightness value. c1, c2, c3, m1 and m2 are constants and can be set to 0.8359, 18.8515, 18.6875, 0.1593 and 78.8437 respectively.

[0091] However, the calculation formula of the above PQ transform is computationally expensive to implement in hardware and requires a lot of resources to implement.

[0092] In order to overcome the above challenges, in this embodiment, the calculation formula of the PQ transformation adopted is:

[0093]

[0094] Among them, L HDR,PQ is the second brightness value, L light1 is the first brightness value, and γ1 is the PQ transformation constant.

[0095] In this embodiment, the calculation formula of the PQ transformation is simple in form, with few parameters, no manual intervention is required to set various parameters, the amount of calculation is small, the calculation cost implemented in hardware is relatively low, it is hardware-friendly, can be easily implemented with limited resources, and can be extended to real-time applications.

[0096] In this embodiment, γ1 is 4. According to extensive simulation experiments, setting γ1 to 4 achieves high visual quality results in both subjective and objective evaluations.

[0097] Step 103: convert the second brightness value into a third brightness value according to the specified global mapping curve equation to obtain a first intermediate image; the third brightness value is the brightness value of the brightness channel of the first intermediate image.

[0098] Flickering is a common problem that can occur during tone mapping, especially when using a local tone mapping algorithm. This is because the local algorithm independently adjusts the brightness of a single pixel or a small group of pixels, which can cause rapid changes in brightness across the entire image. These brightness changes can be particularly noticeable in high-contrast areas and can be distracting and visually jarring. To avoid this problem, the global mapping algorithm (the equation for the global mapping curve) in this application uses a different approach. Instead of adjusting the brightness of a single pixel, it takes the average of the brightness values ​​for the entire image. Because the brightness of the entire image is adjusted as a whole, this approach avoids rapid changes in brightness that can cause flickering.

[0099] In this embodiment, the global mapping curve can be determined according to the curve identifier input by the user. For example, if the curve identifier is 00, the specified global mapping curve is a gamma curve, and if the curve identifier is 01, the specified global mapping curve is a piecewise linear curve.

[0100] In this embodiment, the global mapping curve is a gamma curve, and the equation of the gamma curve is as follows:

[0101]

[0102] Among them, L SDR,PQ is the third brightness value, L HDR,PQ is the second brightness value, c1 and c2 are constants.

[0103] In this embodiment, c2 may be calculated using the following formula:

[0104]

[0105] Then, calculate c1 using the following formula:

[0106]

[0107] Among them, HDR min,PQ is the minimum value of the second brightness value, HDR max,PQ is the maximum value of the second brightness value, SDR min,PQ is the minimum luminance value of the target SDR display in the PQ domain, SDR max,PQ is the maximum luminance value of the target SDR display in the PQ domain.

[0108] In this embodiment, the gamma curve C1 used is as follows: Figure 2 shown.

[0109] In this embodiment, the global mapping curve adopts a gamma curve, which has a simple mathematical form, low computational complexity, and can generate a high-quality SDR image. Moreover, the generated SDR image has more details in brighter areas than in dark areas. If we need to generate an SDR image with more details in brighter areas than in dark areas, it is recommended to use a gamma curve.

[0110] In this embodiment, the mapping curve is constructed by considering the maximum and minimum brightness values ​​of the target SDR display. In this way, the image processing method can generate content that looks pleasing on all types of SDR displays. Therefore, the algorithm provided in this application is display-adaptive and can generate SDR image content that matches the functions of any target SDR display. Therefore, the TMO proposed in this article is display-adaptive and can generate SDR content that matches the functions of any target display.

[0111] Step 104: convert the first intermediate image from the PQ domain to the optical domain to obtain a second intermediate image.

[0112] In this embodiment, the first intermediate image needs to be converted from the PQ domain to the optical domain to display the image content.

[0113] In the related art, the following inverse PQ transformation formula is used to convert the first intermediate image from the PQ domain to the optical domain:

[0114]

[0115] Among them, L light2 is a fourth brightness value of the brightness channel of the pixel of the second intermediate image.

[0116] However, the calculation formula of the above inverse PQ transform is very expensive to implement in hardware and requires a lot of resources to implement. In order to overcome the above challenges, in this embodiment, the calculation formula of the inverse PQ transform is:

[0117]

[0118] Among them, L SDR,PQ is the third brightness value, L light2 is the fourth brightness value of the brightness channel of the pixel of the second intermediate image, and γ2 is the inverse PQ transformation constant.

[0119] In this embodiment, the calculation formula of the inverse PQ transform is simple in form, has few parameters, and has a small amount of calculation. The calculation cost of implementing it in hardware is relatively low, it is hardware-friendly, and can be easily implemented with limited resources.

[0120] In this embodiment, γ2 is 0.25. According to extensive simulation experiments, setting γ2 to 0.25 can achieve high visual quality results in both subjective and objective evaluations.

[0121] Step 105, converting from the original color gamut to the target color gamut according to the color coordinates of the second intermediate image and the color conversion matrix to obtain a target image; wherein the original color gamut is the color gamut of the original image, the target color gamut is the color gamut of the target image, and the target image is a standard dynamic range SDR image.

[0122] In this embodiment, the color gamut of the target image, that is, the target color gamut is BT.709. Because BT.709 is specially designed for SDR images, converting the image to BT.709 can ensure that the generated content displays correct colors.

[0123] Among them, converting an image from the BT.2020 color gamut to the BT.709 color gamut involves converting the color coordinates of a given image or video from one color space to another.

[0124] In one embodiment, a color gamut conversion can be performed using a 3*3 color conversion matrix that maps colors in the source color space (BT.2020) to colors in the target color space (BT.709). The process typically starts by converting an image or video from an RGB color space to the native color representation of the color space. The color conversion matrix is ​​then applied to the color coordinates of the image to convert them to the new color space. The matrix for converting from the BT.2020 color gamut to the BT.709 color gamut can be expressed as:

[0125] R709=T11×R2020+T12×G2020+T13×B2020

[0126] G709=T21×R2020+T22×G2020+T23×B2020

[0127] B709=T31×R2020+T32×G2020+T33×B2020

[0128] Where R, G, and B represent the red, green, and blue channels, respectively. R709, G709, and B709 are the color coordinates of the target image in the BT.709 color gamut, and R2020, G2020, and B2020 are the color coordinates of the second intermediate image in the BT.2020 color gamut. T11, T12, T13, T21, T22, T23, T31, T32, and T33 are conversion coefficients.

[0129] It should be noted that this step is quite complicated and requires very high precision. This color conversion matrix is ​​only an approximation. If accurate conversion is to be performed, a three-dimensional color lookup table (3D-CLUT) needs to be used.

[0130] In this embodiment, since the original image (HDR image) is converted to the optical domain, and the brightness channel is extracted for each pixel to obtain a first brightness value, and then the first brightness value is subjected to perceptual quantization (PQ) conversion to obtain a corresponding second brightness value in the PQ domain, that is, the HDR image is converted to the perceptual domain, it helps to maintain the overall visual impression of the framework. Moreover, the calculation formula of the adopted PQ conversion is It is hardware-friendly and can be easily implemented with limited resources. It has small computational complexity and can reduce computational costs.

[0131] Then, according to the equation of the specified global mapping curve, the second brightness value of each pixel is converted into a corresponding third brightness value to obtain a first intermediate image, wherein the third brightness value is the brightness value of the brightness channel of the first intermediate image. On the one hand, since a global mapping curve is used instead of a local mapping curve, the brightness of each pixel of the entire image of the original image can be converted, that is, all areas of different brightness levels in the HDR image are converted in brightness, rather than only performing local brightness conversion on areas of a certain type of specific brightness level. Therefore, the entire HDR image can be converted into an SDR image, providing high-quality visual quality output for HDR images of different brightness conditions. On the other hand, since the brightness of the entire image of the original image is adjusted as a whole using a global mapping curve, rapid changes in brightness that may cause flickering can be avoided, thereby avoiding flickering artifacts.

[0132] Then, the first intermediate image is converted from the PQ domain to the optical domain to obtain the second intermediate image, and according to the color coordinates and the color conversion matrix of the second intermediate image, it is converted from the original color domain to the target color domain to obtain the target image (SDR image).

[0133] In summary, the technical solution provided in this application can convert the entire HDR image into an SDR image, provide high-quality visual quality output for HDR images with different brightness conditions, and is hardware-friendly, can be easily implemented with limited resources, has a small amount of calculation, and can reduce computing costs.

[0134] Figure 3 is a flow chart of an image processing method according to another exemplary embodiment. Different from the above embodiment, in this embodiment, after converting the second brightness value into the third brightness value according to the equation of the specified global mapping curve to obtain the first intermediate image and before converting the first intermediate image from the PQ domain to the light domain to obtain the second intermediate image, the first intermediate image is color-adjusted according to the specified color adjustment equation for each color channel to keep the color hue unchanged and the brightness basically unchanged. Please refer to Figure 3 , the image processing method may include the following steps:

[0135] Step 301 : converting the original image into a light domain, and extracting a brightness channel for each pixel to obtain a first brightness value, wherein the original image is a high dynamic range HDR image.

[0136] This step is similar to the above-mentioned step 101 and will not be described again here.

[0137] Step 302, performing a perceptual quantization PQ transformation on the first brightness value to obtain a corresponding second brightness value in the PQ domain; wherein the calculation formula of the PQ transformation used is: Among them, L HDR,PQ is the second brightness value, L light1 is the first brightness value, and γ1 is the PQ transformation constant.

[0138] This step is similar to the above-mentioned step 102 and will not be described again here.

[0139] Step 303: According to the specified global mapping curve equation, the second brightness value is converted into a third brightness value to obtain a first intermediate image; the third brightness value is the brightness value of the brightness channel of the first intermediate image.

[0140] In this embodiment, the global mapping curve is a piecewise linear curve. Figure 4As shown, the global mapping curve C2 includes two endpoints D1, D2, two dividing points D3, D4 and three linear curves C21, C22, C23. The three linear curves C21, C22, C23 form a continuous curve. The two endpoints D1, D2 and the two dividing points D3, D4 divide the original image into a dark area, a normal area and a bright area. The three linear curves correspond to the dark area, the normal area and the bright area one by one. Specifically, the linear curve C21 is the curve corresponding to the dark area, the linear curve C22 is the curve corresponding to the normal area, and the linear curve C23 is the curve corresponding to the bright area. x1 and x2 are the horizontal coordinates of the two dividing points D3, D4, y1 and y2 are the vertical coordinates of the two dividing points D3, D4, and s1, s2 and s3 are the slopes of the three linear curves C21, C22, C23 respectively. In other embodiments, the number of segments in the global mapping curve can be extended to more than three.

[0141] In this embodiment, the equation of the global mapping curve is as follows:

[0142]

[0143] Among them, L HDR,PQ is the second brightness value, L SDR,PQ is the third brightness value, s1, s2 and s3 are the slopes of the three linear curves, a1, a2 and a3 are the intercepts of the three linear curves, x1 and x2 are the horizontal coordinates of the two dividing points, HDR min,PQ is the minimum value of the second brightness value, HDR max,PQ is the maximum value of the second brightness value.

[0144] In this embodiment, the above-mentioned piecewise linear curve is used as the equation of the specified global mapping curve to convert the second brightness value into the third brightness value to obtain the first intermediate image. The specific method for performing global mapping is as follows: first, according to the equation C2 of the global mapping curve, two endpoints D1, D2 and two dividing points D3, D4 are determined, and then, according to the two endpoints D1, D2 and the two dividing points D3, D4, the original image is divided into a dark area, a normal area and a bright area, and then, for each area of ​​the original image, the second brightness value is converted into a corresponding third brightness value according to the equation of the global mapping curve. For the dark area of ​​the original image, the second brightness value is converted into a corresponding third brightness value according to the linear curve C21, for the normal area of ​​the original image, the second brightness value is converted into a corresponding third brightness value according to the linear curve C22, and for the bright area of ​​the original image, the second brightness value is converted into a corresponding third brightness value according to the linear curve C23. Doing so helps to convert each area of ​​the HDR image into their corresponding areas in the SDR domain, while helping to maintain the overall artistic impression. Using a piecewise linear curve can make the output image more balanced, thereby achieving the best balance between the overall brightness and contrast of the image.

[0145] In this embodiment, the method for generating the piecewise linear curve is as follows:

[0146] (1) Determine the segmentation method.

[0147] In this step, first, the values ​​of x1 and x2 are calculated to segment the image, and the original image is divided into a dark area, a normal area, and a bright area.

[0148] x1 is calculated by the following formula:

[0149] x1=P1×(HDR max,PQ -HDR min,PQ )+HDR min,PQ ;

[0150] x2 is calculated by the following formula:

[0151] x2=HDR max,PQ -P2×(HDR max,PQ -HDR min,PQ );

[0152] Among them, x1 is the horizontal coordinate of the boundary point between the dark area and the normal area, x2 is the horizontal coordinate of the boundary point between the normal area and the bright area, and P1 and P2 are two constants.

[0153] In this embodiment, P1 is 0.15 and P2 is 0.4. According to extensive research, setting P1 and P2 to 0.15 and 0.4 respectively can achieve good results in segmenting the HDR image into three brightness regions.

[0154] After calculating the values ​​of x1 and x2, determine the values ​​y1 and y2 corresponding to x1 and x2 in the SDR domain. The values ​​of y1 and y2 can be calculated using the following formula:

[0155] y1=P3×(SDR max,PQ -SDR min,PQ )+SDR min,PQ ;

[0156] y2=P4×(SDR max,PQ -SDR min,PQ )+SDR min,PQ ;

[0157] Among them, P3 and P4 are two constants, SDR min,PQ is the minimum luminance value of the target SDR display in the PQ domain, SDR max,PQ is the maximum luminance value of the target SDR display in the PQ domain.

[0158] In this embodiment, P3 is 0.1369 and P4 is 0.7518. According to extensive simulations, setting P3 and P4 to 0.1369 and 0.7518 respectively will produce an SDR image with good visual quality.

[0159] Next, the coordinates of the two endpoints D1 and D2 are obtained.

[0160] Next, the values ​​of s1, s2, s3, a1, a2 and a3 are calculated based on the coordinates of x1, x2, y1, y2 and the two end points D1 and D2.

[0161] (2) Generate piecewise linear curve

[0162] In this embodiment, the above-mentioned piecewise linear curve is generated according to the values ​​of s1, s2, s3, a1, a2 and a3, that is, the equation of the global mapping curve C2 is obtained.

[0163] In this embodiment, the proposed TMO works in the perceptual domain to take into account the sensitivity of the human eye to brightness changes in different regions of the scene (i.e. dark, normal and bright). This helps the TMO to process each region differently based on how the eye perceives them.

[0164] In order to preserve the overall impression of the original HDR image content, the proposed TMO uses a novel segmentation method to divide the HDR image into dark, normal and bright regions. This not only enables the overall brightness of each region to be preserved, but also enables the TMO to process all types of HDR images, which is a problem that most existing TMOs cannot solve.

[0165] Step 304 , for each color channel, color adjustment is performed on the first intermediate image according to a specified color adjustment equation, so as to keep the color hue unchanged and the brightness substantially unchanged, thereby obtaining a third intermediate image.

[0166] An important consideration in designing an efficient TMO is to maintain color accuracy between the input HDR image and the generated SDR image. This is a challenge because color shifts may occur when trying to compress brightness levels from HDR to SDR. Therefore, an effective color adjustment scheme is very important when designing a TMO. The goal of a color adjustment method is to maintain the hue of the color during the transformation while making the change in color brightness negligible.

[0167] In this embodiment, in order to ensure the accuracy of the color of the generated image, the color adjustment equation used is as follows:

[0168]

[0169] Among them, C SDR,PQ is the color saturation of any color channel (red, green, or blue) of the third intermediate image in the PQ domain, C HDR,PQ is the color saturation of any color channel of the first intermediate image in the PQ domain, L HDR,PQ is the second brightness value, L SDR,PQ is the third brightness value, α and β are two constants responsible for controlling the hue and color saturation of the color.

[0170] In this embodiment, since the color adjustment is performed according to the specified color adjustment equation, the color hue is kept unchanged and the brightness is kept basically unchanged, so the color accuracy between the input HDR image and the output SDR image can be guaranteed.

[0171] Additionally, based on extensive simulations, the above color adjustment equation preserves the hue of colors while having a negligible impact on the brightness when mapping an HDR image to an SDR image.

[0172] In this embodiment, by performing color adjustment in the perceptual domain, the image processing method retains the color hue of the HDR image and generates the color of the SDR image that is very close to the color of the HDR image.

[0173] Step 305: convert the third intermediate image from the PQ domain to the optical domain to obtain a second intermediate image.

[0174] This step is similar to the above-mentioned step 104 and will not be described again here.

[0175] Step 306, converting from the original color gamut to the target color gamut according to the color coordinates of the second intermediate image and the color conversion matrix to obtain a target image; wherein the original color gamut is the color gamut of the original image, the target color gamut is the color gamut of the target image, and the target image is a standard dynamic range SDR image.

[0176] This step is similar to the above-mentioned step 105 and will not be described again here.

[0177] In this embodiment, after converting the second brightness value into the third brightness value according to the equation of the specified global mapping curve to obtain the first intermediate image and before converting the first intermediate image from the PQ domain to the light domain to obtain the second intermediate image, the first intermediate image is color-adjusted according to the specified color adjustment equation for each color channel to keep the color hue unchanged and the brightness basically unchanged. By performing color adjustment in the perceptual domain, the image processing method retains the color hue of the HDR image and generates the color of the SDR image that is very close to the color of the HDR image, while making the change in color brightness negligible during the color adjustment process.

[0178] Figure 5 5 is a flowchart of an image processing method according to another exemplary embodiment, and is also a principle block diagram of the TMO proposed in this article. The main part of the method is the part framed by the dotted frame 51. Figure 5 , the image processing method may include the following steps:

[0179] (1) Extract the brightness channel

[0180] This step is the same as step 101 and will not be repeated here.

[0181] (2) PQ transformation

[0182] This step is the same as step 102 and will not be repeated here.

[0183] (3) Select the mapping curve

[0184] In this embodiment, the mapping curve is a global mapping curve, and the global mapping curve includes a gamma curve and a piecewise linear curve. The electronic device can select a mapping curve according to a curve identifier input by a user. For example, if the curve identifier input by the user is 00, the specified global mapping curve is a gamma curve, and if the curve identifier input by the user is 01, the specified global mapping curve is a piecewise linear curve.

[0185] (4) When the user inputs the curve identifier of the gamma curve, the gamma curve is generated and globally mapped.

[0186] In this embodiment, when the curve identifier input by the user is a curve identifier of a gamma curve, a gamma curve is generated and global mapping is performed.

[0187] (5) When the user inputs the curve identifier of a piecewise linear curve, determine the curve segmentation method

[0188] In this embodiment, when the curve identifier input by the user is a curve identifier of a piecewise linear curve, a curve segmentation method is determined.

[0189] This step is the same as the content of determining the curve segmentation method in the above step 303, and will not be repeated here.

[0190] (6) Generate piecewise linear curves and perform global mapping

[0191] In this embodiment, after the curve segmentation method is determined, a piecewise linear curve is generated and global mapping is performed.

[0192] This step is the same as the content of the global mapping method in the above step 303, and will not be repeated here.

[0193] (7) Color adjustment

[0194] This step is the same as step 304 and will not be described again here.

[0195] (8) Reverse PQ conversion

[0196] This step is the same as step 305 and will not be described again here.

[0197] (9) Inverse color gamut conversion

[0198] This step is the same as step 306 and will not be described again here.

[0199] In this embodiment, an HDR image can be converted into a standard dynamic range (SDR) image to achieve high visual quality at all brightness levels. Moreover, it is hardware-friendly and can be implemented in hardware with limited resources.

[0200] Figure 6 FIG. 1 is a block diagram of an image processing apparatus according to an exemplary embodiment. Figure 6 As shown, in this embodiment, the image processing device includes:

[0201] A first acquisition module 61 is configured to convert the original image into a light domain and extract a brightness channel for each pixel to obtain a first brightness value, wherein the original image is a high dynamic range HDR image;

[0202] The second acquisition module 62 is configured to perform a perceptual quantization PQ transformation on the first brightness value to obtain a corresponding second brightness value in the PQ domain; wherein the calculation formula of the PQ transformation used is: Among them, L HDR,PQ is the second brightness value, Llight1 is the first brightness value, γ1 is the PQ transformation constant;

[0203] A first conversion module 63 is configured to convert the second brightness value into a third brightness value according to a specified global mapping curve equation to obtain a first intermediate image; the third brightness value is a brightness value of a brightness channel of the first intermediate image;

[0204] A second conversion module 64 is configured to convert the first intermediate image from the PQ domain to the optical domain to obtain a second intermediate image;

[0205] The third conversion module 65 is configured to convert from the original color gamut to the target color gamut according to the color coordinates of the second intermediate image and the color conversion matrix to obtain a target image; wherein the original color gamut is the color gamut of the original image, the target color gamut is the color gamut of the target image, and the target image is a standard dynamic range SDR image.

[0206] In one embodiment, Figure 7 As shown, the image processing device further includes:

[0207] The color adjustment module 66 is configured to perform color adjustment on the first intermediate image according to a specified color adjustment equation for each color channel, so as to keep the color hue unchanged and the brightness substantially unchanged, thereby obtaining a third intermediate image;

[0208] The second conversion module 64 is further configured to convert the third intermediate image from the PQ domain to the optical domain to obtain a second intermediate image.

[0209] An embodiment of the present application also proposes an electronic device, including a processor and a memory; the memory is used to store a computer program executable by the processor; the processor is used to execute the computer program in the memory to implement the image processing method of any of the above embodiments.

[0210] The embodiments of the present application further provide a computer-readable storage medium, which can implement the image processing method of any of the above embodiments when the executable computer program in the storage medium is executed by a processor.

[0211] Regarding the device in the above embodiment, the specific manner in which the processor performs the operation has been described in detail in the embodiment of the method, and will not be elaborated here.

[0212] Figure 8 8 is a block diagram of an electronic device according to an exemplary embodiment. For example, the electronic device 800 may be provided as a server. Figure 8, the device 800 includes a processing component 822, which further includes one or more processors, and a memory resource represented by a memory 832 for storing instructions that can be executed by the processing component 822, such as an application. The application stored in the memory 832 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 822 is configured to execute instructions to perform the above-mentioned method for image processing.

[0213] The device 800 may also include a power component 826 configured to perform power management of the device 800, a wired or wireless network interface 850 configured to connect the device 800 to a network, and an input / output (I / O) interface 858. The device 800 may operate based on an operating system stored in the memory 832, such as Windows Server™, MacOS X™, Unix™, Linux™, FreeBSD™, or the like.

[0214] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 832 including instructions, which can be executed by the processing component 822 of the device 800 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0215] In the present invention, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance. The term "plurality" refers to two or more than two, unless otherwise clearly defined.

[0216] The above description of the embodiments is to facilitate those of ordinary skill in the art to understand and apply the present application. It is obvious that those familiar with the art can easily make various modifications to these embodiments and apply the general principles described herein to other embodiments without having to pay creative labor. Therefore, the present application is not limited to the embodiments herein, and improvements and modifications made by those skilled in the art based on the contents disclosed in the present application without departing from the scope and spirit of the present application are within the scope of the present application.

Claims

1. An image processing method, characterized in that: include: Converting the original image into a light domain, and extracting a brightness channel for each pixel to obtain a first brightness value, wherein the original image is a high dynamic range (HDR) image; The first brightness value is subjected to a perceptual quantization PQ transformation to obtain a corresponding second brightness value in the PQ domain; wherein the calculation formula of the PQ transformation used is: Among them, L HDR,PQ is the second brightness value, L light1 is the first brightness value, γ1 is the PQ transformation constant; According to the specified global mapping curve equation, the second brightness value is converted into a third brightness value to obtain a first intermediate image; the third brightness value is a brightness value of a brightness channel of the first intermediate image; Converting the first intermediate image from the PQ domain to the optical domain to obtain a second intermediate image; According to the color coordinates of the second intermediate image and the color conversion matrix, convert from the original color gamut to the target color gamut to obtain a target image; wherein the original color gamut is the color gamut of the original image, the target color gamut is the color gamut of the target image, and the target image is a standard dynamic range SDR image; The global mapping curve is a piecewise linear curve, including two endpoints, two dividing points and three linear curves, the three linear curves constitute a continuous curve, the two endpoints and the two dividing points divide the original image into a dark area, a normal area and a bright area; the three linear curves correspond to the dark area, the normal area and the bright area one by one; The step of converting the second brightness value into a third brightness value according to the specified global mapping curve equation to obtain a first intermediate image comprises: Determining the two endpoints and the two dividing points according to the equation of the global mapping curve; According to the two endpoints and the two dividing points, the original image is divided into a dark area, a normal area and a bright area; For each region of the original image, convert the second brightness value into a corresponding third brightness value according to an equation of a global mapping curve to obtain the first intermediate image; The equation of the global mapping curve is: Among them, L HDR,PQ is the second brightness value, L SDR,PQ is the third brightness value, s1, s2 and s3 are the slopes of the three linear curves, a1, a2 and a3 are the intercepts of the three linear curves, x1 and x2 are the horizontal coordinates of the two dividing points, HDR min,PQ is the minimum value of the second brightness value, HDR max,PQ is the maximum value of the second brightness value; x1 is calculated by the following formula: x1=P1×(HDR max，PQ -HDR min，PQ )+HDR min，PQ ; x2 is calculated by the following formula: x2=HDR max，PQ -P2×(HDR max，PQ -HDR min，PQ ); Among them, x1 is the horizontal coordinate of the boundary point between the dark area and the normal area, x2 is the horizontal coordinate of the boundary point between the normal area and the bright area, and P1 and P2 are two constants.

2. The image processing method according to claim 1, characterized in that: P1 is 0.15 and P2 is 0.

4.

3. The image processing method according to claim 1, characterized in that: The calculation methods of s1, s2, s3, a1, a2 and a3 are as follows: Determine the values ​​of y1 and y2 corresponding to x1 and x2 in the SDR domain; Obtaining the coordinates of the two endpoints; According to the coordinates of x1, x2, y1, y2 and the two endpoints, the values ​​of s1, s2, s3, a1, a2 and a3 are calculated.

4. The image processing method according to claim 3, characterized in that: The values ​​of y1 and y2 are calculated using the following formula: y1=P3×(SDR max，PQ -SDR min，PQ )+SDR min，PQ ; y2=P4×(SDR max，PQ -SDR min，PQ )+SDR min，PQ ; Among them, P3 and P4 are two constants, SDR min，PQ is the minimum luminance value of the target SDR display in the PQ domain, SDR max，PQ is the maximum luminance value of the target SDR display in the PQ domain.

5. The image processing method according to claim 3, characterized in that: P3 is 0.1369 and P4 is 0.7518.

6. An image processing device, characterized in that: include: A first acquisition module is configured to convert an original image into a light domain and extract a brightness channel for each pixel to obtain a first brightness value, wherein the original image is a high dynamic range (HDR) image; The second acquisition module is configured to perform a perceptual quantization PQ transformation on the first brightness value to obtain a corresponding second brightness value in the PQ domain; wherein the calculation formula of the PQ transformation used is: Among them, L HDR，PQ is the second brightness value, L light1 is the first brightness value, γ1 is the PQ transformation constant; A first conversion module is configured to convert the second brightness value into a third brightness value according to a specified global mapping curve equation to obtain a first intermediate image; the third brightness value is a brightness value of a brightness channel of the first intermediate image; A second conversion module is configured to convert the first intermediate image from the PQ domain to the optical domain to obtain a second intermediate image; a third conversion module, configured to convert from an original color gamut to a target color gamut according to the color coordinates of the second intermediate image and a color conversion matrix, to obtain a target image; wherein the original color gamut is the color gamut of the original image, the target color gamut is the color gamut of the target image, and the target image is a standard dynamic range SDR image; The global mapping curve is a piecewise linear curve, including two endpoints, two dividing points and three linear curves, the three linear curves constitute a continuous curve, the two endpoints and the two dividing points divide the original image into a dark area, a normal area and a bright area; the three linear curves correspond to the dark area, the normal area and the bright area one by one; The first conversion module is further configured to determine the two endpoints and the two dividing points according to the equation of the global mapping curve, divide the original image into a dark area, a normal area and a bright area according to the two endpoints and the two dividing points, and for each area of ​​the original image, convert the second brightness value into a corresponding third brightness value according to the equation of the global mapping curve to obtain the first intermediate image; The equation of the global mapping curve is: Among them, L HDR，PQ is the second brightness value, L SDR，PQ is the third brightness value, s1, s2 and s3 are the slopes of the three linear curves, a1, a2 and a3 are the intercepts of the three linear curves, x1 and x2 are the horizontal coordinates of the two dividing points, HDR min，PQ is the minimum value of the second brightness value, HDR max，PQ is the maximum value of the second brightness value; x1 is calculated by the following formula: x1=P1×(HDR max，PQ -HDR min，PQ )+HDR min，PQ ; x2 is calculated by the following formula: x2=HDR max，PQ -P2×(HDR max，PQ -HDR min，PQ ); Among them, x1 is the horizontal coordinate of the boundary point between the dark area and the normal area, x2 is the horizontal coordinate of the boundary point between the normal area and the bright area, and P1 and P2 are two constants.

7. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory is used to store a computer program executable by the processor; and the processor is used to execute the computer program in the memory to implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the executable computer program in the storage medium is executed by a processor, the method according to any one of claims 1 to 5 can be implemented.

Images