Image enhancement method and device

Abstract

Disclosed are an image enhancement method and device. First, an image having a first color gamut and / or a first brightness dynamic range can be subjected to image enhancement processing within a first color gamut, such as BT709 and / or a first brightness dynamic range, such as SDR, to obtain a first image. The first image having the first color gamut and / or the first brightness dynamic range is converted into a second image having a second color gamut, such as BT2020 and / or a second brightness dynamic range, such as HDR, and image enhancement processing is performed within the second color gamut and / or the second brightness dynamic range. The second color gamut is wider than the first color gamut, and the upper limit of the second brightness dynamic range is higher than the upper limit of the first brightness dynamic range. Then, the second image is converted into a third image having the first color gamut and / or the first brightness dynamic range. Thus, by utilizing a larger image adjustable space within a wide color gamut and / or a high brightness dynamic range, a better image enhancement effect can be obtained, and the image quality can be significantly improved.

Description

Technical Field

[0001] The present disclosure relates to image processing, and in particular to an enhancement scheme for image color, brightness, contrast, etc. Background Art

[0002] With the rapid development of image capture and image display equipment, as well as the geometric increase in network bandwidth, the image industry has flourished, making images an important information carrier today.

[0003] Given the increasing importance of image content processing and the increasing proportion of image consumption in our lives, in order to enhance users' image viewing experience, it is imperative to provide a framework and algorithm that can significantly improve image quality on various terminal devices.

[0004] On the other hand, as the application scenarios of images in life gradually increase, in order to meet users' increasingly higher demands for image quality, image processing algorithms need to improve the subjective visual effects of images from multiple dimensions.

[0005] However, most of the current stock images, as well as newly generated images, are still standard brightness dynamic range (SDR) images under the standard color gamut (BT709). Moreover, SDR playback devices with standard color gamut still dominate the market.

[0006] Therefore, it is very necessary to design an enhancement scheme for SDR images in the standard color gamut to meet users' demand for high image quality.

[0007] According to the visual characteristics of the human eye, people are most sensitive to changes in image color, brightness, contrast, etc. From the perspective of improving the visual experience, optimizing the brightness level, contrast and color of the image can more effectively improve the overall visual experience of the image.

[0008] Currently, the industry / academia has made many attempts to enhance brightness, contrast, color, etc.

[0009] Brightness and contrast adjustment schemes generally include: global or local contrast adjustment based on histograms; contrast adjustment based on deep learning.

[0010] Color enhancement solutions usually include: color balancing, color adjustment, saturation adjustment, etc. The methods used include those based on traditional algorithms and those based on deep learning.

[0011] The common disadvantage of these solutions is that they usually process on a small color gamut (such as BT601 color gamut, BT709 color gamut) and a standard dynamic range (SDR, 0-100nits). Limited by the small color gamut and low dynamic range, this type of adjustment solution itself has certain limitations, mainly because the effect after processing has limited room for improvement, or there are defects after improvement. In short, the visual effect is far inferior to the feeling of HDR images.

[0012] Therefore, there is still a need for a brightness, contrast and color enhancement solution with superior effects for standard color gamut SDR images to meet users' current demand for high-quality image quality improvement. Summary of the invention

[0013] A technical problem to be solved by the present disclosure is to provide an improved image enhancement method, which can enhance images with lower color gamut and / or lower brightness dynamic range, thereby significantly improving image quality.

[0014] According to a first aspect of the present disclosure, there is provided an image enhancement method, comprising: converting a first image having a first color gamut and / or a first brightness dynamic range into a second image having a second color gamut and / or a second brightness dynamic range, and performing image enhancement processing in the second color gamut and / or the second brightness dynamic range, the second color gamut being wider than the first color gamut, and / or an upper limit of the second brightness dynamic range being higher than an upper limit of the first brightness dynamic range; and converting the second image into a third image having the first color gamut and / or the first brightness dynamic range.

[0015] Optionally, the method may further include: performing image enhancement processing on an image having a first color gamut and / or a first brightness dynamic range in the first color gamut and / or a first brightness dynamic range to obtain a first image.

[0016] Optionally, the step of performing image enhancement processing on an image having a first color gamut and / or a first brightness dynamic range includes: based on a local brightness histogram and / or a global brightness histogram, performing brightness estimation in the RGB color space of the image to obtain brightness distribution information of the image, the brightness distribution information including distribution information of multiple brightness areas of the image, the multiple brightness areas respectively having non-overlapping brightness value ranges; and based on the brightness distribution information, using multiple mapping curves corresponding to multiple brightness value ranges and / or the global image, enhancing the contrast of multiple brightness areas and / or the global image.

[0017] Optionally, the step of performing image enhancement processing on the image having the first color gamut and / or the first brightness dynamic range further includes: performing color saturation enhancement processing on the image for the purpose of protecting skin color.

[0018] Optionally, the step of performing color saturation enhancement processing on the contrast-enhanced image for the purpose of skin color protection includes: calculating skin color similarity for the contrast-enhanced image; determining, based on the skin color similarity, the skin color probability that the color of at least one area of ​​the image is skin color; and adjusting the color saturation of at least one area of ​​the image based on the skin color probability to achieve the purpose of skin color protection.

[0019] Optionally, the steps of converting a first image having a first color gamut and / or a first brightness dynamic range into a second image having a second color gamut and / or a second brightness dynamic range, and performing image enhancement processing within the second color gamut and / or the second brightness dynamic range include: using an electro-optical conversion function corresponding to the first brightness dynamic range to convert a nonlinear brightness signal value of the first image into a linear brightness signal value to obtain a linear brightness image; converting the linear brightness image from the first color gamut to the second color gamut to obtain a second color gamut image; converting the brightness dynamic range of the second color gamut image to the second brightness dynamic range to obtain a second brightness dynamic range image; performing color enhancement processing on the second brightness dynamic range image based on the Lab color space; and using a photoelectric conversion function corresponding to the second brightness dynamic range to convert the color-enhanced second brightness dynamic range image into a nonlinear brightness image with nonlinear brightness as the second image.

[0020] Optionally, the step of converting the brightness dynamic range of the second color gamut image to the second brightness dynamic range includes: converting the second color gamut image to the YUV color space; decomposing the Y channel of the second color gamut image in the YUV color space to obtain a content part and a noise part; adjusting the brightness of the content part based on the human eye's perception characteristics of content; adjusting the brightness of the noise part based on the human eye's perception characteristics of noise; and performing brightness fusion on the brightness-adjusted content part and the brightness-adjusted noise part; and converting the brightness-fused image from the YUV color space back to the second color gamut to obtain a second brightness dynamic range image.

[0021] Optionally, the step of adjusting the brightness of the content part includes: converting the content part from the YUV color space to the Yxy color space; in the Yxy color space, using a brightness mapping curve based on the human eye's perception characteristics of the content, performing brightness mapping on the Y channel, converting the brightness range to a second brightness dynamic range, and based on a set maximum brightness requirement, correcting and / or mapping the maximum brightness and highlight part of the content part to obtain a content part in the second brightness dynamic range; and converting the content part in the second brightness dynamic range from the Yxy color space back to the YUV color space.

[0022] Optionally, the step of converting the second image into a third image having a first color gamut and / or a first brightness dynamic range includes: converting the second image into a brightness linear space using a perceptual quantization curve; linearly stretching the brightness of the second image in the brightness linear space; performing nonlinear reverse mapping on the second image in the brightness linear space using two weighted hable curves; fusing the result of the nonlinear reverse mapping with the result of the linear reverse mapping to obtain a first brightness dynamic range image of the first brightness dynamic range; converting the first brightness dynamic range image from the second color gamut to the first color gamut to obtain a first color gamut image; and using a photoelectric conversion function corresponding to the first brightness dynamic range to convert the first color gamut image into a nonlinear brightness image with nonlinear brightness as a reverse mapping result image.

[0023] Optionally, the step of converting the second image into a third image having a first color gamut and / or a first brightness dynamic range also includes: performing skin color and / or edge detection based on the first image; based on the results of the skin color and / or edge detection, fusing the first image and the reverse mapping result image to repair at least one of the skin color, light and dark area details, and contrast in the reverse mapping result image, thereby obtaining a third image.

[0024] According to a second aspect of the present disclosure, a method for enhancing a video with a standard brightness dynamic range is provided, comprising: converting a first video with a standard color gamut and / or a standard brightness dynamic range into a second video with a wide color gamut and / or a high brightness dynamic range, performing video enhancement processing within the wide color gamut and / or the high brightness dynamic range; and converting the second video into a third video with a standard color gamut and / or a standard brightness dynamic range.

[0025] According to a third aspect of the present disclosure, an image enhancement device is provided, comprising: a first enhancement device, for performing image enhancement processing on an image having a first color gamut and / or a first brightness dynamic range within a first color gamut and / or a first brightness dynamic range to obtain a first image having the first color gamut and / or the first brightness dynamic range; a second enhancement device, for converting the first image into a second image having a second color gamut and / or a second brightness dynamic range, and performing image enhancement processing within the second color gamut and / or the second brightness dynamic range, the second color gamut being wider than the first color gamut, and / or an upper limit of the second brightness dynamic range being higher than an upper limit of the first brightness dynamic range; and a reverse mapping device, for converting the second image into a third image having the first color gamut and / or the first brightness dynamic range.

[0026] According to a fourth aspect of the present disclosure, a computing device is provided, comprising: a processor; and a memory on which executable codes are stored, and when the executable codes are executed by the processor, the processor executes the method described in the first or second aspect above.

[0027] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising an executable code, which, when executed by a processor of an electronic device, enables the processor to execute the method described in the first or second aspect above.

[0028] According to a sixth aspect of the present disclosure, a non-temporary machine-readable storage medium is provided, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor executes the method described in the first or second aspect above.

[0029] Therefore, the image enhancement scheme according to the present invention takes into account the design limitations of traditional brightness and color enhancement schemes, overcomes the limitations of small color gamut and brightness dynamic range under traditional schemes, can provide brightness, contrast and color enhancement effects that exceed traditional schemes, and also provides a new design idea for brightness, contrast and color enhancement algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, wherein like reference numerals generally represent like components in the exemplary embodiments of the present disclosure.

[0031] Figure 1 is a schematic block diagram of an image enhancement device according to the present disclosure.

[0032] Figure 2 is a schematic flow chart of an image enhancement method according to the present disclosure.

[0033] Figure 3 is a schematic flowchart of the first image enhancement processing according to an embodiment of the present disclosure.

[0034] Figure 4 It is a schematic flowchart of a color saturation enhancement processing method for the purpose of skin color protection according to an embodiment of the present disclosure.

[0035] Figure 5 is a schematic flowchart of the second image enhancement process according to an embodiment of the present disclosure.

[0036] Figure 6 is a schematic flow chart of a method for performing brightness dynamic range conversion according to an embodiment of the present disclosure.

[0037] Figure 7 It is a schematic flowchart of a method for adjusting the brightness of a content portion according to an embodiment of the present disclosure.

[0038] Figure 8 is a schematic flowchart of a reverse mapping method according to an embodiment of the present disclosure.

[0039] Fig. 9 A schematic diagram of the structure of a computing device that can be used to implement the above-mentioned image enhancement method according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0040] The preferred embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0041] The present disclosure proposes an image enhancement solution, which can achieve excellent brightness, contrast and color enhancement for images with a relatively low brightness dynamic range by using image enhancement processing within a relatively high brightness dynamic range. As a result, images with a relatively low brightness dynamic range, which are still in large quantities at present, can be presented with better image quality on display devices with a relatively low brightness dynamic range, which are still in large quantities at present.

[0042] For example, when the image enhancement scheme of the present invention is applied to SDR (Standard Dynamic Range, standard brightness dynamic range, brightness range of 0-100 nits) images, the brightness level, global and local contrast, and color expression of the SDR images can be significantly improved, making the effect close to the shock and impact of HDR (High Dynamic Range, high brightness dynamic range, brightness range of 0-1000 nits or 0-10000 nits) images.

[0043] It should be understood that the image in the context of the present disclosure can be either a static picture or a dynamic video. In other words, the image enhancement solution of the present disclosure can be applied to both static pictures and dynamic videos.

[0044] In particular, the image enhancement solution of the present disclosure can be applied to various videos, such as short videos, live videos, chat videos, conference videos, etc.

[0045] Taking SDR (standard brightness dynamic range) images as an example, the present invention can achieve superior brightness, contrast and color enhancement by performing enhancement based on standard color gamut and standard brightness dynamic range, enhancement based on wide color gamut and high brightness dynamic range, and reverse mapping of color gamut and brightness.

[0046] Figure 1 is a schematic block diagram of an image enhancement device 10 according to the present disclosure.

[0047] like Figure 1 As shown, the image enhancement device 10 according to the present disclosure may include a first enhancement device 100 , a second enhancement device 200 and a reverse mapping device 300 .

[0048] The first enhancement device 100 is used to perform image enhancement based on a first color gamut (eg, a standard color gamut BT709) and / or a first brightness dynamic range (eg, a standard brightness dynamic range SDR).

[0049] In this way, the brightness, contrast and color of the image can be first enhanced within the standard color gamut and standard brightness dynamic range to obtain a preliminarily enhanced image.

[0050] Here, for specific application scenarios, a brightness, contrast and color enhancement algorithm with good performance can be selected. For example, based on the first color gamut and / or the first brightness dynamic range, various image enhancement processes such as global and local contrast enhancement, color enhancement, color correction, color adjustment, and defogging can be performed.

[0051] The present disclosure hereinafter refers to Figure 3 and Figure 4 A method for performing image enhancement in a first color gamut and / or a first brightness dynamic range is provided. Those skilled in the art should understand that the image enhancement scheme of the present disclosure is not limited thereto.

[0052] In addition, it should be understood that the image enhancement scheme of the present disclosure may not require the first enhancement device 100, that is, the image enhancement may not be performed in the first color gamut and / or the first brightness dynamic range, and the image enhancement processing of the second enhancement device 200 may be directly performed. Alternatively, the image that has been image-enhanced in the first color gamut and / or the first brightness dynamic range may be used as the input of the image enhancement scheme of the present disclosure, and the image enhancement processing of the second enhancement device 200 may be directly performed.

[0053] The second enhancement device 200 is used to perform image enhancement based on a second color gamut and / or a second brightness range (eg, a wide color gamut and a high brightness dynamic range).

[0054] Here, a conversion algorithm is used to convert, for example, an image that has undergone preliminary enhancement into a wide color gamut and a high brightness dynamic range, thereby achieving the effect of significantly improving the brightness, contrast, and color of the image.

[0055] With the help of the stronger expressive capabilities of wide color gamut and high-brightness dynamic range, the image is adjusted for the second time to obtain an image with a wide color gamut (such as BT2020 color gamut) and high-brightness dynamic range (HDR) with better brightness, contrast and color effects.

[0056] Then, the reverse mapping device 300 performs reverse mapping of color gamut and / or brightness on the image transformed and enhanced by the second enhancement device 200. That is, the reverse mapping device 300 reversely maps the image with a higher color gamut and a higher brightness dynamic range mapped by the second enhancement device 200 back to the same standard color gamut and standard brightness dynamic range as the original image.

[0057] During the reverse mapping conversion process, the superiority of the image obtained by the second enhancement device 200 in terms of brightness, contrast and color can be maintained as much as possible, so that the final result (for example, it can be called "SDR Pro") has a significant improvement in visual effect compared with the original image (SDR), and becomes a standard brightness dynamic range image with better brightness, contrast and color expression.

[0058] Reference below Figure 2 The following describes an image enhancement method that can be performed by the image enhancement device 10 of the present disclosure.

[0059] Figure 2 is a schematic flow chart of an image enhancement method according to the present disclosure.

[0060] like Figure 2 As shown, in step S100, for example, the first enhancement device 100 can perform image enhancement processing on an image (initial image) having a first color gamut and / or a first brightness dynamic range in a first color gamut and / or a first brightness dynamic range to obtain a first image having a first color gamut and / or a first brightness dynamic range.

[0061] As described above, the image enhancement method of the present disclosure may also not need to execute step S100, but directly enter step S200.

[0062] As described above, for specific application scenarios, a brightness, contrast, and color enhancement algorithm with good performance may be selected. For example, based on the first color gamut and / or the first brightness dynamic range, various image enhancement processes such as global and local contrast enhancement, color enhancement, color correction, color adjustment, and defogging may be performed.

[0063] Reference below Figure 3 An image enhancement scheme based on a first color gamut and / or a first brightness dynamic range according to an embodiment of the present disclosure is described in detail, which is particularly suitable for contrast and color enhancement of Internet video images, for example.

[0064] Figure 3 It is a schematic flowchart of the first image enhancement processing according to an embodiment of the present disclosure, wherein the initial image is subjected to brightness estimation, contrast enhancement and color saturation enhancement to obtain a first image that has undergone preliminary enhancement processing.

[0065] like Figure 3 As shown, in step S110, based on the local brightness histogram and / or the global brightness histogram, brightness estimation is performed in the RGB color space of the (initial) image to obtain brightness distribution information of the (initial) image.

[0066] The brightness estimation value of each pixel of the image can be estimated, and then the brightness estimation values ​​of a large number of pixels on the entire image are analyzed to obtain the above brightness distribution information.

[0067] Here, the brightness distribution information may include distribution information of multiple brightness regions of the image. The multiple brightness regions may include, for example, dark regions, middle regions, and bright regions of the image, each having a non-overlapping brightness value range (dark, middle, bright).

[0068] The brightness estimation is completed in the RGB color space of the image by combining the local histogram with the global histogram information.

[0069] In step S120, based on the brightness distribution information, a plurality of mapping curves corresponding to a plurality of brightness value ranges and / or the entire image are used to enhance the contrast of a plurality of brightness regions and / or the entire image.

[0070] Here, the multiple mapping curves may have different divisions of labor, and respectively play the role of adjusting the dark area, middle area, bright area and global brightness of the image.

[0071] In addition, considering that skin color is more sensitive to saturation changes, skin color protection logic can be further designed. In this way, in step S130, the image can be subjected to color saturation enhancement processing based on the purpose of skin color protection.

[0072] Figure 4 It is a schematic flowchart of a color saturation enhancement processing method for the purpose of skin color protection in step S130 according to an embodiment of the present disclosure.

[0073] like Figure 4 As shown, in step S131, the skin color similarity is calculated for the contrast enhanced image.

[0074] For example, the first skin color similarity may be calculated based on H parameter statistics and S parameter statistics in the HSV space.

[0075] By training a large number of skin color images offline, we obtained several typical distributions of skin color about HS, and found that they are approximately multivariate Gaussian distributions. Therefore, the mean and covariance of these distributions can be used for classification.

[0076] Assume that we get K distributions, and their mean is m hs [k], covariance is covhs [k], k = 0, 1, ..., K-1. For an input hs vector x, its basic skin color similarity is:

[0077] max(exp(-(xm hs [k])cov hs -1 (xm hs [k]) T / 2)|k=0,1,2…,K-1).

[0078] Then, according to (xm hs [k])cov hs -1 (xm hs [k]) T The basic skin color similarity is normalized by the size of to obtain a normalized first skin color similarity skin_sim_hs, which ranges from [0,1].

[0079] In addition, the second skin color similarity may also be calculated based on the Cb parameter statistics and the Cr parameter statistics in the YCbCr space.

[0080] Similar to the above description, we also obtained several typical distributions of CbCr of skin color by offline training of a large number of skin color images, and found that they are also approximately multivariate Gaussian distributions. Therefore, the mean and covariance of these distributions can be used for classification.

[0081] Assume that we get K distributions, and their mean is m cbcr [k], covariance is cov cbcr [k], k = 0, 1, ..., K-1. For an input CBCR vector x, its basic skin color similarity is:

[0082] max(exp(-(xm cbcr [k])cov cbcr -1 (xm cbcr [k]) T / 2)|k=0,1,2…,K-1).

[0083] Then, according to (xm cbcr [k])cov cbcr -1 (xm cbcr [k]) T The basic skin color similarity is normalized by the size of to obtain a normalized second skin color similarity skin_sim_cbcr, which ranges from [0,1].

[0084] In step S132, based on the skin color similarity, a skin color probability that the color of at least one region of the image is skin color is determined.

[0085] Here, the first skin color similarity and the second skin color similarity may be further fused to obtain a fused skin color similarity for determining a final skin color probability.

[0086] A simple fusion method may be to take the smaller of the first skin color similarity and the second skin color similarity as the fused skin color similarity. That is, for a pixel, its fused skin color similarity skin_sim may be:

[0087] skin_sim=min(skin_sim_hs,skin_sim_cbcr).

[0088] It should be understood that the first skin color similarity and the second skin color similarity may be calculated in the HSV space and the YCbCr space, respectively, so as to merge the two for subsequent skin color probability calculation.

[0089] Alternatively, the first skin color similarity may be calculated only in the HSV space, or the second skin color similarity may be calculated only in the YCbCr space, and only one of the similarities may be used to calculate the skin color probability to adjust the color saturation.

[0090] Then, in step S133, based on the skin color probability calculated above, the color saturation of at least one region of the image is adjusted to achieve the purpose of skin color protection. The color saturation adjustment here can be adaptive, that is, adapted to the skin color probability.

[0091] Here, color enhancement may only consider saturation enhancement. The basic principle of enhancement may be: for high-saturation pixels close to skin color, skin color pixels, and very low-saturation pixels, no major adjustments are made.

[0092] Thus, the result after color saturation enhancement processing can be obtained and output.

[0093] return Figure 2 In step S200, for example, the first image can be converted into a second image having a second color gamut and / or a second brightness dynamic range by the second enhancement device 200, and image enhancement processing can be performed in the second color gamut and / or the second brightness dynamic range. Thus, the image quality can be further enhanced.

[0094] Here, the second color gamut (eg, BT2020 color gamut) is wider than the first color gamut (eg, BT709 color gamut). And / or, the upper limit of the second brightness dynamic range (eg, HDR) is higher than the upper limit of the first brightness dynamic range (eg, SDR).

[0095] Regarding image enhancement based on the second color gamut and the second brightness dynamic range, various existing solutions can be used, such as various existing SDR to HDR solutions, such as solutions based on deep learning. Alternatively, in addition to the BT202 color gamut and HDR brightness dynamic range mentioned in the above example, adjustments can also be made in other high color gamuts and brightness dynamic ranges, such as the DCI-P3 color gamut, 0-10000nits brightness dynamic range, etc.

[0096] In addition, regarding the image enhancement processing performed in the second color gamut and / or the second brightness dynamic range, it can be performed after the first image is converted into a second image with the second color gamut and / or the second brightness dynamic range, and can also be implemented in the process of converting the first image into a second image with the second color gamut and / or the second brightness dynamic range.

[0097] Reference below Figure 5 An image enhancement scheme based on wide color gamut and high brightness dynamic range provided in an embodiment of the present disclosure is described, wherein image enhancement processing performed in the second color gamut and / or the second brightness dynamic range is implemented in the process of converting a first image into a second image having a second color gamut and / or a second brightness dynamic range.

[0098] Figure 5 It is a schematic flowchart of the second image enhancement processing according to an embodiment of the present disclosure, wherein image enhancement is performed based on a wide color gamut (eg, BT2020 color gamut) and a high brightness dynamic range (eg, HDR).

[0099] Here, for example, we can take advantage of the brightness dynamic range of HDR and the wide color gamut of BT2020 color gamut to convert images with standard color gamut (BT709) and standard brightness dynamic range (SDR) into wide color gamut (BT2020 color gamut, with a brightness range of 0-1000nits within the HDR range) using a conversion algorithm based on visual perception and semantic analysis, and further adjust the color of the image within this color gamut and brightness dynamic range.

[0100] like Figure 5 As shown, in step S210, the nonlinear brightness signal value of the first image is converted into a linear brightness signal value using an electro-optical transfer function (EOTF) corresponding to the first brightness dynamic range to obtain a linear brightness image.

[0101] Here, an electro-optical transfer function (EOTF) is used to map the electrical signal value of the image to the corresponding light brightness signal value.

[0102] In step S220, the linear luminance image is converted from the first color gamut (eg, the BT709 color gamut) to the second color gamut (eg, the BT2020 color gamut) to obtain a second color gamut image.

[0103] In step S230, the brightness dynamic range of the second color gamut image is converted to a second brightness dynamic range to obtain a second brightness dynamic range image.

[0104] Here, a conversion algorithm based on visual perception and semantic analysis can be used to convert the brightness range of the image to the HDR range of 0-1000nits.

[0105] Figure 6 is a schematic flowchart of a method for performing brightness dynamic range conversion in step S230 according to an embodiment of the present disclosure.

[0106] In step S231, the second color gamut image is converted into a YUV color space.

[0107] In step S232, the Y channel of the second color gamut image is decomposed in the YUV color space to obtain a content part and a noise part.

[0108] In step S233, the brightness of the content portion is adjusted based on the perception characteristics of the human eye to the content.

[0109] Figure 7 It is a schematic flowchart of a method for adjusting the brightness of a content portion according to an embodiment of the present disclosure.

[0110] In step S2331, the content part is converted from the YUV color space to the Yxy color space.

[0111] In step S2332, in the Yxy color space, brightness mapping is performed on the Y channel using a brightness mapping curve based on the human eye's perception characteristics of the content, and the brightness range is converted to a second brightness dynamic range.

[0112] In step S2333, based on the set maximum brightness requirement, the maximum brightness of the content part and the highlight part are corrected and / or mapped to obtain the content part in the second brightness dynamic range.

[0113] In step S2334, the content portion of the second luminance dynamic range is converted from the Yxy color space back to the YUV color space.

[0114] return Figure 6 In step S234, the brightness of the noise part is adjusted based on the human eye's perception characteristics of noise.

[0115] It should be understood that the execution order of step S233 and step S234 can be arbitrarily adjusted, or these two steps can also be executed in parallel.

[0116] In step S235, brightness fusion is performed on the brightness-adjusted content portion and the brightness-adjusted noise portion.

[0117] In step S236, the brightness-fused image is converted from the YUV color space back to the second color gamut to obtain a second brightness dynamic range image.

[0118] On this basis, color enhancement processing based on the Lab color space can be adopted.

[0119] return Figure 5 In step S240, color enhancement processing is performed on the second brightness dynamic range image based on the Lab color space.

[0120] Then, in step S250, the second brightness dynamic range image after color enhancement is converted into a nonlinear brightness image with nonlinear brightness by using an optical-electrical conversion function (OETF) corresponding to the second brightness dynamic range as the second image.

[0121] Here, the photoelectric conversion function OETF, also known as the inverse EOTF, is used to map the brightness signal value of the image to the corresponding electrical signal value.

[0122] Thus, image enhancement processing is performed in the second color gamut and / or the second brightness dynamic range, and the first image with the first color gamut and / or the first brightness dynamic range is converted into a second image with the second color gamut and / or the second brightness dynamic range.

[0123] return Figure 2 In step S300 , for example, the second image may be converted into a third image having a first color gamut and / or a first brightness dynamic range by means of a reverse mapping device 300 .

[0124] Here, a variety of existing reverse mapping schemes can be used to map the image from the second color gamut and / or the second brightness dynamic range to the first color gamut and / or the first brightness dynamic range, such as a tone mapping method based on curve mapping or deep learning, reverse mapping based on ColorLUT, etc.

[0125] Figure 8 It is a schematic flowchart of a method for reverse mapping color gamut and brightness according to an embodiment of the present disclosure.

[0126] In step S310, the second image, for example in a non-linear RGB format, is converted into a luminance linear space using a perceptual quantization (PQ) curve.

[0127] The PQ curve is a type of EOTF function curve. Currently, the peak display brightness of most SDR displays is around 100 nits, and its EOTF uses a gamma curve. HDR needs to be able to display a peak brightness of 10,000 nits. If SDR's EOTF continues to be used, 14-bit pixels will be required for encoding, which is extremely disadvantageous for transmission and storage. Considering that the human eye is not as sensitive to high-brightness areas as to dark areas, the PQ curve is proposed by simulating the physiological characteristics of the human eye, which can encode HDR content with 10-bit or 12-bit pixels without introducing artifacts.

[0128] Next, in order to ensure the image effects of brightness and contrast of the final output result image ("SDR Pro"), in step S320, the second image may be linearly stretched in brightness in a brightness linear space.

[0129] In step S330, in the brightness linear space, two hable curves are used to weight the second image to perform nonlinear reverse mapping.

[0130] Here, by using an improved hable curve for mapping, that is, performing reverse mapping by weighting two hable curves, the overall brightness of the nonlinear mapping result can be improved and the contrast of the bright area can be stretched.

[0131] In step S340, the result of the nonlinear reverse mapping and the result of the linear reverse mapping are fused to obtain a first brightness dynamic range image of a first brightness dynamic range. By fusing the result of the nonlinear reverse mapping and the result of the linear reverse mapping, the color and the details of the bright and dark areas can be better preserved.

[0132] In step S350, the first brightness dynamic range image is converted from the second color gamut to the first color gamut to obtain a first color gamut image.

[0133] In step S360, the first color gamut image is converted into a nonlinear brightness image with nonlinear brightness using an optical-electrical conversion function (OETF) corresponding to the first brightness dynamic range as a reverse mapping result image.

[0134] Since the reverse mapping result may have a certain degree of detail loss, contrast loss, and skin color oversaturation problems, the present disclosure may further introduce the first enhancement result (first image) output by step S100, or the input image of step S200, to repair the above reverse mapping result.

[0135] For example, the first enhancement result (first image) and the reverse mapping result (reverse mapping result image) may be first transformed into an HSV color space (not shown in the figure) so as to perform the following repair operation in the HSV color space.

[0136] In step S370, skin color and / or edge detection may be performed based on the first image.

[0137] Here, skin color and / or edge detection may be performed on the H and V channels of the first image, respectively.

[0138] Then, in step S380, based on the results of skin color and / or edge detection, the first image and the reverse mapping result image are fused to restore at least one of skin color, bright and dark area details, and contrast in the reverse mapping result image, thereby obtaining a third image.

[0139] Here, by using the results of skin color and / or edge detection to guide the fusion of the first image and the reverse mapping result image, the repair effect of at least one of the skin color, details of light and dark areas (such as texture), and contrast in the reverse mapping result can be achieved.

[0140] Then, the fusion result can be converted from the HSV space back to the RGB space (not shown in the figure).

[0141] Then, it is converted into YUV format to obtain the third image of the present invention after image enhancement, namely the "SDR Pro" image.

[0142] So far, the image enhancement scheme according to the present disclosure has been described in detail with reference to the accompanying drawings.

[0143] In an embodiment of the present disclosure, taking video enhancement as an example, an enhancement method for a standard dynamic range (SDR) video may be provided.

[0144] First, a first video with a standard color gamut (e.g., BT709) and / or a standard brightness dynamic range (SDR) is converted into a second video with a wide color gamut (BT2020) and / or a high brightness dynamic range (HDR), and video enhancement processing is performed within the wide color gamut (BT2020) and / or the high brightness dynamic range (HDR).

[0145] Then, the second video is converted into a third video (SDRPro) having a standard color gamut (eg, BT709) and / or a standard brightness dynamic range (SDR).

[0146] In this way, based on the preliminary adjustment of the standard color gamut and standard brightness dynamic range, the image is further converted from the standard color gamut and standard brightness dynamic range to the wide color gamut and high brightness dynamic range for further enhancement, overcoming the common limitations of the existing solutions described above.

[0147] In wide color gamut and high brightness dynamic range, the image enhancement scheme has a larger adjustable space and can achieve better adjustment effects. In the reverse mapping of color gamut and brightness, the adjustment effect of the previous step can be retained as much as possible, so that the final image enhancement result can achieve better enhancement effect compared with the existing scheme.

[0148] Fig. 9 A schematic diagram of the structure of a computing device that can be used to implement the above-mentioned image enhancement method according to an embodiment of the present invention is shown.

[0149] See also Fig. 9 , the computing device 900 includes a memory 910 and a processor 920 .

[0150] The processor 920 may be a multi-core processor or may include multiple processors. In some embodiments, the processor 920 may include a general-purpose main processor and one or more special coprocessors, such as a graphics processing unit (GPU), a digital signal processor (DSP), etc. In some embodiments, the processor 920 may be implemented using a customized circuit, such as an application-specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0151] The memory 910 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. Among them, ROM can store static data or instructions required by the processor 920 or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that does not lose the stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as a permanent storage device. In some other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as a dynamic random access memory. The system memory may store some or all instructions and data required by the processor at run time. In addition, the memory 910 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, the memory 910 may include a readable and / or writable removable storage device, such as a laser disc (CD), a read-only digital versatile disc (e.g., DVD-ROM, double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. The computer-readable storage medium does not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0152] The memory 910 stores executable codes, and when the executable codes are processed by the processor 920 , the processor 920 can execute the image enhancement method mentioned above.

[0153] The image enhancement scheme according to the present invention has been described above in detail with reference to the accompanying drawings.

[0154] In addition, the method according to the present invention may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing the above steps defined in the above method of the present invention.

[0155] Alternatively, the present invention may also be implemented as a non-temporary machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) on which executable code (or computer program, or computer instruction code) is stored. When the executable code (or computer program, or computer instruction code) is executed by a processor of an electronic device (or computing device, server, etc.), the processor executes the various steps of the above-mentioned method according to the present invention.

[0156] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or a combination of both.

[0157] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system and method according to multiple embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or code, and the part of the module, program segment or code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0158] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terms used herein are selected to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. An image enhancement method, comprising: Converting a first image having a first color gamut and / or a first brightness dynamic range into a second image having a second color gamut and / or a second brightness dynamic range, and performing image enhancement processing in the second color gamut and / or the second brightness dynamic range, wherein the second color gamut is wider than the first color gamut, and / or an upper limit of the second brightness dynamic range is higher than an upper limit of the first brightness dynamic range; as well as The second image is converted into a third image having a first color gamut and / or a first brightness dynamic range.

2. The method according to claim 1, further comprising: In a first color gamut and / or a first brightness dynamic range, image enhancement processing is performed on an image having a first color gamut and / or a first brightness dynamic range to obtain the first image.

3. The method according to claim 2, wherein: The step of performing image enhancement processing on an image having a first color gamut and / or a first brightness dynamic range comprises: Based on the local brightness histogram and / or the global brightness histogram, brightness estimation is performed in the RGB color space of the image to obtain brightness distribution information of the image, wherein the brightness distribution information includes distribution information of multiple brightness regions of the image, and the multiple brightness regions respectively have brightness value ranges that do not overlap with each other; and Based on the brightness distribution information, a plurality of mapping curves respectively corresponding to the plurality of brightness value ranges and / or the entire image are used to enhance the contrast of the plurality of brightness regions and / or the entire image.

4. The method according to claim 3, wherein: The step of performing image enhancement processing on the image having the first color gamut and / or the first brightness dynamic range also includes: The image is subjected to color saturation enhancement processing based on the purpose of protecting skin color.

5. The method according to claim 4, wherein: The step of performing color saturation enhancement processing on the contrast enhanced image for the purpose of skin color protection comprises: For the contrast enhanced image, calculating skin color similarity; Determining a skin color probability that a color of at least one region of the image is skin color based on the skin color similarity; and Based on the skin color probability, the color saturation of the at least one region of the image is adjusted to achieve the purpose of skin color protection.

6. The method according to claim 1, wherein: The step of converting a first image having a first color gamut and / or a first brightness dynamic range into a second image having a second color gamut and / or a second brightness dynamic range, and performing image enhancement processing in the second color gamut and / or the second brightness dynamic range comprises: Using an electro-optical conversion function corresponding to the first brightness dynamic range, converting the nonlinear brightness signal value of the first image into a linear brightness signal value to obtain a linear brightness image; Converting the linear brightness image from the first color gamut to the second color gamut to obtain a second color gamut image; Converting the brightness dynamic range of the second color gamut image to a second brightness dynamic range to obtain a second brightness dynamic range image; Performing color enhancement processing on the second brightness dynamic range image based on the Lab color space; and The second brightness dynamic range image after color enhancement is converted into a nonlinear brightness image with nonlinear brightness using a photoelectric conversion function corresponding to the second brightness dynamic range as the second image.

7. The method according to claim 6, wherein: The step of converting the brightness dynamic range of the second color gamut image to the second brightness dynamic range comprises: Convert the second color gamut image to a YUV color space; In a YUV color space, decomposing a Y channel of the second color gamut image to obtain a content part and a noise part; Based on the human eye's perception of content, adjust the brightness of the content part; Based on the human eye's perception of noise, the brightness of the noise part is adjusted; and performing brightness fusion on the brightness-adjusted content portion and the brightness-adjusted noise portion; and The brightness-fused image is converted from the YUV color space back to the second color domain to obtain the second brightness dynamic range image.

8. The method according to claim 7, wherein: The steps of adjusting the brightness of the content portion include: Converting the content portion from a YUV color space to a Yxy color space; In the Yxy color space, the brightness mapping curve based on the human eye's perception characteristics of the content is used to map the brightness of the Y channel and convert the brightness range into the second brightness dynamic range. Based on the set maximum brightness requirement, correcting and / or mapping the maximum brightness and highlight portion of the content portion to obtain a content portion with a second brightness dynamic range; and The content portion of the second luminance dynamic range is converted from the Yxy color space back to the YUV color space.

9. The method according to claim 1, wherein: The step of converting the second image into a third image having a first color gamut and / or a first brightness dynamic range comprises: converting the second image to a brightness linear space using a perceptual quantization curve; In the brightness linear space, linearly stretching the brightness of the second image; In the brightness linear space, two hable curves are used to weight the second image to perform nonlinear reverse mapping; fusing the result of the nonlinear reverse mapping and the result of the linear reverse mapping to obtain a first brightness dynamic range image with a first brightness dynamic range; Converting the first brightness dynamic range image from the second color gamut to the first color gamut to obtain a first color gamut image; The first color gamut image is converted into a nonlinear brightness image with nonlinear brightness using a photoelectric conversion function corresponding to the first brightness dynamic range as a reverse mapping result image.

10. The method according to claim 9, wherein: The step of converting the second image into a third image having a first color gamut and / or a first brightness dynamic range further comprises: Perform skin color and / or edge detection based on the first image; Based on the results of skin color and / or edge detection, the first image and the reverse mapping result image are fused to restore at least one of skin color, bright and dark area details, and contrast in the reverse mapping result image, thereby obtaining the third image.

11. A method for enhancing a standard brightness dynamic range video, comprising: Converting a first video with a standard color gamut and / or a standard brightness dynamic range into a second video with a wide color gamut and / or a high brightness dynamic range, and performing video enhancement processing in the wide color gamut and / or the high brightness dynamic range; as well as The second video is converted into a third video having a standard color gamut and / or a standard brightness dynamic range.

12. An image enhancement device, comprising: A first enhancement device performs image enhancement processing on an image having a first color gamut and / or a first brightness dynamic range within a first color gamut and / or a first brightness dynamic range to obtain a first image having a first color gamut and / or a first brightness dynamic range; a second enhancing device, configured to convert the first image into a second image having a second color gamut and / or a second brightness dynamic range, and perform image enhancement processing in the second color gamut and / or the second brightness dynamic range, wherein the second color gamut is wider than the first color gamut, and / or an upper limit of the second brightness dynamic range is higher than an upper limit of the first brightness dynamic range; as well as The reverse mapping device is used to convert the second image into a third image with a first color gamut and / or a first brightness dynamic range.

13. A computing device comprising: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to execute the method according to any one of claims 1 to 11.

Images