Hdr tone mapping based on creative intent metadata and ambient light

Abstract

One embodiment provides a method comprising determining multi-dimensional metadata corresponding to an input image, and determining ambient light information indicative of an ambient light level in a surrounding environment of a display device. The multi-dimensional metadata includes a distribution function of pixels in the input image. The method further comprises determining one or more gains to adaptively compensate for the ambient light level in the surrounding environment based on the ambient light information. The method further comprises generating a tone mapping function based on the multi-dimensional metadata and the one or more gains, and applying the tone mapping function to the input image to generate a tone mapped image that is adaptively compensated for the ambient light level in the surrounding environment. The tone mapped image is provided to the display device for presentation.

Description

Technical Field

[0001] One or more embodiments generally relate to device setups for consumer electronics products, and more specifically, to methods and systems for high dynamic range (HDR) tone mapping based on creative intent metadata and ambient light. Background Technology

[0002] Consumer electronic devices (such as smart TVs, smartphones, etc.) are now equipped with state-of-the-art displays (such as QLED and OLED) that offer ultra-high picture quality. Summary of the Invention

[0003] Technical solution

[0004] One embodiment provides a method including determining multidimensional metadata corresponding to an input image and determining ambient light information indicating ambient light levels in an environment surrounding a display device. The multidimensional metadata includes distribution functions of pixels in the input image. The method further includes determining one or more gains based on the ambient light information to adaptively compensate for ambient light levels in the surrounding environment. The method also includes generating a tone mapping function based on the multidimensional metadata and the one or more gains, and applying the tone mapping function to the input image to generate a tone-mapped image that adaptively compensates for ambient light levels in the surrounding environment. The tone-mapped image is provided to a display device for rendering.

[0005] Another embodiment provides a method including determining multidimensional metadata corresponding to an input image and determining ambient light information indicating ambient light levels in the environment surrounding a display device. The multidimensional metadata includes the cumulative distribution function (CDF) of pixels in the input image. The method further includes determining one or more gains based on the multidimensional metadata and the ambient light information to adaptively compensate for ambient light levels in the surrounding environment. The method also includes generating a tone mapping function based on the one or more gains and applying the tone mapping function to the input image to generate a tone-mapped image that adaptively compensates for ambient light levels in the surrounding environment. The tone-mapped image is provided to a display device for rendering.

[0006] In another embodiment, the tone mapping function preserves the color hue of the input image.

[0007] In another embodiment, the input image is a frame of the input video.

[0008] Another embodiment provides a system including: at least one processor; and a processor-readable storage device storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations. The operations include: determining multidimensional metadata corresponding to an input image, and determining ambient light information indicating ambient light levels in an environment surrounding a display device. The multidimensional metadata includes distribution functions of pixels in the input image. The operations further include determining one or more gains based on the ambient light information to adaptively compensate for ambient light levels in the surrounding environment. The operations further include generating a tone mapping function based on the multidimensional metadata and the one or more gains, and applying the tone mapping function to the input image to generate a tone-mapped image that adaptively compensates for ambient light levels in the surrounding environment. The tone-mapped image is provided to a display device for rendering.

[0009] Another embodiment provides a system including: at least one processor; and a processor-readable storage device storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations. The operations include: determining multidimensional metadata corresponding to an input image, and determining ambient light information indicating ambient light levels in an environment surrounding a display device. The multidimensional metadata includes CDFs of pixels in the input image. The operations further include determining one or more gains based on the ambient light information to adaptively compensate for ambient light levels in the surrounding environment. The operations further include generating a tone mapping function based on the one or more gains and applying the tone mapping function to the input image to generate a tone-mapped image that adaptively compensates for ambient light levels in the surrounding environment. The tone-mapped image is provided to a display device for rendering.

[0010] Another embodiment provides a system including at least one processor and a processor-readable storage device storing instructions that, when executed by at least one processor, cause at least one other processor to perform operations. The operations include generating a tone mapping function and one or more parameters corresponding to the tone mapping function based on a knee point modifier, a lower curve modifier, and an upper curve modifier.

[0011] In another embodiment, the tone mapping function preserves the hue of the input image.

[0012] In another embodiment, the input image is a frame of the input video.

[0013] Another embodiment provides a processor-readable medium including a program that, when executed by a processor, performs a method comprising: determining multidimensional metadata corresponding to an input image, and determining ambient light information indicating ambient light levels in an environment surrounding a display device. The multidimensional metadata includes a distribution function of pixels in the input image. The method further includes determining one or more gains based on the ambient light information to adaptively compensate for ambient light levels in the surrounding environment. The method further includes generating a tone mapping function based on the multidimensional metadata and the one or more gains, and applying the tone mapping function to the input image to generate a tone-mapped image that adaptively compensates for ambient light levels in the surrounding environment. The tone-mapped image is provided to a display device for rendering.

[0014] Another embodiment provides a processor-readable medium including a program that, when executed by a processor, performs a method comprising: determining multidimensional metadata corresponding to an input image, and determining ambient light information indicating ambient light levels in an environment surrounding a display device. The multidimensional metadata includes CDFs of pixels in the input image. The method further includes determining one or more gains based on the multidimensional metadata and the ambient light information to adaptively compensate for ambient light levels in the surrounding environment. The method further includes generating a tone mapping function based on the one or more gains and applying the tone mapping function to the input image to generate a tone-mapped image that adaptively compensates for ambient light levels in the surrounding environment. The tone-mapped image is provided to a display device for rendering.

[0015] In another embodiment, for each percentile of the CDF, the multidimensional metadata includes a corresponding pair of values, including a percentile luminance value and a pixel percentage value.

[0016] In another embodiment, the tone mapping function represents a Bézier curve comprising multiple parts, each of which has a corresponding adjustment point along the Bézier curve.

[0017] Another embodiment provides a processor-readable medium including a program that, when executed by a processor, performs a method comprising: determining an inflection point modifier based on one or more gain modifiers, a lower curve modifier for generating a lower portion of a Bézier curve, and an upper curve modifier for generating an upper portion of a Bézier curve; and generating a tone mapping function and one or more parameters corresponding to the tone mapping function based on the inflection point modifier, the lower curve modifier, and the upper curve modifier.

[0018] These and other aspects and advantages of one or more embodiments will become apparent from the following detailed description, which, when taken in conjunction with the accompanying drawings, illustrates the principles of one or more embodiments by way of example. Attached Figure Description

[0019] To gain a more complete understanding of the nature and advantages of the embodiments and the preferred modes of use, reference should be made to the following detailed description, which should be read in conjunction with the accompanying drawings, wherein:

[0020] Figure 1 An example computational architecture is shown in one or more embodiments for implementing high dynamic range (HDR) tone mapping for rendering HDR content on a display device;

[0021] Figure 2 An example workflow for implementing HDR tone mapping for rendering HDR content on a display device is shown in one or more embodiments;

[0022] Figure 3 An example HDR tone map with an ambient light compensation system is shown in one or more embodiments;

[0023] Figure 4 An example ambient light compensation system is shown in one or more embodiments;

[0024] Figure 5 It is a graph showing the percentage difference between the input percentile curve and the uniform percentile curve in one or more embodiments;

[0025] Figure 6 It is a graph showing the ramp function of SubProbMidToneDiffRatio in one or more embodiments;

[0026] Figure 7 This illustrates the gain hiph P in one or more embodiments. ratio The curve of the slope function;

[0027] Figure 8 This illustrates the gain low P in one or more embodiments. ratio The curve of the slope function;

[0028] Figure 9 This illustrates ΔP in one or more embodiments. high (i) is a graph of the ramp function;

[0029] Figure 10 This illustrates the ΔP of the curve modifier in one or more embodiments. low (i) A graph of the ramp function;

[0030] Figure 11 It is a graph showing the ramp function of ΔK(i) in one or more embodiments;

[0031] Figure 12An example tone mapping curve application system is shown in one or more embodiments;

[0032] Figure 13 An example ambient light compensation development system is shown in one or more embodiments;

[0033] Figure 14A This is an example mastered image viewed in one or more embodiments in a reference surrounding environment with low levels of ambient light.

[0034] Figure 14B These are example tone-mapped images viewed in an environment with high degree / level ambient light in one or more embodiments;

[0035] Figure 14C This is shown in one or more embodiments with Figures 14A-14B Example diagram of the histograms corresponding to the main image and tone-mapped image;

[0036] Figure 15A This is an example main image viewed in one or more embodiments in a reference surrounding environment with low levels of ambient light;

[0037] Figure 15B These are example tone-mapped images viewed in an environment with a high degree / level of ambient light in one or more embodiments;

[0038] Figure 15C This is shown in one or more embodiments with Figures 15A-15B Example diagram of the histograms corresponding to the main image and tone-mapped image;

[0039] Figure 16A A dark image is an example image (“dark image”) with a large amount of dark detail in one or more embodiments;

[0040] Figure 16B This is another example image (“bright image”) with a large amount of bright detail in one or more embodiments;

[0041] Figure 16C This is shown in one or more embodiments. Figures 16A-16B Example diagrams of ambient light compensation curves for dark and bright images;

[0042] Figure 16D This is shown in one or more embodiments. Figures 16A-16B Example diagram of histograms for dark and light images;

[0043] Figure 16E This is shown in one or more embodiments. Figures 16A-16BExample graphs of CDF curves for dark and bright images;

[0044] Figure 17 This is a flowchart in one or more embodiments for implementing an example process of HDR tone mapping for rendering HDR content on a display device; and

[0045] Figure 18 This is a high-level block diagram illustrating an information processing system including a computer system for implementing the disclosed embodiments. Detailed Implementation

[0046] The following description is for illustrative purposes only and is not intended to limit the inventive concepts claimed herein. Furthermore, in each of the various possible combinations and arrangements, the specific features described herein may be used in combination with other described features. Unless otherwise expressly defined herein, all terms should be given the broadest possible interpretation, including the meaning implied in the specification, the meaning understood by those skilled in the art, and / or the meaning defined in dictionaries, papers, etc.

[0047] One or more embodiments generally relate to device settings for consumer electronics products, and more specifically, to methods and systems for high dynamic range (HDR) tone mapping based on creative intent metadata and ambient light. One embodiment provides a method including determining multidimensional metadata corresponding to an input image, and determining ambient light information indicating ambient light levels in the environment surrounding a display device. The multidimensional metadata includes distribution functions of pixels in the input image. The method further includes determining one or more gains based on the ambient light information to adaptively compensate for ambient light levels in the surrounding environment. The method further includes generating a tone mapping function based on the multidimensional metadata and one or more gains, and applying the tone mapping function to the input image to generate a tone-mapped image that adaptively compensates for ambient light levels in the surrounding environment. The tone-mapped image is provided to a display device for rendering.

[0048] Another embodiment provides a system including: at least one processor; and a processor-readable storage device storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations. The operations include: determining multidimensional metadata corresponding to an input image, and determining ambient light information indicating ambient light levels in an environment surrounding a display device. The multidimensional metadata includes the distribution of pixels in the input image (also referred to as a distribution function). The operations further include determining one or more gains based on the ambient light information to adaptively compensate for ambient light levels in the surrounding environment. The operations further include generating a tone mapping function based on the multidimensional metadata and the one or more gains, and applying the tone mapping function to the input image to generate a tone-mapped image that adaptively compensates for ambient light levels in the surrounding environment. The tone-mapped image is provided to a display device for rendering.

[0049] Another embodiment provides a processor-readable medium including a program that, when executed by a processor, performs a method including determining multidimensional metadata corresponding to an input image and determining ambient light information indicating ambient light levels in an environment surrounding a display device. The multidimensional metadata includes a distribution function of pixels in the input image. The method further includes determining one or more gains based on the ambient light information to adaptively compensate for ambient light levels in the surrounding environment. The method also includes generating a tone mapping function based on the multidimensional metadata and the one or more gains, and applying the tone mapping function to the input image to generate a tone-mapped image that adaptively compensates for ambient light levels in the surrounding environment. The tone-mapped image is provided to a display device for rendering.

[0050] For ease of explanation, the terms "tone mapping curve" and "tone mapping function" are used interchangeably in this specification. For illustrative purposes, the term "ambient light compensation curve" as used herein generally refers to a tone mapping curve with ambient light compensation.

[0051] Studios typically provide color tones for content rendered on display devices in very dark environments. For example, a studio colorist might perform color grading on content in a very dark environment. However, people rarely watch television or any other display device in complete darkness. During content rendering on a display device, if the ambient light level around the display device increases, the perceived screen quality gradually decreases.

[0052] The term "creative intent" generally refers to a specific visualization of an image that a content provider or content creator (e.g., a colorist in a studio) wants the audience to see.

[0053] One or more embodiments provide an ambient light compensated HDR tone map based on creative intent metadata (also known as multidimensional metadata) to maintain creative intent across varying ambient light levels. In one embodiment, the HDR tone map includes dynamically correcting image quality based on the creative intent metadata.

[0054] One or more embodiments provide a method and system for HDR video tone mapping. In one embodiment, HDR video tone mapping includes: (1) receiving an input video for rendering on a display device; (2) receiving metadata that at least partially represents frame or scene statistics of the input video; (3) determining an adaptive / custom tone mapping function with ambient light compensation based at least on the metadata, the ambient light level in the environment of the display device, and characteristics of the display device; and (4) applying the tone mapping function to the input video to generate a tone-mapped video with ambient light compensation, wherein the tone-mapped video is provided to the display device for rendering on the display device.

[0055] One or more embodiments utilize an adaptive explicit Bézier curve with multiple parts to provide ambient light compensation, wherein multiple or more parts of the tone mapping curve have multiple adjustment points along the Bézier curve for improving control over tone mapping.

[0056] Figure 1 An example computing architecture 100 for implementing HDR tone mapping for rendering HDR content on a display device 60 is illustrated in one or more embodiments. The computing architecture 100 includes an electronics device 110, which includes resources such as one or more processor units 120 and one or more storage units 130. One or more applications can utilize the resources of the electronics device 110 to perform / operate on the electronics device 110.

[0057] In one embodiment, one or more applications on electronic device 110 include an HDR tone mapping system 200 configured to implement HDR tone mapping for rendering HDR content integrated into or coupled to electronic device 110 on display device 60. In one embodiment, HDR tone mapping system 200 is configured to provide ambient light compensation based on creative intent metadata and ambient light. As described in detail later herein, HDR tone mapping system 200 is configured to: (1) receive an input video (e.g., HDR video) for rendering on display device 60, (2) receive multidimensional creative intent metadata corresponding to the input video, wherein the creative intent metadata indicates the intent of the content creator / content provider of the input video, and (3) improve picture quality during rendering of the input video on display device 60 based on creative intent metadata and ambient light, thereby reducing or minimizing image degradation caused by ambient light. In one embodiment, picture quality is improved by providing ambient light compensation based on creative intent metadata, modifying tone mapping curves based on ambient light compensation, and / or providing hue preservation.

[0058] In one embodiment, the creative intent metadata corresponding to the input content includes per-frame or scene statistics encompassing the entire input video (e.g., the entire HDR video). For example, in one embodiment, for each image of the input video (e.g., an HDR image), the creative intent metadata (also referred to as multidimensional metadata) includes luminance percentile information corresponding to the image. The luminance percentile information corresponding to the image represents the distribution (i.e., the number) of pixels in the image. For example, in one embodiment, the luminance percentile information corresponding to the image includes one or more percentiles of the CDF of the pixels in the image. In one embodiment, the CDF indicates one or more of the following: whether the image is dark or bright, which dynamic range of the CDF has dark detail and the darkness level of the dark detail, or which dynamic ranges of the CDF have bright detail and the brightness level of the bright detail. For example, if the CDF includes a specific range that is steeper than at least one other range of the CDF, then there is more detail (i.e., more pixels) in that specific range compared to at least one other range. As another example, if the CDF includes a specific range that is flatter than at least one other range of the CDF, then there is less detail (i.e., fewer pixels) in that specific range compared to at least one other range.

[0059] In one embodiment, for each image of the input content (e.g., an HDR image), the HDR tone mapping system 200 is configured to generate a tone mapping curve corresponding to the image based on the CDF of the pixels in the image. The tone mapping curve includes multiple portions that preserve contrast. For example, if the CDF includes a specific range that is steeper than at least one other range of the CDF, the tone mapping curve includes a portion corresponding to the steeper range, where the corresponding portion is steeper than at least one other portion of the tone mapping curve because more detail needs to be preserved. As another example, if the CDF includes a specific range that is flatter than at least one other range of the CDF, the tone mapping curve includes a portion corresponding to the flatter range, where the corresponding portion is flatter than at least one other portion of the tone mapping curve because less detail needs to be preserved. Based on the CDF of the pixels in the image of the input video, the HDR tone mapping system 200 is able to distinguish images and apply different ambient light compensation curves based on the CDF (i.e., ambient light compensation is adapted to creative intent metadata). For example, dark details in a bright image are further improved to prevent dark details from becoming fragmented.

[0060] In one embodiment, the luminance percentile information is represented as two-dimensional (2D) data. For example, in one embodiment, each percentile of the CDF curve included in the luminance percentile information is represented as a pair of values ​​{x, y}, where x represents the percentile luminance value and y represents the pixel percentile value (e.g., {x = 100, y = 25%} means that 25% of the pixels are below 100 nits, i.e., the 25th percentile is 100 nits). Noise can be removed (i.e., noisy pixels can be removed) using CDF-based tone mapping curves.

[0061] Let h(i) typically represent a histogram, where i is the i-th column (ii) of the histogram. th Let cdf(i) generally represent CDF. In one embodiment, the HDR tone mapping system 200 is configured to: (1) receive an RGB image (e.g., an image from an input video) as input, (2) determine a maxRGB image based on the RGB image, (3) determine a histogram h(i) corresponding to the maxRGB image, and (4) determine the CDF cdf(i) of the pixels in the RGB image based on the histogram h(i). In one embodiment, the histogram h(i) has 1024 bins and 10 bits of input. In one embodiment, the CDF cdf(i) is calculated according to formula (1) provided below:

[0062]

[0063] Examples of electronic devices 110 include, but are not limited to, televisions (e.g., smart TVs), mobile electronic devices (e.g., tablets, smartphones, laptops, etc.), wearable devices (e.g., smartwatches, smart bands, head-mounted displays, smart glasses, etc.), set-top boxes, Internet of Things (IoT) devices, etc.

[0064] In one embodiment, electronic device 110 includes one or more sensor units 150 integrated in or coupled to electronic device 110, such as a camera, microphone, GPS, motion sensor, etc. In one embodiment, HDR tone mapping system 200 utilizes at least one of the one or more sensor units 150 to capture sensor data, including one or more readings / measurements related to the surrounding environment of display device 60 (e.g., HDR display), such as the degree / level of ambient light in the surrounding environment.

[0065] In one embodiment, electronic device 110 includes one or more I / O units 140 integrated in or coupled to electronic device 110. In one embodiment, the one or more I / O units 140 include, but are not limited to, physical user interfaces (PUIs) and / or GUIs, such as keyboards, keypads, touch interfaces, touchscreens, knobs, buttons, displays, etc. In one embodiment, a user can utilize at least one I / O unit 140 to configure one or more user preferences, configure one or more parameters, provide input, etc.

[0066] In one embodiment, one or more applications on electronic device 110 may further include one or more applications 170 (also referred to as software mobile applications) loaded or downloaded to electronic device 110, such as camera applications, social media applications, video streaming applications, etc. The software mobile application 170 on electronic device 110 may exchange data with system 200.

[0067] In one embodiment, electronic device 110 includes communication unit 160 configured to exchange data with display device 60 (e.g., receiving data including peak brightness level D). nitThe display characteristics of the display device 60. The communication unit 160 is also configured to exchange data (e.g., receive video streams from the HDR content hosting environment 300) with a remote computing environment such as the HDR content hosting environment 300 via a communication network / connection 50 (e.g., a wireless connection, such as a Wi-Fi connection or a cellular data connection, a wired connection, or a combination of both). The communication unit 160 may include any suitable communication circuitry operable to connect to the communication network and exchange communication operations and media between the electronic device 110 and other devices connected to the same communication network 50. The communication unit 160 can be used to interface with the communication network using any suitable communication protocol, such as Wi-Fi (e.g., the IEEE 802.11 protocol). High-frequency systems (e.g., 900MHz, 2.4GHz, and 5.6GHz communication systems), infrared, GSM, GSM plus EDGE, CDMA, quad-band and other cellular protocols, VoIP, TCP-IP, or any other suitable protocol.

[0068] In one embodiment, the HDR content mastering environment 300 includes resources such as one or more servers 310 and one or more storage units 320. One or more applications 330 providing more advanced services can utilize the resources of the HDR content mastering environment 300 to execute / operate on the HDR content mastering environment 300. For example, in one embodiment, the HDR content mastering environment 300 provides an online platform for hosting one or more online services (e.g., video streaming services, etc.) and / or distributing one or more software mobile applications 170. As another example, the HDR tone mapping system 200 can be loaded onto or downloaded to the electronic device 110 from the HDR content mastering environment 300, which maintains and distributes updates to the HDR tone mapping system 200. As yet another example, the HDR content mastering environment 300 may include a cloud computing environment that provides a shared pool of configurable computing system resources and advanced services.

[0069] Figure 2 An example workflow for implementing HDR tone mapping for HDR content rendered on display device 60 is illustrated in one or more embodiments. In one embodiment, HDR content master environment 300 represents a computational environment for color grading at a studio. For example, in one embodiment, one or more applications 330 deployed on HDR content master environment 300 include a color grading unit 340 configured to: (1) receive a raw video, and (2) perform color grading on the raw video based on input from a user (e.g., a colorist in the studio) to produce a mastered video.

[0070] Let Q typically represent the number of quantization bits. In one embodiment, one or more applications 330 deployed on an HDR content master environment 300 include a quantizer unit 350 configured to: (1) receive a master video (e.g., from a color grading unit 340) and (2) perform quantization on the master video to generate a quantized signal of the master video.

[0071] In one embodiment, the HDR content mastering environment 300 includes a mastering monitor 370 configured to: (1) receive a quantization signal of the master video (e.g., from the quantizer unit 350), and (2) provide visual feedback to a user (e.g., a colorist in a studio) by displaying the master video based on the quantization signal, providing visual feedback on one or more color grading adjustments (i.e., adjustments to the original video produced by color grading).

[0072] In one embodiment, one or more applications 330 deployed on the HDR content mastering environment 300 include a tone mastering unit 360 configured to: (1) receive a master video (e.g., from a color grading unit 340), and (2) generate creative intent metadata corresponding to the master video. In one embodiment, the tone mastering unit 360 automatically generates the creative intent metadata. In another embodiment, the tone mastering unit 360 generates the creative intent metadata based on input from a user (e.g., a content creator in a studio).

[0073] In one embodiment, one or more applications 330 deployed on the HDR content master environment 300 include an encoder unit 380 configured to: (1) receive a quantized signal of a master video (e.g., from a quantizer unit 350), (2) receive creative intent metadata corresponding to the master video (e.g., from a tone master unit 360), (3) encode the quantized signal to produce an encoded video combined with the creative intent metadata, and (4) provide the encoded video for transmission via a communication network 50.

[0074] In one embodiment, the HDR tone mapping system 200 includes a decoder unit 210 deployed on an electronic device 110. In one embodiment, the decoder unit 210 is configured to: (1) receive encoded video (e.g., from an HDR content master environment 300) transmitted via a communication network 50; (2) perform decoding on the encoded video to generate a quantized signal of the main video (e.g., to a dequantizer unit 220); and (3) extract creative intent metadata from the encoded video that corresponds to the main video (e.g., to an HDR tone mapping 240 with an ambient light compensation system).

[0075] In one embodiment, the HDR tone mapping system 200 includes a dequantizer unit 220 deployed on an electronic device 110. In one embodiment, the dequantizer unit 220 is configured to: (1) receive a quantized signal of a main video (e.g., from a decoder unit 210), and (2) perform dequantization on the quantized signal video to generate the main video.

[0076] In one embodiment, the HDR tone mapping system 200 includes an ambient light signaling unit 230 deployed on an electronic device 110. In one embodiment, the ambient light signaling unit 230 is configured to: (1) receive sensor data captured by at least one sensor unit 150 of the electronic device 110, and (2) determine ambient light information based on the sensor data that indicates the degree / level of ambient light in the surrounding environment of the display device 60.

[0077] In one embodiment, the HDR tone mapping system 200 includes an HDR tone mapping system 240 with an ambient light compensation system deployed on an electronic device 110. In one embodiment, the HDR tone mapping system 240 with an ambient light compensation system is configured to: (1) receive a main video (e.g., from dequantizer unit 220), (2) receive creative intent metadata corresponding to the main video (e.g., from decoder unit 210), (3) receive ambient light information indicating the degree / level of ambient light in the surrounding environment of the display device 60 (e.g., from ambient light signal unit 230), (4) determine an adaptive / custom tone mapping function (i.e., tone mapping curve) with ambient light compensation based at least on the creative intent metadata and the ambient light information, (5) apply the tone mapping function to the main video to produce a tone-mapped video with ambient light compensation, and (6) provide the tone-mapped video to the display device 60 for rendering on the display device 60.

[0078] Figure 3 An example HDR tone map 240 with an ambient light compensation system is shown in one or more embodiments. In one embodiment, the HDR tone map 240 with an ambient light compensation system includes a metadata parser unit 250 deployed on an electronic device 110. In one embodiment, the metadata parser unit 250 is configured to: (1) receive creative intent metadata corresponding to a main video (e.g., from decoder unit 210), and (2) parse the creative intent metadata into two sets of metadata, specifically tone map metadata corresponding to the main video (e.g., to tone map curve generation unit 260) and ambient light-related metadata corresponding to the main video (e.g., to ambient light compensation system 270).

[0079] In one embodiment, the HDR tone mapping 240 with an ambient light compensation system includes a tone mapping curve generation unit 260 deployed on an electronic device 110. In one embodiment, the tone mapping curve generation unit 260 is configured to: (1) receive tone mapping metadata corresponding to the main video, and (2) generate a dynamic basic tone mapping function (i.e., a basic tone mapping curve) and corresponding parameters based on the tone mapping metadata.

[0080] In one embodiment, the HDR tone mapping 240 with an ambient light compensation system includes an ambient light compensation system 270 deployed on an electronic device 110. In one embodiment, the ambient light compensation system 270 is configured to: (1) receive ambient light-related metadata corresponding to the main video; (2) receive ambient light information (e.g., from ambient light signal unit 230) indicating the degree / level of ambient light in the surrounding environment of the display device 60; (3) receive a basic tone mapping function (i.e., a basic tone mapping curve) and corresponding parameters (e.g., from tone mapping curve generation unit 260); and (4) modify the basic tone mapping function based on the ambient light-related metadata and ambient light information to take into account the degree / level of ambient light in the surrounding environment, thereby generating a modified tone mapping function (i.e., a modified tone mapping curve) and corresponding parameters. The modified tone mapping function adaptively compensates for the degree / level of ambient light in the surrounding environment. The modified tone mapping function preserves the content creator's intent when considering the ambient light-related metadata.

[0081] In one embodiment, the HDR tone mapping 240 with an ambient light compensation system includes a tone mapping curve application system 280 deployed on the electronic device 110. In one embodiment, the tone mapping curve application system 280 is configured to: (1) receive a main video (e.g., from the dequantizer unit 220), (2) receive a modified tone mapping function (i.e., a tone mapping curve) and corresponding parameters (e.g., from the ambient light compensation system 270), and (3) apply the modified tone mapping function to the main video to produce a tone-mapped video with ambient light compensation (i.e., the tone-mapped video adaptively compensates for the degree / level of ambient light in the surrounding environment of the display device 60).

[0082] In one embodiment, the HDR tone map 240 with an ambient light compensation system includes a quantizer unit 290 deployed on an electronic device 110.

[0083] In one embodiment, the quantizer unit is configured to: (1) receive a tone-mapped video with ambient light compensation (e.g., from tone-mapped curve application system 280), (2) perform quantization on the tone-mapped video to generate a quantized signal of the tone-mapped video, and (3) provide the quantized signal to a display device 60 for rendering the tone-mapped video on the display device 60.

[0084] In one embodiment, the dequantizer unit 220, tone mapping curve application system 280, quantizer unit 290, and display device 60 are implemented using one or more HDR10+ standard processing pipelines available on the SoC (System-on-Chip) (i.e., without additional hardware), so that the SoC incurs no additional hardware cost.

[0085] Figure 4 An example ambient light compensation system 270 is illustrated in one or more embodiments. In one embodiment, the ambient light compensation system 270 is configured to determine three distinct factors corresponding to shadow pixels, midtone pixels, and highlight pixels of the main video, based on ambient light-related metadata corresponding to the main video.

[0086] Let [0, high] L This typically indicates a range of brightness levels, where high... L This represents the highest brightness level within the range. In one embodiment, the range [0, high] is the highest brightness level. L It is divided into S levels.

[0087] In one embodiment, the ambient light compensation system 270 includes a midtone factor unit 272 deployed on an electronic device 110. In one embodiment, the midtone factor unit 272 is configured to: (1) receive ambient light-related metadata (e.g., from a metadata parser unit 250) corresponding to the main video, and (2) determine a factor SubProbMidToneDiff corresponding to midtone pixels of the main video based on the ambient light-related metadata. In one embodiment, the factor SubProbMidToneDiff is based on the percentage difference between S levels of input percentile curves and uniform percentile curves (i.e., the low-range brightness difference between interpolated percentiles and uniform histogram percentiles). For example, in one embodiment, the factor SubProbMidToneDiff is determined according to formula (2) provided below:

[0088]

[0089] In one embodiment, the midtone factor unit 272 is configured to determine the factor SubProbMidToneDiffRatio using a ramp function based on SubProbMidToneDiff control.

[0090] In one embodiment, the ambient light compensation system 270 includes a shadow factor unit 271 deployed on an electronic device 110. In one embodiment, the shadow factor unit 271 is configured to: (1) receive ambient light-related metadata corresponding to the main video (e.g., from a metadata parser unit 250), and (2) determine a factor SubProbsShadowDiff corresponding to shadow pixels of the main video based on the ambient light-related metadata. In one embodiment, the factor SubProbsShadowDiff is based on the percentage difference between S levels of input percentile curves and uniform percentile curves (i.e., the low-range brightness difference between interpolated percentiles and uniform histogram percentiles).

[0091] In one embodiment, the ambient light compensation system 270 includes a highlight factor unit 273 deployed on an electronic device 110. In one embodiment, the highlight factor unit 273 is configured to: (1) receive ambient light-related metadata corresponding to the main video (e.g., from a metadata parser unit 250), and (2) determine a factor SubProbsHighlightDiff corresponding to the highlight pixels of the main video based on the ambient light-related metadata. In one embodiment, the factor SubProbsHighlightDiff is based on the average pLumHighAvg of the three highest percentiles of pLumHigh. For example, in one embodiment, pLumHighAvg is determined according to formula (3) provided below:

[0092]

[0093] Where L is the percentile in ambient light-related metadata. In one embodiment, the factor SubProbsHighlightDiff is determined using a ramp function controlled by pLumHighAvg.

[0094] In one embodiment, the ambient light compensation system 270 includes an ambient light compensation gain unit 274 deployed on the electronic device 110. In one embodiment, the ambient light compensation gain unit 274 is configured to: (1) receive ambient light information (e.g., from the ambient light signal unit 230) indicating the degree / level of ambient light in the surrounding environment of the display device 60, and (2) determine one or more gains based on the ambient light information for adaptive ambient light compensation in the surrounding environment. In one embodiment, the one or more gains include low Pratio and high P ratio This is used to determine the lower curve modifier and the upper curve modifier, respectively. In one embodiment, low P... ratio and high P ratio The ramp function is determined using a ramp function that controls the degree / level of ambient light in the surrounding environment.

[0095] In one embodiment, the ambient light compensation system 270 includes a combination unit 275 deployed on an electronic device 110. In one embodiment, the combination unit 275 is configured to: (1) receive a factor SubProbShadowDiff (e.g., from shadow factor unit 271) corresponding to shadow pixels of the main video; (2) receive a factor SubProbMidToneDiffRatio (e.g., from midtone factor unit 272) corresponding to midtone pixels of the main video; (3) receive a factor SubProbHighlightDiff (e.g., from highlight factor unit 273) corresponding to highlight pixels of the main video; and (4) determine a combination factor subProbAll based on each received factor. In one embodiment, the combination factor subProbAll is determined according to formula (4) provided below:

[0096] subProbAll=SubProbShadowDiff+SubProbMidToneDiffRatio+SubProbHighlightDiff (4)

[0097] In one embodiment, the ambient light compensation system 270 includes a mapping function unit 276 deployed on the electronic device 110. In one embodiment, the mapping function unit 276 is configured to: (1) receive a combination factor subpProbAll (e.g., from the combination unit 275), and (2) receive a gain lowP for ambient light compensation in the surrounding environment of the display device 60. ratio and high P ratio (e.g., from ambient light compensation gain unit 274), and (3) based on combination factor subpProbAll and gain low P ratio and high P ratio One or more modifiers are determined for modifying the basic tone mapping function (i.e., the basic tone mapping curve). In one embodiment, the one or more modifiers determined by the mapping function unit 276 include an inflection point modifier Δk(i) and a lower curve modifier ΔP for modifying the lower portion of the basic tone mapping function. low (i) and the upper curve modifier ΔP used to modify the upper part of the basic tone mapping function. high(i), where i∈(1,N). In one embodiment, the modifiers Δk(i) and ΔP low (i) and ΔP high (i) is determined using the ramp function.

[0098] In one embodiment, the ambient light compensation system 270 includes a combination unit 277 deployed on an electronic device 110. In one embodiment, the combination unit 277 is configured to: (1) receive a basic tone mapping function (i.e., a basic tone mapping curve) having corresponding parameters K(i) and P(i), and (2) receive modifiers Δk(i) and ΔP. low (i) and ΔP high (i) (e.g., from mapping function unit 276), and (3) based on modifier Δk(i), ΔP low (i) and ΔP high (i) Modify the basic tone mapping function to generate a function with corresponding parameters K′(i) and P′. low (i) and P′ high (i) Modified tone mapping function (i.e., modified tone mapping curve).

[0099] In one embodiment, the modified tone mapping function is an explicit Bézier curve that adaptively compensates for ambient light. The modified tone mapping function has multiple parts, each with multiple adjustment points along the Bézier curve to improve control over tone mapping. For example, in one embodiment, the parameter P′... low (i) and P′ high (i) are the adjustment points for the lower and upper portions (i.e., the curve) of the Bézier curve.

[0100] In one embodiment, the combination unit 277 is configured to optimize the lower curve modifier ΔP. low (i) to minimize parameter P′ low (i) and manual tuning parameter P tuned The difference between (i). For example, in one embodiment, the mapping function unit 276 determines the parameter P′ according to the formula (5) provided below. low (i):

[0101] P′ low (i)=P low (i)+lowP ratio ×ΔP low (i) (5)

[0102] Among them, P low (i) are the parameters corresponding to the basic tone mapping function at zero level of ambient light. Low P is determined using a ramp function controlled by the level of ambient light in the surrounding environment.ratio .

[0103] In one embodiment, the combination unit 277 is configured to optimize the upper curve modifier ΔP. high (i) to minimize parameter P high (i) and manual tuning parameter P high The difference between (i). For example, in one embodiment, the mapping function unit 276 determines the parameter P according to the formula (6) provided below. high (i):

[0104] P′ high (i)=P high (i)+highP ratio ×ΔP high (i) (6)

[0105] Among them, P high (i) are the parameters corresponding to the basic tone mapping function at zero level of ambient light. High P is determined using a ramp function controlled by the level of ambient light in the surrounding environment. ratio .

[0106] In one embodiment, the combination unit 277 is configured to optimize the inflection point modifier to minimize the parameter K′(i) and the manual tuning parameter K. tuned The difference between (i). For example, in one embodiment, the mapping function unit 276 determines the parameter K′(i) according to the formula (7) provided below:

[0107] K′(i)=K(i)+high P ratio ×ΔK(i) (7)

[0108] Where K(i) is a parameter corresponding to the basic tone mapping function at zero level of ambient light.

[0109] Figure 5 This is a graph 400 illustrating the percentage difference between the input percentile curve and the uniform percentile curve in one or more embodiments. The horizontal axis of graph 400 represents percentages. The vertical axis of graph 400 represents the percentiles of pLumHigh. Graph 400 includes the input percentile curve and the uniform percentile curve. Figure 5 As shown, the range is [0, high]. L The ambient light compensation system 270 is divided into S levels. At each of the S levels, the ambient light compensation system 270 is configured to determine a corresponding difference ΔpProbdock(k) 503, which represents the percentage difference between the input percentile curve 501 and the uniform percentile curve 502. For example, as... Figure 5 As shown, at the percentiles At this point, the corresponding difference ΔpProbdock(k)503 represents the percentage of the input percentile curve 501 PHL. k Percentage of uniform percentile curve 502 PHL uk The difference between them.

[0110] Figure 6 This is a graph 450 showing the ramp function of subProbMidToneDiffRatio in one or more embodiments. The horizontal axis of graph 450 represents subProbMidToneDiff. The vertical axis of graph 450 represents subProbMidToneDiffRatio. Figure 6 As shown, the subProbMidToneDiffRatio is determined using a ramp function controlled by subProbMidToneDiff.

[0111] Figure 7 This illustrates the gain high P in one or more embodiments. ratio The slope function is plotted in graph 500. The horizontal axis of graph 500 represents the intensity / level of ambient light in the surrounding environment (“ambient light level”). The vertical axis of graph 500 represents high P. ratio .like Figure 7 As shown, high P is determined using a ramp function controlled by the degree / level of ambient light in the surrounding environment. ratio .

[0112] Figure 8 This illustrates the gain low P in one or more embodiments. ratio The slope function is plotted in graph 550. The horizontal axis of graph 550 represents the degree / level of ambient light in the surrounding environment (“ambient light level”). The vertical axis of graph 550 represents low P. ratio .like Figure 8 As shown, low P is determined using a ramp function controlled by the degree / level of ambient light in the surrounding environment. ratio .

[0113] Figure 9 This illustrates ΔP in one or more embodiments. high (i) is represented by the slope function curve 600. The horizontal axis of curve 600 represents the combination factor subProbAll. The vertical axis of curve 600 represents ΔP. high (i). like Figure 9 As shown, ΔP is determined using a ramp function controlled by the combination factor subProbAll. high (i).

[0114] Figure 10This illustrates the curve modifier ΔP in one or more embodiments. low (i) is represented by curve 650, which is the ramp function. The horizontal axis of curve 650 represents the combination factor subProbAll. The vertical axis of curve 650 represents ΔP. low (i). like Figure 10 As shown, ΔP is determined using a ramp function controlled by the combination factor subProbAll. low (i).

[0115] Figure 11 This is a graph 700 showing the ramp function of ΔK(i) in one or more embodiments. The horizontal axis of graph 700 represents the combination factor subProbAll. The vertical axis of graph 700 represents ΔK(i). Figure 11 As shown, ΔK(i) is determined using a ramp function controlled by the combination factor subProbAll.

[0116] Figure 12 An example tone mapping curve application system 280 is illustrated in one or more embodiments. In one embodiment, tone mapping curve application system 280 provides hue preservation through tone mapping. Specifically, in one embodiment, tone mapping curve application system 280 includes a maximum value function unit 281. The maximum function unit 281 is configured to: (1) receive a master video (e.g., from dequantizer 220) including red (R) code values, green (g) code values, and blue (B) code values, and (2) determine the maximum code value maxRGB of the R, G, B code values ​​by applying a maximum (i.e., max) function to the R, G, and B code values ​​(e.g., max(R, G, B)).

[0117] In one embodiment, tone mapping curve application system 280 includes an electro-optical transfer function (EOTF) unit 282. EOTF unit 282 is configured to: (1) receive a maximum code value maxRGB (e.g., from maximum function unit 281), and (2) determine a linear luminance value X by applying the EOTF to the maximum code value maxRGB.

[0118] In one embodiment, tone mapping curve application system 280 includes tone mapping unit 283. Tone mapping unit 283 is configured to: (1) receive a linear luminance value X (e.g., from EOTF unit 282), (2) receive a modified tone mapping function (i.e., a modified tone mapping curve) (e.g., from ambient light compensation system 270), and (3) determine (i.e., find) the tone mapping value Y (e.g., tmLUT(X)) to which the modified tone mapping function maps the linear luminance value X.

[0119] In one embodiment, the tone mapping curve application system 280 includes a tone mapping ratio application unit 284. The tone mapping ratio application unit 284 is configured to: (1) receive a linear luminance value X (e.g., from EOTF unit 282), (2) receive a tone mapping value Y (e.g., from tone mapping unit 283), and (3) determine a tone mapping ratio r based on the linear luminance value X and the tone mapping value Y, wherein... By applying EOTF to each of the R, G, and B code values ​​respectively, linear values ​​EOTF(R), EOTF(G), and EOTF(B) are determined, and (5) a tone mapping ratio r is applied to each of the linear values ​​EOTF(R), EOTF(G), and EOTF(B) to produce scaled linear values ​​EOTF(R)*r, EOTF(G)*r, and EOTF(B)*r respectively.

[0120] In one embodiment, tone mapping curve application system 280 includes photoelectric transfer function (OETF) unit 285. OETF unit 285 is configured to: (1) receive scaling linear values ​​EOTF(R)*r, EOTF(G)*r, and EOTF(B)*r (e.g., from tone mapping ratio application unit 284), and (2) determine the R' code value, G' code value, and B' code value of tone-mapped video by applying OETF to each of the scaling linear values ​​EOTF(R)*r, EOTF(G)*r, and EOTF(B)*r, thereby producing tone-mapped video with ambient light compensation and color hue preservation (i.e., the x, y coordinates in the CIE xyY color space remain unchanged, thus preserving the hue).

[0121] Table 1 below provides example pseudocode for hue preservation implemented by tone mapping curve application system 280 in one or more embodiments.

[0122]

[0123]

[0124] Figure 13 An example ambient light compensation development system 800 is shown in one or more embodiments. In one embodiment, the ambient light compensation development system 800 is used to develop an algorithm for ambient light compensation that maintains creative intent in different ambient environments. Specifically, in one embodiment, the ambient light compensation development system 800 includes: (1) a reference monitor 810 having a peak brightness level M deployed on a first environment (“reference ambient environment”) used as a reference; (2) a display device 60 having a peak brightness level N deployed on a controlled second ambient environment (“controlled non-reference ambient environment”) different from the reference ambient environment; and (3) an image adjustment unit 820.

[0125] Let P g A tone mapping curve (“ground truth curve”) typically represents the ground truth value of ambient light compensation. The development of the ambient light compensation algorithm includes the following steps: (1) displaying an image of the main video (“main image”) on a reference monitor 810 to provide the operator with creative intent viewing in the reference surrounding environment; (2) displaying an image of the tone-mapped video (“tone-mapped image”) on a display device 60 to provide the operator with adjusted content viewing in a controlled non-reference environment, wherein the tone-mapped image is the main image with ambient light compensation; (3) the operator compares the tone-mapped image with the main image, and generates a tone-mapped image based on the comparison by an image adjustment unit 820; and (4) generating a ground truth curve P based on the adjustment by an image adjustment unit 820. g .

[0126] Let P a This typically refers to an ambient light compensation curve (“automatic curve”) automatically generated based on an ambient light compensation algorithm. In one embodiment, the ambient light compensation development system 800 includes a fine-tuning unit 830 and a difference unit 840. The development of the ambient light compensation algorithm further includes the steps of: (1) determining the difference θ between the ground truth curve and the automatic curve via the difference unit 840, and (2) an operator (or developer) fine-tuning the algorithm to minimize the difference θ. In one embodiment, the resulting fine-tuning algorithm is implemented in the ambient light compensation system 270 (e.g., deployed in the ambient light correction gain unit 274) to correct the content (e.g., main video) provided to the display device 60, thereby compensating for the effects of ambient light in the environment of the display device 60.

[0127] In one embodiment, the difference θ is determined according to the formula (8) provided below:

[0128] θ = argmin(|P g -P a |)(8)

[0129] Figure 14A This is an example main image 900 viewed in a reference environment with low levels of ambient light in one or more embodiments. In one embodiment, the reference ambient environment refers to the surrounding environment in a studio suitable for color grading. For example, the reference ambient environment is dark, and the colorist views the main image 900 on a master monitor 370.

[0130] Figure 14BExample tone-mapped image 910 is viewed in one or more embodiments in an environment with high level / level ambient light. In one embodiment, the ambient environment refers to the surrounding environment around the display device 60. For example, the ambient environment is bright, and a user views tone-mapped image 910 on display device 60. HDR tone mapping 240 with an ambient light compensation system generates an ambient light compensation curve based on creative intent metadata corresponding to the main image 900. Figure 14A Furthermore, based on the ambient light compensation curve, ambient light is applied to the main image 900 to produce a tone-mapped image 910 with compensation for a bright surrounding environment. The user's perception of viewing the tone-mapped main image in a bright surrounding environment is similar to the perception of a colorist viewing the main image in a dark reference surrounding environment.

[0131] Figure 14C This is shown in one or more embodiments with Figures 14A-14B Example Figure 920 shows the histograms corresponding to the main image 900 and the tone-mapped image 910. The horizontal axis of Figure 920 represents brightness in nits. The vertical axis of Figure 920 represents the pixel count. Figure 920 includes: (1) a first histogram 921 corresponding to the main image 900, and (2) a second histogram 922 corresponding to the tone-mapped image 910.

[0132] Figure 15A Example main image 950 is viewed in one or more embodiments in a reference environment with low levels of ambient light. In one embodiment, the reference surrounding environment refers to the surrounding environment in a studio suitable for color grading. For example, the reference surrounding environment is dark, and the colorist is viewing the main image on a master monitor 370.

[0133] Figure 15B Example tone-mapped image 960 is viewed in one or more embodiments in an ambient environment with a high degree / level of ambient light. In one embodiment, the ambient environment refers to the surrounding environment of display device 60. For example, the ambient environment is bright, and a user views tone-mapped image 960 on display device 60. HDR tone mapping 240 with an ambient light compensation system generates an ambient light compensation curve based on creative intent metadata corresponding to the main image 950. Figure 15A Furthermore, based on the ambient light compensation curve, ambient light compensation is applied to the main image 950 to produce a tone-mapped image 960 with compensation for a bright surrounding environment. The user's perception of viewing the tone-mapped main image in a bright surrounding environment is similar to the colorist's perception of viewing the main image in a dark reference surrounding environment.

[0134] Figure 15C This is shown in one or more embodiments with Figures 15A-15BExample Figure 970 shows the histograms corresponding to the main image and the tone-mapped image. The horizontal axis of Figure 970 represents brightness in nits. The vertical axis of Figure 970 represents the pixel count. Figure 970 includes: (1) a first histogram 971 corresponding to the main image, and (2) a second histogram 972 corresponding to the tone-mapped image.

[0135] In one embodiment, for the same level of ambient light in the surrounding environment, the ambient light compensation system 270 generates different ambient light compensation curves for different images (i.e., frames / scenes) based on creative intent metadata corresponding to the image.

[0136] Figure 16A Example image 1000 (“dark image”) with a large amount of dark detail in one or more embodiments. Figure 16B This is another example image 1010 (“bright image”) with a large amount of bright detail in one or more embodiments. Figures 16A-16B As shown, dark image 1000 has more dark details than bright image 1010, and bright image 1010 has more bright details than dark image 1000.

[0137] Figure 16C This is shown in one or more embodiments. Figures 16A-16B Example Figure 1020 shows the ambient light compensation curves for the dark image 1000 and the bright image 1010. The horizontal axis of Figure 1020 represents the input brightness in nits. The vertical axis of Figure 1020 represents the output brightness in nits. Figure 1020 includes: (1) a first ambient light compensation curve 1021 generated by the ambient light compensation system 270 for the dark image 1000 based on creative intent metadata and ambient light information (i.e., the degree / level of ambient light) corresponding to the dark image 1000, and (2) a second ambient light compensation curve 1022 generated by the ambient light compensation system 270 for the bright image 1010 based on creative intent metadata and the same ambient light information (i.e., the same degree / level of ambient light) corresponding to the bright image 1010.

[0138] like Figure 16C As shown, the first ambient light compensation curve 1021 takes into account a higher number of dark details in the dark image 1000, and the second ambient light compensation line 1022 takes into account a lower number of dark details in the bright image 1010. More specifically, because the dark image 1000 has a large number of dark details, it provides better illumination at low input brightness (e.g., in...). Figure 16C In a dark image 1000 (with an input brightness value of 1 / 10 nits), the difference between the output brightness and the input brightness is greater than the difference between the output brightness and the input brightness of a bright image 1010 at low input brightness. Conversely, because the bright image 1010 has a large amount of bright detail, the difference is greater at high input brightness (e.g., ...). Figure 16CThe difference between the output brightness and input brightness of a bright image 1010 (with an input brightness value of 10 nits) is greater than the difference between the output brightness and input brightness of a dark image 1000 with a high input brightness.

[0139] Figure 16D This is shown in one or more embodiments. Figures 16A-16B Example Figure 1030 shows histograms of dark image 1000 and bright image 1010. The horizontal axis of Figure 1030 represents the input brightness in nits. The vertical axis of Figure 1030 represents the pixel count. Figure 1030 includes: (1) a first histogram 1031 corresponding to the dark image 1000, and (2) a second histogram 1032 corresponding to the bright image 1010. Figure 16D As shown, dark image 1000 has more dark details than bright image 1010, and bright image 1010 has more bright details than dark image 1000.

[0140] Figure 16E This is shown in one or more embodiments. Figures 16A-16B Example Figure 1040 shows the CDF curves of the dark image 1000 and the bright image 1010. The horizontal axis of Figure 1040 represents the input luminance in nits. The vertical axis of Figure 1040 represents percentages. Figure 1040 includes: (1) a first CDF curve 1041 representing the pixel distribution in the dark image 1000, wherein the distribution is included in the creative intent metadata corresponding to the dark image 1000, and (2) a second CDF curve 1042 representing the pixel distribution in the bright image 1010, wherein the distribution is included in the creative intent metadata corresponding to the bright image 1010. Figure 16E As shown, dark image 1000 has more dark details than bright image 1010, and bright image 1010 has more bright details than dark image 1000.

[0141] In one embodiment, for different levels / degrees of ambient light in different surrounding environments, the ambient light compensation system 270 generates different ambient light compensation curves for the same image (i.e., frame / scene) based on creative intent metadata corresponding to the image. For example, suppose a first surrounding environment has a first level / degree of ambient light, and a second environment has a second level / degree of ambient light that is brighter than the first level / degree of ambient light (i.e., the second surrounding environment is brighter). For the same image, the ambient light compensation system 270 generates a first ambient light compensation curve based on the first level / degree of ambient light, and generates a second ambient light compensation curve based on the second level / degree of ambient light, wherein the second ambient light compensation curve is brighter than the first ambient light compensation curve.

[0142] Figure 17This is a flowchart of an example process 1100 for implementing HDR tone mapping for rendering HDR content on a display device, in one or more embodiments. Operation 1110 includes determining multidimensional metadata corresponding to an input image, wherein the multidimensional metadata includes a distribution function of pixels in the input image. Operation 1120 includes determining ambient light information indicating the level of ambient light in the environment surrounding the display device. Operation 1130 includes determining one or more gains based on the ambient light information to adaptively compensate for the level of ambient light in the surrounding environment. Operation 1140 includes generating a tone mapping function based on the multidimensional metadata and one or more gains. The operation includes applying the tone mapping function to the input image to generate a tone-mapped image that adaptively compensates for the level of ambient light in the surrounding environment, wherein the tone-mapped image is provided to the display device for rendering.

[0143] In one embodiment, operations 1110-1150 may be performed by one or more components of the HDR tone mapping system 200, such as ambient light signal unit 230 and HDR tone mapping 240 with an ambient light compensation system, including metadata parser unit 250, ambient light compensation system 270 and tone mapping curve application unit 280.

[0144] Figure 18 This is a high-level block diagram illustrating an information processing system including a computer system 1200 for implementing the disclosed embodiments. System 200 may be incorporated into computer system 1200. Computer system 1200 includes one or more processors 1201 and may further include an electronic display device 1202 (for displaying video, graphics, text, and other data), main memory 1203 (e.g., random access memory (RAM)), storage device 1204 (e.g., hard disk drive), removable storage device 1205 (e.g., removable storage drive, removable storage module, tape drive, optical disk drive, computer-readable medium storing computer software and / or data), viewer interface device 1206 (e.g., keyboard, touchscreen, keypad, pointing device), and communication interface 1207 (e.g., modem, network interface (such as Ethernet card), communication port, or PCMCIA slot and card). Communication interface 1207 allows the transfer of software and data between the computer system and external devices. System 1200 also includes communication infrastructure 1208 (e.g., communication bus, crossover cable, or network) to which the aforementioned devices / modules 1201 to 1207 are connected.

[0145] Information transmitted via communication interface 1207 may be in the form of signals, such as electronic, electromagnetic, optical, or other signals that can be received by communication interface 1207 via a signal-carrying communication link, and may be implemented using wired or cable, fiber optic, telephone line, cellular telephone link, radio frequency (RF) link, and / or other communication code values. Computer program instructions representing the block diagram and / or flowchart herein may be loaded onto a computer, programmable data processing apparatus, or processing device to cause a series of operations to be performed thereon, thereby generating a computer-implemented process. In one embodiment, for processing 1100 ( Figure 17 The processing instructions can be stored as program instructions on memory 1203, storage device 1204 and / or removable storage device 1205 for execution by processor 1201.

[0146] Embodiments have been described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products. Each block or combination thereof in such illustrations / diagrams can be implemented by computer program instructions. When provided to a processor, the computer program instructions generate machinery such that the instructions, executed via the processor, create means for implementing the functions / operations specified in the flowchart illustrations and / or block diagrams. Each block in the flowchart / block diagram may represent a hardware and / or software module or logic. In alternative implementations, the functions marked in the blocks may occur out of order, simultaneously, etc.

[0147] The terms "computer program medium," "computer-usable medium," "computer-readable medium," and "computer program product" generally refer to media such as main memory, secondary storage, removable storage drives, hard disks installed in hard disk drives, and signals. These computer program products are means for providing software to a computer system. A computer-readable medium allows a computer system to read data, instructions, messages, or message packets, and other computer-readable information from it. For example, a computer-readable medium may include non-volatile memory such as floppy disks, ROM, flash memory, disk drive memory, CD-ROM, and other permanent storage. It can be used, for example, to transfer information such as data and computer instructions between computer systems. Computer program instructions may be stored in a computer-readable medium and may instruct a computer, other programmable data processing apparatus, or other device to operate in a particular manner, causing the instructions stored in the computer-readable medium to produce an article of manufacture including instructions that implement the functions / actions specified in flowcharts and / or one or more block diagram blocks.

[0148] As those skilled in the art will understand, aspects of the embodiments may be embodied as a system, method, or computer program product. Therefore, aspects of the embodiments may take the form of a completely hardware embodiment, a completely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, which may generally be referred to herein as a “circuit,” “module,” or “system.” Furthermore, aspects of the embodiments may take the form of a computer program product embodied in one or more computer-readable media having computer-readable program code embodied thereon.

[0149] Any combination of one or more computer-readable media may be used. A computer-readable medium can be a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media will include the following: electrical connections having one or more wires, portable computer floppy disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable optical disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In the context of this document, a computer-readable storage medium can be any tangible medium that can contain or store programs for use by or in connection with an instruction execution system, apparatus, or device.

[0150] Computer program code for performing operations of one or more embodiments can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, etc., and traditional procedural programming languages ​​such as the "C" programming language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer, partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet through an Internet service provider).

[0151] The flowcharts and / or block diagrams of the methods, apparatus (systems), and computer program products described above illustrate aspects of one or more embodiments. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a special-purpose computer or other programmable data processing apparatus to produce a machine, such that the instructions, executable via a processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in the flowchart and / or one or more block diagram blocks.

[0152] These computer program instructions may also be stored in a computer-readable medium that can instruct a computer, other programmable data processing apparatus or other device to operate in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture including instructions that implement the functions / actions specified in a flowchart and / or one or more block diagram blocks.

[0153] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to perform a series of operational steps on the computer, other programmable data processing apparatus or other device to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide processing for implementing the functions / actions specified in the flowchart and / or one or more block diagram blocks.

[0154] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions comprising one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions shown in the blocks may not appear in the order shown in the figures. For example, in fact, two blocks shown consecutively may be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order, depending on the functions involved. It will also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a system based on dedicated hardware that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0155] Unless explicitly stated otherwise, references to singular elements in the claims do not imply "uniqueness" but rather "one or more". All structural and functional equivalents of the elements of the exemplary embodiments described above, now or hereafter known to those skilled in the art, are intended to be covered by these claims.

[0156] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the technology disclosed. As used herein, the singular forms “a,” “an,” and “the” also include the plural forms unless the context clearly indicates otherwise. It should be further understood that when the terms “comprising” and / or “including” are used in this specification, the presence of the stated features, integers, steps, operations, elements, and / or components is specified, but the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or combinations thereof is not excluded.

[0157] All the means or steps plus functional elements in the following claims are intended to include any structure, material, action, and equivalent for performing a function in combination with other claimed elements of the specific claim. The description of the embodiments is for illustrative and descriptive purposes only and is not intended to be exhaustive or limited to the embodiments of the disclosed forms. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the disclosed technology.

[0158] Although embodiments have been described with reference to certain versions of the examples, other versions are also possible. Therefore, the spirit and scope of the appended claims should not be limited to the preferred versions described herein.

Claims

1. A method for presenting an image, comprising: Determine the multidimensional metadata corresponding to the input image, wherein the multidimensional metadata includes the distribution function of the pixels in the input image; Determine ambient light information to indicate the ambient light level in the surrounding environment of the display device; Based on ambient light information, determine one or more gains to adaptively compensate for the ambient light level in the surrounding environment; One or more gain-based tone mapping functions are generated based on multidimensional metadata and adaptive compensation for ambient light levels in the surrounding environment; and A tone mapping function is applied to the input image to generate a tone-mapped image that adaptively compensates for ambient light levels in the surrounding environment, wherein the tone-mapped image is provided to a display device for rendering.

2. The method according to claim 1, wherein, Distribution functions include the cumulative distribution function; and For each percentile of the cumulative distribution function, the multidimensional metadata includes a corresponding pair of values, including a percentile brightness value and a pixel percentage value.

3. The method of claim 1, wherein, The tone mapping function represents a Bézier curve comprising multiple parts, each of which has a corresponding adjustment point along the Bézier curve.

4. The method of claim 3, wherein, The functions for generating tone maps include: Based on one or more gains, determine the inflection point modifier, the lower curve modifier for generating the lower portion of the Bézier curve, and the upper curve modifier for generating the upper portion of the Bézier curve.

5. The method of claim 4, wherein, The tone mapping generation function also includes: Based on the inflection point modifier, lower curve modifier, and upper curve modifier, generate a tone mapping function and one or more parameters corresponding to the tone mapping function.

6. The method of claim 5, wherein, One or more parameters include an inflection point, a first adjustment point corresponding to the lower part of the Bézier curve, and a second adjustment point corresponding to the upper part of the Bézier curve.

7. The method of claim 4, wherein, The functions for generating tone maps include: Generate basic tone mapping functions based on tone mapping metadata; and Tone mapping functions are generated based on the basic tone mapping function, inflection point modifier, lower curve modifier, and upper curve modifier.

8. The method according to claim 1, wherein, Providing tone-mapped images to display devices includes: Perform quantization on the tone-mapped image and generate the quantized signal of the tone-mapped image; and The quantized signal of the tone-mapped image is provided to the display device for rendering.

9. The method according to claim 1, wherein, Determining one or more gains includes: Based on ambient light information, one or more gains are determined using a ramp function based on the ambient light level in the surrounding environment.

10. A system for presenting an image, comprising: At least one processor; as well as A processor-readable storage device that stores instructions, which, when executed by at least one processor, cause at least one processor to perform an operation, includes: Determine the multidimensional metadata corresponding to the input image, wherein the multidimensional metadata includes the distribution function of the pixels in the input image; Determine ambient light information to indicate the ambient light level in the surrounding environment of the display device; Based on ambient light information, determine one or more gains to adaptively compensate for the ambient light level in the surrounding environment; One or more gain-based tone mapping functions are generated based on multidimensional metadata and adaptive compensation for ambient light levels in the surrounding environment; and A tone mapping function is applied to the input image to generate a tone-mapped image that adaptively compensates for ambient light levels in the surrounding environment, wherein the tone-mapped image is provided to a display device for rendering.

11. The system according to claim 10, wherein, Distribution functions include the cumulative distribution function; and For each percentile of the cumulative distribution function of a pixel, the multidimensional metadata includes a corresponding pair of values, including a percentile luminance value and a pixel percentage value.

12. The system according to claim 10, wherein, The tone mapping function represents a Bézier curve comprising multiple parts, each of which has a corresponding adjustment point along the Bézier curve.

13. The system according to claim 12, wherein, The functions for generating tone maps include: Based on one or more gains, determine the inflection point modifier, the lower curve modifier for generating the lower portion of the Bézier curve, and the upper curve modifier for generating the upper portion of the Bézier curve.

14. The system according to claim 13, wherein, The tone mapping generation function also includes: Generate basic tone mapping functions based on tone mapping metadata; and Tone mapping functions are generated based on the basic tone mapping function, inflection point modifier, lower curve modifier, and upper curve modifier.

15. A processor-readable medium having a computer program stored thereon, the computer program performing the method according to claims 1 to 9 when executed by a processor.

Images