HDR tone mapping based on creative intent metadata and ambient lighting

Abstract

One embodiment provides a method comprising the operation of determining multidimensional metadata corresponding to an input image and the operation of determining ambient lighting information representing the degree of ambient lighting within the surrounding environment of a display device. The multidimensional metadata includes a dispersion function of pixels within the input image. The method further comprises the operation of determining one or more gains that adaptively compensate for the degree of ambient lighting within the surrounding environment based on the ambient lighting information. The method further comprises the operation of generating a tone mapping function based on the multidimensional metadata and one or more gains, and the operation of applying the tone mapping function to the input image to generate a tone-mapped image that adaptively compensates for the degree of ambient lighting within the surrounding environment. The tone-mapped image is provided to a display device for display.

Description

Technology Field

[0001] One or more embodiments generally relate to settings for consumer electronics devices, and in particular to a method and system for high-dynamic range (HDR) tone mapping based on creative intent metadata and ambient lighting. Background Technology

[0002] Consumer electronics (e.g., smart televisions, smartphones, etc.) are currently equipped with the latest display screens (e.g., QLED, OLED) that provide ultra-high definition. means of solving the problem

[0003] One embodiment provides a method comprising the steps of determining multidimensional metadata corresponding to an input image and determining ambient lighting information representing the degree of ambient lighting within the surrounding environment of a display device. The multidimensional metadata includes a distributed function of pixels within the input image. The method further comprises the step of determining one or more gains that adaptively compensate for the degree of ambient lighting within the surrounding environment based on the ambient lighting information. The method further comprises the steps of generating a tone mapping function based on the multidimensional metadata and one or more gains, and applying the tone mapping function to the input image to generate a tone-mapped image that adaptively compensates for the degree of ambient lighting within the surrounding environment. The tone-mapped image is provided to a display device for display.

[0004] Another embodiment provides a method comprising the steps of determining multidimensional metadata corresponding to an input image and determining ambient light information representing the degree of ambient light in the surrounding environment of a display device. The multidimensional metadata includes a cumulative variance function (CDF) of pixels in the input image. The method further comprises the step of determining one or more gains that adaptively compensate for the degree of ambient light in the surrounding environment based on the multidimensional metadata and ambient light information. The method further comprises the steps of generating a tone mapping function based on one or more gains, and applying the tone mapping function to the input image to generate a tone-mapped image that adaptively compensates for the degree of ambient light in the surrounding environment. The tone-mapped image is provided to a display device for display.

[0005] In another embodiment, the tone mapping function maintains the hue of the input image.

[0006] In another embodiment, the input image is a frame of the input video.

[0007] Another embodiment provides a system comprising at least one processor and a processor-readable memory device that stores instructions for the processor to perform operations when executed by at least one processor. The operations include an operation of determining multidimensional metadata corresponding to an input image and an operation of determining ambient light information indicating the degree of ambient light in the surrounding environment of a display device. The multidimensional metadata includes a variance function of pixels in the input image. The operations further include an operation of determining one or more gains that adaptively compensate for the degree of ambient light in the surrounding environment based on the ambient light information. The operations further include an operation of generating a tone mapping function based on the multidimensional metadata and one or more gains, and an operation of applying the tone mapping function to the input image to generate a tone-mapped image that adaptively compensates for the degree of ambient light in the surrounding environment. The tone-mapped image is provided to a display device for display.

[0008] Another embodiment provides a system comprising at least one processor and a processor-readable memory device that stores instructions for the processor to perform operations when executed by at least one processor. The operations include an operation of determining multidimensional metadata corresponding to an input image and an operation of determining ambient light information indicating the degree of ambient light in the surrounding environment of a display device. The multidimensional metadata includes the CDF of pixels in the input image. The operations further include an operation of determining one or more gains that adaptively compensate for the degree of ambient light in the surrounding environment based on the ambient light information. The operations further include an operation of generating a tone mapping function based on one or more gains, and an operation of applying the tone mapping function to the input image to generate a tone-mapped image that adaptively compensates for the degree of ambient light in the surrounding environment. The tone-mapped image is provided to a display device for display.

[0009] Another embodiment provides a system comprising at least one processor and a processor-readable memory device that stores instructions that cause the processor to perform operations when executed by at least one processor. The operations include generating a tone mapping function and one or more parameters corresponding to the tone mapping function based on a knee point modifier, a lower curve modifier, and an upper curve modifier.

[0010] In another embodiment, the tone mapping function maintains the hue of the input image.

[0011] In another embodiment, the input image is a frame of the input video.

[0012] Another embodiment provides a processor-readable medium comprising a program that performs a method, when executed by a processor, including the step of determining multidimensional metadata corresponding to an input image and the step of determining ambient light information indicating the degree of ambient light in the surrounding environment of a display device. The multidimensional metadata includes a distributed function of pixels in the input image. The method further comprises the step of determining one or more gains that adaptively compensate for the degree of ambient light in the surrounding environment based on the ambient light information. The method further comprises the step of generating a tone mapping function based on the multidimensional metadata and one or more gains, and the step of applying the tone mapping function to the input image to generate a tone-mapped image that adaptively compensates for the degree of ambient light in the surrounding environment. The tone-mapped image is provided to a display device for display.

[0013] Another embodiment provides a processor-readable medium comprising a program that performs a method, when executed by a processor, including the step of determining multidimensional metadata corresponding to an input image and the step of determining ambient light information indicating the degree of ambient light in the surrounding environment of a display device. The multidimensional metadata includes the CDF of pixels in the input image. The method further comprises the step of determining one or more gains that adaptively compensate for the degree of ambient light in the surrounding environment based on the multidimensional metadata and ambient light information. The method further comprises the step of generating a tone mapping function based on one or more gains, and the step of applying the tone mapping function to the input image to generate a tone-mapped image that adaptively compensates for the degree of ambient light in the surrounding environment. The tone-mapped image is provided to a display device for display.

[0014] In another embodiment, multidimensional metadata includes, for each percentile of the CDF, a pair of corresponding values ​​including a percentile luminance value and a pixel percentage value.

[0015] In another embodiment, the tone mapping function represents a Bezier curve comprising a plurality of sections, each of which has a corresponding adjustment point along the Bezier curve.

[0016] Another embodiment provides a processor-readable medium comprising a program that performs a method, which, when executed by a processor, determines, based on one or more gains, a knee point modifier, a sub-curve modifier for generating a sub-section of a Bézier curve, and an upper-curve modifier for generating an upper section of a Bézier curve; and based on the knee point modifier, the sub-curve modifier, and the upper-curve modifier, generates a tone mapping function and one or more parameters corresponding to the tone mapping function.

[0017] These and other aspects and advantages of one or more embodiments will become apparent from the following, which is presented as an example of the principles of one or more embodiments in conjunction with the drawings. Brief explanation of the drawing

[0018] For a more complete understanding of the characteristics and benefits of the embodiments as well as the preferred modes of use, please refer to the following detailed description in conjunction with the attached drawings. FIG. 1 illustrates an exemplary computing structure for implementing HDR tone mapping of high-dynamic range (HDR) content to be displayed on a display device in one or more embodiments. FIG. 2 illustrates an exemplary workflow for implementing HDR tone mapping of HDR content to be displayed on a display device in one or more embodiments. FIG. 3 illustrates an exemplary HDR tone mapping system that compensates for ambient light in one or more embodiments. FIG. 4 illustrates an exemplary ambient light compensation system in one or more embodiments. FIG. 5 is a graph showing the percentage difference between the input percentile curve and the uniform percentile curve in one or more embodiments. FIG. 6 is a graph illustrating the ramp function for SubProbMidToneDiffRatio in one or more embodiments. FIG. 7 shows, in one or more embodiments, the gain highP ratio This is a graph plotting the ramp function for . FIG. 8 shows the gain lowP in one or more embodiments. ratio This is a graph plotting the ramp function for . FIG. 9 shows ΔP in one or more embodiments. high This is a graph showing the ramp function for (i). FIG. 10 shows, in one or more embodiments, a sub-curve modifier ΔP low This is a graph showing the ramp function for (i). FIG. 11 is a graph showing the ramp function for ΔK(i) in one or more embodiments. FIG. 12 illustrates an exemplary tone mapping curve application system in one or more embodiments. FIG. 13 illustrates an exemplary ambient light compensation development system in one or more embodiments. FIG. 14a is an exemplary mastered image shown in one or more embodiments in a reference ambient environment with low degree / illumination ambient lighting. FIG. 14b is an exemplary tone-mapped image shown in one or more embodiments in an ambient environment with high degree / illumination ambient lighting. FIG. 14c is an exemplary graph illustrating histograms corresponding to the mastered images and tone-mapped images of FIG. 14a-14b in one or more embodiments. FIG. 15a is an exemplary mastered image shown in one or more embodiments in a reference ambient environment with low degree / illumination ambient lighting. FIG. 15b is an exemplary tone-mapped image shown in one or more embodiments in an ambient environment with high degree / illumination ambient lighting. FIG. 15c is an exemplary graph illustrating histograms corresponding to the mastered images and tone-mapped images of FIG. 15a-15b in one or more embodiments. FIG. 16a is an exemplary image ("dark image") with a large amount of dark detail in one or more embodiments. FIG. 16b is an exemplary image ("bright image") with a large amount of bright detail in one or more embodiments. FIG. 16c is an exemplary graph illustrating ambient light compensation curves for the dark and bright images of FIG. 16a-16b in one or more embodiments. FIG. 16d is an exemplary graph illustrating histograms for the dark and bright images of FIG. 16a-16b in one or more embodiments. FIG. 16e is an exemplary graph illustrating CDF curves for the dark and bright images of FIG. 16a-16b in one or more embodiments. FIG. 17 is a flowchart of an exemplary process for implementing HDR tone mapping of HDR content to be displayed on a display device in one or more embodiments. FIG. 18 is a top-level block diagram illustrating an information processing system including a computer system used to implement the disclosed embodiments. Specific details for implementing the invention

[0019] The following description is intended to illustrate the general principles of one or more embodiments and is not intended to limit the concepts of the invention claimed herein. Furthermore, specific configurations described herein may be used in combination with other configurations described in each of the various possible combinations and substitutions. Unless otherwise specifically defined in this specification, all terms shall be interpreted in the broadest possible sense, including not only the meanings implied in this specification but also the meanings understood by those skilled in the art and / or the meanings defined in dictionaries, treaties, etc.

[0020] One or more embodiments generally relate to the configuration of consumer electronics devices, and in particular to methods and systems for high-dynamic range (HDR) tone mapping based on creative intent metadata and ambient lighting. One embodiment provides a method comprising the steps of determining multidimensional metadata corresponding to an input image and determining ambient lighting information representing the degree of ambient lighting within the surrounding environment of a display device. One embodiment provides a method comprising the steps of determining multidimensional metadata corresponding to an input image and determining ambient lighting information representing the degree of ambient lighting within the surrounding environment of a display device. The method further comprises the step of determining one or more gains that adaptively compensate for the degree of ambient lighting within the surrounding environment based on the ambient lighting information. The method further comprises the steps of generating a tone mapping function based on the multidimensional metadata and one or more gains, and applying the tone mapping function to an input image to generate a tone-mapped image that adaptively compensates for the degree of ambient lighting within the surrounding environment. The tone-mapped image is provided to a display device for display.

[0021] Another embodiment provides a system comprising at least one processor and a processor-readable memory device that stores instructions for the processor to perform operations when executed by at least one processor. The operations include an operation of determining multidimensional metadata corresponding to an input image and an operation of determining ambient light information representing the degree of ambient light in the surrounding environment of a display device. The multidimensional metadata includes the variance of pixels in the input image (also referred to as a variance function). The operations further include an operation of determining one or more gains that adaptively compensate for the degree of ambient light in the surrounding environment based on the ambient light information. The operations further include an operation of generating a tone mapping function based on the multidimensional metadata and one or more gains, and an operation of applying the tone mapping function to the input image to generate a tone-mapped image that adaptively compensates for the degree of ambient light in the surrounding environment. The tone-mapped image is provided to a display device for display.

[0022] Another embodiment provides a processor-readable medium comprising a program that performs a method, when executed by a processor, including the step of determining multidimensional metadata corresponding to an input image and the step of determining ambient light information indicating the degree of ambient light in the surrounding environment of a display device. The multidimensional metadata includes the distribution of pixels in the input image. The method further comprises the step of determining one or more gains that adaptively compensate for the degree of ambient light in the surrounding environment based on the ambient light information. The method further comprises the step of generating a tone mapping function based on the multidimensional metadata and one or more gains, and the step of applying the tone mapping function to the input image to generate a tone-mapped image that adaptively compensates for the degree of ambient light in the surrounding environment. The tone-mapped image is provided to a display device for display.

[0023] For the purposes of explanation, the terms "tone mapping curve" and "tone mapping function" are used interchangeably in this specification. For the purposes of explanation, as used herein, "ambient light compensation curve" generally refers to a tone mapping curve that compensates for ambient light.

[0024] Studios provide tone-mastered content to be displayed on display devices in very dark environments. For example, the environment in which a colorist in a studio selects the colors of the content is very dark. However, people rarely watch TV or any other device in complete darkness. While content is being displayed on a display device, if the illuminance of the ambient lighting in the environment surrounding the display device increases, the perceived on-screen image quality on the display device steadily degrades.

[0025] The term "creative intent" can generally be said to refer to a specific visualization of an image that a content provider or content creator (e.g., a studio colorist) intends for the audience to see.

[0026] One or more embodiments provide ambient light compensation and HDR tone mapping based on creative intent metadata (also referred to as multidimensional metadata) to maintain creative intent in ambient lighting of varying illuminance. In one embodiment, HDR tone mapping includes dynamically correcting image quality based on creative intent metadata.

[0027] One or more embodiments provide a method and system for HDR video tone mapping. In one embodiment, HDR video tone mapping comprises: (1) receiving an input video to be displayed on a display device; (2) receiving metadata representing at least partially frame or scene statistical information for the input video; (3) determining an adaptive / custom tone mapping function that performs ambient light compensation based on at least the metadata, the illuminance of ambient light within the environment of the display device, and the characteristics of the display device; and (4) applying the tone mapping function to the input video to generate a tone-mapped video with ambient light compensation, wherein the tone-mapped video is provided to the display device for display on the display device.

[0028] One or more embodiments provide ambient light compensation using an adaptive and explicit Bezier curve having multiple sections, and said multiple sections or multiple curves of the tone mapping curve have multiple adjustment points along the Bezier curve for improved tone mapping control.

[0029] FIG. 1 illustrates an exemplary computing structure (100) for implementing HDR tone mapping of HDR content to be displayed on a display device (60) in one or more embodiments. The computing structure (100) includes an electronic device (110) that includes resources such as one or more processor units (120) and one or more storage units (130). One or more applications may be executed / operated on the electronic device (110) by utilizing the resources of the electronic device (110).

[0030] In one embodiment, one or more applications on an electronic device (110) include an HDR tone mapping system (200) configured to implement HDR tone mapping of HDR content to be displayed on a display device (60) that is included in or connected to the electronic device (110). In one embodiment, the HDR tone mapping system (200) is configured to provide ambient light compensation based on creative intent metadata and ambient light. As described in detail later in this specification, the HDR tone mapping system (200) is configured to (1) receive an input video (e.g., HDR video) to be displayed on the display device (60), (2) receive multidimensional creative intent metadata corresponding to the input video, wherein the creative intent metadata represents the intent of the content creator / content provider of the input video, and (3) reduce or minimize image degradation caused by ambient light by improving image quality during the display of the input video on the display device (60) based on the creative intent metadata and ambient light. In one embodiment, image quality is improved by providing ambient light compensation based on creative intent metadata, modifying the tone mapping curve based on ambient light compensation and / or providing tone preservation.

[0031] In one embodiment, creative intent metadata corresponding to input content includes content statistics per frame or scene for the entire input video (e.g., the entire HDR video). For example, in one embodiment, creative intent metadata (also referred to as multidimensional metadata) includes luminance percentile information corresponding to each image (e.g., an HDR image) of the input video. The luminance percentile information corresponding to an image represents the distribution (i.e., number) of pixels within the image. For example, in one embodiment, the luminance percentile information corresponding to an image includes one or more percentiles for the CDF of the pixels within the image. In one embodiment, the CDF represents one or more of whether the image is dark or bright, which dynamic range of the CDF has dark details and the degree / level of darkness of those dark details, or which dynamic range of the CDF has bright details and the degree / level of brightness of those bright details. For example, if a CDF includes a specific range that is steeper than at least one other range of the CDF, this specific range contains more details (i.e., more pixels) compared to that at least one other range. As another example, if a CDF includes a specific range that is flatter than at least one other range of the CDF, this specific range contains fewer details (i.e., fewer pixels) compared to that at least one other range.

[0032] In one embodiment, for each image of input content (e.g., an HDR image), the HDR tone mapping system (200) is configured to generate a tone mapping curve corresponding to the image based on the CDF of the pixels within the image. The tone mapping curve includes several sections that maintain contrast. For example, if the CDF includes a specific range that is steeper than at least one other range of the CDF, the tone mapping curve includes a section corresponding to that steeper range, which is steeper than at least one other section of the tone mapping curve because there is more detail to be maintained. As another example, if the CDF includes a specific range that is flatter than at least one other range of the CDF, the tone mapping curve includes a section corresponding to that flatter range, which is flatter than at least one other section of the tone mapping curve because there is less detail to be maintained. Based on the CFDs of pixels within the images of the input video, the HDR tone mapping system (200) can distinguish the images and apply different ambient light compensation curves based on the CDFs (i.e., the ambient light compensation is adaptive to the creative intent metadata). For example, dark details within brighter images are improved to prevent the dark details from becoming distorted.

[0033] In one embodiment, luminance percentile information is represented as two-dimensional (2D) data. For example, in one embodiment, each percentile of the CDF curve included in the luminance percentile information is represented by a pair of values ​​{x, y}, where x represents the percentile luminance value and y represents the pixel percentile value (for example, {x = 100, y = 25%} indicates that 25% of the pixels are less than 100 nits, i.e., the 25th percentile is 100 nits). Noise removal can be performed using a tone mapping curve based on the CDF (i.e., noisy pixels are removed).

[0034] h(i) generally represents a histogram, where i is the i-th bin of the histogram. cdf(i) generally represents a CDF. In one embodiment, an HDR tone mapping system (200) is configured to (1) receive an RGB image (e.g., an image from an input video) as input, (2) determine a maxRGB image based on the RGB image, (3) determine a histogram h(i) corresponding to maxRGB, and (4) determine the CDF, cdf(i), of pixels within the RGB image based on the histogram h(i). In one embodiment, the histogram h(i) has 1024 bins and a 10-bit input. In one embodiment, the CDF, cdf(i) is calculated according to Equation 1 provided below.

[0035]

[0036] Examples of electronic devices (110) include, but are not limited to, televisions (e.g., smart televisions), mobile electronic devices (e.g., tablets, smartphones, laptops, etc.), wearable devices (e.g., smart watches, smart bands, head-mounted displays, smart glasses, etc.), set-top boxes, IoT (Internet of Things) devices, etc.

[0037] In one embodiment, the electronic device (110) includes one or more sensor units (150) that are included in or connected to the electronic device (110), such as a camera, a microphone, a GPS, a motion sensor, etc. In one embodiment, the HDR tone mapping system (200) uses one or more sensor units (150) to acquire sensor data including one or more readings / measurements related to the surrounding environment of the display device (60) (e.g., an HDR display), such as the degree / illumination of ambient light within the surrounding environment.

[0038] In one embodiment, the electronic device (110) includes one or more I / O units (140) that are included in or connected to the electronic device (110). In one embodiment, the one or more I / O units (140) include, but are not limited to, a physical user interface (PUI) and / or GUI such as a keyboard, keypad, touch interface, touch screen, knob, button, display screen, etc. In one embodiment, a user configures one or more user settings, sets one or more parameters, and provides input, etc. using at least one I / O unit (140).

[0039] In one embodiment, one or more applications on the electronic device (110) may further include one or more applications (170) (referred to as software mobile applications) that are installed on or downloaded to the electronic device (110), such as a camera application, a social media application, a video streaming application, etc. The software mobile application on the electronic device (110) may exchange data with the system (200).

[0040] In one embodiment, the electronic device (110) exchanges data with the display device (60) (e.g., peak brightness level D nitIt includes a communication unit (16) configured to receive display characteristics of a display device (60) including. The communication unit (160) is further configured to exchange data (e.g., receive video streams from an HDR content mastering environment (300)) with a wireless computing environment, such as an HDR content mastering environment (300), through a communication network / connection (50) (e.g., a wireless connection such as a Wi-Fi connection or a cellular connection, a wired connection, or a combination of both). The communication unit (160) may include any suitable communication circuit connected to the communication network and operating to exchange communication operations and media between the electronic device (110) and other devices connected to the same communication network (50). The communication unit (160) can operate to interface with a communication network using any suitable communication protocol, such as Wi-Fi (e.g., IEEE 802.11 protocol), Bluetooth, high frequency systems (e.g., 900 MHz, 2.4 GHz, and 5.6 GHz communication systems), infrared, GSM, GSM plus EDGE, CDMA, quadband, and other cellular protocols, VOIP, TCP-IP, or any other suitable protocol.

[0041] In one embodiment, the HDR content mastering environment (300) includes resources such as one or more servers (310) and one or more storage units (320). One or more applications (330) providing higher-level services may be executed / operated on the HDR content mastering environment (300) using the resources of the HDR content mastering environment (300). For example, in one embodiment, the HDR content mastering environment (300) provides an online platform for hosting one or more online services (e.g., video streaming services, etc.) and / or distributing one or more software mobile applications (170). As another example, the HDR tone mapping system (200) may be loaded or downloaded to an electronic device (110) from the HDR content mastering environment (300) which maintains and distributes updates to the HDR tone mapping system (200). As yet another example, the HDR tone mastering environment (300) may include a cloud computing environment that provides configurable computing system resources and shared pools of higher-level services.

[0042] FIG. 2 illustrates an exemplary workflow for implementing HDR tone mapping of HDR content to be displayed on a display device (60) in one or more embodiments. In one embodiment, the HDR content mastering environment (300) represents a computing environment used for color selection in a studio. For example, in one embodiment, one or more applications (330) placed on the HDR content mastering environment (300) include a color selection unit (340) configured to (1) receive an original video and (2) perform color selection on the original video based on input from a user (e.g., a colorist in the studio) to produce a mastered video.

[0043] Q can generally represent the number of quantization bits. In one embodiment, one or more applications (330) placed on an HDR content mastering environment (300) include a quantization unit (350) configured to (1) receive a mastered video (e.g., from a color selection unit (340)) and (2) perform quantization on the mastered video to obtain a quantized signal of the mastered video.

[0044] In one embodiment, the HDR content mastering environment (300) includes a mastering monitor (370) configured to provide visual feedback to a user (e.g., a colorist in a studio) on one or more color selection adjustment values ​​(i.e., adjustment values ​​for the original video resulting from color selection) by (1) receiving a quantization signal of a mastered video (e.g., from a quantization unit (350)) and (2) displaying the mastered video based on the quantization signal.

[0045] In one embodiment, one or more applications (330) placed on an HDR content mastering environment (300) include a tone mastering unit (360) configured to (1) receive a mastered video (e.g., from a color selection unit (340)) and (2) generate creative intent metadata corresponding to the mastered video. In one embodiment, the tone mastering unit (360) automatically generates creative intent metadata. In another embodiment, the tone mastering unit (360) generates creative intent metadata based on input from a user (e.g., a content creator in a studio).

[0046] In one embodiment, one or more applications (330) placed in an HDR content mastering environment (300) include (1) receiving a quantization signal of a mastered video (e.g., from a quantization unit (350)), (2) receiving creative intent metadata corresponding to the mastered video (e.g., from a tone mastering unit (360)), (3) producing an encoded video combined with the creative intent metadata by performing encoding on the quantization signal, and (4) an encoding unit (380) configured to provide the encoded video to be transmitted through a communication network (50).

[0047] In one embodiment, the HDR tone mapping system (200) includes a decoding unit (210) placed on an electronic device (110). In one embodiment, the decoding unit (210) is configured to (1) receive an encoded video transmitted through a communication network (50) (e.g., from an HDR content mastering environment (300)), (2) derive a quantized signal of a mastered video (e.g., for a dequantization unit (220)) by performing decoding on the encoded video, and (3) extract creative intent metadata corresponding to the mastered video from the encoded video (e.g., for an HDR tone mapping system (240) that performs ambient light compensation).

[0048] In one embodiment, the HDR tone mapping system (200) includes a dequantization unit (220) disposed on an electronic device (110). In one embodiment, the dequantization unit (220) is configured to (1) receive a quantization signal of a mastered video (e.g., from a decoding unit (210)) and (2) produce a mastered video by performing dequantization on the quantization signal video.

[0049] In one embodiment, the HDR tone mapping system (200) includes an ambient light signal unit (230) placed on an electronic device (110). In one embodiment, the ambient light signal unit (230) is configured to (1) receive sensor data captured by at least one sensor unit (150) of the electronic device (110) and (2) determine ambient light information indicating the degree / illumination of ambient light within the surrounding environment of the display device (60) based on the sensor data.

[0050] In one embodiment, the HDR tone mapping system (200) includes an HDR tone mapping system (240) that performs ambient light compensation and is placed on an electronic device (110). In one embodiment, the HDR tone mapping system (240) is configured to (1) receive a mastered video (e.g., from a dequantization unit (220)), (2) receive creative intent metadata corresponding to the mastered video (e.g., from a decoding unit (210)), (3) receive ambient light information (e.g., from an ambient light signal unit (230)) indicating the degree / illumination of ambient light within the surrounding environment of the display device (60), (4) determine an adaptive / custom tone mapping function (i.e., a tone mapping curve) that performs ambient light compensation based on at least the creative intent metadata and ambient light information, (5) produce an ambient light compensated tone-mapped video by applying the tone mapping function to the mastered video, and (6) provide the tone-mapped video to the display device (60) for display on the display device (60).

[0051] FIG. 3 illustrates an exemplary HDR tone mapping system (240) that performs ambient light compensation in one or more embodiments. In one embodiment, the HDR tone mapping system (240) that performs ambient light compensation includes a metadata analysis unit (250) placed on an electronic device (110). In one embodiment, the metadata analysis unit (250) is configured to (1) receive creative intent metadata corresponding to a mastered video (e.g., from a decoding unit (210)) and (2) classify the creative intent metadata into two sets of metadata, specifically tone mapping metadata corresponding to the mastered video (e.g., for a tone mapping curve generation unit (260)) and ambient light-related metadata corresponding to the mastered video (e.g., for an ambient light compensation system (270)).

[0052] In one embodiment, an HDR tone mapping system (240) for ambient lighting compensation includes a tone mapping curve generation unit (260) placed on an electronic device (110). In one embodiment, the tone mapping curve generation unit (260) is configured to (1) receive tone mapping metadata corresponding to a mastered video and (2) generate a dynamic base tone mapping function (i.e., a base tone mapping curve) and corresponding parameters based on the tone mapping metadata.

[0053] In one embodiment, the HDR tone mapping system (240) for ambient light compensation includes an ambient light compensation system (270) placed on an electronic device (110). In one embodiment, the ambient light compensation system (270) is configured to (1) receive ambient light-related metadata corresponding to a mastered video, (2) receive ambient light information (e.g., from an ambient light signal unit (230)) indicating the degree / illumination of ambient light in the surrounding environment of the display device (60), (3) receive a base tone mapping function (i.e., a base tone mapping curve) and corresponding parameters (e.g., from a tone mapping curve generation unit (260)), and (4) obtain a modified tone mapping function (i.e., a modified tone mapping curve) and corresponding parameters by modifying the base tone mapping function to describe the degree / illumination of ambient light in the surrounding environment based on the ambient light-related metadata and ambient light information. The modified tone mapping function adaptively compensates for the degree / illumination of ambient light in the surrounding environment. The modified tone mapping function preserves the content creator's intent because it takes ambient lighting-related metadata into account.

[0054] In one embodiment, the HDR tone mapping system (240) for ambient light compensation includes a tone mapping curve application system (280) placed on an electronic device (110). In one embodiment, the tone mapping curve application system (280) is configured to (1) receive a mastered video (e.g., from a dequantization unit (220)), (2) receive a modified tone mapping function (i.e., a tone mapping curve) and corresponding parameters (e.g., from an ambient light compensation system (270)), and (3) derive an ambient light compensated tone mapping video by applying the modified tone mapping function to the mastered video (i.e., the tone mapping video adaptively compensates for the degree / illumination of ambient light within the surrounding environment of the display device (60)).

[0055] In one embodiment, an HDR tone mapping system (240) that compensates for ambient light includes a quantization unit (290) placed on an electronic device (110).

[0056] In one embodiment, the quantization unit is configured to (1) receive a tone-mapped video with ambient light compensation (e.g., from a tone mapping curve application system (280)), (2) derive a quantization signal of the tone-mapped video by performing quantization on the tone-mapped video, and (3) provide the quantization signal to a display device (60) to display the tone-mapped video on a display device (60).

[0057] In one embodiment, the dequantization unit (220), tone mapping curve application system (280), quantization unit (290), and display device (60) are implemented using one or more HDR10+ standard processing pipelines available on the system on chip (SoC) (i.e., no additional hardware is required), thereby avoiding additional hardware costs for the SoC.

[0058] FIG. 4 illustrates an exemplary ambient light compensation system (270) in one or more embodiments. In one embodiment, the ambient light compensation system (270) is configured to determine three different elements corresponding to shadow pixels of the mastered video, midtone pixels of the mastered video, and highlight pixels of the mastered video based on ambient light-related metadata corresponding to the mastered video.

[0059] [0, high L ] generally represents a range of luminance levels, where high L is the brightest luminance level in that range. In one embodiment, the range [0, high L ] is divided into S levels.

[0060] In one embodiment, the ambient light compensation system (270) includes a midtone element unit (272) placed on an electronic device (110). In one embodiment, the midtone element unit (272) is configured to (1) receive ambient light-related metadata corresponding to the mastered video (e.g., from a metadata analysis unit (250)) and (2) determine the element SubProbMidToneDiff corresponding to the midtone pixels of the mastered video based on the ambient light-related metadata. In one embodiment, the element SubProbMidToneDiff is based on the percentage difference between the input percentile curve and the uniform percentile curve at S levels (i.e., the low-range luminance difference between the interpolated percentiles and the uniform histogram percentiles). For example, in one embodiment, the element SubProbMidToneDiff is determined according to Equation 2 provided below.

[0061]

[0062] In one embodiment, the midtone element unit (272) is configured to determine the element SubProbMidToneDiffRatio using a ramp function controlled based on SubProbMidToneDiff.

[0063] In one embodiment, the ambient light compensation system (270) includes a shadow element unit (271) placed on an electronic device (110). In one embodiment, the shadow element unit (271) is configured to (1) receive ambient light-related metadata corresponding to the mastered video (e.g., from a metadata analysis unit (250)) and (2) determine a factor SubProbsShadowDiff corresponding to the shadow pixels of the mastered video based on the ambient light-related metadata. In one embodiment, the factor SubProbsShadowDiff is based on the percentage difference between the input percentile curve and the uniform percentile curve at S levels (i.e., the low-range luminance difference between the interpolated percentiles and the uniform histogram percentiles).

[0064] In one embodiment, the ambient light compensation system (270) includes a highlight element unit (273) placed on an electronic device (110). In one embodiment, the highlight element unit (273) is configured to (1) receive ambient light-related metadata corresponding to a mastered video (e.g., from a metadata analysis unit (250)) and (2) determine an element SubProbsHighlightDiff corresponding to highlight pixels of the mastered video based on the ambient light-related metadata. In one embodiment, the element SubProbsHighlightDiff is based on the average of three highest percentiles of pLumHigh, pLumHighAvg. For example, in one embodiment, pLumHighAvg is determined by Equation 3 provided below.

[0065]

[0066] L is the number of percentiles of ambient lighting-related metadata. In one embodiment, the element SubProbsHighlightDiff is determined using a ramp function controlled based on pLumHighAvg.

[0067] In one embodiment, the ambient light compensation system (270) includes an ambient light compensation gain unit (274) disposed on an electronic device (110). In one embodiment, the ambient light compensation gain unit (274) is configured to (1) receive ambient light information (e.g., from an ambient light signal unit (230)) indicating the degree / illumination of ambient light in the surrounding environment of the display device (60), and (2) determine one or more gains for adaptive ambient light compensation in the surrounding environment based on the ambient light information. In one embodiment, the one or more gains are lowP for determining a lower curve modifier and an upper curve modifier, respectively. ratio and highP ratio Includes. In one embodiment, lowP ratio and highP ratio It is determined using lamp functions controlled based on the degree / illumination of ambient lighting within the surrounding environment.

[0068] In one embodiment, the ambient light compensation system (270) includes a coupling unit (275) disposed on an electronic device (110). In one embodiment, the coupling unit (275) is configured to (1) receive an element SubProbShadowDiff corresponding to shadow pixels of a mastered video (e.g., from a shadow element unit (271)), (2) receive an element SubProbMidToneDiffRatio corresponding to midtone pixels of a mastered video (e.g., from a midtone element unit (272)), (3) receive an element SubProbHighlightDiff corresponding to highlight pixels of a mastered video (e.g., from a highlight element unit (273)), and (4) determine a coupling element subProbAll based on each of the received elements. In one embodiment, the combination element subProbAll is determined by Equation 4, such as subProbAll=SubProbShadowDiff=SubProbMidToneDiffRatio + SubProbMidToneDiffRatio + SubProbHighlightDiff.

[0069] In one embodiment, the ambient light compensation system (270) includes a mapping function unit (276) placed on an electronic device (110). In one embodiment, the mapping function unit (276) receives (1) a coupling element subpProbAll (e.g., from a coupling unit (275)) and (2) gains lowP (e.g., from an ambient light compensation gain unit (274)) for ambient light compensation within the surrounding environment of the display device (60). ratio and highP ratio Receives, (3) combination element subpProbAll and gains lowP ratio and highP ratioBased on, it is configured to determine one or more modifiers for modifying a base tone mapping function (i.e., a base tone mapping curve). In one embodiment, the one or more modifiers determined by the mapping function unit (276) are a knee point modifier ΔK(i), a sub-curve modifier ΔP for modifying a sub-section of the base tone mapping function. low (i) and upper curve modifier ΔP for modifying the upper section of the basic tone mapping function high Includes (i), where i ∈ (1,N). In one embodiment, modifiers ΔK(i), ΔP low (i), and ΔP high (i) is determined using ramp functions.

[0070] In one embodiment, the ambient light compensation system (270) includes a coupling unit (277) disposed on an electronic device (110). In one embodiment, the coupling unit (277) receives (1) a base tone mapping function (i.e., a base tone mapping curve) along with corresponding parameters K(k) and P(i), and (2) modifiers ΔK(i), ΔP (e.g., from a mapping function unit (276)). low (i), and ΔP high Receive (i), and (3) modifiers ΔK(i), ΔP low (i), and ΔP high By modifying the basic tone mapping function based on (i), the corresponding parameters K'(i), P' low (i), and P' high It is configured to derive a modified tone mapping function (i.e., a modified tone mapping curve) together with (i).

[0071] In one embodiment, the modified tone mapping function is an explicit Bezier curve that adaptively compensates for ambient light. The modified tone mapping function has multiple sections, and the multiple sections have multiple adjustment points along the Bezier curve for improved tone mapping control. For example, in one embodiment, parameters P'low (i) and P' high (i) are adjustment points for the lower and upper sections (i.e., curves) of the Bézier curve.

[0072] In one embodiment, the coupling unit (277) is parameter P' low (i) and manually tuned parameter P tuned (i) To minimize the difference between the subcurve modifier ΔP low (i) is configured to optimize. For example, in one embodiment, the mapping function unit (276) is Equation 5 provided below. = + parameter P' according to low Determine (i).

[0073] P low (i) is a parameter corresponding to the basic tone mapping function at 0 illuminance of ambient light. lowP ratio It is determined using a ramp function controlled based on the degree / illumination of ambient lighting in the surrounding environment.

[0074] In one embodiment, the coupling unit (277) is parameter P' high (i) and manually tuned parameter P tuned (i) To minimize the difference between the upper curve modifier ΔP high (i) is configured to optimize. For example, in one embodiment, the mapping function unit (276) is Equation 6 provided below. = + parameter P' according to high Determine (i).

[0075] P high (i) is a parameter corresponding to the basic tone mapping function at 0 illuminance of ambient light. highP ratio It is determined using a ramp function controlled based on the degree / illumination of ambient lighting in the surrounding environment.

[0076] In one embodiment, the coupling unit (277) has parameter K'(i) and manually tuned parameter K named It is configured to optimize the knee point modifier ΔK(i) to minimize the difference between (i). For example, in one embodiment, the mapping function unit (276) is Equation 7 provided below. = + Determine the parameter K'(i) according to .

[0077] K(i) is a parameter corresponding to the basic tone mapping function at 0 illuminance of ambient light.

[0078] FIG. 5 is a graph (400) illustrating a 400 percent difference between an input percentile curve and a uniform percentile curve in one or more embodiments. The horizontal axis of the graph (400) represents the percentage. The vertical axis of the graph (400) represents the percentile of pLumHigh. The graph (400) includes an input percentile curve and a uniform percentile curve. As shown in FIG. 5, the range [0, high L ] is divided into S levels. The ambient light compensation system (270) is configured to determine, at each of the S levels, the corresponding difference ΔpProbdock(k) (503), which represents the percentage difference between the input percentile curve (501) and the uniform percentile curve (502). For example, as shown in FIG. 5, the percentile K·high L In / S, the corresponding difference ΔpProbdock(k) (503) is the percentage PHL of the input percentile curve (501). k and percentage PHL of the uniform percentile curve (502) uk It indicates the difference between.

[0079] FIG. 6 is a graph (450) illustrating a ramp function for subProbMidToneDiffRatio in one or more embodiments. The horizontal axis of the graph (450) represents subProMidToneDiff. The vertical axis of the graph (450) represents subProbMidToneDiffRatio. As shown in FIG. 6, subProbMidToneDiffRatio is determined using a ramp function controlled based on subProbMidToneDiff.

[0080] FIG. 7 shows, in one or more embodiments, the gain highP ratio This is a graph (500) plotting the ramp function for. The horizontal axis of the graph (500) represents the degree / illumination of ambient lighting within the surrounding environment ("ambient lighting illuminance"). The vertical axis of the graph (500) represents highP ratio It represents. As shown in Fig. 7, highP ratio It is determined using a ramp function controlled based on the degree / illumination of ambient lighting in the surrounding environment.

[0081] FIG. 8 shows the gain lowP in one or more embodiments. ratio This is a graph (550) plotting the ramp function for. The horizontal axis of the graph (550) represents the degree / illumination of ambient lighting within the surrounding environment ("ambient lighting illuminance"). The vertical axis of the graph (550) represents lowP ratio It represents. As shown in Fig. 8, lowP ratio It is determined using a ramp function controlled based on the degree / illumination of ambient lighting in the surrounding environment.

[0082] FIG. 9 shows ΔP in one or more embodiments. high This is a graph (600) plotting the ramp function for (i). The horizontal axis of the graph (600) represents the combination element subProbAll. The vertical axis of the graph (600) is ΔP high (i) represents. As shown in FIG. 9, ΔP high(i) is determined using a ramp function controlled based on the combination element subProbAll.

[0083] FIG. 10 shows, in one or more embodiments, a sub-curve modifier ΔP low This is a graph (650) plotting the ramp function for (i). The horizontal axis of the graph (650) represents the combination element subProbAll. The vertical axis of the graph (650) represents ΔP low (i) represents. As shown in FIG. 10, ΔP low (i) is determined using a ramp function controlled based on the combination element subProbAll.

[0084] FIG. 11 is a graph (700) illustrating a ramp function for ΔK(i) in one or more embodiments. The horizontal axis of the graph (700) represents the coupling element subProbAll. The vertical axis of the graph (700) represents ΔK(i). As shown in FIG. 11, ΔK(i) is determined using a ramp function controlled based on the coupling element subProbAll.

[0085] FIG. 12 illustrates an exemplary tone mapping curve application system (280) in one or more embodiments. In one embodiment, the tone mapping curve application system (280) provides tone preservation through tone mapping. Specifically, in one embodiment, the tone mapping curve application system (280) includes a maximum function unit (281). The maximum function unit (281) is configured to (1) receive a mastered video (e.g., from a dequantization unit (220)) containing red (R) code values, green (G) code values, and blue (B) code values, and (2) determine a maximum code value maxRGB among the R, G, and B code values ​​by applying a maximum (i.e., max) function to the R, G, and B code values ​​(e.g., max(R,G,B)).

[0086] In one embodiment, the tone mapping curve application system (280) includes an electro-optical conversion function (EOTF) unit (282). The EOTF unit (282) is configured to (1) receive a maximum code value maxRGB (e.g., from a maximum function unit (281)) and (2) determine a linear luminance value X by applying the EOTF to the maximum code value maxRGB.

[0087] In one embodiment, the tone mapping curve application system (280) includes a tone mapping unit (283). The tone mapping unit (283) is configured to (1) receive a linear luminance value X (e.g., from an EOTF unit (282)), (2) receive a modified tone mapping function (i.e., a modified tone mapping curve) (e.g., from an ambient light compensation system (270)), and (3) determine (i.e., find) a tone mapping value Y (e.g., tmLUT(X)) to which the modified tone mapping function maps the linear luminance value x.

[0088] In one embodiment, the tone mapping curve application system (280) includes a tone mapping ratio application unit (284). The tone mapping ratio application unit (284) is configured to (1) receive a linear luminance value X (e.g., from an EOTF unit (282)), (2) receive a tone mapping value Y (e.g., from a tone mapping unit (283)), (3) determine a tone mapping ratio r based on the linear luminance value X and the tone mapping value Y (r=X / Y), (4) determine linear values ​​EOTF(R), EOTF(G), and EOTF(B) respectively by applying EOTF to each of the R, G, and B code values, and (5) derive scaled linear values ​​EOTF(R)*r, EOTF(G)*r, and EOTF(B)*r respectively by applying the tone mapping ratio r to each of the linear values ​​EOTF(R), EOTF(G), and EOTF(B).

[0089] In one embodiment, the tone mapping curve application system (280) includes an optoelectric conversion function (OETF) unit (285). The OETF unit (285) is configured to derive a tone-mapped video that maintains ambient light compensation and hue (i.e., maintains hue by keeping the x,y coordinates of the CIE xyY color space unchanged) by (1) receiving scaled linear values ​​EOTF(R)*r, EOTF(G)*r, and EOTF(B)*r (e.g., from the tone mapping ratio application unit (284)) and (2) determining the R code value, G code value, and B code value of the tone-mapped video by applying OETF to each of the scaled linear values ​​EOTF(R)*r, EOTF(G)*r, and EOTF(B)*r.

[0090] Table 1 below provides exemplary pseudocode implemented by a tone mapping curve application system (280) for tone maintenance in one or more embodiments.

[0091]

[0092] FIG. 13 illustrates an exemplary ambient light compensation development system (800) in one or more embodiments. In one embodiment, the ambient light compensation development system (800) is used to develop an algorithm for ambient light compensation that maintains creative intent in various ambient environments. Specifically, in one embodiment, the ambient light compensation development system (800) comprises (1) a reference monitor (810) having a peak luminance level M placed in a first ambient environment ("reference ambient environment") used as a reference, (2) a display device (60) having a peak luminance level N placed in a second ambient environment ("controlled non-reference ambient environment") that is controlled and different from the reference ambient environment, and (3) an image adjustment unit (820).

[0093] P ggenerally represents a tone mapping curve ("ground truth curve") representing ground truth information for ambient light compensation. The development of an algorithm for ambient light compensation involves (1) displaying an image of a mastered video ("mastered image") on a reference monitor (810) to provide the operator with a creative intent view within a reference ambient environment, (2) displaying an image of a tone-mapped video ("tone-mapped image") on a display device (60) to provide the operator with a controlled non-reference ambient environment view of adjusted content, wherein the tone-mapped image is a mastered image that compensates for ambient light, (3) the operator compares the tone-mapped image with the mastered image and adjusts the tone-mapped image based on the comparison through an image adjustment unit (820), and (4) through the image adjustment unit (820), ground truth curve P based on the adjustment g It involves steps to generate.

[0094] P a Generally, it represents an ambient light compensation curve ("automatic curve") that is automatically generated based on an algorithm for ambient light compensation. In one embodiment, the ambient light compensation development system (800) includes a fine tuning unit (830) and a difference unit (840). The development of the algorithm for ambient light compensation further includes steps of (1) determining a difference θ between the ground actual information curve and the automatic curve through the difference unit (840), and (2) fine-tuning the algorithm so that the operator (or developer) minimizes the difference θ. In one embodiment, the resulting fine-tuning algorithm is implemented to compensate for the influence of ambient light in the environment of the display device (60) by correcting content (e.g., mastered video) provided from the ambient light compensation system (270) (e.g., placed in the ambient light compensation gain unit (274)).

[0095] In one embodiment, the difference θ is Equation 8 It is determined by.

[0096] FIG. 14a is an exemplary mastered image (900) shown in a reference ambient environment with low degree / illumination ambient lighting in one or more embodiments. In one embodiment, the reference ambient environment represents the ambient environment in a studio suitable for color grading. For example, the reference ambient environment is dark, and a colorist is viewing the mastered image (900) on a mastering monitor (370).

[0097] FIG. 14b is an exemplary tone-mapped image (910) shown in one or more embodiments in an ambient environment with high degree / illumination ambient lighting. In one embodiment, the ambient environment refers to the surrounding environment of a display device (60). For example, the ambient environment is bright, and a user is viewing the tone-mapped image (910) on the display device (60). An HDR tone-mapping system (240) that performs ambient lighting compensation generates an ambient lighting compensation curve based on creative intent metadata corresponding to a mastered image (900) (Fig. 14a), and by applying ambient lighting compensation to the mastered image (900) based on the ambient lighting compensation curve, derives a tone-mapped image (910) that is compensated for in a bright ambient environment. The cognitive emotion of a user viewing a tone-mastered image in a bright ambient environment is similar to the cognitive emotion of a colorist viewing a mastered image in a dark reference ambient environment.

[0098] FIG. 14c is an exemplary graph (920) illustrating histograms corresponding to the mastered image (900) and tone-mapped image (910) of FIG. 14a-14b in one or more embodiments. The horizontal axis of the graph (920) represents luminance in nits. The vertical axis of the graph (920) represents the number of pixels. The graph (920) includes (1) a first histogram (921) corresponding to the mastered image (900) and (2) a second histogram (922) corresponding to the tone-mapped image (910).

[0099] FIG. 15a is an exemplary mastered image (950) shown in a reference ambient environment with low degree / illumination ambient lighting in one or more embodiments. In one embodiment, the reference ambient environment represents the ambient environment in a studio suitable for color selection. For example, the reference ambient environment is dark, and a colorist is viewing the mastered image on a mastering monitor (370).

[0100] FIG. 15b is an exemplary tone-mapped image (960) shown in one or more embodiments in an ambient environment with high degree / illumination ambient lighting. In one embodiment, the ambient environment represents the ambient environment of a display device (60). For example, the ambient environment is bright, and a user is viewing the tone-mapped image (960) on the display device (60). An HDR tone-mapping system (240) that performs ambient lighting compensation generates an ambient lighting compensation curve based on creative intent metadata corresponding to a mastered image (950) (Fig. 15a), and by applying ambient lighting compensation to the mastered image (950) based on the ambient lighting compensation curve, derives a tone-mapped image (960) that is compensated for in a bright ambient environment. The cognitive emotion of a user viewing a tone-mastered image in a bright ambient environment is similar to the cognitive emotion of a colorist viewing a mastered image in a dark reference ambient environment.

[0101] FIG. 15c is an exemplary graph (970) illustrating histograms corresponding to the mastered image and tone-mapped image of FIG. 15a-15b in one or more embodiments. The horizontal axis of the graph (970) represents luminance in nits. The vertical axis of the graph (970) represents the number of pixels. The graph (970) includes (1) a first histogram (971) corresponding to the mastered image and (2) a second histogram (972) corresponding to the tone-mapped image.

[0102] In one embodiment, for ambient lighting of the same degree / illumination within an ambient environment, the ambient lighting compensation system (270) generates different ambient lighting compensation curves for the images based on creative intent metadata corresponding to the different images (i.e., frames / scenes).

[0103] FIG. 16a is an exemplary image (1000) ("dark image") with a large amount of dark detail in one or more embodiments. FIG. 16b is an exemplary image (1010) ("bright image") with a large amount of bright detail in one or more embodiments. As illustrated in FIG. 16a-16b, the dark image (1000) has darker details than the bright image (1010), and the bright image (1010) has brighter details than the dark image (1000).

[0104] FIG. 16c is an exemplary graph (1020) illustrating ambient light compensation curves for the dark image (1000) and bright image (1010) of FIG. 16a-16b in one or more embodiments. The horizontal axis of the graph (1020) represents the input luminance in nits. The vertical axis of the graph (1020) represents the output luminance in nits. The graph (1020) includes (1) a first ambient light compensation curve (1021) generated by an ambient light compensation system (270) for a dark image (1000) based on creative intent metadata and ambient light information (i.e., degree / illumination of ambient light) corresponding to a dark image (1000), and (2) a second ambient light compensation curve (1022) generated by an ambient light compensation system (270) for a bright image (1010) based on creative intent metadata and ambient light information (i.e., degree / illumination of ambient light) corresponding to a bright image (1010).

[0105] As illustrated in FIG. 16c, the first ambient light compensation curve (1021) includes a greater amount of dark details as a factor in the dark image (1000), and the second ambient light compensation curve (1022) includes a smaller amount of dark details as a factor in the bright image (1010). More specifically, due to the dark image (1000) having a greater amount of dark details, the difference between the input luminance and output luminance of the dark image (1000) with low input luminance (e.g., the input luminance value is 1 / 10 nit in FIG. 16c) is greater than the difference between the input luminance and output luminance of the bright image (1010) with low input luminance. Conversely, due to the bright image (1010) having a larger amount of bright details, the difference between the input brightness and output brightness of the bright image (1010) (e.g., the input brightness value is 10 nits in FIG. 16c) is greater than the difference between the input brightness and output brightness of the dark image (1000) of the high input brightness.

[0106] FIG. 16d is an exemplary graph (1030) illustrating histograms for the dark image (1000) and the bright image (1010) of FIG. 16a-16b in one or more embodiments. The horizontal axis of the graph (1030) represents the input luminance in nits. The vertical axis of the graph (1030) represents the number of pixels. The graph (1030) includes (1) a first histogram (1031) corresponding to the dark image (1000) and (2) a second histogram (1032) corresponding to the bright image (1010). As illustrated in FIG. 16d, the dark image (1000) has darker details than the bright image (1010), and the bright image (1010) has brighter details than the dark image (1000).

[0107] FIG. 16e is an exemplary graph (1040) illustrating CDF curves for the dark image (1000) and the bright image (1010) of FIG. 16a-16b in one or more embodiments. The horizontal axis of the graph (1040) represents input luminance in nits. The vertical axis of the graph (1040) represents percentages. The graph (1040) includes (1) a first CDF curve (1041) representing the distribution of pixels within the dark image (1000) (wherein the distribution is included in the creative intent metadata corresponding to the dark image (1000)), and (2) a second CDF curve (1042) representing the distribution of pixels within the bright image (1010) (wherein the distribution is included in the creative intent metadata corresponding to the bright image (1000)). As illustrated in FIG. 16e, the dark image (1000) has darker details than the bright image (1010), and the bright image (1010) has brighter details than the dark image (1000).

[0108] In one embodiment, for different degrees / illuminations of ambient lighting in different ambient environments, the ambient lighting compensation system (270) generates different ambient lighting compensation curves for the image based on creative intent metadata corresponding to the same image (i.e., frame / scene). For example, assume a first ambient environment with a first degree / illumination of ambient lighting, and a second ambient environment with a second degree / illumination of ambient lighting that is higher than the first degree / illumination of ambient lighting (i.e., the second ambient environment is brighter). For the same image, the ambient lighting compensation system (270) generates a first ambient lighting compensation curve based on the first degree / illumination of ambient lighting and a second ambient lighting compensation curve based on the second degree / illumination of ambient lighting, wherein the second ambient lighting compensation curve is brighter than the first ambient lighting compensation curve.

[0109] FIG. 17 is a flowchart of an exemplary process (1100) for implementing HDR tone mapping of HDR content to be displayed on a display device in one or more embodiments. Operation 1110 includes determining multidimensional metadata corresponding to an input image, wherein the multidimensional metadata includes a distribution function of pixels within the input image. Operation 1120 includes determining ambient light information indicating the degree of ambient light in the surrounding environment of the display device. Operation 1130 includes determining ambient light information indicating the degree of ambient light in the surrounding environment of the display device. Operation 1140 includes generating a tone mapping function based on the multidimensional metadata and one or more gains. Operation 1150 includes applying the tone mapping function to the input image to generate a tone-mapped image that adaptively compensates for the illuminance of ambient light in the surrounding environment, wherein the tone-mapped image is provided to a display device for display.

[0110] In one embodiment, operations 1110-1150 may be performed by one or more components of a tone mapping system (200), such as an HDR tone mapping system (240) that performs ambient light compensation, including an ambient light signal unit (230) and a metadata analysis unit (250), an ambient light compensation system (270), and a tone mapping curve application unit (280).

[0111] FIG. 18 is a top-level block diagram illustrating an information processing system including a computer system (1200) used to implement the disclosed embodiments. The system (200) may be incorporated into the computer system (1200). The computer system (1200) may further include one or more processors (1201), an electronic display device (1202) (for displaying video, graphics, text and other data), main memory (1203) (e.g., RAM (random access memory)), a storage device (1204) (e.g., a hard disk drive), a removable storage device (1205) (e.g., a removable storage drive, a removable memory module, a magnetic tape drive, an optical disc drive, a computer-readable medium storing computer software and / or data), a viewer interface device (1206) (e.g., a keyboard, a touch screen, a keypad, a pointing device), and a communication interface (1207) (e.g., a modem, a network interface (such as an Ethernet card), a communication port, or a PCMCIA slot and card). The communication interface (1207) enables software and data to be transmitted between the computer system and external devices. The system (1200) is the above-described The devices / modules (1201 to 1207) are connected to a communication infrastructure (1208) (e.g., a communication bus, a crossover bar, or a network).

[0112] Information transmitted through the communication interface (1207) may be in the form of electronic signals, electromagnetic signals, optical signals, or other signals that the communication interface (1207) can receive through a communication link carrying the signal, and may be implemented using wired or cable, optical fiber, telephone line, mobile phone link, radio frequency (RF) link and / or other communication code values. Computer program instructions representing block diagrams and / or flowcharts in this specification may be loaded onto a computer, a programmable data processing unit, or processing units so that a series of operations performed thereon may generate a computer-implemented process. In one embodiment, processing instructions for the process (1100) (Fig. 17) may be stored as program instructions on a memory (1203), a storage device (1204), and / or a removable storage device (1205) so as to be executed by a processor (1201).

[0113] Embodiments have been described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products. Each block, or combination thereof, of such exemplary diagrams / drawings may be implemented by computer program instructions. When provided by a processor, computer program instructions create a machine that generates means for instructions executed through the processor to implement the functions / operations specified in the flowcharts and / or block diagrams. Each block within the flowcharts / block diagrams may represent a hardware and / or software module or logic. In other embodiments, the functions indicated within the blocks may occur concurrently or otherwise, in a different order than that indicated in the drawings.

[0114] The terms "computer program medium," "computer-usable medium," "computer-readable medium," and "computer program product" are generally used to refer to media and signals such as main memory, secondary memory, removable storage drives, and hard disks installed in hard disk drives. These computer program products are means for providing software to a computer system. A computer-readable medium enables a computer system to read data, instructions, messages or message packets, and other computer-readable information from the computer-readable medium. For example, a computer-readable medium may include non-volatile memory such as floppy disks, ROM, flash memory, disk drive memory, CD-ROM, and other permanent storage devices. It is useful for transferring information, such as data and computer instructions, between computer systems, for example. Computer program instructions are stored on a computer-readable medium that can cause a computer, other programmable data processing device, or other electronic device to function in a specific way, so that the instructions stored on the computer-readable medium produce a product containing instructions that implement a specific function / behavior in blocks or blocks of a flowchart and / or block diagram.

[0115] As can be expected by those skilled in the art, the embodiments of the above examples may be implemented as a system, method, or computer program product. Accordingly, the embodiments of the above examples may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or a combination of software and hardware embodiments, all of which may generally be referred to herein as a “circuit,” “module,” or “system.” Additionally, the embodiments of the above examples may take the form of a computer program product stored on one or more computer-readable media(s) on which computer-readable program code is implemented.

[0116] Any combination of one or more computer-readable media(s) may be used. A computer-readable medium may be a computer-readable storage medium. A computer-readable storage medium may be, for example, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, apparatus, or any suitable combination of the above, provided that it is not limited. More specific examples of a computer-readable storage medium may include: electrical connections having one or more wires, portable computer diskettes, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or flash memory, fiber optics, portable CD-ROM (compact disc read-only memory), optical storage devices, magnetic storage devices, or any suitable combination of the above. In the context of this document, a computer-readable storage medium may be any type (tangible) medium capable of containing / storing a program to be used by or in connection with an instruction execution system, device, or apparatus.

[0117] Computer program code for executing operations for aspects of one or more embodiments may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, etc., and ordinary procedural programming languages ​​such as the "C" programming language or a similar programming language. The program code may be executed in its entirety on a user computer, in part on a user computer, as a standalone software package, in part on a user computer and in part on a remote computer, or in its entirety on a remote computer or server. In the latter case, the remote computer may be connected to the user computer through any type of network including a LAN (local area network) or a WAN (wide area network), or connection to an external electronic device may be made (e.g., via the Internet using an Internet service provider).

[0118] Aspects of one or more embodiments have been described above with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products. Each block of the flowcharts and / or block diagrams, and combinations of blocks of the flowcharts and / or block diagrams, may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a special-purpose computer or other programmable data processing device to create a machine, so that instructions executed through the processor of the computer or other programmable data processing device may create means for implementing specific functions / behaviors in the blocks of the flowcharts and / or block diagrams or in the blocks.

[0119] These computer program instructions are stored on a computer-readable medium capable of causing a computer, other programmable data processing device, or other electronic device to function in a specific way, so that the instructions stored on the computer-readable medium produce a manufactured product that includes instructions implementing a specific function / behavior in a block or block of a flowchart and / or block diagram.

[0120] Computer program instructions also provide processes for implementing specific functions / behaviors in blocks of a flowchart and / or block diagram when said instructions are executed on a computer, other programmable data processing device, or other device by causing a series of operational steps to be executed on the computer, other programmable device, or other device to generate a computer implementation process.

[0121] The flowcharts and block diagrams in the drawings illustrate the structure, function, and operation of possible embodiments of systems, methods, and computer program products according to various embodiments. In this regard, each block of the flowcharts or block diagrams may represent a module, segment, or part of instructions containing one or more executable instructions for implementing a specific logical function(s). In some other alternative embodiments, the operations indicated within the blocks may occur differently from the order shown in the drawings. For example, depending on the related function, two blocks shown consecutively may be executed substantially concurrently, or the blocks may often be executed in the reverse order. It should also be noted that each block of the block diagrams and / or flowcharts, and combinations of blocks of the block diagrams and / or flowcharts, may be implemented by special hardware-based systems that perform specific functions or behaviors or execute combinations of special hardware and computer instructions.

[0122] In the claims, references to a singular component are intended to mean "one or more" rather than "only" unless explicitly stated otherwise. All structural and functional equivalents of the components of the exemplary embodiments described above, which are currently known to or will be known to those skilled in the art, are to be covered by the claims.

[0123] The terms used herein are for the purpose of describing specific embodiments only and are not intended to limit the technology of this disclosure. As used herein, the singular form is intended to include the plural form unless the relevant context clearly indicates otherwise. In this specification, terms such as “comprising” or “having” are intended to indicate the presence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and do not preclude the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0124] The structures, materials, actions, and equivalents of functional elements added to all means or steps in the following claims are to include any structures, materials, or actions for performing the corresponding function in combination with other specifically claimed elements. The embodiments are presented for illustrative and illustrative purposes only and should not be the only or limited to the embodiments of the disclosed form. It will be obvious to those skilled in the art that various modifications and variations may be made without departing from the scope and concept of the disclosed art.

[0125] Although the embodiments have been described with reference to a specific version, other versions are possible. Accordingly, the spirit and scope of the appended claims should not be limited to the description of the preferred versions included in this specification.

Claims

Claim 1 A method comprising: determining multidimensional metadata corresponding to an input image, wherein the multidimensional metadata includes a distribution function of pixels within the input image; determining ambient lighting information representing the illuminance of ambient lighting within the surrounding environment of a display device; determining a plurality of gains that adaptively compensate for the illuminance of the ambient lighting within the surrounding environment based on the ambient lighting information; determining a plurality of modifiers for each of a plurality of sections included in a basic tone mapping function based on the plurality of gains; generating a tone mapping function by modifying the basic tone mapping function using the multidimensional metadata and the plurality of modifiers; and generating a tone-mapped image that adaptively compensates for the illuminance of the ambient lighting within the surrounding environment by applying the generated tone mapping function to the input image, wherein the tone-mapped image is provided to the display device for display. Claim 2 A method according to claim 1, wherein the distribution function includes a cumulative distribution function, and the multidimensional metadata includes a corresponding pair of values ​​including a percentile luminance value and a pixel percentage value for each percentile of the cumulative distribution function. Claim 3 A method according to claim 1, wherein the tone mapping function represents a Bezier curve comprising a plurality of sections, and each of the plurality of sections has a corresponding adjustment point along the Bezier curve. Claim 4 A method according to claim 3, wherein the operation of determining the plurality of modifiers includes, based on the plurality of gains, the operation of determining a knee point modifier, a sub-curve modifier for modifying a sub-section of the Bézier curve, and an upper-curve modifier for modifying an upper section of the Bézier curve. Claim 5 In claim 4, the operation of generating the tone mapping function further includes the operation of generating the tone mapping function and one or more parameters corresponding to the tone mapping function based on the knee point modifier, the lower curve modifier, and the upper curve modifier. Claim 6 A method according to claim 5, wherein the one or more parameters include a knee point, a first adjustment point corresponding to the lower section of the Bézier curve, and a second adjustment point corresponding to the upper section of the Bézier curve. Claim 7 delete Claim 8 A method according to claim 1, further comprising: performing quantization on the tone-mapped image and generating a quantization signal of the tone-mapped image; and providing the quantization signal of the tone-mapped image to the display device for display. Claim 9 A method according to claim 1, wherein the operation of determining the plurality of gains includes the operation of determining the plurality of gains using a ramp function based on the illuminance of the ambient light within the ambient environment based on the ambient lighting information. Claim 10 A system comprising: at least one processor; and a processor-readable memory for storing a plurality of instructions; wherein, when the plurality of instructions are executed by the at least one processor, the at least one processor determines multidimensional metadata corresponding to an input image, wherein the multidimensional metadata includes a distribution function of pixels within the input image and determines ambient lighting information representing the illuminance of ambient lighting within the surrounding environment of a display device, determines a plurality of gains that adaptively compensate for the illuminance of the ambient lighting within the surrounding environment based on the ambient lighting information, determines a plurality of modifiers for each of a plurality of sections included in a basic tone mapping function based on the plurality of gains, and generates a tone mapping function by modifying the basic tone mapping function using the multidimensional metadata and the plurality of modifiers, and generates a tone-mapped image that adaptively compensates for the illuminance of the ambient lighting within the surrounding environment by applying the generated tone mapping function to the input image, and performs operations provided to the display device for the tone-mapped image to be displayed. Claim 11 A system according to claim 10, wherein the distribution function includes a cumulative distribution function, and the multidimensional metadata includes a corresponding pair of values ​​including a percentile luminance value and a pixel percentage value for each percentile of the cumulative distribution function of the pixels. Claim 12 In claim 10, the tone mapping function represents a Bezier curve comprising a plurality of sections, and each of the plurality of sections includes a corresponding adjustment point along the Bezier curve, a system. Claim 13 delete Claim 14 delete Claim 15 A processor-readable medium storing a computer program that performs the method according to any one of claims 1 through 6, 8, and 9 when executed by at least one processor.

Images