EDR-aware, hue-preserving, soft-clip for HDR content

Abstract

Disclosed herein are systems and techniques for performing an extended dynamic range (EDR)-aware, hue-preserving, soft-clipping operation. According to some embodiments, the techniques may first convert every media type to an EDR representation by way of the media type's own reference display-referred mapping. Knowing the EDR "headroom" availability of the source content and destination display device, a perceptual model may be employed to control soft-clip tone-mapping parameters, which are optimized for both the source's and destination's available EDR headroom. Further, by knowing the brightness of the source EDR headroom that the current soft-clip parameterization will maintain, it is possible preserve the hue of the soon-to-be clipped input values. In some embodiments, the parameterization of the soft-clipping operation may implement a series of stages of cascading (and continuous) solutions over different ranges of available destination EDR headroom values, with each stage attempting to provide the least amount of source media distortion possible.

Description

Title EDR-AWARE, HUE-PRESERVING, SOFT-CLIP FOR HDR CONTENTInventors GREENEBAUM, et al.BACKGROUND

[0001] Today, consumer electronic devices with display screens are used in many different environments with many different lighting conditions, e.g., the office, the home, home theaters, inside head-mounted displays (HMD), and outdoors. Devices typically need to be designed so that — no matter what the user’s viewing environment is at any given moment — there is only minimal (or, ideally, no) color banding perceivable to the viewer, and the displayed content has consistent appearance and tonality. Many content items are authored for particular display devices and viewing environments. For example, movies are often authored for Rec.709 displays to be viewed in in dark viewing environments. Devices typically need to be able to adapt content items to many different types of intended display devices and many different viewing environments, such that the content items appear as the content authors intended them to be perceived — no matter what the current viewing conditions around the display device are like.

[0002] As more and more so-called high dynamic range (HDR) content enters the ecosystem via a number of sources, such as: user-captured HDR camera content (cameras capturing both video and stills to hybrid log-gamma (HLG), PQ, and other proprietary HDR containers), user- authored content via HDR creative apps, or professionally-authored HDR video, game, and / or still image content, there is a great need to adapt this content to be viewed on not only SDR systems, but also systems of varying dynamic ranges.

[0003] Further, it is important to be able to transcode between file formats of varying dynamic range. This conversion is usually performed by way of a “tone-mapping” operation. However,existing tone maps are sub-optimal for various reasons. For example, naive tone maps might simply clip input values that are not representable in the output format. Such a hard clipping operation might clip the subject — leading to a jarring result. Hard clips also typically clip the color of a pixel to the native white point of the display or file format (e.g., D65 white), which might not be characteristic of the actual lighting in the scene.

[0004] Further, even when these tone-maps provide soft-clipping to maintain highlight detail, a lack of control often renders the results incompatible with other content. That is, tone maps tend to be designed for only one media format, as viewed in a single viewing environment, and they don’t maintain fidelity well with other media. For example, the subject matter of tonemapped content might appear overly dim in comparison to other media (e.g., due to compression applied in order to make room for highlights to be displayed on the display device), highlights can be over-compressed and lose detail, and the dynamic range of highlights in the content can be exaggerated beyond that of the source — making them incongruous to other content.

[0005] For these reasons and more, it would be desirable to have an EDR-aware, huepreserving, soft-clip tone-mapping operation, which yields results that are consistent for all media types, while respecting the reference characteristics of each medium.

[0006] In some cases, it may be further desirable to map each content item to a shared, systemlevel viewing environment, also referred to herein as a “common compositing space,” and then from the shared, system-level viewing environment to the current viewing environment, e.g., via the implementation of a dynamic system -level optical-to-optical transfer function (DS- OOTF) that is capable of utilizing an ambient conditions model (and a number of other display- related factors) to automatically adjust a display’s overall content adaptation process to provide a so-called “perceptual reference,” such that, as the current viewing scenario departs from the “reference” viewing scenario, the DS- OOTF adapts, so as to provide the viewer with as closeto the perceptual effect in the “non-reference” viewing scenario as they would have experienced in the “reference” viewing scenario. Further details regarding the use of a DS- OOTF may be found in the ‘298 application.SUMMARY

[0007] Human perception is not absolute; rather, it is relative. In other words, a human viewer’s perception of a displayed image changes based on what surrounds the image, the image itself, and what brightness and white point the viewer is presently adapted to. Many color-management systems attempt to consistently map the content to the display, such that the content’s encoding and the display’s reproduction do not influence the resulting displayed content, thus providing consistency across content encoding and displays. However, these color-management systems require fixed viewing conditions, such as always using the intended display and suggested “reference” viewing environment.

[0008] According to various embodiments described herein, systems and techniques are disclosed for performing an extended dynamic range (EDR)-aware, hue-preserving, soft- clipping operation. According to some embodiments, the techniques may first convert every media type to an EDR representation by way of the media type’s own reference display- referred mapping. Knowing the EDR “headroom” availability of the source content and destination display device, a perceptual model may be employed to control soft-clip tonemapping parameters, which are optimized for both the source’s and destination’s available EDR headroom. Further, by knowing the brightness of the source EDR headroom that the current soft-clip parameterization will maintain, it is possible preserve the hue of the soon-to- be clipped input values. In some embodiments, the parameterization of the soft-clipping operation may implement a series of stages of cascading (and continuous) solutions over different ranges of available destination EDR headroom values, with each stage attempting to provide the least amount of source media distortion possible.

[0009] In some embodiments, a first stage (“Stage D”) may keep input content represented in a 1 : 1 fashion in the output, while a second stage (“Stage C”) may partially compress the input content, and a third stage (“Stage B”) may at least partially compress and clip the input content, and a fourth stage (“Stage D”) may compress the input content reference white value (up to a limit) as well as the input content highlight values.

[0010] A perceptual model may be applied during the tone mapping operation that solves for the amount of perceptual information that could be encoded in a given source or destination highlight representation, as well as the perceptual loss due to highlight compression and provides for selection of stage to apply, as well as the parameters for each.

[0011] Such techniques may yield superior results to the algorithms as presently used across the industry, especially in relationship to compatibility with other media types, as well as adaptation to devices with dynamic levels of headroom (e.g., EDR and HDR display systems).

[0012] Thus, according to some embodiments, a method of adapting content for display on a display device, comprising: receiving data indicative of a first content item; converting the first content item into an extended dynamic range (EDR) representation based, at least in part, on a reference di splay -ref erred mapping for the first content item; applying a perceptual model to the EDR representation of the first content item to generate an adapted version of the first content item, wherein the perceptual model is based, at least in part, on an amount of EDR headroom available in the first content item and an amount of EDR headroom available on the display device, and wherein the perceptual model is configured to control soft clipping and tone mapping parameters for the first content item; and displaying the adapted version of the first content item on the display device.

[0013] According to some such embodiments, the method further comprises: receiving data indicative of a second content item; converting the second content item into an EDR representation based, at least in part, on a reference display-referred mapping for the secondcontent item; applying the perceptual model to the EDR representation of the second content item to generate an adapted version of the second content item, wherein the perceptual model is based, at least in part, on an amount of EDR headroom available in the second content item and an amount of EDR headroom available on the display device, and wherein the perceptual model is configured to control soft clipping and tone mapping parameters for the second content item; and displaying the adapted version of the second content item concurrently with the adapted version of the first content item on the display device.

[0014] According to some such embodiments, the first content item and the second content item are of different (or the same) media types.

[0015] According to some embodiments, the perceptual model is further configured to preserve hues in the first content item, e.g., by determining that a color component of at least a first pixel in the first content item having a first hue is not representable on the display device; and solving for a set of brightest color component values for the first pixel that preserve the first hue.

[0016] According to some embodiments, for a first range of values for the amount of EDR headroom available on the display device, applying the perceptual model further comprises: maintaining a reference white value and a dynamic range of the first content item, wherein the first range of values comprises values wherein the amount of EDR headroom available on the display device is equal to or greater than the amount of EDR headroom available in the first content item.

[0017] According to some such embodiments, for a second range of values for the amount of EDR headroom available on the display device, applying the perceptual model further comprises: compressing at least a portion of highlight values of the first content item, wherein the second range of values comprise values wherein the amount of EDR headroom available on the display device is less than the amount of EDR headroom available in the first content item but greater than a first threshold value.

[0018] According to some such embodiments, for a third range of values for the amount of EDR headroom available on the display device, applying the perceptual model further comprises: compressing at least a first portion of highlight values of the first content item; and clipping at least a second portion of highlight values of the first content item, wherein the third range of values comprises values wherein the amount of EDR headroom available on the display device is less than the first threshold value but greater than a second threshold value, and wherein the second threshold value is lower than the first threshold value.

[0019] According to some such embodiments, for a fourth range of values for the amount of EDR headroom available on the display device, applying the perceptual model further comprises: compressing at least a third portion of highlight values of the first content item; clipping at least a fourth portion of highlight values of the first content item; and compressing a reference white value of the first content item, wherein the fourth range of values comprises values wherein the amount of EDR headroom available on the display device is less than the second threshold value.

[0020] According to some embodiments, the method further comprises: comparing an estimated perception of a viewer of a reference rendition of the first content item to an estimated perception of a viewer of the adapted version of the first content item; and applying an additional correction to at least a portion of the adapted version of the first content item to maintain a desired perception of the at least a portion of the adapted version of the first content item for the viewer.

[0021] According to other embodiments, the the perceptual model is further configured to cause a continuous tone mapping curve to be applied to the first content item over each of: a first range of values for the amount of EDR headroom available in the first content item; and a second range of values for the amount of EDR headroom available on the display device.

[0022] In still other embodiments, the aforementioned techniques may be embodied in instructions stored on non-transitory program storage devices and / or implemented on electronic devices having display devices, e.g., a mobile phone, PDA, HMD, monitor, television, or a laptop, desktop, or tablet computer.BRIEF DESCRIPTION OF THE DRAWINGS

[0023] FIG. 1A illustrates the properties of ambient lighting, diffuse reflection off a display device, and other environmental conditions influencing a display device.

[0024] FIG. IB illustrates the additive effects of unintended light on a display device.

[0025] FIG. 2 illustrates a system for performing gamma adjustment utilizing a look up table.

[0026] FIG. 3 illustrates a Framebuffer Gamma Function and an exemplary Native Display Response.

[0027] FIG. 4 illustrates graphs representative of a LUT transformation and a Resultant Gamma Function, as well as a graph indicative of a perceptual transformation due to environmental conditions.

[0028] FIG. 5 illustrates a unified display model system for performing display adjustment based on a dynamic system OOTF, in accordance with one or more embodiments.

[0029] FIG. 6 illustrates a simplified functional block diagram of an ambient conditions model, in accordance with one or more embodiments.

[0030] FIG. 7A illustrates, in flowchart form, a process for performing an EDR-aware, huepreserving soft-clipping operation, in accordance with one or more embodiments.

[0031] FIG. 7B illustrates, in flowchart form, another process for performing an EDR-aware, hue-preserving soft-clipping operation, in accordance with one or more embodiments.

[0032] FIG. 8 illustrates, in graph form, various variables used in the performance of an EDR- aware, hue-preserving soft-clipping operation, in accordance with one or more embodiments.

[0033] FIG. 9A illustrates, in graph form, a tone mapping curve used in the performance of an EDR-aware, hue-preserving soft-clipping operation for a first range of output EDR headroom values available on a display device, in accordance with one or more embodiments.

[0034] FIG. 9B illustrates, in graph form, a tone mapping curve used in the performance of an EDR-aware, hue-preserving soft-clipping operation for a second range of output EDR headroom values available on a display device, in accordance with one or more embodiments.

[0035] FIG. 9C illustrates, in graph form, a tone mapping curve used in the performance of an EDR-aware, hue-preserving soft-clipping operation for a third range of output EDR headroom values available on a display device, in accordance with one or more embodiments.

[0036] FIG. 9D illustrates, in graph form, a tone mapping curve used in the performance of an EDR-aware, hue-preserving soft-clipping operation for a fourth range of output EDR headroom values available on a display device, in accordance with one or more embodiments.

[0037] FIG. 10 illustrates a simplified functional block diagram of a device possessing a display, in accordance with one embodiment.DETAILED DESCRIPTION

[0038] As mentioned above, naive tone mapping operations that perform hard clipping might clip the subject of image content — leading to a jarring result, such as very bright colors in the content (e.g., pixels representing the sky in an image) clipping to the display native white point, which might not be characteristic of the actual lighting in the scene.

[0039] Further, even when these tone-maps provide soft-clipping to maintain highlight detail, a lack of control often renders the results incompatible with other content. For example, the subject matter of tone-mapped content might appear overly dim in comparison to other media (e.g., due to compression applied in order to make room for highlights to be displayed on the display device), highlights can be over-compressed and lose detail, and the dynamic range ofhighlights in the content can be exaggerated beyond that of the source — making them incongruous to other content.

[0040] By contrast, the techniques disclosed herein may apply an EDR-aware, hue-preserving soft-clip operation that is consistent for all media types, while respecting the media. For example, the techniques may uses each medium’s reference display-referred mapping, scale the content to the appropriate level of nits, and then divide by each medium’s reference white value to obtain an EDR representation of the content. Once represented in EDR, all media may be processed by the same tone map, thereby maintaining compatibility between the media.

[0041] According to some embodiments, EDR-aware mapping that is based on both the source and destination EDR headroom may respect reference white values and dynamic range to maintain compatibility with other systems and other media. Further, a perceptual quality model may be used to control the soft-clip tone-map. Finally, in order to make the clipping operation hue-preserving, if a give source pixel value is not representable in the output (e.g., the R, G, or B component exceeds the output’s available EDR headroom), according to some embodiments, values for the brightest pixel of the same hue may be solved for and used by the display device.

[0042] The techniques disclosed herein are applicable to any number of electronic devices: such as digital cameras, digital video cameras, mobile phones, personal data assistants (PDAs), head-mounted display (HMD) devices, monitors, televisions, digital projectors (including cinema projectors), as well as desktop, laptop, and tablet computer displays.

[0043] In the interest of clarity, not all features of an actual implementation are described in this specification. It will, of course, be appreciated that in the development of any such actual implementation (as in any development project), numerous decisions must be made to achieve the developers’ specific goals (e.g., compliance with system- and business-related constraints), and that these goals will vary from one implementation to another. It will be appreciated that such development effort might be complex and time-consuming, but they would neverthelessbe a routine undertaking for those of ordinary skill having the benefit of this disclosure. Moreover, the language used in this disclosure has been principally selected for readability and instructional purposes, and it therefore may not have been selected to delineate or circumscribe the inventive subject matter, with resort to the claims being necessary to determine such inventive subject matter. Reference in the specification to “one embodiment” or to “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiments is included in at least one embodiment of the invention, and multiple references to “one embodiment” or “an embodiment” should not be understood as necessarily all referring to the same embodiment.

[0044] Background on Exemplary Display Device Properties and Ambient Viewing Conditions

[0045] Referring now to FIG. 1A, the properties of ambient lighting, diffuse reflection off a display device 102, and other environmental conditions influencing the display device are shown via the depiction of a side view of a viewer 116 of the display device 102 in a particular ambient lighting environment. As shown in FIG. 1A, viewer 116 is looking at display device 102, which, in this case, is a typical desktop computer monitor. Dashed lines 110 represent the viewing angle of viewer 116. The ambient environment, as depicted in FIG. 1A, is lit by environmental light source 100, which casts light rays 108 onto all of the objects in the environment, including wall 112, as well as the display surface 114 of display device 102. As shown by the multitude of small arrows 109 (representing reflections of light rays 108), a certain percentage of incoming light radiation will reflect off of the surface that it shines upon. Diffuse reflection may be defined as the reflection of light from a surface such that an incident light ray is reflected at many angles, and it has a particular effect on a viewer’s perception of display device 102.

[0046] When the brightness of reflected light and / or the brightness of light leakage from display device 102 is greater than the brightness of pixels driven by the display device for a content item, the viewer may not be able to perceive low tonal details in the content item. This effect is illustrated by dashed line 106 in FIG. 1A, which indicates a threshold brightness level. When the brightness of pixels in the emissive display surface 114 is less than the threshold brightness level indicated by dashed line 106, the pixels are not perceived as intended. When the brightness of pixels in the emissive display surface 114 is greater than the threshold brightness level, the pixels are perceived as intended. The dashed line 106 and the threshold brightness level may be adjusted to account for each of the reflected light and light leakage from the display device 102, either alone or in combination. The influence of reflected light and light leakage from the display device on the viewer’s perception of displayed content is described further herein with respect to FIG. IB. Information regarding diffuse reflection and other ambient light in the current viewing environment may be used to inform an ambient conditions model that suggests which adaptation processes to perform on content to compensate for environmental conditions and / or suggests modifications to adaptation processes already being performed.

[0047] The information regarding diffuse reflection and other ambient light may be based off of light level readings recorded by one or more optical sensors, e.g., ambient light sensor 104. Dashed line 118 represents data indicative of the light source being collected by ambient light sensor 104. Optical sensor 104 may be used to collect information about the ambient conditions in the environment of the display device and may comprise, e.g., an ambient light sensor, an image sensor, or a video camera, or some combination thereof. A front-facing image sensor provides information regarding how much light (and, in some embodiments, what color of light) is hitting the display surface 114. This information may be used in conjunction with a model of the reflective and diffuse characteristics of the display to inform the ambientconditions model about the particular lighting conditions that the display is currently in and that the user is currently adapted to. Although optical sensor 104 is shown as a “front-facing” image sensor, i.e., facing in the general direction of the viewer 116 of the display device 102, other optical sensor types, placements, positioning, and quantities are possible. For example, one or more “back-facing” image sensors alone (or in conjunction with one or more front facing sensors) could give even further information about light sources and the color in the viewer’s environment. The back-facing sensor collects light from emissive sources or re-reflected off objects behind the display, and it may be used to determine the brightness of the display’s surroundings, i.e., what the user sees beyond the display. This information may also be used for the ambient conditions model. For example, the color of wall 112, if it is close enough behind display device 102 could have a profound effect on the viewer’s perception. Likewise, in the example of an outdoor environment, the color and intensity of light surrounding the viewer can make the display appear different than it would an indoor environment with, e.g., incandescent (colored) lighting.

[0048] In one embodiment, the optical sensor 104 may comprise a video camera (or other devices) capable of capturing spatial information, color information, as well as intensity information. With regard to spatial information, a video camera or other device(s) may also be used to determine a viewing user’s distance from the display, e.g., to further model how much of the user’s field of view the display fills and, correspondingly, how much influence the display / environment will have on the user’s perception of displayed content. In some embodiments, a video camera may be configured to capture images of the surrounding environment for analysis at some predetermined time interval, e.g., every two minutes, such that the ambient conditions model may be gradually updated or otherwise changed as the ambient conditions in the viewer’s environment change.

[0049] Additionally, a back-facing video camera used to model the surrounding environment could be designed to have a field of view roughly consistent with the calculated or estimated field of view of the viewer of the display. Once the field of view of the viewer is calculated or estimated, e.g., based on the size or location of the viewer’s facial features as recorded by a front-facing camera, assuming the native field of view of the back-facing camera is known and is larger than the field of view of the viewer, the system may then determine what portion of the back-facing camera image to use in the surround computation.

[0050] In still other embodiments, one or more cameras or depth sensors may be used to further estimate the distance of particular surfaces from the display device. This information could, e.g., be used to further inform the ambient conditions model based on the likely composition of the viewer’s surround and the perceptual impacts thereof. For example, a display with a 30” diagonal sitting 18” from a user will have a greater influence on the user’ s vision than the same display sitting 48” away from the user, filling less of the user’s field of view.

[0051] Referring now to FIG. IB, the additive effects of unintended light on a display device are shown in more detail. For example, the light rays 155 emitting from display representation 150 represent the amount of light that the display is intentionally driving the pixels to produce at a given moment in time. Likewise, light rays 165 emitting from display representation 160 represent the amount of light leakage from the display at the given moment in time, and light rays 109 reflecting off display representation 170 represent the aforementioned diffuse reflection of ambient light rays off the surface of the display at the given moment in time. There may be more diffuse reflection off of non-glossy displays than off of glossy displays, in displays of stacked components compared to laminated components, or off of clean displays compared to dusty or otherwise dirty displays. Finally, display representation 180 represents the summation of the three forms of light illustrated in display representations 150, 160, and170

[0052] As illustrated in FIG. IB, the light rays 185 emitting from display representation 180 represent the actual amount of light that is perceived by a viewer of the display device, which may be different than the initial amount of light 155 pixels in the display were intentionally driven with in order to produce the desired content. The unintended light from display leakage, diffuse reflections, and the like may desaturate perceived colors compared to the content’s intended color. The darker or dimmer the intended color is, the more pronounced the desaturation appears to a viewer. Thus, accounting for the effects of these various phenomenon may help to achieve a more consistent and content-accurate perceptual experience across viewing environments.

[0053] Thus, in one or more embodiments disclosed herein, an ambient conditions model may be employed as part of a unified display model for dynamically selecting which environmental adaptations to perform or adjusting environmental adaptations already being performed may compensate for unintended light, such that the dimmest colors are not masked by light leakage and / or the predicted diffuse reflection levels and all the colors are not perceived as desaturated compared to the intended colors. A model of the display device characteristics may be used to determine an amount of light leakage from the display device under the current display parameters. The model of the display device characteristics may also be used in combination with information from ambient light sensor 104 to estimate an amount of diffuse reflection off the display device. A perceptual model may be used to estimate an amount of desaturation from unintended light, such that the ambient conditions model may determine a recommended resaturation and environmental adaptations to achieve the recommended resaturation.

[0054] Background on System Gamma and Perceived Gamma for Exemplary Display Devices

[0055] Referring now to FIG. 2, a typical system 212 for performing gamma adjustment utilizing a Look Up Table (LUT) 210 is shown. Element 200 represents the source content,created by, e.g., a source content author, that viewer 116 wishes to view. Source content 200 may comprise an image, video, or other displayable content type. Element 202 represents the source profile, that is, information describing the color profile and display characteristics of the device on which source content 200 was authored by the source content author. Source profile 202 may comprise, e.g., an International Color Consortium (ICC) profile of the author’s device or color space (which will be described in further detail below), or other related information.

[0056] Information relating to the source content 200 and source profile 202 may be sent to viewer 116’s device containing the system 212 for performing gamma adjustment utilizing a LUT 210. Viewer 116’s device may comprise, for example, a mobile phone, PDA, HMD, monitor, television, or a laptop, desktop, or tablet computer, or the like. Upon receiving the source content 200 and source profile 202, system 212 may perform a color adaptation process 206 on the received data, e.g., for performing gamut mapping, i.e., color matching across various color spaces. For instance, gamut matching tries to preserve (as closely as possible) the relative relationships between colors (e.g., as authored / approved by the content author on the display described by the source ICC profile), even if all the colors must be systematically changed or adapted in order to get them to display on the destination device.

[0057] Once the color profiles of the source and destination have been appropriately adapted, image values may enter the so-called "framebuffer” 208. In some embodiments, image values, e.g., pixel luma values, enter the framebuffer having come from an application or applications that have already processed the image values to be encoded with a specific implicit gamma. A framebuffer may be defined as a video output device that drives a video display from a memory buffer containing a complete frame of, in this case, image data. The implicit gamma of the values entering the framebuffer can be visualized by looking at the “Framebuffer Gamma Function,” as will be explained further below in relation to FIG. 3. Ideally, this FramebufferGamma Function is the exact inverse of the display device’s “Native Display Response” function, which characterizes the luminance response of the display to input.

[0058] Because the inverse of the Native Display Response isn’t always exactly the inverse of the framebuffer, a LUT, sometimes stored on a video card or in other memory, may be used to account for the imperfections in the relationship between the encoding gamma and decoding gamma values, as well as the display’s particular luminance response characteristics. Thus, if necessary, system 212 may then utilize LUT 210 to perform a so-called “gamma adjustment process.” LUT 210 may comprise a two-column table of positive, real values spanning a particular range, e.g., from zero to one. The first column values may correspond to an input image value, whereas the second column value in the corresponding row of the LUT 210 may correspond to an output image value that the input image value will be “transformed” into before being ultimately being displayed on display 102. LUT 210 may be used to account for the imperfections in the display 102’ s luminance response curve, also known as the “display transfer function.” In other embodiments, a LUT may have separate channels for each primary color in a color space, e.g., a LUT may have Red, Green, and Blue channels in the sRGB color space.

[0059] The transformation applied by the LUT to the incoming framebuffer data before the data is output to the display device may be used to ensure that a desired 1.0 gamma boost is applied to the eventual display device. The system shown in FIG. 2 is generally a good system, although it does not take into account the effect of differences or changes in ambient light conditions on the perceived gamma, or gamma adjustments already encoded in the source content 200 by the source author to compensate for differences between the source content capture environment and the source content 200’ s intended viewing environment. In other words, the 1.0 gamma boost for encoding and decoding content is only achieved / appropriate in one ambient lighting environment, and this environment is typically brighter than a normaloffice environment. For example, content captured in a bright environment won’t require a gamma boost, e.g., due to the “simultaneous contrast” phenomenon, if viewed in the identical (i.e., bright) environment. For another example, content captured and edited in a bright environment but intended for viewing in a dim environment (e.g., a dark surround, such as a movie theater) may already include gamma adjustments in the source content 200 received by system 212. Additional gamma boost based on LUT 210 may thus distort the gamma adjustments already provided in the source content 200 and cause the displayed content to differ from the source author’ s intent.

[0060] As mentioned above, in some embodiments, the goal of this gamma adjustment system 212 is to have an overall 1.0 system gamma applied to the content that is being displayed on the display device 102. An overall 1.0 system gamma corresponds to a linear relationship between the input encoded luma values and the output luminance on the display device 102. Ideally, an overall 1.0 system gamma will cause the displayed content to appear largely as the source author intended, despite the intervening encoding and decoding of the content, and other color management processes used to adapt the content to the particular display device 102. However, as will be described later, this overall 1.0 gamma may only be properly perceived in one particular set one set of ambient lighting conditions, thus necessitating the need for a dynamic display adjustment system to accommodate different ambient lighting conditions and adjust the overall system gamma to achieve a perceived system gamma of 1.0. Further, gamma adjustment is only one kind of correction for environmental conditions, and environmental adaptations described herein include gamma adjustment as well as resaturation, black point and white point adjustment, and the like.

[0061] Referring now to FIG. 3, a Framebuffer Gamma Function 300 and an exemplary Native Display Response 302 is shown. Gamma adjustment, or, as it is often simply referred to, “gamma,” is the name given to the nonlinear operation commonly used to encode luma valuesand decode luminance values in video or still image systems. Gamma, y, may be defined by the following simple power-law expression: Lout = Lin7, where the input and output values, Lin and Lout, respectively, are non-negative real values, typically in a predetermined range, e.g., zero to one. A gamma value greater than one is sometimes called an “encoding gamma,” and the process of encoding with this compressive power-law nonlinearity is called “gamma compression;” conversely, a gamma value less than one is sometimes called a “decoding gamma,” and the application of the expansive power-law nonlinearity is called “gamma expansion.” Gamma encoding of content helps to map the content data into a more perceptually-uniform domain.

[0062] Another way to think about the gamma characteristic of a system is as a power-law relationship that approximates the relationship between the encoded luma in the system and the actual desired image luminance on whatever the eventual user display device is. In existing systems, a computer processor or other suitable programmable control device may perform gamma adjustment computations for a particular display device it is in communication with based on the native luminance response of the display device, the color gamut of the device, and the device’s white point (which information may be stored in an ICC profile), as well as the ICC color profile and other content indicators that the source content’s author attached to the content to specify the content’s “rendering intent.”

[0063] The ICC profile is a set of data that characterizes a color input or output device, or a color space, according to standards promulgated by the International Color Consortium. ICC profiles may describe the color attributes of a particular device or viewing requirement by defining a mapping between the device source or target color space and a profile connection space (PCS), usually the CIE XYZ color space. ICC profiles may be used to define a color space generically in terms of three main pieces: 1) the color primaries that define the gamut; 2) the transfer function (sometimes referred to as the gamma function); and 3) the white point.ICC profiles may also contain additional information to provide mapping between a display’s actual response and its “advertised” response, i.e., its tone response curve (TRC), for instance, to correct or calibrate a given display to a perfect 2.2 gamma response.

[0064] In some implementations, the ultimate goal of the gamma adjustment process is to have an eventual overall 1.0 gamma boost, i.e., so-called “unity” or “no boost,” applied to the content as it is displayed on the display device. An overall 1.0 system gamma corresponds to a linear relationship between the input encoded luma values and the output luminance on the display device, meaning there is actually no amount of gamma “boosting” being applied, and the gamma encoding process is undone by the gamma decoding process, without further adjustment.

[0065] Classically, a gamma encoding is optimized for a particular environment, dynamic range of content, and dynamic range of display, such that the encoding and display codes are well-spaced across the intended range and the content appears as intended (e.g., not banded, without crushed highlights or blacks, and with correct contrast — sometimes called tonality, etc.). 8-bit 2.2 gamma is an example of an acceptable representation for encoding SDR (standard dynamic range) content to be displayed on a 1 / 2.45 gamma Rec.709 CRT in a bright- office viewing environment.

[0066] However, the example SDR content will not have the intended appearance when viewed in an environment that is brighter or dimmer than the intended, bright-office viewing environment, even when displayed on its intended Rec.709 display. When the current viewing environment differs from the suggested viewing environment, for instance, if it is brighter than the suggested viewing environment, the user’s vision adapts to the current, brighter viewing environment, such that the user perceives fewer distinguishable details in the darker portions of the content. The display may only be able to modulate a small range of the user’s vision asadapted to the current, brighter viewing environment. Further, the display’s fixed maximum brightness may be dim compared to the brightness of the current viewing environment.

[0067] The current, brighter viewing environment prevents the user from perceiving the darker portions in the content that the source author intended the viewer to perceive when the content is viewed on the suggested Rec.709 display in the suggested, bright-office viewing environment. In other words, “shadow detail” is “crushed” to black. This effect is magnified when ambient light from the viewing environment is reflected off the display and / or light from display leakage, collectively called unintended light, further limit how dark the content is perceived by the viewer. The lowest codes in the content are spaced apart in brightness based on the suggested viewing environment and may be too closely spaced to be differentiable in the current, brighter viewing environment.

[0068] The perceived, overall tonality of the content differs when the current viewing environment differs from the suggested viewing environment as well. For example, the content may appear lower in contrast when the current viewing environment is brighter than the suggested viewing environment. The content may also appear desaturated, with an unintended color cast, due to unintended light from reflections off the display and / or display leakage, or when the white point of the suggested viewing environment differs from the white point of the current viewing environment.

[0069] Even when viewed on the suggested Rec.709 display in the suggested, bright-office viewing environment, the tonality of the content may be perceived differently based on what other content is displayed at the same time, in an effect referred to as “simultaneous contrast.” Some devices display multiple content items at a time, for example, a user’s work computer may display multiple documents and a video at the same time. The different content items may be tailored for different suggested viewing environments, such that each content item uses a different gamma encoding and / or a different gamma boost. Display devices that implementthe same gamma boost to all the content items may end up distorting the individual content items away from their intended appearances.

[0070] For instance, Rec.709 content has an overall 1.22 gamma boost from the intentional mismatch between the content’s encoding gamma and the display’s decoding gamma, to compensate for bright-surround content being viewed in a dim-surround environment. In contrast, DCI P3 content directly encodes the compensation for bright-surround content being viewed in a dim-surround environment into the pixels themselves, such that no gamma boost is needed, that is, a 1.0 gamma is sufficient. No single gamma boost is appropriate for both the Rec.709 content and the DCI P3 content in any viewing environment. While this example describes differences in gamma boost, similar differences may be found in other kinds of content adaptation, such as tone mapping, re- saturation, black point and / or white point adjustments, modified transfer functions for the display, and combinations thereof. As used herein, “surround environment” refers to ambient lighting conditions and the like in the environment around the display device. A “viewing environment” refers to the surround environment around the display device and display characteristics, such as display device light leakage, that may further influence how a user perceives content displayed on the display device.

[0071] Returning now to FIG. 3, the x-axis of Framebuffer Gamma Function 300 represents input image values spanning a particular range, e.g., from zero to one. The y-axis of Framebuffer Gamma Function 300 represents output image values spanning a particular range, e.g., from zero to one. As mentioned above, in some embodiments, image values may enter the framebuffer 208 already having been processed and have a specific implicit gamma. As shown in graph 300 in FIG. 3, the encoding gamma is roughly 1 / 2.2, or 0.45. That is, the line in graph 300 roughly looks like the function, LOUT = LIN0'45. Gamma values around 1 / 2.2, or0.45, are typically used as encoding gammas because the native display response of manydisplay devices have a gamma of roughly 2.2, that is, the inverse of an encoding gamma of 1 / 2.2. In other cases, a gamma of, e.g., 1 / 2.45, may be applied to 1.96 gamma encoded content when displayed on a conventional 1 / 2.45 gamma CRT display, in order to provide the 1.25 gamma “boost” (i.e., 2.45 divided by 1.96), required to compensate for the simultaneous contrast effect causing bright content to appear low-contrast when viewed in a dim surround environment (i.e., the area beyond the display is typically more dim), such as the 16 lux Rec.709 intended viewing environment. If the content already includes additional gamma boost because the source author intended the bright content to be viewed in a dim surround environment and framebuffer 208 does not account for this encoded gamma boost, the resulting gamma boost will differ from the source author’s rendering intent.

[0072] The x-axis of Native Display Response Function 302 represents input image values spanning a particular range, e.g., from zero to one. The y-axis of Native Display Response Function 302 represents output image values spanning a particular range, e.g., from zero to one. In theory, systems in which the decoding gamma is the inverse of the encoding gamma should produce the desired overall 1.0 system gamma. However, this fails to account for ambient light in the environment around the display device and / or the gamma boost already encoded into the source content. Thus, the desired overall 1.0 system gamma is only achieved in one ambient lighting environment, e.g., the authoring lighting environment or, where gamma boost is already encoded into the source content, in the intended viewing environment. These systems do not dynamically adapt to environmental conditions surrounding the display device, or according to user preferences.

[0073] Referring now to FIG. 4, graphs representative of a LUT transformation and a Resultant Gamma Function are shown, as well as a graph indicative of a perceptual transformation due to environmental conditions. The graphs in FIG. 4 show how, in an ideal system, a LUT may be utilized to account for the imperfections in the relationship between the encoding gammaand decoding gamma values, as well as the display’s particular luminance response characteristics at different input levels. The graphs in FIG. 4 also illustrate how the environmental conditions surrounding the display device may then distort perception of the content such that the perceived gamma differs from the Resultant Gamma Function. The x-axis of native display response graph 400 represents input image values spanning a particular range, e.g., from zero to one. The y-axis of native display response graph 400 represents output image values spanning a particular range, e.g., from zero to one. The non-straight line nature of graph 400 represents the minor peculiarities and imperfections in the exemplary display’s native response function. The x-axis of LUT graph 410 represents input image values spanning the same range of input values the display is capable of responding to, e.g., from zero to one. The y-axis of LUT graph 410 represents the same range of output image values the display is capable of producing, e.g., from zero to one. In an ideally-calibrated display device, the display response 400 will be the inverse of the LUT response 410, such that, when the LUT graph is applied to the input image data, the Resultant Gamma Function 420 reflects a desired overall system 1.0 gamma response, i.e., resulting from the adjustment provided by the LUT and the native (nearly) linear response of the display, and the content is perceived as the source author intended. The x-axis of Resultant Gamma Function 420 represents input image values as authored by the source content author spanning a particular range, e.g., from zero to one. The y-axis of Resultant Gamma Function 420 represents output image values displayed on the resultant display spanning a particular range, e.g., from zero to one. The slope of 1.0, reflected in the line in graph 420, indicates that luminance levels intended by the source content author will be reproduced at corresponding luminance levels on the ultimate display device.

[0074] Ideally, the Resultant Gamma Function 420 reflects a desired overall 1.0 system gamma on the resultant display device, indicating that the tone response curves (i.e., gamma) are matched between the source and the display, that the gamma encoding of the content has beenundone by the gamma decoding process without further adjustment, and that the image on the display is likely being displayed more or less as the source’s author intended. However, this calculated overall 1.0 system gamma does not take into account the effect of ambient lighting conditions on the viewer’s perception of the gamma boost. In other words, due to perceptual transformations caused by ambient conditions in the viewer’s environment 425, the viewer does not perceive the content as the source author intended and does not perceive an overall 1.0 gamma in all lighting conditions. The calculated overall 1.0 gamma may further fail to take into account the effect on the viewer’s current adaptation to the ambient light conditions. As described above, a user’s ability to perceive changes in light intensity (as well as the overall range of light intensities that their eyes may be able to perceive) is further based on what levels of light the user’s eyes have been around (and thus adjusted to) over a preceding window of time (e.g., 30 seconds, 5 minutes, 15 minutes, etc.) The calculated overall 1.0 gamma may also fail to take into account a gamma boost already encoded into the source content by the source author based on the source capture and editing environments and the intended viewing environment. For example, a video may be filmed in a bright environment but have been edited for viewing in a dim environment, with a gamma boost matching this transition already encoded into the video. If a system tries to further adjust the already adjusted gamma boost, the resultant gamma differs from the source author’s rendering intent.

[0075] As is shown in graph 430, the dashed line indicates a perceived 1.0 gamma boost, i.e., the viewer’s actual perception of the achieved system gamma, which corresponds to an overall gamma boost that is greater than 1.0. The ambient conditions in the viewing surround transformed the achieved system gamma of greater than 1.0 into a perceived system gamma of equal to 1.0. Thus, a unified display model for dynamically adjusting a display’s characteristics according to one or more embodiments disclosed herein may be able to account for the perceptual transformation due to the viewer’s current environmental conditions, cause thedisplay to boost the achieved system gamma above the intended 1.0 system gamma, and thus present the viewer with what he or she will perceive as an overall 1.0 system gamma, causing the content to be perceived as the source author intended. As explained in more detail below, such unified display models may also have a non-uniform time constant for how stimuli affect the viewer’s instantaneous adaptation over time. In other words, the model may attempt to predict changes in a user’s perception due to changes in the viewer’s ambient conditions.

[0076] A Unified Display Model Utilizing Dynamic System OOTF

[0077] Referring now to FIG. 5, a unified display model system 500 for performing display adjustment based on a dynamic system OOTF or “DS-OOTF” is illustrated, in accordance with one or more embodiments. A given display, e.g., display 102, may be said to have the capability to “modulate” (that is, adapt or adjust to) only a certain percentage of possible surround environments at any given moment in time. For instance, if the environment is much brighter than the display, such that the display is reflecting a lot of light at its minimum display output level, then the display may have a relatively high “pedestal” value, and thus, even at its maximum display output level, only be able to modulate a fraction of the ambient lighting conditions.

[0078] Unified display model system 500 may thus be used to apply a transformation(s) for warping the source content 200 (e.g., high precision source content) into the viewer’s adapted visual perception of display 102 in a given viewing environment. As described above, warping the original source content signal to the perception of the viewer of the display and the display’s environment may be based, e.g., on the predicted viewing environment conditions received an ambient conditions model, as will be described further with reference to FIG. 6. For example, the ratio of display 102’ s diffuse white brightness in nits to the brightness of the user’s view beyond display 102, called the surround, also in nits may be used to apply a gamma boost, color saturation correction, or similar algorithm to compensate for the perceptual effect ofviewing content in a surround with a different brightness than the surround associated with source content 200 during capture, editing, or approval.

[0079] According to some embodiments, the unified display model system 500 may consider one or more dynamic display characteristics 502, such as: information obtained from forwardfacing ambient light sensors (ALS) 504; information obtained from rear-facing ALS 510; histogram information for the currently-displayed content 506; and / or the display device’s current overall brightness level 508.

[0080] According to some embodiments, unified display model system 500 may also consider one or more static display characteristics 512 when determining how to modify displayed content, such as: information regarding the percentage of light leakage experienced by the display 514; information regarding the percentage of light reflection of the surface of the display 516; information regarding the display device’s color primaries 518; information regarding the display device’s native white point 520; and / or information regarding the display device’s native response 522.

[0081] According to some embodiments, the unified display model system 500 may combine information from both the dynamic display characteristics 502 and static display characteristics 512 in a perceptual model 530. According to some such embodiments, the perceptual model 530 may comprise a perceptual visual adaptation model 532 configured to model a viewer’s likely adaptation level, given the current dynamic display characteristics 502 and static display characteristics 512. In some embodiments, the perceptual visual adaptation model 532 may be based, at least in part, on a color appearance model (CAM), such as the CIECAM02 color appearance model, and may be used to further inform an ambient conditions model 600 regarding the appropriate amount of gamma boost to apply with the display’s modified transfer function. The CAM may, e.g., be based on the brightness and white point of the viewer’s surround, as well as the field of view of the display subtended by the viewer’s field of vision.

[0082] In some embodiments, knowledge of the size of the display and the distance between the display and the viewer may also serve as useful inputs to the unified display model 500. Information about the distance between the display and the user could be retrieved from a frontfacing image sensor, such as front-facing camera 104. For example, the brightness and white point of the viewer’s surround may be used to determine a ratio of diffuse white brightness to the viewing surround brightness. Based on the determined ratio, a particular gamma boost may be applied. For example, for pitch black ambient environments, an additional gamma boost of about 1.5 imposed by the LUT may be appropriate, whereas a 1.0 gamma boost (i.e., unity, or no boost) may be appropriate for a bright or sun-lit environment. For intermediate surrounds, appropriate gamma boost values to be imposed by the LUT may be interpolated between the values of 1.0 and about 1.5. A more detailed model of surround conditions is provided by the CIECAM02 specification.

[0083] According to some embodiments, the perceptual visual adaptation model 532 may also be used to predict a current lowest perceivable light level for the viewer using model 534 as well as to perceptually map the display and the environment to the viewer’s current perception using model 536. Using this information, and optionally after adapting the luma and / or chroma display data to an XYZ color space (or another device-invariant color space) a perceptual distance model 540 may employ a perceptual color model 542 (e.g., based on the CIELAB color space) to determine, at block 544, a perceptual threshold below which the viewer may not currently be able to perceive changes in tonality and / or the steps (i.e., changes) needed to modify the display’s response based on the viewer’s predicted perceptual adaptation level under the current viewing conditions.

[0084] The output of perceptual model 530 may then be transmitted to a color math model 550 that is used to calculate and configure the modifications to the display’s response to achieve the desired perceptual reference. According to some embodiments, color math model 550 maycomprise: a module 552 for matching the displayed content values to the viewer’s current color perception; a module 554 for performing white point adaptation; a module 556 for performing color matching to the display device’s color gamut; a module 558 for performing white point adaptation; and / or a module 560 for calculating a gamma matching response for the display device. The output of modules 552 / 554 / 556 / 558 / 560 may be combined into one more matrices 562, e.g., a mesopic matrix, chromatic adaptation matrix, etc., and / or one or more combined look up tables (LUTs) 564 to efficiently store the values embodying the changes determined by the color math model 550 to be applied to the display device. The aforementioned matrices 562 and / or LUTs 564 may then be normalized and passed to a display pipeline 580.

[0085] Display pipeline 580 may perform one or more functions of: compositing multiple content items for simultaneous display 582; linearizing content item color data 584; applying the color changes as determined by the color math model 550, e.g., via the application of one or more 3x3 matrices 586; performing any necessary brightness compensation 588 as determined by the unified display model; and gamma encoding 590 the modified content for final display to the viewer 116.

[0086] In some embodiments, the modifications to the combined LUTs 564 may be implemented gradually (e.g., over a determined interval of time), via an animation engine or similar control element in display pipeline 580. According to some such embodiments, display pipeline 580 may be configured to adjust the combined LUTs 564 based on the rate at which it is predicted the viewer’s vision will adapt to the changes.

[0087] In some embodiments, the black level for a given ambient environment is determined, e.g., by using an ambient light sensor 104 or by taking measurements of the actual panel and / or diffuser of the display device. As mentioned above in reference to FIG. 1A, diffuse reflection of ambient light off the surface of the device may add to the intended display values and affect the user’s ability to perceive the darkest display levels (a phenomenon also known as “blackcrush”). In other environments, light levels below a certain brightness threshold will simply not be visible to the viewer. Once this level is determined, the black point may be adjusted accordingly.

[0088] In another embodiment, the white point, i.e., the color a user perceives as white for a given ambient environment, may be determined similarly, e.g., by using one or more optical sensors 104 to analyze the lighting and color conditions of the ambient environment. The white point for the display device may then be chromatically adapted to be the determined white point from the viewer’s surround. Additionally, it is noted that modifications to the white point may be asymmetric between the LUT’s Red, Green, and Blue channels, thereby moving the relative RGB mixture, and hence the white point.

[0089] As described above, in some embodiments, unified display model 500 may first adapt source content 200 to its reference environment using specified adaptation algorithms included in source profile 202, if necessary. For example, RGB-based gamma for Rec.709 video, as classically applied via a mismatch between content encoding gamma and display 102’ s decoding response may be applied. Once source content 200 is adapted to its reference environment using its specified algorithms, unified display model 500 may adapt source content 200 into a shared, system-level viewing environment, or common compositing space, using best practices. The common compositing space may be dynamically changed to match the user’s current viewing environment, or it may be held constant. In implementations in which the common compositing space is held constant, unified display model 500 may globally adapt all content items in the common compositing space to adapt the fixed common compositing space to the current viewing environment. Any appropriate techniques may be used to adapt source content 200 from its reference environment to the common compositing space, and from the common compositing space to the current viewing environment. This function may be particularly useful where multiple content items from multiple source authorsare to be displayed at a time. The unique content adaptations already encoded in each content item may be adjusted without influencing content adaptations applied to other content items. Then, the common compositing space for all content items may be adjusted based on the particular viewing surround for display 102. In the embodiments described immediately above, the combined LUTs 564 may serve as a useful and efficient place for unified display model system 500 to impose these environmentally-aware display transfer function adaptations. In some embodiments, the unified display model system 500 may generate an ICC profile that represents the native response of the display as the true native response of the display divided by the desired system gamma, based on the viewing surround. The ICC profile may include fixed “presets” where each preset represents a particular viewing surround and the corresponding environmental adaptations needed for content to be perceived correctly in the particular viewing surround. Unified display model 500 may then determine an appropriate preset based on analysis of the obtained ambient conditions and apply the corresponding environmental adaptations to source content 200 — either directly or to the common compositing space.

[0090] Referring now to FIG. 6, a simplified functional block diagram of an example ambient conditions model 600 is shown. As alluded to above, the ambient conditions model 600 may consider various factors, e.g.: predictions from a color appearance / perception model 610; information regarding the ambient environment, e.g., from ambient light sensor(s) / image sensor(s) 620; information regarding the display’s current brightness level and / or brightness history 630 (e.g., knowing how bright the display has been and for how long may influence the user’s adaptation level); information and characteristics from the display device’s profile 640; and / or information based on historically displayed content / predictions based on upcoming content 650.

[0091] Color appearance model 610 may comprise, e.g., the CIECAM02 color appearance model or the CIECAM97s model. Color appearance models may be used to perform chromatic adaptation transforms and / or for calculating mathematical correlates for the six technically defined dimensions of color appearance: brightness (luminance), lightness, colorfulness, chroma, saturation, and hue.

[0092] Display characteristics 640 may comprise information from display profile 204 regarding the display device’s color space, native display response characteristics or abnormalities, reflectiveness, leakage, or even the type of screen surface used by the display. For example, an “anti-glare” display with a diffuser will “lose” many more black levels at a given (non-zero) ambient light level than a glossy display will.

[0093] Historical model 650 may take into account both the instantaneous brightness levels of content and the cumulative brightness of content over a period of time. In other embodiments, the model 650 may also perform an analysis of upcoming content, e.g., to allow the ambient conditions model to begin to adjust a display’s transfer function over time, such that it is in a desired state by the time (or within a threshold amount of time) that the upcoming content is displayed to the viewer. The biological / chemical speeds of visual adaptation in humans may also be considered when the ambient conditions model 600 determines how quickly to adjust the display to account for the upcoming content. In some cases, content may itself already be adaptively encoded, e.g., by the source content creator. For example, one or more frames of the content may include a customized transfer function associated with respective frame or frames. In some embodiments, the customized transfer function for a given frame may be based only on the given frame’s content, e.g., a brightness level of the given frame. In other embodiments, the customized transfer function for a given frame may be based, at least in part, on at least one of: a brightness level of one or more frames displayed prior to the one or more frames of content; and / or a brightness level of one or more frames displayed after the one ormore frames of content. In cases where the content itself has been adaptively encoded, the ambient conditions model 600 may first implement the adaptively encoded adjustments, moving the content into a common compositing space according to content indicators included in source profile 202. Then, ambient conditions model 600 may attempt to further modify the display’s transfer function during the display of particular frames of the encoded content, e.g., based on the other various environment factors, e.g., 610 / 620 / 630 / 640, that may have been obtained at the display device.

[0094] According to some embodiments, modifications determined by the ambient conditions model 600 may be implemented by changing existing table values (e.g., as stored in one or more calibration LUTs, i.e., tables configured to give the display a ‘perfectly’ responding tone response curve). Such changes may be performed via looking up the value for the transformed value in the original table, or by modifying the original table ‘in place’ via a warping technique. For example, the aforementioned black level (and / or white level) adaptation processes may implemented via a warped compression of the values in the table up from black (and / or down from white). In other embodiments, a “re-gamma” and / or a “re-saturation” of the LUTs may be applied in response to the adjustments determined by the ambient conditions model 600.

[0095] As is to be understood, the exact manner in which ambient conditions model 600 processes information 610 / 620 / 630 / 640 / 650 received from the various sources optical sensors 104, display brightness 508, display profile 204, and indicators in content source profile 202, and how it modifies the resultant display response curve, e.g., by modifying LUT values, including how quickly such modifications take place, are up to the particular implementation and desired effects of a given system.

[0096] According to some embodiments, the ambient conditions model 600 may be used to consider the various factors described above with reference to FIG. 6 that may have an impact on the viewer’s perception at the given moment in time. Then, based on the output of theambient conditions model 600, an updated display transfer function may be determined for driving the display 102. The display transfer function may be used to convert between the input signal data values and the voltage values that can be used to drive the display to generate a pixel brightness corresponding to the perceptual bin that the transfer function has mapped the input signal data value to at the given moment in time. One goal of the ambient conditions model 600 is to: determine the viewer’s current surround; determine what region of the adapted range the content and / or display is modulating; and then map to the transfer function corresponding to that portion of the adapted range, so as to optimally use the display codes (and the bits needed to enumerate them).

[0097] EDR-aware, Hue-preserving, Soft-clipping Tone Mapping Operation

[0098] Referring now to FIG. 7A, one embodiment of a process 700 for performing an EDR- aware, hue-preserving soft-clipping operation is illustrated, in accordance with one or more embodiments. First, at Step 705, the process 700 may receive source content data in a first format (e.g., HLG). Next, at Step 710, the process 700 may use the source medium’s reference standard to map the source data to display-referred nits. Next, at Step 715, the process 700 may scale by the display-referred reference white value nits to generate EDR content. As EDR content, all further steps in the tone-mapping operation may be handled in the same manner, regardless of their source format. That is, only one tone-mapping algorithm is required, which allows for simple and consistent application across a wide variety of content and conditions. Finally, at Step 720, the aforementioned tone-mapping operations may be performed by process 700.

[0099] Referring now to FIG. 7B, another embodiment of a process 750 for performing an EDR-aware, hue-preserving soft-clipping operation is illustrated, in accordance with one or more embodiments. First, at Step 755, the process 750 may receive data indicative of a first content item (e.g., HLG-encoded content). Next, at Step 760, the process 750 may convert thefirst content item into an extended dynamic range (EDR) representation based, at least in part, on a reference display-referred mapping for the first content item. Next, at Step 765, the process 760 may apply a perceptual model to the EDR representation of the first content item to generate an adapted version of the first content item, wherein the perceptual model is based, at least in part, on: an amount of EDR headroom available in the first content item and an amount of EDR headroom available on the display device, and wherein the perceptual model is configured to control soft clipping and tone mapping parameters for the first content item. Finally, at Step 770, the process 700 may display the adapted version of the first content item on the display device.

[0100] Referring now to FIG. 8, a graph 800 is shown, illustrating various variables used in the performance of an EDR-aware, hue-preserving soft-clipping operation, in accordance with one or more embodiments. In the example of graph 800, the vertical or y-axis 804 reflects the values of relevant soft-clipping operation parameters (in this case: an input clip value 810, a highlight compression value 812, and an SDR knee value 814) for input (or “source”) content having 4.9x EDR headroom with respect to the output (or “display”) device EDR headroom, which is reflected on the horizontal or x-axis 802 values of graph 800. It is to be understood that graph 800 is merely exemplary, and the mapping of input content with 4.9x EDR headroom with respect to the output EDR headroom is but one example of the type of input content that may be adapted.

[0101] As illustrated, the horizontal axis 802 of graph 800 is divided into four distinct stages or “modes” 806, which represent tone mapping techniques that may be applied to input content, based on the output EDR headroom available. The modes will now be discussed in decreasing order of preference.

[0102] First, Mode D (806D) may begin to be applied for output EDR headroom values greater than or equal to amount of EDR headroom available in the input content (in thisexample, 4.9x, as shown by the dashed line 826 on graph 800). Thus, for output display devices with 4.9x or greater EDR headroom: the input clip value (810D) may remain at 4.9, and the highlight compression (812D) and SDR knee value (814D) may remain at 1. In other words, Mode D (806D) is the most preferred mode, because it represents the input content in a 1 : 1 fashion, without adaptation. Thus, as will be illustrated further in FIG. 9A, in Mode D (806D), the reference white value and the dynamic range of the input content is maintained, and no compression or clipping is applied to the input content signal.

[0103] Next, Mode C (806c) may begin to be applied for output EDR headroom values less than the amount of EDR headroom available in the input content but greater than a first threshold value (in this example, ~1 ,7x, as shown by the dashed line 824 on graph 800). Thus, for output display devices with between 1.7x and 4.9x EDR headroom: the input clip value (810c) may remain at 4.9, while some highlight compression is applied as required (812c) and SDR knee value (814c) remains at 1. In other words, Mode C (806c) is the second-most preferred mode, because the full range of input content can be represented without excessive perceptual losses. Thus, as will be illustrated further in FIG. 9B, in Mode C (806c), the adapted content maintains compatibility with the system and other displayed SDR and HDR content, and the minimal necessary amount of compression (and no clipping) is applied to the input content signal.

[0104] Next, Mode B (806B) may begin to be applied for output EDR headroom values less than the first threshold value (1 ,7x, shown by line 824) but greater than a second threshold value, wherein the second threshold value is lower than the first threshold value (in this example, ~1.3x, as shown by the dashed line 822 on graph 800). Thus, for output display devices with between 1.3x and 1.7x EDR headroom: the input clip value (810B) may begin to be clipped down from 4.9, while a decreasing amount of highlight compression is applied moving towards the lower end of Mode B (812B) and SDR knee value (814B) remains at 1. Inother words, Mode B (806B) is the third-most preferred mode, because highlight clipping (preferably hue-preserving clipping) — as well as some amount of highlight compression — is used to avoid excessive perceptual losses in the highlight regions. Thus, as will be illustrated further in FIG. 9C, in Mode B (806B), the adapted content maintains compatibility with the system and other displayed SDR and HDR content, and the minimal necessary amount of compression and hue-preserving highlight clipping is applied to the input content signal.

[0105] Finally, Mode A (806A) may begin to be applied for output EDR headroom values less than the second threshold value (1.3x, shown by line 822) all the way down to a values of 1.0 output EDR headroom (i.e., an SDR display(?), as shown by the dashed line 820 on graph 800). Thus, for output display devices with between l.Ox and 1.3x EDR headroom: the input clip value (810A) may be clipped to a fixed level (e.g., 1.4), while an increasing amount of highlight compression is applied moving towards the lower end of Mode A (812A) and SDR knee value (814A) decrease below 1 to some minimal amount of permissible reference white compression (e.g., 0.8, in the example of graph 800). In other words, Mode A (806A) is the least preferred mode, because it involves compressing at least a portion of highlight values of the input content, clipping at least a portion of highlight values of the input content (preferably, in a hue-preserving fashion), and compressing (preferably, adaptively compressing) a reference white value of the first content item. Thus, as will be illustrated further in FIG. 9D, in Mode A (806A), the adapted content maintains as much compatibility as possible with the system and other displayed SDR and HDR content (while still providing some headroom), and some amount of compression and hue-preserving highlight clipping is applied to the input content signal, so that more headroom than otherwise expected is maintained.

[0106] Referring now to FIG. 9 A, a graph 900 is shown, illustrating a tone mapping curve used in the performance of an EDR-aware, hue-preserving soft-clipping operation for a first range of output EDR headroom values available on a display device, in accordance withone or more embodiments. Graph 900 illustrates an exemplary tone mapping curve 906 that may be applied by an EDR-aware, hue-preserving soft-clipping operation for a first range of output EDR headroom values (referred to above as Mode D (806D)), e.g., output EDR headroom values greater than or equal to amount of EDR headroom available in the input content. In the example of graph 900, the vertical or y-axis 904 reflects the output EDR values, and the horizontal or x-axis 902 reflects the input EDR values.

[0107] In the example of graph 900, an exemplary tone curve 906 is shown for a hypothetical display device having 6. Ox output EDR headroom (i.e., a value that is higher than the 4.9x EDR headroom of the exemplary input source content). As described above with reference to FIG. 8, in Mode D (806D), which is illustrated in graph inset 950, a tone mapping curve 906 is derived that represents the input content in a 1 : 1 fashion, i.e., without adaptation. As a consequence, the reference white value and the dynamic range of the input content is maintained, and no compression or clipping is applied to the input content signal.

[0108] Referring now to FIG. 9B, a graph 910 is shown, illustrating a tone mapping curve used in the performance of an EDR-aware, hue-preserving soft-clipping operation for a second range of output EDR headroom values available on a display device, in accordance with one or more embodiments. Graph 910 illustrates an exemplary tone mapping curve made of components 916 and 918 that may be applied by an EDR-aware, hue-preserving soft-clipping operation for a second range of output EDR headroom values (referred to above as Mode C (806c)), e.g., output EDR headroom values less than the amount of EDR headroom available in the input content but greater than a first threshold value.

[0109] In the example of graph 910, an exemplary tone curve made of components 916 and 918 is shown for a hypothetical display device having 3.2x output EDR headroom (i.e., a value that is lower than the 4.9x EDR headroom of the exemplary input source content, but higher than a 1 ,7x first threshold value). As described above with reference to FIG. 8, in ModeC (806c), which is illustrated in graph inset 960, a tone mapping curve made of components 916 (e.g., a pre-smoothed SDR segment for input content having values 1.0 and under) and 918 (e.g., a pre-smoothed HDR segment for input content having values greater than 1.0) is derived that represents the full range of input content (i.e., the output EDR values go to the maximum hypothetical display device output EDR headroom of 3.2x, shown at dashed line 920) without excessive perceptual losses and using the minimal necessary amount of compression (and no clipping) applied to the input content signal.

[0110] In some embodiments, control parameters for the soft-clipping operation may comprise: a parameter representative of the compression amount for the reference white value (e.g., a percentage); a parameter representative of the amount of the input peak value that is maintained (e.g., a percentage); and / or a parameter representative of the amount of input peak compression (e.g., a percentage or scalar value).

[0111] For example, as shown in FIG. 9B, the soft-clipped SDR segment 916 goes from a coordinate value of (0,0) on graph 910 to a coordinate value of (referencewhite, compressedReferenceWhite). In the example of graph 910, there is no compression of the reference white value, so the soft-clipped SDR segment 916 goes from a coordinate value of (0,0) to a coordinate value of (1, 1). As also shown in FIG. 9B, the soft-clipped HDR segment 918 goes from a coordinate value of (referencewhite, compressedReferenceWhite) on graph 910 to a coordinate value of (inputHeadroom, maintainedlnputPeak * maintainedlnputPeakCompression). In the example of graph 910, the inputHeadroom is 3.2x, and there is no compression of the maintained input peak, so the soft-clipped HDR segment 918 goes from a coordinate value of (1,1) to a coordinate value of (4.9, 3.2). In some embodiments, a look up table (LUT) may be created and (optionally smoothed) to stores the values representing the aforementioned SDR and HDR segments of the derived tone-mapping curve.

[0112] In some embodiments, a Barten Loss value may be evaluated against an acceptable amount of perceptual loss. (Barten distance may be measured in different units as desired by a given implementation, e.g., PQ10 steps.) Beyond this acceptable loss value, the tone mapping operation may transition into the next most preferable Mode, e.g., moving from Mode C to Mode B, as will be discussed further with reference to FIG. 9C, below. In some embodiments, a smoothing parameter (e.g., via a hyperbolic interpolation parameters) may be used to transition smoothly between the segment functions of the tone mapping curve, e.g., by controlling the center and / or width of the transition portion of the tone mapping curve.

[0113] Referring now to FIG. 9C, a graph 920 is shown, illustrating a tone mapping curve used in the performance of an EDR-aware, hue-preserving soft-clipping operation for a third range of output EDR headroom values available on a display device, in accordance with one or more embodiments. Graph 920 illustrates an exemplary tone mapping curve made of components 922 and 924 that may be applied by an EDR-aware, hue-preserving soft-clipping operation for a second range of output EDR headroom values (referred to above as Mode B (806B)), e.g., output EDR headroom values between the first threshold value and a second threshold value.

[0114] In the example of graph 920, an exemplary tone curve made of components 922 and 924 is shown for a hypothetical display device having 1.7x output EDR headroom (i.e., a value that is between the 1 ,3x and 1 ,7x EDR headroom threshold values for the output display device). As described above with reference to FIG. 8, in Mode B (806B), which is illustrated in graph inset 970, a tone mapping curve made of components 922 (e.g., a pre-smoothed SDR segment for input content having values 1.0 and under) and 924 (e.g., a pre-smoothed HDR segment for input content having values greater than 1.0) is derived that performs some degree of (preferably hue-preserving) highlight clipping (e.g., at the output EDR value of 1.7x, asshown at dashed line 920) — as well as some amount of highlight compression (as shown by line segment 924) — in order to avoid excessive perceptual losses in the highlight regions.

[0115] In some embodiments, Mode B (806B) may represent a region of output EDR headroom values where the Barten Loss value is too high for compression alone, meaning that some efforts to adaptively limit highlights (while maintaining hues) may be performed. As with Mode C, the amount of compression may be configured not to exceed a Barten loss criteria value, and a smoothing parameter (e.g., via a hyperbolic interpolation parameters) may be used to transition smoothly between the segment functions of the tone mapping curve.

[0116] For example, in some embodiments, the tone-mapping operation may solve for an input clip that smoothly interpolates between the clip that is applied at the “end” of Mode A (i.e., maintaining a minimum amount of Barten headroom without compressing the reference white value) and the “beginning” of Mode C (i.e., wherein no headroom clipping is required). The input clip value may be used to solve (e.g., using the secant method) for the required headroom compression (i.e., at the kept input peak).

[0117] In some such embodiments, the perceptual model may be further configured to ensure that a continuous tone mapping curve is applied to the input content as the model transitions between the various Modes, as well as over the full range of values for the amount of EDR headroom available in input content and the amount of EDR headroom available on the display device. In other words, a continuous tone mapping curve ensures a gradual change in the display device’s brightness levels. Moreover, because an “auto brightness” algorithm could simultaneously be modulating the brightness of display device, it may have an unintended consequence of changing the amount of headroom available for the display, which could cause discontinuity / flashing in the compression curves, if not accounted for.

[0118] Referring now to FIG. 9D, a graph 930 is shown, illustrating a tone mapping curve used in the performance of an EDR-aware, hue-preserving soft-clipping operation for afourth range of output EDR headroom values available on a display device, in accordance with one or more embodiments. Graph 930 illustrates an exemplary tone mapping curve made of components 928 and 930 that may be applied by an EDR-aware, hue-preserving soft-clipping operation for a second range of output EDR headroom values (referred to above as Mode A (806A)), e.g., output EDR headroom values below the second threshold value.

[0119] In the example of graph 930, an exemplary tone curve made of components 928 and 930 is shown for a hypothetical display device having l. lx output EDR headroom (i.e., a value that is below the 1.3x EDR headroom threshold of Mode B). As described above with reference to FIG. 8, in Mode A (806A), which is illustrated in graph inset 980, a tone mapping curve made of components 928 (e.g., a pre-smoothed SDR segment for input content having values 1.0 and under) and 930 (e.g., a pre-smoothed HDR segment for input content having values greater than 1.0) is derived that performs some degree of (preferably hue-preserving) highlight clipping (e.g., at the output EDR value of 1. lx, as shown at dashed line 932) — as well as some amount of highlight compression (as shown by line segment 930), as well as moving the reference white point us to some maximum amount (e.g., in this case reference white may be adaptively compressed up to a limit of 20%, or a value of 0.8). Preferably, the reference white point is compressed by no farther than is necessary to maintain the minimum Barten Loss headroom, and no more than the aforementioned predetermined maximum amount. According to some such embodiments, the “knee” value of the compressed reference white may also be solved for, so that the input peak to be maintained may be solved for (i.e., an input peak that still allows the remaining input content values to be compressed by no more than the Barten compression limit).

[0120] In some embodiments, Mode A (806A) may represent a region of output EDR headroom values where the Barten Loss value is too high for compression and clipping alone, meaning that some efforts to adaptively compress the reference white value may also beperformed. By maintaining as much compatibility as possible with the system and other displayed SDR and HDR content (while still providing some headroom), and providing some amount of compression and hue-preserving highlight clipping to the input content signal, more headroom than would otherwise be expected is maintained in the output image (e.g., providing the ability to maintain accurate hue in highlight regions of the content, such as bright sky regions).

[0121] Hue Preservation

[0122] According to some embodiments, a display may not be capable of representing a pixel’s hue accurately at a calculated brightness level (e.g., the pixel’s R, G, and / or B component may exceed the display’s EDR headroom). In such instances, the tone mapping system may instead solve for the brightest pixel value of the same hue and use those values for the pixel’s mapped color components.

[0123] For example, a hue preservation process may begin by finding pixels with components exceeding the peak kept input headroom, and then compute L*a*b* values for these pixels. The process may then solve for the L*a*b* values at the peak kept input headroom value and replace the pixel’ s values with the solved-for values. Thus, when the aforementioned hue-preservation techniques are combined with EDR-aware soft-clipping operations (such as those described above), brightness gradients will smoothly transition to the maximum representable brightness values, while maintaining a consistent hue representation.

[0124] Post-Tone Mapping Perceptual Adjustments

[0125] In some situation, simply performing tone-mapping (or even highlight clipping) could have unintended consequence on a user’ s perception of the viewed content. For example, consider a scenario wherein a region of bright sky pixels are clipped to a dimmer value. The presence of the bright sky in the scene affects the viewer’s perception, e.g., limiting their ability to perceive shadow details, which is likely to be the author’s intention for the content.However, the aforementioned process of clipping (or otherwise limiting) the sky’s brightness via tone-mapping may allow shadow detail in the content to be more apparent than the author intended. Thus, in some embodiments, a post-tone mapping perceptual adjustment operation may be performed, wherein a perceptual model of the user’ s adaptation may be used to compare the reference rendition with the tone-mapped rendition of the content. Then, a correction(s) may be applied to the content to maintain the author’s intended perception of the rest of the content in the displayed image(s). In other words, the tone-mapping operation may apply an additional correction to at least a portion of the adapted version of the content (e.g., just the sky pixels, or the like), in order to maintain a desired perception of the at least a portion of the adapted version of the content for the user.

[0126] As may now be appreciated, while prior art tone-mapping operations may incorporate fixed curves to attempt to compensate for perceptual side-effects, some of the techniques disclosed herein allow for deterministic curves to be applied, which allow for further sharpening (or other techniques) to be applied to the adapted content — thereby providing dynamic correction of the perceptual effect on the user based on the actual content being viewed.

[0127] Exemplary Electronic Device

[0128] Referring now to FIG. 10, a simplified functional block diagram of a representative electronic device possessing a display is shown, in accordance with some embodiments. Electronic device 1000 could be, for example, a mobile telephone, personal media device, HMD, portable camera, or a tablet, notebook or desktop computer system. As shown, electronic device 1000 may include processor 1005, display 1010, user interface 1015, graphics hardware 1020, device sensors 1025 e.g., proximity sensor / ambient light sensor, accelerometer and / or gyroscope), microphone 1030, audio codec(s) 1035, speaker(s) 1040, communications circuitry 1045, image sensor / camera circuitry 1050, which may, e.g.,comprise multiple camera units / optical sensors having different characteristics (as well as camera units that are housed outside of, but in electronic communication with, device 1000), video codec(s) 1055, memory 1060, storage 1065, and communications bus 1070.

[0129] Processor 1005 may execute instructions necessary to carry out or control the operation of many functions performed by device 1000 (e.g., such as the generation and / or processing of signals in accordance with the various embodiments described herein). Processor 1005 may, for instance, drive display 1010 and receive user input from user interface 1015. User interface 1015 can take a variety of forms, such as a button, keypad, dial, a click wheel, keyboard, display screen and / or a touch screen. User interface 1015 could, for example, be the conduit through which a user may view a captured image or video stream and / or indicate particular frame(s) that the user would like to have played / paused, etc., or have particular adjustments applied to (e.g., by clicking on a physical or virtual button at the moment the desired frame is being displayed on the device’s display screen).

[0130] In one embodiment, display 1010 may display a video stream as it is captured, while processor 1005 and / or graphics hardware 1020 evaluate an ambient conditions model to determine modifications to the display’s transfer function or gamma boost, optionally storing the video stream in memory 1060 and / or storage 1065. Processor 1005 may be a system-on- chip such as those found in mobile devices and include one or more dedicated graphics processing units (GPUs). Processor 1005 may be based on reduced instruction-set computer (RISC) or complex instruction-set computer (CISC) architectures or any other suitable architecture and may include one or more processing cores. Graphics hardware 1020 may be special purpose computational hardware for processing graphics and / or assisting processor 1005 perform computational tasks. In one embodiment, graphics hardware 1020 may include one or more programmable graphics processing units (GPUs).

[0131] Image sensor / camera circuitry 1050 may comprise one or more camera units configured to capture images, e.g., images which indicate ambient lighting conditions in the viewing environment and may have an effect on the output of the ambient conditions model, e.g., in accordance with this disclosure. Output from image sensor / camera circuitry 1050 may be processed, at least in part, by video codec(s) 1055 and / or processor 1005 and / or graphics hardware 1020, and / or a dedicated image processing unit incorporated within circuitry 1050. Images so captured may be stored in memory 1060 and / or storage 1065. Memory 1060 may include one or more different types of media used by processor 1005, graphics hardware 1020, and image sensor / camera circuitry 1050 to perform device functions. For example, memory 1060 may include memory cache, read-only memory (ROM), and / or random access memory (RAM). Storage 1065 may store media e.g., audio, image and video files), computer program instructions or software, preference information, device profile information, and any other suitable data. Storage 1065 may include one more non-transitory storage mediums including, for example, magnetic disks (fixed, floppy, and removable) and tape, optical media such as CD-ROMs and digital video disks (DVDs), and semiconductor memory devices such as Electrically Programmable Read-Only Memory (EPROM), and Electrically Erasable Programmable Read-Only Memory (EEPROM). Memory 1060 and storage 1065 may be used to retain computer program instructions or code organized into one or more modules and written in any desired computer programming language. When executed by, for example, processor 1005, such computer program code may implement one or more of the methods described herein. Power source 1075 may comprise a rechargeable battery (e.g., a lithium-ion battery, or the like) or other electrical connection to a power supply, e.g., to a mains power source, that is used to manage and / or provide electrical power to the electronic components and associated circuitry of electronic device 1000.

[0132] The foregoing description of preferred and other embodiments is not intended to limit or restrict the scope or applicability of the inventive concepts conceived of by the Applicants. In exchange for disclosing the inventive concepts contained herein, the Applicants desire all patent rights afforded by the appended claims. Therefore, it is intended that the appended claims include all modifications and alterations to the full extent that they come within the scope of the following claims or the equivalents thereof.

Claims

CLAIMSWhat is claimed is:

1. A method of adapting content for display on a display device, comprising: receiving data indicative of a first content item; converting the first content item into an extended dynamic range (EDR) representation based, at least in part, on a reference display-referred mapping for the first content item; applying a perceptual model to the EDR representation of the first content item to generate an adapted version of the first content item, wherein the perceptual model is based, at least in part, on an amount of EDR headroom available in the first content item and an amount of EDR headroom available on the display device, and wherein the perceptual model is configured to control soft clipping and tone mapping parameters for the first content item; and displaying the adapted version of the first content item on the display device.

2. The method of claim 1, further comprising: receiving data indicative of a second content item; converting the second content item into an EDR representation based, at least in part, on a reference display-referred mapping for the second content item; applying the perceptual model to the EDR representation of the second content item to generate an adapted version of the second content item, wherein the perceptual model is based, at least in part, on an amount of EDR headroom available in the second content item and an amount of EDR headroom available on the display device, and wherein the perceptual model is configured to control soft clipping and tone mapping parameters for the second content item; and displaying the adapted version of the second content item concurrently with the adapted version of the first content item on the display device.

3. The method of claim 2, wherein the first content item and the second content item are of different media types.

4. The method of claim 1, wherein the perceptual model is further configured to preserve hues in the first content item.

5. The method of claim 4, wherein preserving hues in the first content item comprises: determining that a color component of at least a first pixel in the first content item having a first hue is not representable on the display device; and solving for a set of brightest color component values for the first pixel that preserve the first hue.

6. The method of claim 1 , wherein, for a first range of values for the amount of EDR headroom available on the display device, applying the perceptual model further comprises: maintaining a reference white value and a dynamic range of the first content item, wherein the first range of values comprises values wherein the amount of EDR headroom available on the display device is equal to or greater than the amount of EDR headroom available in the first content item.

7. The method of claim 6, wherein, for a second range of values for the amount of EDR headroom available on the display device, applying the perceptual model further comprises: compressing at least a portion of highlight values of the first content item, wherein the second range of values comprise values wherein the amount of EDR headroom available on the display device is less than the amount of EDR headroom available in the first content item but greater than a first threshold value.

8. The method of claim 7, wherein, for a third range of values for the amount of EDR headroom available on the display device, applying the perceptual model further comprises: compressing at least a first portion of highlight values of the first content item; and clipping at least a second portion of highlight values of the first content item, wherein the third range of values comprises values wherein the amount of EDR headroom available on the display device is less than the first threshold value but greater than a second threshold value, and wherein the second threshold value is lower than the first threshold value.

9. The method of claim 8, wherein, for a fourth range of values for the amount of EDR headroom available on the display device, applying the perceptual model further comprises: compressing at least a third portion of highlight values of the first content item; clipping at least a fourth portion of highlight values of the first content item; and compressing a reference white value of the first content item, wherein the fourth range of values comprises values wherein the amount of EDR headroom available on the display device is less than the second threshold value.

10. The method of claim 1, further comprising: comparing an estimated perception of a viewer of a reference rendition of the first content item to an estimated perception of a viewer of the adapted version of the first content item; and applying an additional correction to at least a portion of the adapted version of the first content item to maintain a desired perception of the at least a portion of the adapted version of the first content item for the viewer.

11. The method of claim 1, wherein the perceptual model is further configured to cause a continuous tone mapping curve to be applied to the first content item over each of: a first range of values for the amount of EDR headroom available in the first content item; and a second range of values for the amount of EDR headroom available on the display device.

12. A non-transitory program storage device comprising instructions stored thereon to cause one or more processors to: receive data indicative of a first content item; convert the first content item into an extended dynamic range (EDR) representation based, at least in part, on a reference display-referred mapping for the first content item; apply a perceptual model to the EDR representation of the first content item to generate an adapted version of the first content item, wherein the perceptual model is based, at least in part, on an amount of EDR headroom available in the first content item and an amount of EDR headroom available on a display device, and wherein the perceptual model is configured to control soft clipping and tone mapping parameters for the first content item; andcause the adapted version of the first content item to be displayed on the display device.

13. The non-transitory program storage device of claim 12, wherein the instructions to cause the one or more processors to apply a perceptual model further comprise instructions to cause the one or more processors to apply a perceptual model that is configured to preserve hues in the first content item.

14. The non-transitory program storage device of claim 13, wherein the instructions to cause the one or more processors to apply a perceptual model that is configured to preserve hues in the first content item further comprise instructions to cause the one or more processors to: determine that a color component of at least a first pixel in the first content item having a first hue is not representable on the display device; and solve for a set of brightest color component values for the first pixel that preserve the first hue.

15. The non-transitory program storage device of claim 12, wherein the instructions further cause the one or more processors to: compare an estimated perception of a viewer of a reference rendition of the first content item to an estimated perception of a viewer of the adapted version of the first content item; and apply an additional correction to at least a portion of the adapted version of the first content item to maintain a desired perception of the at least a portion of the adapted version of the first content item for the viewer.

16. A device, comprising: a memory; a display device; and one or more processors operatively coupled to the memory, wherein the one or more processors are configured to execute instructions causing the one or more processors to: receive data indicative of a first content item;convert the first content item into an extended dynamic range (EDR) representation based, at least in part, on a reference display-referred mapping for the first content item; apply a perceptual model to the EDR representation of the first content item to generate an adapted version of the first content item, wherein the perceptual model is based, at least in part, on an amount of EDR headroom available in the first content item and an amount of EDR headroom available on the display device, and wherein the perceptual model is configured to control soft clipping and tone mapping parameters for the first content item; and cause the adapted version of the first content item to be displayed on the display device.

17. The device of claim 16, wherein the instructions to cause the one or more processors to apply a perceptual model further comprise instructions to cause the one or more processors to apply a perceptual model that is configured to preserve hues in the first content item.

18. The device of claim 17, wherein the instructions to cause the one or more processors to apply a perceptual model that is configured to preserve hues in the first content item further comprise instructions to cause the one or more processors to: determine that a color component of at least a first pixel in the first content item having a first hue is not representable on the display device; and solve for a set of brightest color component values for the first pixel that preserve the first hue.

19. The device of claim 16, wherein the instructions further cause the one or more processors to: compare an estimated perception of a viewer of a reference rendition of the first content item to an estimated perception of a viewer of the adapted version of the first content item; and apply an additional correction to at least a portion of the adapted version of the first content item to maintain a desired perception of the at least a portion of the adapted version of the first content item for the viewer.

20. The device of claim 16, wherein the perceptual model is further configured to cause a continuous tone mapping curve to be applied to the first content item over each of: a first range of values for the amount of EDR headroom available in the first content item; and a second range of values for the amount of EDR headroom available on the display device.

Images