Dynamic system light-to-light transfer function (OOTF) for providing perceptual benchmark

Abstract

A dynamic systematic light-to-light transfer function (OOTF) is disclosed that adapts the displayed content to provide a so-called'perceptual benchmark 'effect to a viewer, in which when a current viewing condition deviates from a'benchmark' viewing scene, the current viewing condition does not deviate from the'benchmark 'viewing scene, and when the current viewing condition does not deviate from the'benchmark' viewing scene, the current viewing condition does not deviate from the'benchmark 'viewing scene. The dynamic system OOTF adapts the content and / or display to provide perceptual effects to the viewer that are as close as possible to the "benchmark" viewing scene. The dynamic system OOTF may perform one or more of the following adaptation processes: adaptation of media content from a source color space to a linear XYZ color space; adapting the media content from the linear XYZ color space to a color space of the display device; automatically adjusting the brightness, white points and / or black points of the display equipment; adapting the media content from the expected viewing environment to a fixed viewing environment of the common composite space; and / or adapting media content from a fixed viewing environment of the common composite space to a fixed viewing environment of the current viewing condition of the viewer.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application Serial No. 63 / 376,540 (“the '540 Application”), filed on September 21, 2022. This application is also related to commonly assigned U.S. Patent No. 11,386,875 (“the '875 Patent”). The '540 Application and the '875 Patent are hereby incorporated by reference herein in their entirety. Background Art

[0003] Today, consumer electronic devices with display screens are used in many different environments with many different lighting conditions, such as offices, homes, home theaters, head-mounted displays (HMD) inside and outdoors. Devices generally need to be designed so that no matter what the user's viewing environment is at any given moment, the viewer can perceive only minimal (or, ideally, no) color banding, and the displayed content has a consistent appearance and tone. Many content items are created for specific display devices and viewing environments. For example, movies are often created for Rec.709 displays to be viewed in a dark viewing environment. Devices generally need to be able to adapt content items to many different types of expected display devices and many different viewing environments, so that no matter what the current viewing conditions around the display device are, the content items appear as the content author expects them to be perceived.

[0004] For these reasons and more, it is desirable to map each content item to a shared system-level viewing environment (also referred to herein as a "common composition space"), and then map from that shared system-level viewing environment to the current viewing environment. Thus, there is a need for techniques to implement a dynamic system-level light-to-light transfer function (OOTF) that is capable of utilizing a model of environmental conditions (as well as a number of other display-related factors) to automatically adjust the overall content adaptation process for a display, e.g., to provide a so-called "perceptual benchmark" such that when a dynamic viewing scene corresponds precisely to a "benchmark" viewing scene, a measurably accurate benchmark response is provided by the display device (e.g., as quantifiable via an optical instrument measuring the brightness of the face of the display device), and then, when the viewing scene deviates from the "benchmark" viewing scene, the dynamic system OOTF adapts to provide the viewer with perceptual effects in a "non-benchmark" viewing scene that are as close to what they experience in the "benchmark" viewing scene. Successfully modeling a user's current viewing environment and its impact on the perception of displayed content will allow the user's perception of displayed content to remain relatively independent of the environmental conditions of the display device being viewed and / or any other content items being displayed simultaneously. Summary of the invention

[0005] As mentioned above, human perception is not absolute; rather, it is relative. In other words, a human viewer's perception of a displayed image changes based on what is around the image, the image itself, and what brightness and white point the viewer is currently adapted to. The display can often be positioned in front of a wall. In this case, the ambient lighting (e.g., brightness and color) in the room illuminates the wall behind the display and changes the viewer's perception of the displayed image. Potential changes to the viewer's perception of displayed content include changes in hue (which can be modeled using a gamma function) as well as changes to the white point (i.e., the absolute color perceived as white) and the black point (i.e., the highest brightness level that is indistinguishable from true black).

[0006] Thus, while some devices may attempt to maintain consistent content adaptation on a display device throughout the encoding, decoding, and color management process, this does not take into account the impact that the environmental conditions surrounding the display device may have on a viewer's perception of the displayed content. Many color management systems attempt to consistently map content to a display so that the encoding of the content and the reproduction of the display do not affect the resulting displayed content, thus providing consistency across content encoding and display. However, these color management systems require fixed viewing conditions, such as always using an intended display and a suggested "reference" viewing environment.

[0007] According to various embodiments described herein, a processor in communication with a display device executing a dynamic system OOTF may adapt a variety of constrained parameters to provide a so-called "perceptual baseline" effect to a viewer. For example, the dynamic system OOTF may perform one or more of the following adaptation processes: adapting media content from a source color space to a linear XYZ color space; adapting media content from a linear XYZ color space to a color space of a display device; automatically adjusting display device brightness; automatically adjusting display device white point; automatically adjusting display device black point; adapting media content from an expected viewing environment to a fixed viewing environment of a common synthetic space; and / or adapting media content from a fixed viewing environment of the common synthetic space to a fixed viewing environment of the viewer's actual current viewing conditions.

[0008] Another adaptation process, referred to as a "simultaneous contrast adaptation" process, maps each content item to its suggested viewing environment using the technique indicated in the content item by the content indicator. For example, a content item intended for viewing on a Rec.709 display includes a content indicator that uses RGB spatial gamma. The resulting simultaneous contrast-adapted content item is referred to herein as color space data for the suggested viewing environment.

[0009] The dynamic system OOTF technology disclosed herein provides an extension of classic color management systems (which typically match content to the color space of a provided display device while requiring the viewing environment of the source content to be reproduced in the viewer's actual viewing environment) by providing adaptation to the viewer's actual viewing environment, which can be important, especially for mobile devices used in a wide variety of viewing environments and for film content that may be consumed, for example, in a sunlit living room rather than the intended dark movie theater viewing environment.

[0010] Furthermore, authoring content in a viewing environment that does not match the intended viewing environment may also result in biases being included in the content itself, and therefore resulting in an incorrect appearance when the content is viewed in the intended viewing environment. Authoring content under "non-baseline" environmental conditions, but with the viewer adapted to a so-called "perceptual baseline" (e.g., as provided by the application of the dynamic system OOTF techniques described herein), helps ensure that environmental biases are minimized or avoided, and that the resulting edited content (which may even have been authored in a changing, i.e., dynamic, environment) will have the correct appearance when viewed in the intended viewing environment.

[0011] The technology disclosed herein uses a display device in conjunction with various optical sensors (e.g., an ambient light sensor, a multi-spectral ambient light sensor, an image sensor, or a camera) to collect information about the ambient conditions in the current viewing environment of a viewer of the display device. The use of these various optical sensors can provide more detailed information about the ambient lighting conditions, which a processor can use to evaluate a unified display model including an ambient condition model and / or a perceptual adaptation model based at least in part on the received ambient information and information about the display (such as the display's peak brightness, leakage percentage, reflection percentage, reference brightness (SDR max), white point, and the instantaneous, historical, and even future content itself that is being displayed to the viewer, has been displayed to the viewer, or will be displayed to the viewer).

[0012] Output from the unified display model may be used to adapt content so that a viewer's perception of content displayed on a display device is relatively independent of the ambient viewing conditions in which the display is viewed, what the viewer sees on the display (and beyond the display), and therefore how the viewer's vision is adapted. The output of the unified display model may include modifications to the display's transfer function, gamma boost, tone mapping, resaturation, black point, white point, or combinations thereof.

[0013] Thus, according to some embodiments, a non-transitory program storage device is disclosed that includes stored thereon. When executed, the instructions are configured to cause one or more processors to perform the following operations: receiving data indicating a first content item; linearizing the data indicating the first content item according to an inverse transfer function associated with the first content item; mapping the linearized data indicating the first content item from a first color space gamut associated with the first content item to a second color space gamut associated with a common synthetic space; modifying the mapped, linearized data indicating the first content item based on at least one of the following: (a) a first difference between a first expected viewing condition associated with the first content item and a second expected viewing condition associated with the common synthetic space; and (b) a second difference between a first expected viewer adaptation level associated with the first content item and a first predicted viewer adaptation level; and encoding the modified data indicating the first content item according to a transfer function associated with the common synthetic space.

[0014] In some embodiments, the non-volatile program storage device also includes instructions stored thereon to cause the one or more processors to perform the following operations: re-linearizing the encoded data indicating the first content item according to an inverse transfer function associated with the common synthetic space; mapping the re-linearized data indicating the first content item from the second color space gamut associated with the common synthetic space to a third color space gamut associated with the display device; applying a color adaptation operation to the mapped, re-linearized data indicating the first content item based on a measured white point of current viewing conditions surrounding the display device; performing simultaneous contrast adaptation on the first content item based on a third difference between the second expected viewing condition associated with the common synthetic space and the current viewing condition surrounding the display device; and displaying the first content item on the display device.

[0015] In some embodiments, the non-volatile program storage device also includes instructions stored thereon to cause the one or more processors to receive a second content item and similarly process the second content item into the common synthetic space, for example based on an inverse transfer function and a color space gamut of the second content item, and then further adjust the second content item based on current viewing conditions around the display device, wherein the second content item may include a content item having a different media type, color space, dynamic range, etc. than the first content item displayed simultaneously on the display device. In other words, the non-volatile program storage device may further include instructions stored thereon to cause the one or more processors to perform the following operations: receive data indicating a second content item; linearize the data indicating the second content item according to an inverse transfer function associated with the second content item; map the linearized data indicating the second content item from a fourth color space gamut associated with the second content item to the second color space gamut associated with the common synthetic space; modify the mapped, linearized data indicating the second content item based on at least one of the following: (c) a fourth difference between a third expected viewing condition associated with the second content item and the second expected viewing condition associated with the common synthetic space; and (d) a fifth difference between a second expected viewer adaptation level associated with the second content item and a second predicted viewer adaptation level; and encode the modified data indicating the second content item according to the transfer function associated with the common synthetic space.

[0016] The non-volatile program storage device may then further cause the one or more processors to: re-linearize the encoded data indicating the second content item according to the inverse transfer function associated with the common synthetic space; map the re-linearized data indicating the second content item from the second color space gamut associated with the common synthetic space to the third color space gamut associated with the display device; apply the color adaptation operation to the mapped, re-linearized data indicating the second content item based on the measured white point of the current viewing condition surrounding the display device; perform simultaneous contrast adaptation on the second content item based on the third difference between the second expected viewing condition associated with the common synthetic space and the current viewing condition surrounding the display device; and display the second content item on the display device.

[0017] In other embodiments, the foregoing techniques embodied as instructions stored on a non-volatile program storage device may also be practiced as a method and / or implemented on an electronic device with a display device, such as a mobile phone, a PDA, an HMD, a monitor, a television, or a laptop, a desktop computer, or a tablet computer. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1A Illustrate properties of ambient lighting, diffuse reflectance of a display device, and other environmental conditions that affect a display device.

[0019] Figure 1B The additional effects of unintended light on a display device are illustrated.

[0020] Figure 2 A system that utilizes a lookup table to perform gamma adjustment is illustrated.

[0021] Figure 3 A frame buffer gamma function and an exemplary intrinsic display response are illustrated.

[0022] Figure 4 Graphs representing LUT transformations and resulting gamma functions are illustrated, as well as graphs indicating perceptual transformations due to environmental conditions.

[0023] Figure 5 A unified display model system for performing display adjustments based on a dynamic system OOTF according to one or more embodiments is illustrated.

[0024] Figure 6 A simplified functional block diagram of an environmental condition model is illustrated in accordance with one or more embodiments.

[0025] Figure 7 A process for performing display adjustments based on a dynamic system OOTF according to one or more embodiments is illustrated in flowchart form.

[0026] Figure 8 A process for performing display adjustments based on a dynamic system OOTF according to one or more embodiments is illustrated in flowchart form.

[0027] Fig. 9 A simplified functional block diagram of a device with a display according to one embodiment is illustrated. DETAILED DESCRIPTION

[0028] The disclosed technology uses a display device in combination with various optical sensors (e.g., an ambient light sensor or an image sensor) to collect information about environmental conditions in the environment of a viewer of the display device. The use of ambient environment information; information about the display device and its characteristics; information about the content being displayed, its intended display type, and its recommended viewing environment can provide a more accurate prediction of the viewer's current viewing environment and its impact on how the user perceives the displayed content. A processor in communication with the display device can evaluate an environmental condition model and / or a perceptual adaptation model as part of a unified display model to predict the impact of current environmental viewing conditions (and / or the content itself) on viewer perception. The output of the unified display model can be a suggested modification for performing environmental adaptation on the parameters of the content to be displayed and the display device itself (e.g., a suggested adjustment of gamma, black point, white point, and / or saturation), so that the viewer perceives the adapted display content as expected, while remaining relatively independent of the current environmental conditions.

[0029] The techniques disclosed herein are applicable to any number of electronic devices: such as digital cameras, digital video cameras, mobile phones, personal data assistants (PDAs), head-mounted display (HMD) devices, monitors, televisions, digital projectors (including cinema projectors), and desktop, laptop, and tablet computer displays.

[0030] For the sake of clarity, not all features of an actual implementation are described in this specification. Of course, it should be understood that in the development of any such actual implementation (as in any development project), many decisions must be made to achieve the developer's specific goals (e.g., to meet system and business-related constraints), and these goals will vary between different implementations. It should be understood that such development work may be complex and time-consuming, but nevertheless, it is still their daily work for those of ordinary skill who benefit from this disclosure. In addition, the language used in this disclosure has been selected primarily for readability and instructional purposes, and therefore may not be selected to delineate or limit the subject matter of the present invention, resorting to the necessary claims to determine such invention subject matter. Reference to "one embodiment" or "an embodiment" in the specification means a specific feature, structure or characteristic described in conjunction with the embodiment included in at least one embodiment of the present invention, and multiple references to "one embodiment" or "an embodiment" should not be understood as all necessarily referring to the same embodiment.

[0031] Background showing exemplary device properties and environmental observation conditions

[0032] Reference now Figure 1A, illustrates the properties of ambient lighting, diffuse reflections of the display device 102, and other environmental conditions that affect the display device via a depiction of a side view of a viewer 116 of the display device 102 in a particular ambient lighting environment. Figure 1A As shown, viewer 116 is viewing display device 102, which in this case is a typical desktop computer monitor. Dashed line 110 represents the viewing angle of viewer 116. Figure 1A The surrounding environment depicted in FIG. 1 is illuminated by an ambient light source 100 that projects light rays 108 onto all objects in the environment, including walls 112 and display surface 114 of display device 102. As shown by a plurality of small arrows 109 (representing reflections of light rays 108), a certain percentage of the incident light radiation will be reflected from the surface it strikes. Diffuse reflection can be defined as the reflection of light from a surface such that the incident light rays are reflected at many angles and have a particular effect on the perception of a viewer of display device 102.

[0033] When the brightness of reflected light and / or leakage light from display device 102 is greater than the brightness of the pixels driven by the display device for the content item, the viewer may not be able to perceive low-tone details in the content item. This effect is caused by Figure 1A 106, which indicates a threshold brightness level. When the brightness of a pixel in the emissive display surface 114 is less than the threshold brightness level indicated by the dashed line 106, the pixel is not perceived as expected. When the brightness of a pixel in the emissive display surface 114 is greater than the threshold brightness level, the pixel is perceived as expected. The dashed line 106 and the threshold brightness level can be adjusted to account for each of reflected light and leakage light from the display device 102, either individually or in combination. Figure 1B The effects of reflected light and light leakage from a display device on a viewer's perception of displayed content are further described. Information about diffuse reflections and other ambient light in the current viewing environment can be used to inform an environmental condition model that suggests which adaptation processes to perform on the content to compensate for the environmental conditions and / or suggests modifications to adaptation processes that have already been performed.

[0034] Information about diffuse reflection and other ambient light can be based on light level readings recorded by one or more optical sensors (e.g., ambient light sensor 104). Dashed line 118 represents data indicating the light source collected by ambient light sensor 104. Optical sensor 104 can be used to collect information about ambient conditions in the environment of the display device, and can include, for example, ambient light sensors, image sensors, or cameras, or some combination thereof. The front image sensor provides information about how much light (and in some embodiments, what color of light) is hitting the display surface 114. This information can be used in conjunction with a model of the reflective and diffuse properties of the display to inform the ambient condition model about the specific lighting conditions in which the display is currently located and the user is currently adapting. Although optical sensor 104 is shown as a "front" image sensor, i.e., facing the approximate direction of the viewer 116 of the display device 102, other optical sensor types, placements, positioning, and quantities are also possible. For example, one or more "rear" image sensors alone (or in conjunction with one or more front sensors) can provide further information about the color of the light source and the viewer's environment. The rear sensor collects light that is re-reflected from the emission source or from objects behind the display, and it can be used to determine the brightness of the display's surroundings, i.e., what the user sees outside the display. This information can also be used in models of environmental conditions. For example, if the color of the wall 112 is close enough to the back of the display device 102, the color of the wall may have a profound effect on the viewer's perception. Similarly, in the example of an outdoor environment, the color and intensity of the light surrounding the viewer can make the display look different than an indoor environment with, for example, incandescent (colored) lighting.

[0035] In one embodiment, the optical sensor 104 may include a camera (or other device) capable of capturing spatial information, color information, and intensity information. With respect to spatial information, the camera or other device may also be used to determine the distance of the viewing user from the display, for example to further simulate how much of the user's field of view the display fills, and accordingly, how much influence the display / environment will have on the user's perception of the displayed content. In some embodiments, the camera may be configured to capture images of the surrounding environment at some predetermined time interval (e.g., every two minutes) for analysis, so that the environmental condition model may be gradually updated or otherwise changed as environmental conditions in the viewer's environment change.

[0036] Additionally, the rear-facing camera used to model the surrounding environment can be designed to have a field of view that is roughly consistent with the calculated or estimated field of view of the viewer of the display. Once the viewer's field of view is calculated or estimated, for example based on the size or position of the viewer's facial features recorded by the front-facing camera, the system can then determine which portion of the rear-facing camera image to use in the surround calculations, assuming that the intrinsic field of view of the rear-facing camera is known and larger than the viewer's field of view.

[0037] In other embodiments, one or more cameras or depth sensors may be used to further estimate the distance of a particular surface from the display device. This information may be used, for example, to further inform a model of environmental conditions based on the likely composition of the viewer's environment and its perceived impact. For example, a display with a diagonal position of 18" from a user at 30" will have a greater impact on the user's vision than the same display position of 48" away from the user, filling less of the user's field of view.

[0038] Reference now Figure 1B , showing in more detail the additional effects of unintended light on a display device. For example, light 155 emitted from display representation 150 represents the amount of light produced by the intentionally driven pixels of the display at a given moment. Similarly, light 165 emitted from display representation 160 represents the amount of light leaking from the display at a given moment, and light 109 reflected from display representation 170 represents the aforementioned diffuse reflection of ambient light from the display surface at a given moment. In displays that are stacked assemblies (as compared to laminated assemblies), the diffuse reflections of a non-glossy display may be more than those of a glossy display, or the diffuse reflections of a non-glossy display may be more than those of a clean display (as compared to a dusty or otherwise dirty display). Finally, display representation 180 represents the sum of the three forms of light illustrated in display representations 150, 160, and 170.

[0039] like Figure 1B , the light rays 185 emitted from the display representation 180 represent the actual amount of light perceived by a viewer of the display device, which may be different from the initial amount of light 155 that intentionally drives the pixels in the display in order to produce the desired content. Unintended light from display leakage, diffuse reflections, etc. may desaturate the perceived colors (compared to the intended colors of the content). The darker or more dim the intended colors, the more noticeable the desaturation appears to the viewer. Therefore, accounting for the effects of these various phenomena can help achieve a more consistent and content-accurate perceived experience across viewing environments.

[0040] Thus, in one or more embodiments disclosed herein, an environmental condition model may be used as part of a unified display model for dynamically selecting which environmental adaptations will be performed or adjusting the environmental adaptations that have been performed to compensate for unexpected light so that the darkest colors are not obscured by light leakage and / or predicted diffuse reflectance levels and all colors are not perceived as desaturated (compared to expected colors). A model of display device characteristics may be used to determine the amount of light leakage of the display device under current display parameters. The model of display device characteristics may also be used in combination with information from ambient light sensor 104 to estimate the amount of diffuse reflectance of the display device. A perceptual model may be used to estimate the amount of desaturation from unexpected light so that the environmental condition model may determine recommended resaturation and environmental adaptation to achieve the recommended resaturation.

[0041] Background on System Gamma and Perceived Gamma of an Exemplary Display Device

[0042] Reference now Figure 2 , shows an exemplary system 212 for performing gamma adjustment using a lookup table (LUT) 210. Element 200 represents source content created by, for example, a source content author that a viewer 116 wishes to view. Source content 200 may include images, videos, or other displayable content types. Element 202 represents a source profile, i.e., information describing the color profile and display characteristics of the device on which the source content author created source content 200. Source profile 202 may include, for example, an International Color Consortium (ICC) profile of the author's device or color space (described in further detail below), or other relevant information.

[0043] Information about the source content 200 and the source profile 202 may be transmitted to a device of the viewer 116 including a system 212 for performing gamma adjustment using the LUT 210. The device of the viewer 116 may include, for example, a mobile phone, a PDA, an HMD, a monitor, a television, or a laptop, a desktop computer, or a tablet computer, etc. Upon receiving the source content 200 and the source profile 202, the system 212 may perform a color adaptation process 206 on the received data, for example, for performing gamut mapping, i.e., color matching across various color spaces. For example, gamut matching attempts to maintain (as closely as possible) the relative relationships between colors (e.g., as created / approved by the content author on the display described by the source ICC profile), even if all colors must be systematically changed or adapted in order for them to be displayed on the target device.

[0044] Once the color profiles of the source and destination have been appropriately adapted, the image values ​​may enter a so-called "frame buffer" 208. In some embodiments, image values ​​(e.g., pixel brightness values) enter the frame buffer from one or more applications that have processed image values ​​to be encoded with a particular implicit gamma. A frame buffer may be defined as a video output device that drives a video display from a storage buffer containing a complete frame of image data (in this case). The implicit gamma of the values ​​entering the frame buffer can be visualized by looking at a "frame buffer gamma function," as will be described below with respect to Figure 3 Ideally, the frame buffer gamma function is the exact inverse of the display device's "intrinsic display response" function, which characterizes the display's brightness response to an input.

[0045] Because the inverse function of the intrinsic display response is not always exactly the inverse function of the frame buffer, sometimes a LUT stored on a video card or other memory can be used to account for imperfections in the relationship between the encoded gamma value and the decoded gamma value and the specific brightness response characteristics of the display. Therefore, if necessary, the system 212 can then utilize the LUT 210 to perform a so-called "gamma adjustment process". The LUT 210 may include two columns of positive real values ​​spanning a specific range (e.g., from 0 to 1). The first column value may correspond to the input image value, and the second column value in the corresponding row of the LUT 210 may correspond to the output image value, to which the input image value will be "transformed" before being finally displayed on the display 102. The LUT 210 can be used to account for imperfections in the brightness response curve (also called the "display transfer function") of the display 102. In other embodiments, the LUT may have a separate channel for each primary color in the color space, for example, the LUT may have a red channel, a green channel, and a blue channel in the sRGB color space.

[0046] The transformation applied by the LUT to the input frame buffer data before the data is output to the display device can be used to ensure that the desired 1.0 gamma boost is applied to the final display device. Figure 2 The system shown in is generally a good system, despite not taking into account the effects of differences or changes in ambient light conditions on perceived gamma, or gamma adjustments that have been encoded in the source content 200 by the source author to compensate for differences between the source content capture environment and the intended viewing environment of the source content 200. In other words, the 1.0 gamma boost used to encode and decode content is only implemented / appropriate in one ambient lighting environment, and that environment is typically brighter than a normal office environment. For example, content captured in a bright environment does not require a gamma boost if viewed in the same (i.e., bright) environment, for example due to the phenomenon of "simultaneous contrast." For another example, content captured and edited in a bright environment but intended for viewing in a dim environment (e.g., a dark environment such as a movie theater) may already include a gamma adjustment in the source content 200 received by the system 212. The additional gamma boost based on the LUT 210 may therefore distort the gamma adjustment already provided in the source content 200 and cause the displayed content to be different from what the source author intended.

[0047] As described above, in some embodiments, the goal of the gamma adjustment system 212 is to apply an overall 1.0 system gamma to the content being displayed on the display device 102. An overall 1.0 system gamma corresponds to a linear relationship between input encoded brightness values ​​and output brightness on the display device 102. Ideally, an overall 1.0 system gamma will cause the displayed content to appear largely as intended by the source author, regardless of the intermediate encoding and decoding of the content and other color management processes used to adapt the content to a particular display device 102. However, as will be described later, this overall 1.0 gamma may only be appropriately perceived in a particular set of ambient lighting conditions, so a dynamic display adjustment system is needed to adapt to different ambient lighting conditions and adjust the overall system gamma to achieve a perceived system gamma of 1.0. In addition, gamma adjustment is only a correction for ambient conditions, and the ambient adaptation described herein includes gamma adjustment as well as resaturation, black point and white point adjustment, etc.

[0048] Reference now Figure 3 , shows a frame buffer gamma function 300 and an exemplary intrinsic display response 302. Gamma adjustment, or as it is often referred to simply as "gamma", is the name given to a non-linear operation commonly used for encoding and decoding luminance values ​​in video or still image systems. Gamma, γ, can be defined by the following simple power law expression: L out =L in γ , where the input and output values ​​are L in and L out Gamma is a non-negative real value, typically within a predetermined range, such as 0 to 1. Gamma values ​​greater than 1 are sometimes called "encoding gamma", and the process of encoding with this compressing power-law nonlinearity is called "gamma compression"; conversely, gamma values ​​less than 1 are sometimes called "decoding gamma", and the application of an expansion power-law nonlinearity is called "gamma expansion". Gamma encoding of content helps map the content data into a more perceptually uniform domain.

[0049] Another way to think of the gamma characteristics of a system is as a power law relationship that approximates the relationship between the encoded brightness in the system and the actual desired image brightness on the end-user display device. In existing systems, a computer processor or other suitable programmable control device can perform gamma adjustment calculations for a particular display device with which it is communicating, based on the intrinsic brightness response of the device, the color gamut of the device, and the white point of the device (information of which can be stored in an ICC profile), as well as the ICC color profile and other content indicators that the author of the source content attached to the content to specify the "rendering intent" of the content.

[0050] An ICC profile is a set of data that characterizes a color input or output device or color space according to standards published by the International Color Consortium. An ICC profile can describe the color properties of a specific device or viewing requirement by defining a mapping between a device source or target color space and a Profile Connection Space (PCS), typically the CIE XYZ color space. An ICC profile can be used to define a color space generally in terms of three main parts: 1) the primary colors that define the color gamut; 2) the transfer function (sometimes called the gamma function); and 3) the white point. An ICC profile may also contain additional information to provide a mapping between the actual response of a display and its "advertised" response, i.e., its tone response curve (TRC), for example, to correct or calibrate a given display to an ideal 2.2 gamma response.

[0051] In some implementations, the ultimate goal of the gamma adjustment process is to apply a final overall 1.0 gamma boost (i.e., so-called "unity boost" or "no boost") to the content when it is displayed on a display device. An overall 1.0 system gamma corresponds to a linear relationship between input encoded luminance values ​​and output luminance on a display device, meaning that there is effectively no amount of gamma "boost" applied, and the gamma encoding process is canceled by the gamma decoding process without further adjustment.

[0052] Traditionally, gamma encoding is optimized for a specific environment, the dynamic range of the content, and the dynamic range of the display, so that the encoding and display codes are well spaced throughout the expected range, and the content appears as expected (e.g., not banded, not crushed highlights or blacks, and has the correct contrast sometimes referred to as tint, etc.). 8-bit 2.2 gamma is an example of an acceptable representation for encoding SDR (standard dynamic range) content that would be displayed on a 1 / 2.45 gamma Rec.709 CRT in a bright office viewing environment.

[0053] However, when viewed in an environment that is brighter or darker than the intended, brightly lit office viewing environment, the example SDR content will not have the intended appearance even when displayed on its intended Rec.709 display. When the current viewing environment is different from the suggested viewing environment, for example, if it is brighter than the suggested viewing environment, the user's vision adapts to the current brighter viewing environment, causing the user to perceive less distinguishable detail in the darker portions of the content. The display may be able to adjust only a small range of the user's vision to accommodate the current brighter viewing environment. Furthermore, the fixed maximum brightness of the display may be dim compared to the brightness of the current viewing environment.

[0054] When viewing content on a proposed Rec.709 display in a proposed, brightly lit office viewing environment, the current brighter viewing environment prevents the user from perceiving the darker portions of the content that the source author intended the viewer to perceive. In other words, "shadow detail" is "crushed" into black. This effect is amplified when ambient light from the viewing environment is reflected off the display and / or light leaking from the display (collectively referred to as unintended light) further limits how dark the viewer perceives the content. The lowest codes in the content are spaced apart in luminance based on the proposed viewing environment, and may be too closely spaced to be distinguishable in the current brighter viewing environment.

[0055] When the current viewing environment is also different from the suggested viewing environment, the perceived overall hue of the content is also different. For example, when the current viewing environment is brighter than the suggested viewing environment, the content may appear lower in contrast. Content may also appear desaturated, with an unexpected color cast, due to unexpected light reflections from the display and / or leakage from the display, or when the white point of the suggested viewing environment is different from the white point of the current viewing environment.

[0056] Even when viewed on a recommended Rec.709 display in a recommended, brightly lit office viewing environment, the hue of the content may be perceived differently based on what other content is being displayed at the same time, in an effect known as "simultaneous contrast". Some devices display multiple content items at once, for example, a user's work computer may display multiple documents and videos simultaneously. Different content items may be customized for different recommended viewing environments, such that each content item uses a different gamma encoding and / or a different gamma boost. A display device that implements the same gamma boost for all content items may end up distorting the individual content items away from their intended appearance.

[0057] For example, Rec.709 content has an overall 1.22 gamma boost from the intentional mismatch between the encoded gamma of the content and the decoded gamma of the display to compensate for the bright surround content viewed in a dark surround environment. In contrast, DCIP3 content encodes the compensation of the bright surround content viewed in a dark surround environment directly into the pixel itself, so that no gamma boost is required, that is, 1.0 gamma is sufficient. In any viewing environment, no single gamma boost is applicable to both Rec.709 content and DCIP3 content. Although this example describes the difference in gamma boost, similar differences can be found in other types of content adaptation, such as tone mapping, resaturation, black point and / or white point adjustment, modified transfer functions of displays, and combinations thereof. As used herein, "surrounding environment" refers to ambient lighting conditions in the environment surrounding the display device, etc. "Viewing environment" refers to the surrounding environment and display characteristics around the display device, such as display device light leakage, which can further affect how the user perceives the content displayed on the display device.

[0058] Return now Figure 3 , the x-axis of the frame buffer gamma function 300 represents input image values ​​spanning a particular range, such as from 0 to 1. The y-axis of the frame buffer gamma function 300 represents output image values ​​spanning a particular range, such as from 0 to 1. As described above, in some embodiments, image values ​​may enter the frame buffer 208 already processed and have a particular implicit gamma. Figure 3 As shown in the graph 300 in FIG. 3 , the encoding gamma is approximately 1 / 2.2 or 0.45. That is, the line in the graph 300 roughly looks like the function L OUT =L IN 0.45 . A gamma value of approximately 1 / 2.2 or 0.45 is typically used as the encoding gamma because the intrinsic display response of many display devices has a gamma of approximately 2.2, i.e., the inverse function of the encoding gamma of 1 / 2.2. In other cases, a gamma of, for example, 1 / 2.45 may be applied to 1.96 gamma encoded content when displayed on a conventional 1 / 2.45 gamma CRT display in order to provide a 1.25 gamma "boost" (i.e., 2.45 divided by 1.96) which is necessary to compensate for simultaneous contrast effects that cause bright content to appear low contrast when viewed in a dim ambient environment (i.e., areas outside the display are typically dimmer), such as a 16 lux Rec. 709 intended viewing environment. If the content already includes an additional gamma boost because the source author intended the bright content to be viewed in a dim ambient environment and the frame buffer 208 does not account for this encoded gamma boost, the resulting gamma boost will be different than the source author's rendering intent.

[0059] The x-axis of the intrinsic display response function 302 represents input image values ​​across a specific range, such as from 0 to 1. The y-axis of the intrinsic display response function 302 represents output image values ​​across a specific range, such as from 0 to 1. In theory, a system in which the decoded gamma is the inverse function of the encoded gamma should produce a desired overall 1.0 system gamma. However, this fails to account for ambient light in the environment surrounding the display device and / or gamma boosts that have been encoded into the source content. Therefore, the desired overall 1.0 system gamma is only achieved in one ambient lighting environment, for example, in the creative lighting environment, or where the gamma boost has been encoded into the source content in the intended viewing environment. These systems cannot dynamically adapt to the ambient conditions surrounding the display device, or cannot adapt based on user preferences.

[0060] Reference now Figure 4 , shows a graph representing the LUT transformation and the resulting gamma function, as well as a graph indicating the perceptual transformation due to environmental conditions. Figure 4The graph in shows how a LUT can be used in an ideal system to account for imperfections in the relationship between encoded gamma and decoded gamma values ​​and the specific brightness response characteristics of the display at different input levels. Figure 4 The graph in also illustrates how the environmental conditions surrounding the display device subsequently distort the perception of the content, causing the perceived gamma to be different from the resulting gamma function. The x-axis of the intrinsic display response graph 400 represents input image values ​​across a particular range, e.g., from 0 to 1. The y-axis of the intrinsic display response graph 400 represents output image values ​​across a particular range, e.g., from 0 to 1. The non-linear nature of the graph 400 represents secondary characteristics and defects in the intrinsic response function of the exemplary display. The x-axis of the LUT graph 410 represents input image values ​​across the same range of input values ​​that the display is able to respond to, e.g., from 0 to 1. The y-axis of the LUT graph 410 represents the same range of output image values ​​that the display is able to produce, e.g., from 0 to 1. In a desired calibrated display device, the display response 400 will be the inverse function of the LUT response 410, so that when the LUT graph is applied to the input image data, the resulting gamma function 420 reflects the desired overall system 1.0 gamma response, i.e., produced by the adjustments provided by the LUT and the intrinsic (almost) linear response of the display, and the content is perceived as what the source author intended. The x-axis of the resulting gamma function 420 represents the input image values ​​across a particular range, e.g., from 0 to 1, as authored by the source content author. The y-axis of the resulting gamma function 420 represents the output image values ​​displayed on the resulting display across a particular range, e.g., from 0 to 1. The slope of 1.0 reflected in the line in the graph 420 indicates that the brightness levels intended by the source content author will be reproduced at corresponding brightness levels on the final display device.

[0061] Ideally, the resulting gamma function 420 reflects the desired overall 1.0 system gamma on the resulting display device, indicating that the tone response curve (i.e., gamma) matches between the source and the display, the gamma encoding of the content has been cancelled by the gamma decoding process without further adjustment, and the image on the display may be displayed more or less as the author of the source intended. However, this calculated overall 1.0 system gamma does not take into account the impact of ambient lighting conditions on the viewer's perception of gamma boost. In other words, due to the perceptual shift caused by the ambient conditions in the viewer's environment 425, the viewer will not perceive the content as intended by the source author, and will not perceive an overall 1.0 gamma under all lighting conditions. The calculated overall 1.0 gamma may further fail to take into account the impact of the viewer's current adaptation to the ambient light conditions. As described above, the user's ability to perceive changes in light intensity (and the total range of light intensities that their eyes may be able to perceive) is further based on what level of light the user's eyes have been around (and therefore adjusted to) in the previous time window (e.g., 30 seconds, 5 minutes, 15 minutes, etc.). The calculated overall 1.0 gamma may also fail to take into account the gamma boost that has been encoded into the source content by the source author based on the source capture and editing environment and the intended viewing environment. For example, a video may have been shot in a bright environment, but has been edited for viewing in a dim environment, where a gamma boost matching that transition has been encoded into the video. If the system attempts to further adjust the already adjusted gamma boost, the resulting gamma is different than the source author's rendering intent.

[0062] As shown in graph 430, the dashed line indicates a perceived 1.0 gamma boost, i.e., the viewer's actual perception of the achieved system gamma, which corresponds to an overall gamma boost of greater than 1.0. Ambient conditions in the viewing environment transform the achieved system gamma greater than 1.0 into a perceived system gamma equal to 1.0. Thus, a unified display model for dynamically adjusting the characteristics of a display according to one or more embodiments disclosed herein may be able to take into account perceptual transformations due to the viewer's current ambient conditions, such that the display boosts the achieved system gamma to above the expected 1.0 system gamma, and thus presents to the viewer content that he or she will perceive as an overall 1.0 system gamma, such that the content is perceived as what the source author intended. As described in more detail below, such a unified display model may also have non-uniform time constants for how stimuli affect the viewer's transient adaptation over time. In other words, the model may attempt to predict changes in user perception due to changes in the viewer's ambient conditions.

[0063] Unified display model using dynamic system OOTF

[0064] Reference now Figure 5, illustrates a unified display model system 500 for performing display adjustments based on a dynamic system OOTF according to one or more embodiments. A given display, such as display 102, may be said to have the ability to "modulate" (i.e., adapt or adjust to) only a certain percentage of the possible surrounding environment at any given moment. For example, if the environment is much brighter than the display, such that the display reflects a large amount of light at its minimum display output level, the display may have a relatively high "pedestal" value and, therefore, be able to modulate only a portion of the ambient lighting conditions even at its maximum display output level.

[0065] The unified display model system 500 may thus be used to apply a transformation for warping source content 200 (e.g., high-precision source content) to an adapted visual perception of a viewer of the display 102 in a given viewing environment. As described above, warping the original source content signal to a viewer's perception of the display and display environment may be based, for example, on the predicted viewing environment conditions received by the environment condition model, such as by referring to the Figure 6 For example, the ratio of the diffuse white brightness of the display 102, measured in nits, to the brightness of the user's view outside the display 102 (referred to as the environment), also measured in nits, may be used to apply a gamma boost, color saturation correction, or similar algorithm to compensate for the perceived effects of viewing content in an environment that is different in brightness than the environment associated with the source content 200 during capture, editing, or approval.

[0066] According to some embodiments, the unified display model system 500 may take into account one or more dynamic display characteristics 502, such as: information obtained from a forward ambient light sensor (ALS) 504; information obtained from a rearward ALS 510; histogram information for currently displayed content 506; and / or a current overall brightness level 508 of the display device.

[0067] According to some embodiments, the unified display model system 500 may also consider one or more static display characteristics 512 when determining how to modify displayed content, such as: information 514 about the percentage of light leakage experienced by the display; information 516 about the percentage of light reflection from the display surface; information 518 about the primary colors of the display device; information 520 about the intrinsic white point of the display device; and / or information 522 about the intrinsic response of the display device.

[0068] According to some embodiments, the unified display model system 500 may combine information from both the dynamic display characteristics 502 and the static display characteristics 512 in the perceptual model 530. According to some such embodiments, the perceptual model 530 may include a perceptual visual adaptation model 532 configured to model a possible adaptation level of a viewer given the current dynamic display characteristics 502 and the static display characteristics 512. In some embodiments, the perceptual visual adaptation model 532 may be based at least in part on a color appearance model (CAM), such as the CIECAM02 color appearance model, and may be used to further inform the environmental condition model 600 about the appropriate amount of gamma boost to be applied with the modified transfer function of the display. The CAM may be based, for example, on the brightness and white point of the viewer's environment and the field of view of the display subtended by the viewer's field of view.

[0069] In some embodiments, knowledge about the size of the display and the distance between the display and the viewer may also be used as a useful input to the unified display model 500. Information about the distance between the display and the user may be retrieved from a front-facing image sensor (such as a front-facing camera 104). For example, the brightness and white point of the viewer's environment may be used to determine the ratio of diffuse white brightness to the viewing environment brightness. Based on the determined ratio, a specific gamma boost may be applied. For example, for a pitch-black ambient environment, an additional gamma boost of about 1.5 applied by the LUT may be appropriate, while a 1.0 gamma boost (i.e., uniform boost or no boost) may be suitable for bright or sunlit environments. For intermediate environments, an appropriate gamma boost value to be applied by the LUT may be interpolated between values ​​1.0 and about 1.5. A more detailed model of environmental conditions is provided by the CIECAM02 specification.

[0070] According to some embodiments, the perceptual visual adaptation model 532 may also be used to predict the viewer's current minimum perceptible light level using model 534, and to perceptually map the display and environment to the viewer's current perception using model 536. Using this information, and optionally after adapting the luminance and / or chrominance display data to an XYZ color space (or another device-invariant color space), the perceptual distance model 540 may employ a perceptual color model 542 (e.g., based on a CIELAB color space) to determine, at block 544, a perceptual threshold below which the viewer may not currently be able to perceive changes in hue and / or step size (i.e., changes) required to modify the display's response based on the viewer's predicted perceptual adaptation level under current viewing conditions.

[0071] The output of the perceptual model 530 may then be sent to a color math model 550, which is used to calculate and configure modifications to the response of the display to achieve the desired perceptual benchmark. According to some embodiments, the color math model 550 may include: a module 552 for matching displayed content values ​​to the current color sense of the viewer; a module 554 for performing white point adaptation; a module 556 for performing color matching to the color gamut of the display device; a module 558 for performing white point adaptation; and / or a module 560 for calculating the gamma matching response of the display device. The outputs of the modules 552 / 554 / 556 / 558 / 560 may be combined into one or more matrices 562, such as intermediate image matrices, color adaptation matrices, etc., and / or one or more combined lookup tables (LUTs) 564 to efficiently store values ​​embodying the changes determined by the color math model 550 to be applied to the display device. The aforementioned matrices 562 and / or LUTs 564 may then be normalized and passed to the display pipeline 580.

[0072] The display pipeline 580 may perform one or more of the following functions: synthesizing multiple content items for simultaneous display 582; linearizing content item color data 584; applying color changes 586 determined by the color mathematical model 550 (e.g., via application of one or more 3x3 matrices); performing any necessary brightness compensation 588 determined by the unified display model; and gamma encoding 590 the modified content for ultimate display to the viewer 116.

[0073] In some embodiments, modifications to the combined LUT 564 may be implemented gradually (e.g., over determined time intervals) via an animation engine or similar control element in the display pipeline 580. According to some such embodiments, the display pipeline 580 may be configured to adjust the combined LUT 564 based on a prediction of the rate at which the viewer's vision will adapt to the changes.

[0074] In some embodiments, the black level for a given ambient environment is determined, for example, by using the ambient light sensor 104 or by measuring the actual panel and / or diffuser of the display device. Figure 1A As described above, diffuse reflections of ambient light from device surfaces may increase the expected display value and affect the user's ability to perceive the darkest display level (a phenomenon also known as "black crush"). In other environments, light levels below a certain brightness threshold will simply be invisible to the viewer. Once this level is determined, the black point can be adjusted accordingly.

[0075] In another embodiment, the white point, i.e., the color that a user perceives as white for a given surrounding environment, may be similarly determined, for example, by analyzing the lighting and color conditions of the surrounding environment using one or more optical sensors 104. The white point of the display device may then be chromatically adapted to the white point determined from the viewer's environment. Additionally, note that the modification to the white point may be asymmetric between the red, green, and blue channels of the LUT, thereby shifting the relative RGB mix, and therefore the white point.

[0076] As described above, in some embodiments, if necessary, the unified display model 500 may first use the specified adaptation algorithm included in the source configuration file 202 to adapt the source content 200 to its reference environment. For example, the RGB-based gamma for Rec.709 video may be applied, as traditionally applied via the mismatch between the content encoding gamma and the decoding response of the display 102. Once the source content 200 is adapted to its reference environment using its specified algorithm, the unified display model 500 can use best practices to adapt the source content 200 to a shared system-level viewing environment or a public synthesis space. The public synthesis space can be dynamically changed to match the user's current viewing environment, or it can remain constant. In a specific implementation where the public synthesis space remains constant, the unified display model 500 can globally adapt all content items in the public synthesis space to adapt the fixed public synthesis space to the current viewing environment. Any appropriate technology can be used to adapt the source content 200 from its reference environment to the public synthesis space, and from the public synthesis space to the current viewing environment. This function may be particularly useful in the case where multiple content items from multiple source authors are to be displayed simultaneously. The unique content adaptation that has been encoded in each content item can be adjusted without affecting the content adaptation applied to other content items. Then, the common synthesis space for all content items can be adjusted based on the specific viewing environment for the display 102. In the implementation scheme just described above, the combined LUT 564 can serve as a useful and efficient location for the unified display model system 500 to apply the display transfer function adaptation of these environmental perceptions. In some embodiments, the unified display model system 500 can generate an ICC profile based on the viewing environment, and the ICC profile represents the intrinsic response of the display as the true intrinsic response of the display divided by the desired system gamma. The ICC profile may include a fixed "preset", wherein each preset represents a specific viewing environment and the corresponding environmental adaptation required for the content that will be correctly perceived in the specific viewing environment. The unified display model 500 can then determine the appropriate preset based on the analysis of the environmental conditions obtained, and the corresponding environmental adaptation is directly applied to the source content 200 or to the common synthesis space.

[0077] Reference now Figure 6, shows a simplified functional block diagram of an example environmental condition model 600. As described above, the environmental condition model 600 may take into account various factors, such as: predictions from a color appearance / perception model 610; information about the surrounding environment, such as from an ambient light sensor / image sensor 620; information about the current brightness level and / or brightness history of the display 630 (e.g., knowing how bright the display has been and for how long may affect the user's adaptation level); information and characteristics from a profile of the display device 640; and / or information based on historically displayed content / based on predictions of upcoming content 650.

[0078] The color appearance model 610 may include, for example, the CIECAM02 color appearance model or the CIECAM97s model. The color appearance model may be used to perform color adaptation transformations and / or to calculate mathematical correlations of the six technically defined dimensions of color appearance: brightness (luminance), lightness, colorfulness, chroma, saturation, and hue.

[0079] Display characteristics 640 may include information from display profile 204 about the display device's color space, intrinsic display response characteristics or anomalies, reflectivity, leakage, or even the type of screen surface used by the display. For example, an "anti-glare" display with a diffuser will "lose" more black level than a glossy display at a given (non-zero) ambient light level.

[0080] The historical model 650 may consider both the instantaneous brightness level of the content and the cumulative brightness of the content over a period of time. In other embodiments, the model 650 may also perform an analysis of the upcoming content, such as to allow the environmental condition model to start adjusting the transfer function of the display over time so that the display is in a desired state before the time when the upcoming content is displayed to the viewer (or within a threshold time). When the environmental condition model 600 determines how to quickly adjust the display to illustrate the upcoming content, the biological / chemical speed of human visual adaptation may also be considered. In some cases, the content itself may have been adaptively encoded, for example, by the source content creator. For example, one or more frames of the content may include a customized transfer function associated with the corresponding one or more frames. In some embodiments, the customized transfer function of a given frame may be based only on the content of the given frame, for example, the brightness level of the given frame. In other embodiments, the customized transfer function of a given frame may be based at least in part on at least one of the following: the brightness level of the one or more frames displayed before the one or more frames of the content; and / or the brightness level of the one or more frames displayed after the one or more frames of the content. In the case where the content itself has been adaptively encoded, the environmental condition model 600 may first implement adjustments for the adaptive encoding to move the content into the common composition space according to the content indicators included in the source profile 202. The environmental condition model 600 may then attempt to further modify the transfer function of the display during display of a particular frame of the encoded content (e.g., based on various other environmental factors that may have been obtained at the display device, such as 610 / 620 / 630 / 640).

[0081] According to some embodiments, the modifications determined by the environmental condition model 600 may be implemented by changing existing table values ​​(e.g., stored in one or more calibration LUTs, i.e., tables of tone response curves configured to give a "perfect" response to the display). Such changes may be performed by looking up values ​​in the original table for the transformed values, or by modifying the original table "in place" via a warping technique. For example, the aforementioned black level (and / or white level) adaptation process may be implemented via a warping compression of the values ​​in the table from black upward (and / or from white downward). In other embodiments, "re-gamma" and / or "re-saturation" of the LUT may be applied in response to adjustments determined by the environmental condition model 600.

[0082] It will be appreciated that the exact manner in which the environmental condition model 600 processes the information 610 / 620 / 630 / 640 / 650 received from the various source optical sensors 104, the display brightness 508, the display profile 204, and the indicators in the content source profile 202, and how it modifies the resulting display response curve (e.g., by modifying LUT values) (including how quickly such modifications occur) depends on the specific implementation and desired effect of a given system.

[0083] According to some embodiments, the environmental condition model 600 may be used to take into account the above reference Figure 6 The various factors described above may have an impact on the viewer's perception at a given moment. Then, based on the output of the environmental condition model 600, an updated display transfer function for driving the display 102 may be determined. The display transfer function may be used to convert between input signal data values ​​and voltage values ​​that may be used to drive the display to produce pixel brightness corresponding to the perception interval to which the transfer function has mapped the input signal data values ​​at a given moment. One goal of the environmental condition model 600 is to: determine the current environment of the viewer; determine which region of the adaptation range the content and / or display is modulating; and then map to a transfer function corresponding to that portion of the adaptation range in order to optimally use the display codes (and the bits required to enumerate them).

[0084] Reference now Figure 7, an embodiment of a process 700 for performing display adjustment based on a dynamic system OOTF is shown in the form of a flowchart. The overall goal of some unified display models may be to understand how the source material will be perceived by the viewer on the viewer's display and in the viewer's environment at a given moment. First, the display adjustment process may start by receiving data indicating a first content item (step 705). For example, the first content item may include encoded display data bound to a specific source color space color gamut. In some embodiments, an indicator in the content may specify a specific adaptation algorithm to be used to adapt the content item from the source color space to the display color space and the expected viewing environment. For example, the RGB-based gamma for Rec.709 video, as traditionally applied, usually needs to take into account the mismatch between the encoding gamma of the content and the decoding response of the display. For another example, the video that the viewer wishes to display may have been captured in a bright environment and is intended to be viewed in a dark environment, so the video may include appropriate gamma enhancement to adapt to the dark environment of the expected viewing environment. Next, the process 700 may perform a linearization process to attempt to remove the gamma encoding (step 710). For example, if the data has been encoded with a gamma of (1 / 2.2), the linearization process may attempt to linearize the data by performing a gamma stretch with a gamma of 2.2. After linearization, the process will have a version of the first content item data that approximately represents the data in the source color space. Linearization may be required to perform some operations, such as color management and scaling. In some cases, for example, if an extended dynamic range pixel buffer format (e.g., EDR) is used, the pixel brightness value may also be divided by the desired reference white brightness value before further processing. For example, when SDR and HDR content are to be displayed simultaneously on the same display, it may be necessary to use the EDR format.

[0085] At this point, process 700 may map the linearized data indicating the first content item from a first color space gamut associated with the first content item to a second color space gamut associated with a common synthesis space (step 715). In one embodiment, the gamut mapping may use one or more color adaptation matrices. In other embodiments, a 3DLUT may be applied. In some embodiments, if necessary, one or more pre-synthesized, content-specific tone mapping operations may be applied. For example, some content may have metadata, gain mapping, and / or other available attributes directly associated with the content item itself.

[0086] Once the first content item is mapped to the color space gamut of the common composition space, the mapped, linearized data indicative of the first content item may be further modified based on at least one of: (a) a first difference between a first expected viewing condition associated with the first content item and a second expected viewing condition associated with the common composition space (the first difference may be required to support content items that are not authored for bright surround viewing); and (b) a second difference between a first expected viewer adaptation level associated with the first content item and a first predicted viewer adaptation level (step 720). The common composition space may include any common composition space that encompasses both the color space gamut associated with the common composition space and any expected viewing condition of the common composition space (e.g., dim viewing environment, bright viewing environment, etc.). Any suitable adaptation algorithm may be used to modify each displayed content item to the common composition space. This ensures that, for example, multiple displayed content items having multiple encoded gamma boosts, saturation levels, etc., can be adapted to a single system-wide common composition space. For example, a video that a viewer wishes to display may include a gamma boost corresponding to an expected viewing environment for a dark environment, but a word processing document that a viewer wishes to view on the same display may include a gamma boost corresponding to a bright environment. If adjustments to content items based on the current ambient viewing conditions surrounding the display device are applied without first converting the individual content items into a common synthesis space, the resulting gamma boost for the video will be different from the resulting gamma boost for the document, so that the content items may not be properly adjusted for the current ambient viewing conditions. In some embodiments, a common synthesis space and / or system-wide display parameters may be selected based on the reference environment of one or more content items being displayed. For example, if most of the content items correspond to a bright surround reference environment, a bright surround reference environment may be selected as the common synthesis space. In some embodiments, a common synthesis space may be selected based on the reference environment of the content items determined to be the most important. In some embodiments, a common synthesis space may be selected based on the current viewing environment, thereby reducing the amount of adjustment required to adapt the content items to the current viewing environment. This feature can be used for stable viewing environments with infrequent or small changes. For example, the common synthesis space and the corresponding modified display parameters may be the "average" of the recent ambient conditions. Then, in some embodiments, system-wide display parameters may be adjusted based on environmental conditions, such as based on a model of environmental conditions; device characteristics, such as based on display brightness, reflections, and leakage; and / or according to explicit user settings. For example, for high dynamic range (HDR) content, adjustment of the reference white point may reduce the range of brightness levels dedicated to highlight displays (so-called "headroom") in high dynamic range content. The modified display data resulting from step 720 may then be encoded according to a transfer function associated with the common composition space (step 725).

[0087] Reference now Figure 8 , another embodiment of a process 800 for performing display adjustment based on a dynamic system OOTF is shown in flowchart form. Specifically, Figure 8 An exemplary process of acquiring content from a common synthetic space and adapting the content based on the current viewing conditions around the display device will be described in detail. Figure 7 Continuing with step 725, at step 805, process 800 may begin by re-linearizing the encoded data indicating the first content item according to an inverse transfer function associated with the common synthetic space, i.e., providing linear processing in the common synthetic space. Next, process 800 may map the re-linearized data indicating the first content item from a second color space gamut associated with the common synthetic space to a third color space gamut associated with the display device (step 810). Next, process 800 may apply a color adaptation operation to the mapped, re-linearized data indicating the first content item based on a measured white point of the current viewing conditions surrounding the display device (step 815). For example, a color adaptation operation may be employed to move the white point from a nominal white point value (e.g., D65) to a white point that matches the actual current viewing conditions. Finally, if necessary, process 800 may perform a simultaneous contrast adaptation operation on the first content item based on a third difference between the second expected viewing condition associated with the common synthetic space and the current viewing condition surrounding the display device (step 820).

[0088] In some embodiments, rather than performing a simultaneous contrast adaptation operation based on current viewing conditions surrounding the display device at step 820, a "reference preset" selected, for example, by the viewer or the system may be used. In some cases, for example, if an extended dynamic range pixel buffer format (e.g., EDR) is used, a brightness control mapping may be applied to map a reference white value to a desired reference white brightness value prior to display. Finally, one or more additional environmental adaptation corrections (such as adapting the black point of the display) may be applied to the display data, for example, if necessary, based on a predicted adaptation level of the viewer while the content is being displayed.

[0089] Exemplary Electronic Devices

[0090] Reference now Fig. 9, a simplified functional block diagram of a representative electronic device with a display according to some specific implementations is shown. The electronic device 900 may be, for example, a mobile phone, a personal media device, an HMD, a portable camera, or a tablet, a laptop, or a desktop computer system. As shown, the electronic device 900 may include a processor 905, a display 910, a user interface 915, graphics hardware 920, a device sensor 925 (e.g., a proximity sensor / ambient light sensor, an accelerometer, and / or a gyroscope), a microphone 930, an audio codec 935, a speaker 940, a communication circuit 945, an image sensor / camera circuit 950 (e.g., it may include multiple camera units / optical sensors with different characteristics (and a camera unit housed outside the device 900 but in electronic communication with the device)), a video codec 955, a memory 960, a storage device 965, and a communication bus 970.

[0091] The processor 905 may execute instructions necessary for implementing or controlling the operation of the various functions performed by the device 900 (e.g., such as the generation and / or processing of signals according to the various embodiments described herein). The processor 905 may, for example, drive the display 910 and receive user input from the user interface 915. The user interface 915 may take a variety of forms, such as buttons, keypads, dials, click wheels, keyboards, display screens, and / or touch screens. The user interface 915 may, for example, be a conduit through which a user can view a captured image or video stream and / or indicate a particular frame that the user wants to play / pause, etc., or apply a particular adjustment to it (e.g., by clicking a physical or virtual button at the moment when the desired frame is being displayed on the display screen of the device).

[0092] In one embodiment, the display 910 may display the video stream as it is captured while the processor 905 and / or graphics hardware 920 evaluates the environmental condition model to determine modifications to the transfer function or gamma boost of the display, optionally storing the video stream in the memory 960 and / or storage device 965. The processor 905 may be a system on a chip (such as those found in mobile devices) and may include one or more dedicated graphics processing units (GPUs). The processor 905 may be based on a reduced instruction set computer (RISC) or complex instruction set computer (CISC) architecture or any other suitable architecture and may include one or more processing cores. The graphics hardware 920 may be dedicated computing hardware for processing graphics and / or assisting the processor 905 in performing computing tasks. In one embodiment, the graphics hardware 920 may include one or more programmable graphics processing units (GPUs).

[0093] The image sensor / camera circuit 950 may include one or more camera units configured to capture images, such as images indicating ambient lighting conditions in a viewing environment and, for example, images that may have an impact on the output of an environmental condition model according to the present disclosure. The output from the image sensor / camera circuit 950 may be processed at least in part by: a video codec 955 and / or a processor 905 and / or graphics hardware 920 and / or a dedicated image processing unit incorporated into the circuit 950. Such captured images may be stored in a memory 960 and / or a storage device 965. The memory 960 may include one or more different types of media used by the processor 905, the graphics hardware 920, and the image sensor / camera circuit 950 to perform device functions. For example, the memory 960 may include a memory cache, a read-only memory (ROM), and / or a random access memory (RAM). The storage device 965 may store media (e.g., audio files, image files, and video files), computer program instructions or software, preference information, device profile information, and any other suitable data. The storage device 965 may include one or more non-transitory storage media, including, for example, magnetic disks (fixed disks, floppy disks, and removable disks) and tapes, optical media (such as CD-ROMs and digital video disks (DVDs)), and semiconductor memory devices (such as electrically programmable read-only memories (EPROMs), and electrically erasable programmable read-only memories (EEPROMs)). The memory 960 and the storage device 965 may be used to maintain computer program instructions or codes organized into one or more modules and written in any desired computer programming language. When executed by, for example, the processor 905, such computer program code may implement one or more of the methods described herein. The power supply 975 may include a rechargeable battery (e.g., a lithium-ion battery, etc.) or other electrical connection to a power source (e.g., to a power source) for managing the electronic components and associated circuits of the electronic device 900 and / or providing power to the electronic components and associated circuits of the electronic device.

[0094] The foregoing description of preferred embodiments and other embodiments is not intended to limit or restrict the scope or applicability of the inventive concepts conceived by the applicant. In exchange for disclosing the inventive concepts contained herein, the applicant expects all patent rights provided by the appended claims. Therefore, the appended claims are intended to include all modifications and changes within the scope of the following claims or their equivalents.

Claims

1. A method for displaying content on a display device, the method comprising: receiving data indicative of a first content item; linearizing the data indicative of the first content item according to an inverse transfer function associated with the first content item; mapping the linearized data indicative of the first content item from a first color space gamut associated with the first content item to a second color space gamut associated with a common composition space; modifying the mapped, linearized data indicative of the first content item based on at least one of: (a) a first difference between a first expected viewing condition associated with the first content item and a second expected viewing condition associated with the common composite space; and (b) a second difference between a first expected viewer adaptation level and a first predicted viewer adaptation level associated with the first content item; and The modified data indicative of the first content item is encoded in accordance with a transfer function associated with the common synthesis space.

2. The method according to claim 1, further comprising: re-linearizing the encoded data indicative of the first content item according to an inverse transfer function associated with the common synthesis space; mapping the re-linearized data indicative of the first content item from the second color space gamut associated with the common composition space to a third color space gamut associated with the display device; applying a color adaptation operation to the mapped, re-linearized data indicative of the first content item based on a measured white point of current viewing conditions surrounding the display device; performing simultaneous contrast adaptation on the first content item based on a third difference between the second expected viewing condition associated with the common composite space and the current viewing condition surrounding the display device; and The first content item is displayed on the display device.

3. The method according to claim 2, further comprising: receiving data indicative of a second content item; linearizing the data indicative of the second content item according to an inverse transfer function associated with the second content item; mapping the linearized data indicative of the second content item from a fourth color space gamut associated with the second content item to the second color space gamut associated with a common composition space; modifying the mapped, linearized data indicative of the second content item based on at least one of: (c) a fourth difference between a third expected viewing condition associated with the second content item and the second expected viewing condition associated with the common composite space; and (d) a fifth difference between a second expected viewer adaptation level and a second predicted viewer adaptation level associated with the second content item; and The modified data indicative of the second content item is encoded in accordance with the transfer function associated with the common synthesis space.

4. The method according to claim 3, further comprising: re-linearizing the encoded data indicative of the second content item according to the inverse transfer function associated with the common synthesis space; mapping the re-linearized data indicative of the second content item from the second color space gamut associated with the common composition space to the third color space gamut associated with the display device; applying the color adaptation operation to the mapped, re-linearized data indicative of the second content item based on the measured white point of the current viewing conditions surrounding the display device; performing simultaneous contrast adaptation on the second content item based on the third difference between the second expected viewing condition associated with the common composite space and the current viewing condition surrounding the display device; and The second content item is displayed on the display device.

5. The method according to claim 2, further comprising: Data indicative of ambient light conditions surrounding the display device is received, wherein the current viewing conditions surrounding the display device are based at least in part on the received data indicative of ambient light conditions.

6. The method of claim 2, wherein displaying the first content item on the display device further comprises: An adjustment to a black point, a white point, a system gamma, or a combination thereof of the display device is determined based at least in part on the first predicted viewer adaptation level. The method of claim 4 , wherein the first content item and the second content item are of different media types.

8. A non-transitory program storage device comprising instructions stored thereon to cause one or more processors to: receiving data indicative of a first content item; linearizing the data indicative of the first content item according to an inverse transfer function associated with the first content item; mapping the linearized data indicative of the first content item from a first color space gamut associated with the first content item to a second color space gamut associated with a common composition space; modifying the mapped, linearized data indicative of the first content item based on at least one of: (a) a first difference between a first expected viewing condition associated with the first content item and a second expected viewing condition associated with the common composite space; and (b) a second difference between a first expected viewer adaptation level and a first predicted viewer adaptation level associated with the first content item; and The modified data indicative of the first content item is encoded in accordance with a transfer function associated with the common synthesis space.

9. The non-transitory program storage device of claim 8, further comprising instructions stored thereon to cause the one or more processors to: re-linearizing the encoded data indicative of the first content item according to an inverse transfer function associated with the common synthesis space; mapping the re-linearized data indicative of the first content item from the second color space gamut associated with the common composition space to a third color space gamut associated with the display device; applying a color adaptation operation to the mapped, re-linearized data indicative of the first content item based on a measured white point of current viewing conditions surrounding the display device; performing simultaneous contrast adaptation on the first content item based on a third difference between the second expected viewing condition associated with the common composite space and the current viewing condition surrounding the display device; and The first content item is displayed on the display device.

10. The non-transitory program storage device of claim 9, further comprising instructions stored thereon to cause the one or more processors to: receiving data indicative of a second content item; linearizing the data indicative of the second content item according to an inverse transfer function associated with the second content item; mapping the linearized data indicative of the second content item from a fourth color space gamut associated with the second content item to the second color space gamut associated with a common composition space; modifying the mapped, linearized data indicative of the second content item based on at least one of: (c) a fourth difference between a third expected viewing condition associated with the second content item and the second expected viewing condition associated with the common composite space; and (d) a fifth difference between a second expected viewer adaptation level and a second predicted viewer adaptation level associated with the second content item; and The modified data indicative of the second content item is encoded in accordance with the transfer function associated with the common synthesis space.

11. The non-transitory program storage device of claim 10, further comprising instructions stored thereon to cause the one or more processors to: re-linearizing the encoded data indicative of the second content item according to the inverse transfer function associated with the common synthesis space; mapping the re-linearized data indicative of the second content item from the second color space gamut associated with the common composition space to the third color space gamut associated with the display device; applying the color adaptation operation to the mapped, re-linearized data indicative of the second content item based on the measured white point of the current viewing conditions surrounding the display device; performing simultaneous contrast adaptation on the second content item based on the third difference between the second expected viewing condition associated with the common composite space and the current viewing condition surrounding the display device; and The second content item is displayed on the display device.

12. The non-transitory program storage device of claim 9, further comprising instructions stored thereon to cause the one or more processors to: Data indicative of ambient light conditions surrounding the display device is received, wherein the current viewing conditions surrounding the display device are based at least in part on the received data indicative of ambient light conditions.

13. The non-transitory program storage device of claim 9, wherein the instructions for displaying the first content item on the display device further comprise instructions for causing the one or more processors to: An adjustment to a black point, a white point, a system gamma, or a combination thereof of the display device is determined based at least in part on the first predicted viewer adaptation level.

14. A device, comprising: Memory; Display devices; and one or more processors operatively coupled to the memory, wherein the one or more processors are configured to execute instructions that cause the one or more processors to: receiving data indicative of a first content item; linearizing the data indicative of the first content item according to an inverse transfer function associated with the first content item; mapping the linearized data indicative of the first content item from a first color space gamut associated with the first content item to a second color space gamut associated with a common composition space; modifying the mapped, linearized data indicative of the first content item based on at least one of: (a) a first difference between a first expected viewing condition associated with the first content item and a second expected viewing condition associated with the common composite space; and (b) a second difference between a first expected viewer adaptation level and a first predicted viewer adaptation level associated with the first content item; and The modified data indicative of the first content item is encoded in accordance with a transfer function associated with the common synthesis space.

15. The apparatus of claim 14, further comprising instructions stored in the memory to cause the one or more processors to: re-linearizing the encoded data indicative of the first content item according to an inverse transfer function associated with the common synthesis space; mapping the re-linearized data indicative of the first content item from the second color space gamut associated with the common composition space to a third color space gamut associated with the display device; applying a color adaptation operation to the mapped, re-linearized data indicative of the first content item based on a measured white point of current viewing conditions surrounding the display device; performing simultaneous contrast adaptation on the first content item based on a third difference between the second expected viewing condition associated with the common composite space and the current viewing condition surrounding the display device; and The first content item is displayed on the display device.

16. The apparatus of claim 15, further comprising instructions stored in the memory to cause the one or more processors to: receiving data indicative of a second content item; linearizing the data indicative of the second content item according to an inverse transfer function associated with the second content item; mapping the linearized data indicative of the second content item from a fourth color space gamut associated with the first content item to the second color space gamut associated with a common composition space; modifying the mapped, linearized data indicative of the second content item based on at least one of: (c) a fourth difference between a third expected viewing condition associated with the second content item and the second expected viewing condition associated with the common composite space; and (d) a fifth difference between a second expected viewer adaptation level and a second predicted viewer adaptation level associated with the second content item; and The modified data indicative of the second content item is encoded in accordance with the transfer function associated with the common synthesis space.

17. The apparatus of claim 16, further comprising instructions stored in the memory to cause the one or more processors to: re-linearizing the encoded data indicative of the second content item according to the inverse transfer function associated with the common synthesis space; mapping the re-linearized data indicative of the second content item from the second color space gamut associated with the common composition space to the third color space gamut associated with the display device; applying the color adaptation operation to the mapped, re-linearized data indicative of the second content item based on the measured white point of the current viewing conditions surrounding the display device; performing simultaneous contrast adaptation on the second content item based on the third difference between the second expected viewing condition associated with the common composite space and the current viewing condition surrounding the display device; and The second content item is displayed on the display device.

18. The apparatus of claim 15, further comprising instructions stored in the memory to cause the one or more processors to: Data indicative of ambient light conditions surrounding the display device is received, wherein the current viewing conditions surrounding the display device are based at least in part on the received data indicative of ambient light conditions.

19. The device of claim 15, wherein the instructions for displaying the first content item on the display device further comprise instructions for causing the one or more processors to: An adjustment to a black point, a white point, a system gamma, or a combination thereof of the display device is determined based at least in part on the first predicted viewer adaptation level.

20. The apparatus of claim 17, wherein the first content item and the second content item are different media types.

Images