Dynamic system optical-to-optical transfer functions (OOTF) to provide perceptual criteria

Abstract

A dynamic system-to-optic transfer function is provided to provide a perceptual criterion. A dynamic system optical transfer function (OOTF) is disclosed that adapts displayed content to provide a viewer with a so-called "perceptual baseline" effect, such that as current viewing conditions deviate from the "baseline" viewing scenario, the dynamic system OOTF adapts the content and / or display to provide the viewer with as close as possible to the perceptual effect of the "baseline" viewing scenario. The dynamic system OOTF may perform one or more of the following adaptation processes: adapting media content from a source color space to a linear XYZ color space, adapting media content from a linear XYZ color space to a color space of a display device, automatically adjusting the luminance, white point, and / or black point of a display device, adapting media content from an intended viewing environment to a fixed viewing environment in a common composite space, and / or adapting media content from a fixed viewing environment in a common composite space to the environment of the viewer's current viewing conditions.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Provisional Patent Application No. 63 / 376,540, filed September 21, 2022 (the "'540 Application"). This application is also related to commonly assigned U.S. Patent No. 11,386,875 (the "'875 Patent"). The '540 Application and the '875 Patent are incorporated herein by reference in their entireties. [Background technology]

[0002] Today, consumer electronic devices with display screens are used in many different environments with many different lighting conditions, for example, in offices, homes, home theaters, head-mounted displays (HMDs), and outdoors. Devices typically need to be designed so that the displayed content has a consistent look and tone, with only minimal (or ideally no) color banding perceptible to the viewer, regardless of the user's viewing environment at any given moment. Many content items are created for specific display devices and viewing environments. For example, movies are often created for Rec. 709 displays viewed in dark viewing environments. Devices typically need to be able to adapt content items to many different types of intended display devices and many different viewing environments so that they appear perceived as the content creator intended, regardless of the current viewing conditions around the display device.

[0003] For these and other reasons, it is desirable to map each content item into a shared system-level viewing environment, also referred to herein as a "common composition space," and then map from the shared system-level viewing environment to the current viewing environment. Therefore, there is a need for a technique that utilizes an ambient condition model (and many other display-related factors) to implement a dynamic system-level optical transfer function (OOTF) that can automatically adjust the display's overall content adaptation process, for example, to provide a so-called "perceptual baseline" so that when a dynamic viewing scenario exactly corresponds to a "reference" viewing scenario, a measurably accurate baseline response is provided by the display device (e.g., as may be quantified via optical instruments measuring luminance away from the face of the display device), and then, as the viewing scenario deviates from the "reference" viewing scenario, the dynamic system OOTF adapts to provide the viewer with a perceptual effect in a "non-reference" viewing scenario that is as close as possible to that which they would experience in the "reference" viewing scenario. Successfully modeling a user's current viewing environment and its effect on their perception of the displayed content will allow the user's perception of the displayed content to remain relatively independent of the ambient conditions in which the display device is being viewed and / or any other content items being displayed simultaneously. Summary of the Invention

[0004] As mentioned above, human perception is relative, not absolute. In other words, a human viewer's perception of a displayed image changes based on what surrounds the image, the image itself, and the luminance and white point to which the viewer is currently adapted. A display may typically be placed in front of a wall. In this case, the ambient lighting (e.g., luminance and color) in the room will illuminate the wall behind the display, changing the viewer's perception of the displayed image. Potential changes in the viewer's perception of the displayed content include changes in tonality (which may be modeled using a gamma function), as well as changes to the white point (i.e., the absolute color perceived to be white) and black point (i.e., the highest luminance level indistinguishable from true black).

[0005] Thus, some devices may attempt to maintain consistent content adaptation on a display device throughout the encoding, decoding, and color management processes, but this does not consider the effect that environmental conditions around the display device may have on a viewer's perception of the displayed content. Many color management systems attempt to consistently map content to a display so that the encoding of the content and its reproduction on the display do not affect the resulting displayed content, thus providing consistency throughout the encoding and display of the content. However, these color management systems require fixed viewing conditions, such as always using the intended display and a proposed "reference" viewing environment.

[0006] According to various embodiments described herein, a processor in communication with a display device executing the dynamic system OOTF can adapt a wide variety of constraint parameters to provide a so-called “perceptually based” effect to a viewer. For example, the dynamic system OOTF may perform one or more of the following adaptation processes: adapting media content from a source color space to a linear XYZ color space, adapting media content from a linear XYZ color space to a color space of the display device, automatically adjusting the luminance of the display device, automatically adjusting the white point of the display device, automatically adjusting the black point of the display device, adapting media content from an intended viewing environment to a fixed viewing environment in a common composition space, and / or adapting media content from a fixed viewing environment in a common composition space to an environment of the viewer's actual current viewing conditions.

[0007] Another adaptation process, called a "simultaneous contrast adaptation" process, maps each content item to its proposed viewing environment using a technique indicated in the content item by a content indicator. For example, a content item intended for viewing on a Rec. 709 display includes a content indicator to use RGB space gamma. The resulting simultaneous contrast adaptation content item is referred to herein as the color space data of the proposed viewing environment.

[0008] The dynamic system OOTF techniques disclosed herein provide an extension to traditional color management systems (which typically require the viewing environment of the source content to be reproduced in the viewer's actual viewing environment while matching the content to the color space of the display device on which it is provided) by adapting to the viewer's actual viewing environment, which can be particularly important for mobile devices used in a wide variety of viewing environments, as well as for movie content that may be consumed, for example, in a sunlight-lit living room rather than the intended dark movie theater viewing environment.

[0009] Furthermore, creating content in a viewing environment that does not match the intended viewing environment may introduce biases into the content itself, and therefore may result in an inaccurate appearance when the content is viewed in the intended viewing environment. Creating content under "non-reference" environmental conditions, but while the viewer is adapting to so-called "perceptual references," as provided, for example, by application of the dynamic system OOTF techniques described herein, helps ensure that environmental biases are minimized or avoided, and that the resulting edited content, which may even have been created in a changing, or dynamic, environment, will have an accurate appearance when viewed in the intended viewing environment.

[0010] The techniques disclosed herein use a display device with various optical sensors, such as ambient light sensor(s), multispectral ambient light sensor(s), image sensor(s), or video camera(s), to gather information about the ambient conditions in a viewer's current viewing environment of the display device. The use of these various optical sensors can provide more detailed information about the ambient lighting conditions, which a processor can use to evaluate a unified display model, including an ambient condition model and / or a perceptual adaptation model, based at least in part on the received environmental information and information about the display, such as the display's peak luminance, leakage percentage, reflectance percentage, reference luminance (SDR max), white point, etc., as well as the instantaneous, historical, and even future content itself that is being, has been, or will be displayed to the viewer.

[0011] The output from the unified display model may be used to adapt content so that a viewer's perception of content displayed on a display device is relatively independent of the ambient viewing conditions in which the display is being viewed, what the viewer sees on (and through) the display, and therefore how the viewer's vision is adapted. The output of the unified display model may include modifications to the display's transfer function, gamma boost, tone mapping, resaturation, black point, white point, or combinations thereof.

[0012] Thus, according to some embodiments, a non-transitory program storage device is disclosed that includes instructions stored thereon that, when executed, are configured to cause one or more processors to receive data representing a first content item, linearize the data representing the first content item according to an inverse transfer function associated with the first content item, map the linearized data representing the first content item from a first color space gamut associated with the first content item to a second color space gamut associated with a common composite space, modify the mapped, linearized data representing the first content item based on at least one of (a) a first difference between a first intended viewing condition associated with the first content item and a second intended viewing condition associated with the common composite space, and (b) a second difference between a first intended viewer adaptation level associated with the first content item and a first predicted viewer adaptation level, and encode the modified data representing the first content item according to the transfer function associated with the common composite space.

[0013] In some embodiments, a non-transitory program storage device may include instructions stored thereon that cause one or more processors to relinearize encoded data representing a first content item according to an inverse transfer function associated with the common composite space; map the relinearized data representing the first content item from a second color space gamut associated with the common composite space to a third color space gamut associated with the display device; apply a chromatic adaptation operation to the mapped, relinearized data representing the first content item based on a measured white point of current viewing conditions around the display device; perform simultaneous contrast adaptation on the first content item based on a third difference between the second intended viewing conditions associated with the common composite space and the current viewing conditions around the display device; and display the first content item on the display device.

[0014] In some embodiments, the non-transitory program storage device further includes instructions stored thereon that cause the one or more processors to receive a second content item, similarly process it according to, for example, an inverse transfer function and color space gamut of the second content item into a common composite space, and then further adjust it based on current viewing conditions around the display device, where the second content item may include a content item having a different media type, color space, dynamic range, etc. than the first content item being simultaneously displayed on the display device. In other words, the non-transitory program storage device may further include instructions stored thereon that cause the one or more processors to receive data representing a second content item, linearize the data representing the second content item according to an inverse transfer function associated with the second content item, map the linearized data representing the second content item from a fourth color space gamut associated with the second content item to a second color space gamut associated with the common composite space, modify the mapped linearized data representing the second content item based on at least one of: (c) a fourth difference between a third intended viewing condition associated with the second content item and a second intended viewing condition associated with the common composite space; and (d) a fifth difference between a second intended viewer adaptation level associated with the second content item and a second predicted viewer adaptation level; and encode the modified data representing the second content item according to the transfer function associated with the common composite space.

[0015] The non-transitory program storage device may then cause the one or more processors to relinearize the encoded data representing the second content item according to an inverse transfer function associated with the common composite space, map the relinearized data representing the second content item from a second color space gamut associated with the common composite space to a third color space gamut associated with the display device, apply a chromatic adaptation operation to the mapped, relinearized data representing the second content item based on a measured white point of the current viewing conditions around the display device, perform a simultaneous contrast adaptation on the second content item based on a third difference between the second intended viewing conditions associated with the common composite space and the current viewing conditions around the display device, and display the second content item on the display device.

[0016] In other embodiments, the aforementioned techniques embodied in instructions stored on a non-transitory program storage device may also be practiced as a method and / or implemented on an electronic device having a display device, such as a mobile phone, a PDA, an HMD, a monitor, a television, or a laptop, desktop, or tablet computer. [Brief explanation of the drawings]

[0017] [Figure 1A] It characterizes ambient lighting, diffuse reflection from the display device, and other environmental conditions that affect the display device.

[0018] [Figure 1B] 1 illustrates the additive effect of unintended light on a display device.

[0019] [Figure 2] 1 illustrates a system for performing gamma adjustment utilizing a lookup table.

[0020] [Figure 3] 1 illustrates a frame buffer gamma function and an exemplary native display response.

[0021] [Figure 4] 1 shows graphs depicting LUT transformations and resulting gamma functions, as well as graphs illustrating perceptual transformations due to environmental conditions.

[0022] [Figure 5] 1 illustrates a unified display model system for performing display adjustments based on dynamic system OOTF according to one or more embodiments.

[0023] [Figure 6] FIG. 1 illustrates a simplified functional block diagram of an ambient condition model according to one or more embodiments.

[0024] [Figure 7] 1 illustrates, in flowchart form, a process for performing display adjustments based on dynamic system OOTF, according to one or more embodiments.

[0025] [Figure 8] 1 illustrates, in flowchart form, a process for performing display adjustments based on a dynamic system OOTF, according to one or more embodiments.

[0026] [Figure 9] FIG. 1 illustrates a simplified functional block diagram of a display processing device according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0027] The disclosed techniques use a display device with various optical sensors, such as ambient light sensors or image sensors, to collect information about ambient conditions in the environment of a viewer of the display device. The use of ambient environment information, information about the display device and its characteristics, and information about the displayed content, its intended display type, and its proposed viewing environment can provide a more accurate prediction of the viewer's current viewing environment and its impact on how the user perceives the displayed content. A processor in communication with the display device can evaluate an ambient condition model and / or a perceptual adaptation model as part of a unified display model to predict the impact of the current ambient viewing conditions (and / or the content itself) on the viewer's perception. The output of the unified display model can be proposed modifications (e.g., proposed adjustments to gamma, black point, white point, and / or saturation) used to perform environmental adaptation to the displayed content and parameters of the display device itself, so that the viewer perceives the adapted displayed content as intended, while remaining relatively independent of the current ambient conditions.

[0028] The techniques disclosed herein are applicable to any of many electronic devices, such as digital cameras, digital video cameras, mobile phones, personal digital assistants (PDAs), head-mounted display (HMD) devices, monitors, televisions, digital projectors (including cinema projectors), and displays for desktop, laptop, and tablet computers.

[0029] Also, for clarity, not all features of actual implementations are described in this specification. Of course, it will be understood that the development of any actual implementation (as in any development project) requires numerous decisions to achieve the developer's specific goals (e.g., conformance with system and business-related constraints), and that these goals may vary from implementation to implementation. It will also be understood that such development efforts may be complex and time-consuming, but that they would nevertheless be routine undertakings of one of ordinary skill in the art having the benefit of this disclosure. Moreover, the language used in this disclosure has been chosen primarily for purposes of readability and explanation, and therefore not to limit or restrict the inventive subject matter, which should be determined by the claims. References herein to "one embodiment" or "one embodiment" mean that a particular feature, structure, or characteristic described with respect to an embodiment is included in at least one embodiment of the invention, and multiple references to "one embodiment" or "one embodiment" should not be understood as necessarily all referring to the same embodiment.

[0030] Background Regarding Characteristics of Exemplary Display Devices and Ambient Viewing Conditions

[0031] Referring now to FIG. 1A , the characteristics of ambient lighting, diffuse reflection from display device 102, and other environmental conditions affecting the display device are illustrated through a side-view depiction of viewer 116 of display device 102 in a particular ambient lighting environment. As shown in FIG. 1A , viewer 116 is looking at display device 102, which in this case is a typical desktop computer monitor. Dashed line 110 represents the viewing angle of viewer 116. As depicted in FIG. 1A , the ambient environment is illuminated by ambient light source 100, which casts light rays 108 onto all objects in the environment, including wall 112 and display surface 114 of display device 102. As indicated by the multiple small arrows 109 (representing reflections of light rays 108), a certain percentage of the incoming light radiation reflects off the surface it illuminates. Diffuse reflection may be defined as the reflection of light from a surface such that incident light rays are reflected at many angles, which has a particular effect on the viewer's perception of display device 102.

[0032] If the luminance of reflected light and / or light leakage from display device 102 is greater than the luminance of the pixels driven by the display device for a content item, the viewer may be unable to perceive low-tone details within that content item. This effect is illustrated by dashed line 106 in FIG. 1A , which indicates a threshold luminance level. When the luminance of a pixel within emissive display surface 114 is below the threshold luminance level indicated by dashed line 106, the pixel is not perceived as intended. When the luminance of a pixel within emissive display surface 114 is greater than the threshold luminance level, the pixel is perceived as intended. Dashed line 106 and the threshold luminance level may be adjusted to account for each of reflected light and light leakage from display device 102, either alone or in combination. The effects of reflected light and light leakage from a display device on the viewer's perception of displayed content are further described herein with respect to FIG. 1B . Information about diffuse reflection and other ambient light in the current viewing environment can be used to inform an ambient condition model, which suggests which adaptation processes should be performed on the content to compensate for environmental conditions and / or suggests modifications to adaptation processes already being performed.

[0033] The diffuse reflectance and other ambient light information may be based on light level readings recorded by one or more optical sensors, such as ambient light sensor 104. Dashed line 118 represents data indicative of the light source being collected by ambient light sensor 104. Optical sensor 104 may be used to gather information about the ambient conditions in the display device's environment and may comprise, for example, an ambient light sensor, an image sensor, or a video camera, or some combination thereof. A forward-facing image sensor provides information about how much light (and, in some embodiments, what color light) is falling on display surface 114. This information can be used in conjunction with a model of the display's reflective and diffuse characteristics to inform the ambient condition model about the particular lighting conditions in which the display is currently situated and to which the user is currently adapting. Optical sensor 104 is shown as a "forward-facing" image sensor, i.e., an image sensor facing in the general direction of a viewer 116 of display device 102, although other optical sensor types, placements, positioning, and quantities are possible. For example, one or more "rear-facing" image sensors, alone (or in conjunction with one or more forward-facing sensors), can provide additional information about the light sources and colors in the viewer's environment. The rear-facing sensors collect light from emitting sources or light reflected back from objects behind the display and can be used to determine the brightness of the display's surround, i.e., what the user sees through the display. This information may also be used for the ambient condition model. For example, the color of the wall 112, if it is close enough behind the display device 102, can have a significant effect on the viewer's perception. Similarly, in the example of an outdoor environment, the color and intensity of the light surrounding the viewer can cause the display to appear different than in an indoor environment with, for example, incandescent (colored) lighting.

[0034] In one embodiment, optical sensor 104 may comprise a video camera (or other device) capable of capturing spatial, color, and intensity information. With respect to spatial information, the video camera or other device(s) may also be used to determine the viewing user's distance from the display, e.g., to further model how much of the user's field of view the display occupies and, correspondingly, how much the display / environment affects the user's perception of the displayed content. In some embodiments, the video camera may be configured to capture images of the surround environment for analysis at some predetermined time interval, e.g., every two minutes, so that the ambient condition model can be gradually updated or otherwise modified as ambient conditions in the viewer's environment change.

[0035] Additionally, the rear-facing video camera used to model the surround environment may be designed to have a field of view that closely matches the calculated or estimated field of view of the viewer of the display. For example, once the viewer's field of view is calculated or estimated based on the size or position of the viewer's facial features as recorded by the front-facing camera, the system can then determine which portion of the rear-facing camera's image to use in the surround calculation, assuming the rear-facing camera's native field of view is known and is larger than the viewer's field of view.

[0036] In yet other embodiments, one or more cameras or depth sensors may be used to further estimate the distance of particular surfaces from the display device. This information can be used, for example, to further inform an ambient condition model based on the likely composition of the viewer's surround and its perceptual impact. For example, a display placed 18 inches away from a user and at a 30-inch angle will have a greater impact on the user's vision than the same display placed 48 inches away from the user and occupies less of the user's field of view.

[0037] 1B, the additive effects of unintended light on a display device are shown in more detail. For example, light ray 155 emanating from display representation 150 represents the amount of light the display intentionally drives its pixels to generate at a given moment. Similarly, light ray 165 emanating from display representation 160 represents the amount of light leakage from the display at a given moment, and light ray 109 reflecting from display representation 170 represents the aforementioned diffuse reflection of ambient light from the display's surface at a given moment. In a display of stacked components compared to a laminated component, there may be more diffuse reflection from a matte display than a glossy display, or from a clean display compared to a dusty or otherwise dirty display. Finally, display representation 180 represents the sum of the three forms of light shown in display representations 150, 160, and 170.

[0038] As shown in FIG. 1B , light rays 185 emanating from display representation 180 represent the actual amount of light perceived by a viewer of the display device, which may differ from the initial amount of light 155 to which pixels in the display were intentionally driven to produce the desired content. Unintended light, such as from display leakage and diffuse reflection, can desaturate the perceived color compared to the intended color of the content. The darker or dimmer the intended color, the more noticeable the desaturation that appears to the viewer. Therefore, considering the effects of these various phenomena can help achieve a more consistent and content-accurate perceptual experience across viewing environments.

[0039] Thus, in one or more embodiments disclosed herein, the ambient condition model may be employed as part of an integrated display model to dynamically select which environmental adaptations to implement, or adjusting already implemented environmental adaptations may compensate for unintended light so that the dimmest colors are not masked by light leakage and / or predicted diffuse reflectance levels, and all colors are not perceived as desaturated compared to intended colors. The model of display device characteristics may be used to determine the amount of light leakage from the display device under current display parameters. The model of display device characteristics may also be used in combination with information from the ambient light sensor 104 to estimate the amount of diffuse reflectance from the display device. The perceptual model may be used to estimate the amount of desaturation from unintended light so that the ambient condition model may determine recommended resaturation and environmental adaptations to achieve that recommended resaturation.

[0040] Background on System Gamma and Perceptual Gamma for Exemplary Display Devices

[0041] 2, an exemplary system 212 for performing gamma adjustment utilizing a lookup table (LUT) 210 is shown. Element 200 represents source content, such as that created by a source content creator, that viewer 116 desires to view. Source content 200 may include images, videos, or other displayable content types. Element 202 represents a source profile, i.e., information describing the color profile and display characteristics of the device on which the source content creator created source content 200. Source profile 202 may include, for example, an International Color Consortium (ICC) profile (described in more detail below) of the creator's device or color space, or other related information.

[0042] Information about the source content 200 and the source profile 202 may be transmitted to a device of the viewer 116, which may include a system 212 for performing gamma adjustment using the LUT 210. The device of the viewer 116 may include, for example, a mobile phone, a PDA, an HMD, a monitor, a television, or a laptop, desktop, or tablet computer. Upon receiving the source content 200 and the source profile 202, the system 212 may perform a color adaptation process 206 on the received data to, for example, perform gamut mapping, i.e., color matching across various color spaces. For example, gamut matching attempts to preserve (as closely as possible) the relative relationships between colors (e.g., created / approved by the content creator on the display described by the source ICC profile), even if all colors must be systematically modified or adapted so that they appear on the destination device.

[0043] Once the source and destination color profiles have been properly adapted, image values ​​can enter a so-called "frame buffer" 208. In some embodiments, image values, e.g., pixel luma values, enter the frame buffer coming from a single application or multiple applications that have already processed image values ​​to be encoded using a particular implicit gamma. A frame buffer can be defined as a device with a video output that drives a video display, in this case from a memory buffer containing a complete frame of image data. As further described below in connection with FIG. 3, the implicit gamma of the values ​​entering the frame buffer can be visualized by looking at the "frame buffer gamma function." Ideally, this frame buffer gamma function is the exact inverse of the display device's "native display response" function, which characterizes the display's luminance response to the input.

[0044] Because the inverse of the native display response is not always exactly the inverse of the frame buffer, a LUT, which may be stored on a video card or other memory, may be used to account for imperfections in the relationship between the encoded and decoded gamma values, as well as the particular luminance response characteristics of the display. Thus, if desired, system 212 may utilize LUT 210 to perform a so-called “gamma adjustment process.” LUT 210 may comprise a two-column table of positive real values ​​spanning a particular range, e.g., from 0 to 1. The values ​​in the first column may correspond to input image values, and the values ​​in the second column in the corresponding row of LUT 210 may correspond to output image values ​​to which the input image values ​​are “transformed” before ultimately being displayed on display 102. LUT 210 may be used to account for imperfections in the luminance response curve of display 102, also known as the “display transfer function.” In other embodiments, the LUT may have a separate channel for each primary color in the color space; for example, the LUT may have a red channel, a green channel, and a blue channel in the sRGB color space.

[0045] The transformation applied by the LUT to the incoming frame buffer data before the data is output to the display device may be used to ensure the desired 1.0 gamma boost is applied to the final display device. While the system shown in FIG. 2 is generally a good system, it does not consider the effect of differences or changes in ambient lighting conditions on perceived gamma, or gamma adjustments already encoded into the source content 200 by the source creator to compensate for differences between the source content capture environment and the intended viewing environment of the source content 200. In other words, a 1.0 gamma boost for encoding and decoding content is only achievable / appropriate in one ambient lighting environment, which is typically brighter than a typical office environment. For example, content captured in a bright environment does not require a gamma boost when viewed in the same (i.e., bright) environment, e.g., due to the “simultaneous contrast” phenomenon. In another example, content captured and edited in a bright environment but intended to be viewed in a dimly lit environment (e.g., a dark surround such as a movie theater) may already include a gamma adjustment in the source content 200 received by the system 212. Thus, the additional gamma boost based on the LUT 210 may distort the gamma adjustment already provided in the source content 200, causing the displayed content to differ from the intent of the source creator.

[0046] As mentioned above, in some embodiments, the goal of this gamma adjustment system 212 is to apply an overall 1.0 system gamma to content being displayed on the display device 102. An overall 1.0 system gamma corresponds to a linear relationship between input coded luma values ​​and output luminance on the display device 102. Ideally, an overall 1.0 system gamma would cause the displayed content to appear largely as intended by the source creator, despite intervening content encoding and decoding, and other color management processes used to adapt the content to the particular display device 102. However, as described below, this overall 1.0 gamma may only be perceived appropriately under one specific set of ambient lighting conditions, thus necessitating the need for a dynamic display adjustment system to adjust the overall system gamma to accommodate different ambient lighting conditions and achieve a perceived 1.0 system gamma. Furthermore, gamma adjustment is only one type of compensation for environmental conditions, and environmental adaptation as described herein includes gamma adjustment as well as resaturation adjustment, black point adjustment, white point adjustment, etc.

[0047] 3, a frame buffer gamma function 300 and an exemplary native display response 302 are shown. Gamma adjustment, or often simply referred to as "gamma," is the name given to a non-linear operation commonly used to encode luma values ​​and decode luminance values ​​in video or still image systems. Gamma γ is calculated using the simple power law formula L out =L in γ where the input value L in and output value L outare each non-negative real values, typically within a predetermined range, e.g., 0 to 1. Gamma values ​​greater than 1 are sometimes called "encoded gamma," and the process of encoding using this compressive power-law nonlinearity is called "gamma compression." Conversely, gamma values ​​less than 1 are sometimes called "decoded gamma," and the application of an extended power-law nonlinearity is called "gamma expansion." Gamma encoding of content serves to map the content data into a more perceptually uniform domain.

[0048] Another way to think of the gamma characteristics of a system is as similar to a power law relationship that approximates the relationship between the encoded luma in the system and the actual desired image luminance, whatever the ultimate user display device. In existing systems, a computer processor or other suitable programmable control device may perform gamma adjustment calculations for the particular display device with which it is communicating, based on the display device's native luminance response, the device's color gamut, and the device's white point (which information may be stored in an ICC profile), as well as ICC color profiles and other content indicators that the source content creator attached to the content to specify the content's "rendering intent."

[0049] An ICC profile is a set of data that characterizes a color input or output device or color space according to standards promulgated by the International Color Consortium (ICC). An ICC profile can describe the color attributes or viewing requirements of a particular device by defining a mapping between the device's source or target color space and a profile connection space (PCS), typically the CIE XYZ color space. ICC profiles can be used to define a color space generally in terms of three main parts: 1) the primaries that define the gamut, 2) a transfer function (sometimes called a gamma function), and 3) a white point. ICC profiles may also contain additional information to provide a mapping between a display's actual response and its "advertised" response, i.e., its tone response curve (TRC), for example, to correct or calibrate a given display to a perfect 2.2 gamma response.

[0050] In some implementations, the ultimate goal of the gamma adjustment process is to have a final overall gamma boost of 1.0 applied to the content when it is displayed on a display device, i.e., so-called "unity" or "no boost." An overall system gamma of 1.0 corresponds to a linear relationship between the input encoded luma values ​​and the output luminance on the display device, meaning that there is effectively no amount of gamma "boosting" applied and the gamma encoding process is not carried out by the gamma decoding process without further adjustment.

[0051] Traditionally, gamma encoding is optimized for a particular environment, content dynamic range, and display dynamic range, so that the encoding and display codes are spaced far enough across the intended range to make the content appear as intended (e.g., unbanged, no crushed highlights or blacks, with the correct contrast, sometimes called tonality, etc.) A gamma of 2.2 at 8 bits is an example of an acceptable representation for encoding standard dynamic range (SDR) content displayed on a 1 / 2.45 gamma Rec. 709 CRT in a bright office viewing environment.

[0052] However, even when displayed on its intended Rec. 709 display, the exemplary SDR content will not have the intended appearance when viewed in an environment brighter or darker than the intended bright office viewing environment. If the current viewing environment differs from the proposed viewing environment, e.g., if it is brighter than the proposed viewing environment, the user's vision will adapt to the current brighter viewing environment, and the user will perceive less distinguishable detail in the darker portions of the content. The display may be capable of modulating only a narrow range of the user's vision to adapt to the current brighter viewing environment. Furthermore, the display's fixed maximum brightness may be dark compared to the brightness of the current viewing environment.

[0053] Current brighter viewing environments prevent users from perceiving the darker portions of the content that the creator of the source intended viewers to perceive when the content is viewed on the proposed Rec. 709 display in the proposed bright office viewing environment. In other words, "shadow detail" is "crushed" to black. This effect is magnified when ambient light from the viewing environment is reflected off the display and / or light from display leakage, collectively referred to as unintended light, further limits how dark the content is perceived by the viewer. The lowest codes in the content are spaced apart in luminance based on the proposed viewing environment and may be spaced too closely together to be distinguishable in the current brighter viewing environment.

[0054] The perceived overall tonality of the content also differs when the current viewing environment differs from the proposed viewing environment. For example, the content may appear lower in contrast if the current viewing environment is brighter than the proposed viewing environment. The content may also appear desaturated, with unintended color casts due to unintended light from reflections and / or display leakage from the display, or when the white point of the proposed viewing environment differs from the white point of the current viewing environment.

[0055] Even when viewed on a proposed Rec. 709 display in a proposed bright office viewing environment, the tonality of the content may be perceived differently based on what other content is simultaneously displayed, an effect known as "simultaneous contrast." Some devices display multiple content items at once; for example, a user's work computer may simultaneously display multiple documents and a video. Different content items may be tailored to different proposed viewing environments, with each content item using different gamma encoding and / or different gamma boost. A display device that implements the same gamma boost for all content items may distort individual content items from their intended appearance.

[0056] For example, Rec. 709 content has an overall gamma boost of 1.22 due to an intentional mismatch between the content's encoding gamma and the display's decoding gamma, compensating for bright surround content viewed in a dim surround environment. In contrast, DCI P3 content encodes compensation for bright surround content viewed in a dim surround environment directly into the pixels themselves, so that no gamma boost is needed—i.e., a gamma of 1.0 is sufficient. A single gamma boost is not appropriate for both Rec. 709 and DCI P3 content in any viewing environment. While this example illustrates the difference in gamma boost, similar differences can be seen in other types of content adaptation, such as tone mapping, resaturation, black and / or white point adjustment, modified transfer functions for the display, and combinations thereof. As used herein, "surround environment" refers to the ambient lighting conditions, etc., in the environment around the display device. "Viewing environment" refers to the ambient environment around a display device and display characteristics, such as light leakage, of a display device that can further affect how a user perceives content displayed on the display device.

[0057] As shown in Figure 3, the x-axis of frame buffer gamma function 300 represents input image values ​​that span a particular range, e.g., from 0 to 1. The y-axis of frame buffer gamma function 300 represents output image values ​​that span a particular range, e.g., from 0 to 1. As noted above, in some embodiments, image values ​​are input to frame buffer 208 already processed and may have a particular implicit gamma. As shown in graph 300 of Figure 3, the coding gamma is approximately 1 / 2.2, or 0.45. That is, the line in graph 300 is approximately the function L OUT =L IN 0.45 Because the native display response of many display devices has a gamma of roughly 2.2, i.e., the inverse of the 1 / 2.2 encoding gamma, gamma values ​​around 1 / 2.2 or 0.45 are typically used as the encoding gamma. In other cases, for example, a gamma of 1 / 2.45 may be applied to 1.96 gamma-encoded content when displayed on a conventional 1 / 2.45 gamma CRT display to provide a gamma "boost" of 1.25 (i.e., 2.45 divided by 1.96), which is needed to compensate for the simultaneous contrast effect that causes bright content to appear lower in contrast when viewed in a dim surround environment (i.e., the area beyond the display is typically dimmer), such as the intended viewing environment for Rec. 709 at 16 lux. Even if the content already contains additional gamma boost because the source creator intended the bright content to be viewed in a dim surround environment, the frame buffer 208 does not take into account this encoded gamma boost, and the resulting gamma boost will differ from the rendering intent of the source creator.

[0058] The x-axis of the native display response function 302 represents input image values ​​spanning a particular range, e.g., from 0 to 1. The y-axis of the native display response function 302 represents output image values ​​spanning a particular range, e.g., from 0 to 1. Theoretically, a system in which the decoded gamma is the inverse of the encoded gamma should produce a desired overall 1.0 system gamma. However, this does not take into account ambient light in the environment around the display device and / or gamma boost already encoded in the source content. Therefore, the desired overall 1.0 system gamma is only achieved in one ambient lighting environment, e.g., the production lighting environment, or, if gamma boost is already encoded in the source content, the intended viewing environment. These systems do not dynamically adapt to the environmental conditions surrounding the display device or according to user preferences.

[0059] Referring now to FIG. 4, a graph is shown illustrating a LUT transformation and the resulting gamma function, as well as a graph illustrating perceptual transformation due to environmental conditions. The graph in FIG. 4 illustrates how an LUT can be utilized to account for imperfections in the relationship between encoded and decoded gamma values ​​in an ideal system, as well as the specific luminance response characteristics of a display at different input levels. The graph in FIG. 4 also illustrates how environmental conditions surrounding a display device can distort the perception of content, such that the perceived gamma differs from the resulting gamma function. The x-axis of native display response graph 400 represents input image values ​​over a specific range, e.g., from 0 to 1. The y-axis of native display response graph 400 represents output image values ​​over a specific range, e.g., from 0 to 1. The nonlinear nature of graph 400 illustrates minor idiosyncrasies and imperfections in the native response function of an exemplary display. The x-axis of LUT graph 410 represents input image values ​​over the same range of input values ​​to which the display can respond, e.g., from 0 to 1. The y-axis of LUT graph 410 represents the same range of output image values ​​that the display can produce, e.g., from 0 to 1. In an ideally calibrated display device, the display response 400 would be the inverse of the LUT response 410; therefore, when the LUT graph is applied to input image data, the resulting gamma function 420 reflects the desired overall system gamma response of 1.0, i.e., that resulting from the adjustments provided by the LUT and the display's native (nearly) linear response, such that the content is perceived as intended by the source creator. The x-axis of the resulting gamma function 420 represents the input image values ​​created by the source content creator, spanning a particular range, e.g., from 0 to 1. The y-axis of the resulting gamma function 420 represents the output image values ​​displayed on the resulting display, spanning a particular range, e.g., from 0 to 1. The slope of 1.0 reflected in the line in graph 420 indicates that the brightness levels intended by the source content creator will be reproduced at the corresponding luminance levels on the final display device.

[0060] Ideally, the resulting gamma function 420 reflects the desired overall 1.0 system gamma on the resulting display device, indicating that the tone response curves (i.e., gamma) are aligned between the source and the display, that the gamma encoding of the content is undone by the gamma decoding process without further adjustment, and that the image on the display is likely to appear approximately as the source creator intended. However, this calculated overall 1.0 system gamma does not account for the effects of ambient lighting conditions on the viewer's perception of the gamma boost. In other words, due to perceptual transformations caused by ambient conditions in the viewer's environment 425, the viewer will not perceive the content as the source creator intended and will not perceive an overall 1.0 gamma in all lighting conditions. The calculated overall 1.0 gamma may also fail to account for the effects of the viewer's current adaptation to ambient lighting conditions. As mentioned above, a user's ability to perceive changes in light intensity (as well as the overall range of light intensities their eyes can perceive) is further based on what range of light levels the user's eyes have been in (and therefore adjusted to) over a preceding time window (e.g., 30 seconds, 5 minutes, 15 minutes, etc.). A calculated overall gamma of 1.0 may also fail to account for gamma boost already encoded into the source content by the source creator based on the source capture and editing environment and the intended viewing environment. For example, a video may be shot in a bright environment but edited to be viewed in a dimly lit environment, and the gamma boost would match this transition already encoded into the video. If the system attempts to further adjust the already adjusted gamma boost, the resulting gamma will differ from the source creator's rendering intent.

[0061] As shown in graph 430, the dashed line indicates the viewer's actual perception of the achieved system gamma, which corresponds to a perceived gamma boost of 1.0, i.e., an overall gamma boost greater than 1.0. Ambient conditions in the viewing surround transformed the achieved system gamma greater than 1.0 into a perceived system gamma equal to 1.0. Thus, a unified display model for dynamically adjusting display characteristics according to one or more embodiments disclosed herein can account for perceptual transformations resulting from the viewer's current environmental conditions and cause the display to boost the achieved system gamma above the intended 1.0 system gamma, thus presenting the viewer with what is perceived as an overall 1.0 system gamma and allowing the viewer to perceive the content as intended by the source creator. As described in more detail below, such a unified display model can also have a non-uniform time constant for how stimuli affect the viewer's instantaneous adaptation over time. In other words, the model can attempt to predict changes in the user's perception due to changes in the viewer's ambient conditions.

[0062] Unified display model utilizing dynamic system OOTF

[0063] 5, a unified display model system 500 for performing display adjustments based on the dynamic system OOTF is shown, according to one or more embodiments. A given display, e.g., display 102, can be said to have the ability to "modulate" (i.e., adapt or adjust to) only a certain percentage of the possible surround environment at any given moment. For example, if the environment is much brighter than the display, such that the display reflects a lot of light at its minimum display output level, the display may have a relatively high "pedestal" value and therefore can only modulate a portion of the ambient lighting conditions, even at its maximum display output level.

[0064] The unified display model system 500 may therefore be used to apply a transform or transforms to distort the source content 200 (e.g., high-precision source content) to the adapted visual perception of a viewer of the display 102 in a given viewing environment. As explained above, distorting the original source content signal to the perception of the viewer of the display and the display environment may be based on predicted viewing environment conditions, for example, receiving an ambient condition model, as further explained with reference to FIG. 6. For example, the ratio between the diffuse white luminance of the display 102 in nits and the luminance of the user's vision through the display 102, referred to as the surround, also in nits, may be used to apply gamma boost, color saturation correction, or similar algorithms to compensate for the perceptual effect of viewing content in a surround having a different luminance than the surround associated with the source content 200 during capture, editing, or approval.

[0065] According to some embodiments, the unified display model system 500 may take into account one or more dynamic display characteristics 502, such as, for example, information obtained from a forward-facing ambient light sensor (ALS) 504, information obtained from a rear-facing ALS 510, histogram information about currently displayed content 506, and / or a current overall brightness level 508 of the display device.

[0066] According to some embodiments, the unified display model system 500 may also consider one or more static display characteristics 512 when determining how to modify the displayed content, such as, for example, information about the rate of light leakage experienced by the display 514, information about the light reflectance of the surface of the display 516, information about the primary colors of the display device 518, information about the native white point of the display device 520, and / or information about the native response of the display device 522.

[0067] According to some embodiments, the unified display model system 500 may combine information from both the dynamic display features 502 and the static display features 512 into a perceptual model 530. According to some such embodiments, the perceptual model 530 may include a perceptual visual adaptation model 532 configured to model a viewer's likely level of adaptation given the current dynamic display features 502 and static display features 512. In some embodiments, the perceptual visual adaptation model 532 may be based at least in part on a color appearance model (CAM), such as the CIECAM02 color appearance model, and may be used to further inform the ambient condition model 600 regarding the appropriate amount of gamma boost to apply in conjunction with the display's modified transfer function. The CAM may be based, for example, on the luminance and white point of the viewer's surround, as well as the display's field of view subtended by the viewer's field of view.

[0068] In some embodiments, knowledge of the size of the display and the distance between the display and the viewer may also provide useful input to the unified display model 500. Information about the distance between the display and the user may be retrieved from a forward-facing image sensor, such as the forward-facing camera 104. For example, the luminance and white point of the viewer's surround may be used to determine the ratio of diffuse white luminance to the luminance of the viewing surround. Based on the determined ratio, a particular gamma boost may be applied. For example, for a pitch-black ambient environment, an additional gamma boost of approximately 1.5 imposed by the LUT may be appropriate, while a gamma boost of 1.0 (i.e., unity or no boost) may be appropriate for a bright or sunlit environment. For an intermediate surround, the appropriate gamma boost value to be imposed by the LUT may be interpolated between values ​​of 1.0 and approximately 1.5. A more detailed model of surround conditions is provided by the CIECAM02 specification.

[0069] According to some embodiments, the perceptual visual adaptation model 532 may also be used to predict 534 the viewer's current minimum perceptible light level, and to perceptually map 536 the display and environment to the viewer's current perception using the model. Using this information, the perceptual distance model 540, optionally after adapting the luma and / or chroma display data to the XYZ color space (or another device-invariant color space), may employ a perceptual color model 542 (e.g., based on the CIELAB color space) to determine, in block 544, a perceptual threshold below which the viewer may not currently be able to perceive the change in tonality and / or the step (i.e., change) necessary to modify the display's response based on the viewer's predicted perceptual adaptation level under current viewing conditions.

[0070] The output of the perceptual model 530 may then be sent to a color mathematical model 550, which is used to calculate and configure modifications to the display's response to achieve the desired perceptual criteria. According to some embodiments, the color mathematical model 550 may include: a module 552 for matching displayed content values ​​to a viewer's current color perception, a module 554 for performing white point adaptation, a module 556 for performing color matching to the display device's color gamut, a module 558 for performing white point adaptation, and / or a module 560 for calculating a gamma matching response for the display device. The outputs of modules 552 / 554 / 556 / 558 / 560 may be combined into one or more matrices 562, e.g., mesopic matrices, chromatic adaptation matrices, etc., and / or one or more combined look-up tables (LUTs) 564, to efficiently store values ​​embodying the changes determined by the color mathematical model 550 to be applied to the display device. These matrices 562 and / or LUTs 564 may then be normalized and sent to a display pipeline 580.

[0071] The display pipeline 580 may perform one or more of the following functions: compositing multiple content items for simultaneous display 582; linearizing content item color data 584; applying color changes determined by the color mathematical model 550, for example via application of one or more 3x3 matrices 586; performing any necessary brightness compensation 588 as determined by the unified display model; and gamma encoding 590 the modified content for ultimate display to the viewer 116.

[0072] In some embodiments, modifications to combined LUT 564 may be implemented gradually (e.g., over a determined time interval) via an animation engine or similar control element within display pipeline 580. According to some such embodiments, display pipeline 580 may be configured to adjust combined LUT 564 based on the rate at which the viewer's vision is predicted to adapt to those changes.

[0073] In some embodiments, the black level for a given ambient environment is determined, for example, by using an ambient light sensor 104 or by taking measurements of the actual panel and / or diffuser of the display device. As discussed above with reference to FIG. 1A, diffuse reflection of ambient light from the surface of the device can increase the intended display value and affect the user's ability to perceive the darkest display levels (a phenomenon also known as "black crush"). In other environments, light levels below a certain brightness threshold are simply invisible to the viewer. Once this level is determined, the black point can be adjusted accordingly.

[0074] In another embodiment, the white point, i.e., the color that a user perceives as white for a given ambient environment, may similarly be determined by analyzing the lighting and color conditions of the ambient environment, for example, using one or more optical sensors 104. The white point for the display device may then be chromatically adapted to be the determined white point from the viewer's surround. Additionally, note that modifications to the white point may be asymmetric between the red, green, and blue channels of the LUT, thereby shifting the relative RGB mix and therefore the white point.

[0075] As described above, in some embodiments, the unified display model 500 may first adapt the source content 200 to its reference environment, if necessary, using a specified adaptation algorithm included in the source profile 202. For example, an RGB-based gamma for Rec. 709 video, as traditionally applied through a mismatch between the content encoding gamma and the display 102's decoding response, may be applied. Once the source content 200 has been adapted to its reference environment using its specified algorithm, the unified display model 500 may use best practices to adapt the source content 200 to a shared system-level viewing environment, or common composition space. The common composition space may be dynamically changed to match the user's current viewing environment or may be held constant. In implementations in which the common composition space is held constant, the unified display model 500 may globally adapt all content items within the common composition space to adapt the fixed common composition space to the current viewing environment. Any suitable technique may be used to adapt the source content 200 from its reference environment to the common composition space and from the common composition space to the current viewing environment. This feature can be particularly useful when multiple content items from multiple source creators are displayed at once. Unique content adaptations already encoded in each content item can be adjusted without affecting the content adaptations applied to other content items. A common composite space for all content items can then be adjusted based on the particular viewing surround for the display 102. In the most recently described embodiment, the combined LUT 564 can serve as a useful and efficient place for the unified display model system 500 to impose these environmentally aware display transfer function adaptations. In some embodiments, the unified display model system 500 may generate an ICC profile that represents the native response of the display based on the viewing surround as the true native response of the display divided by the desired system gamma.An ICC profile can contain fixed "presets," each representing a particular viewing surround and the corresponding environmental adaptations required for the content to be correctly perceived in the particular viewing surround. The unified display model 500 can then determine the appropriate preset based on an analysis of the captured ambient conditions and apply the corresponding environmental adaptations to the source content 200, either directly or in a common synthesis space.

[0076] 6, there is shown a simplified functional block diagram of an exemplary ambient conditions model 600. As alluded to above, the ambient conditions model 600 may take into account various factors, such as: predictions from a color appearance / perception model 610, information about the ambient environment from, for example, ambient light sensor(s) / image sensor(s) 620, information about the display's current brightness level and / or brightness history 630 (e.g., knowing how bright the display has been and for how long may affect the user's adaptation level), information and features from the display device's profile 640, and / or predictions based on historically displayed / upcoming content 650.

[0077] The color appearance model 610 may include, for example, the CIECAM02 color appearance model or the CIECAM97s model, which may be used to perform chromatic adaptive transformations and / or to calculate mathematical correlations for six technically defined dimensions of color appearance: brightness (luminance), lightness, chroma, saturation, and hue.

[0078] Display characteristics 640 may include information about the color space of the display device, native display response characteristics or anomalies, reflectivity, leakage, or even the type of screen surface used by the display from display profile 204. For example, an "anti-glare" display with a diffuser "loses" more black levels than a glossy display at a given (non-zero) ambient light level.

[0079] The history model 650 can take into account both the instantaneous brightness level of the content and the cumulative brightness of the content over a period of time. In other embodiments, the model 650 may also perform an analysis of upcoming content, for example, to enable the ambient condition model to begin adjusting the transfer function of the display over time so that the desired state is reached by the time the upcoming content is displayed to the viewer (or within a threshold of time). The biological / chemical rate of visual adaptation in humans may also be taken into account when the ambient condition model 600 determines how quickly the display should adjust to account for upcoming content. In some cases, the content itself may already be adaptively coded, for example, by the creator of the source content. For example, one or more frames of the content may include a customized transfer function associated with the individual frame or multiple frames. In some embodiments, the customized transfer function for a given frame may be based solely on the content of the given frame, for example, the brightness level of the given frame. In other embodiments, the customized transfer function for a given frame may be based, at least in part, on at least one of the luminance levels of one or more frames displayed before the one or more frames of content and / or the luminance levels of one or more frames displayed after the one or more frames of content. If the content itself is adaptively coded, the ambient condition model 600 may first implement the adaptively coded adjustments and move the content into a common composition space according to the content indicators included in the source profile 202. The ambient condition model 600 may then attempt to further modify the transfer function of the display during the display of a particular frame of the coded content based, for example, on various other environmental factors, e.g., 610 / 620 / 630 / 640, that may have been acquired at the display device.

[0080] According to some embodiments, the modifications determined by ambient condition model 600 may be implemented by modifying existing table values ​​(e.g., as stored in one or more calibration LUTs, i.e., tables configured to provide a "perfect" tone response curve for the display). Such modifications may be performed via value lookups against transformed values ​​in the original tables, or by modifying the original tables "in place" via distortion techniques. For example, the black level (and / or white level) adaptation process described above may be implemented via distortion compression of values ​​in the tables above black (and / or below white). In other embodiments, "regammation" and / or "resaturation" of the LUTs may be applied in response to adjustments determined by ambient condition model 600.

[0081] As will be understood hereinafter, the exact manner in which the ambient condition model 600 processes the information 610 / 620 / 630 / 640 / 650 received from the various source optical sensors 104, the display luminance 508, the display profile 204, and the indicators in the content source profile 202, and how it modifies the resulting display response curve, for example, by modifying LUT values, will depend on the particular implementation and the desired effect of a given system, including how quickly such modifications are made.

[0082] According to some embodiments, the ambient condition model 600 may be used to consider the various factors described above with reference to FIG. 6 that may affect the viewer's perception at a given moment. Then, based on the output of the ambient condition model 600, an updated display transfer function may be determined to drive the display 102. The display transfer function may be used to convert between input signal data values ​​and voltage values ​​that may be used to drive the display, generating pixel intensities that correspond to the perceptual bins to which the transfer function maps the input signal data values ​​at a given moment. One of the goals of the ambient condition model 600 is to determine the viewer's current surround, determine which region of the adaptation range the content and / or display is modulating, and then map that portion of the adaptation range to a transfer function that optimally uses the display codes (and the bits required to enumerate them).

[0083] Referring now to FIG. 7 , one embodiment of a process 700 for performing display adjustments based on the dynamic system OOTF is illustrated in flowchart form. The overall goal of some unified display models may be to understand how source material will be perceived by a viewer, on the viewer's display, and in the viewer's surround at a given moment. The display adjustment process can begin by receiving data indicative of a first content item (step 705). For example, the first content item may include encoded display data bound to a particular source color space gamut. In some embodiments, an indicator within the content may specify a particular adaptation algorithm to be used to adapt the content item from the source color space to the display color space and the intended viewing environment. For example, RGB-based gamma for Rec. 709 video, as traditionally applied, often needs to account for mismatches between the content's encoded gamma and the display's decoding response. As another example, the video a viewer wants to view may have been captured in a bright surround and be intended to be viewed in a dark surround, and therefore may include an appropriate gamma boost to accommodate the dark surround of the intended viewing environment. Next, process 700 may perform a linearization process to attempt to remove the gamma encoding (step 710). For example, if the data is encoded with a gamma of (1 / 2.2), the linearization process may attempt to linearize the data by performing gamma expansion with a gamma of 2.2. After linearization, the process will have a version of the first content item data that approximately represents the data as it was in the source color space. Linearization may be required to perform some operations, such as color management and scaling. In some cases, for example, when an extended dynamic range pixel buffer format (e.g., extended dynamic range, EDR) is used, the pixel luminance value may also be divided by a desired reference white luminance value before further processing.The use of the EDR format may be necessary, for example, when SDR and HDR content are to be shown simultaneously on the same display.

[0084] At this point, process 700 may map the linearized data representing the first content item from a first color space gamut associated with the first content item to a second color space gamut associated with the common synthesis space (step 715). In one embodiment, the gamut mapping may use one or more color adaptation matrices. In other embodiments, a 3DLUT may be applied. In some embodiments, one or more pre-synthesized and content-specific tone mapping operations may be applied, as needed. For example, some content may have metadata, gain maps, and / or other affordances associated directly with the content item itself.

[0085] Once the first content item has been mapped to the color space gamut of the common composite space, the mapped, linearized data representing the first content item may be further modified based on at least one of the following: (a) a first difference between a first intended viewing condition associated with the first content item and a second intended viewing condition associated with the common composite space (which may be required to support content items not created for viewing in bright surround), and (b) a second difference between a first intended viewer adaptation level associated with the first content item and a first predicted viewer adaptation level (step 720). The common composite space may include any common composite space that encompasses both the color space gamut associated with that common composite space and any intended viewing conditions for that common composite space (e.g., dimly lit and brightly lit viewing environments, etc.). Any suitable adaptation algorithm may be used to modify each displayed content item relative to the common composite space. This ensures that multiple displayed content items, for example, with multiple encoded gamma boosts and saturation levels, can be adapted to a single system-wide common composite space. For example, a video that a viewer wants to display may include a gamma boost corresponding to an intended viewing environment that is a dark surround, while a word processing document that the viewer wants to view on the same display may include a gamma boost corresponding to a bright surround. If adjustments are applied to content items based on the current ambient viewing conditions around the display device without first transitioning the individual content items to a common composite space, the resulting gamma boost for the video may differ from the resulting gamma boost for the document, and the content items may not be appropriately adjusted for the current ambient viewing conditions. In some embodiments, the common composite space and / or system-wide display parameters may be selected based on the reference environment of one or more content items being displayed. For example, if a majority of the content items correspond to a reference environment with a bright surround, the bright surround reference environment may be selected as the common composite space.In some embodiments, the common composite space may be selected based on the reference environment of the content item determined to be most important. In some embodiments, the common composite space may be selected based on the current viewing environment, reducing the amount of adjustment required to adapt the content item to the current viewing environment. This feature may be useful for stable viewing environments where changes are infrequent or small. For example, the common composite space and corresponding modified display parameters may be an “average” of recent environmental conditions. In some embodiments, system-wide display parameters may then be adjusted based on ambient conditions, e.g., based on an ambient condition model, based on device characteristics, e.g., based on display luminance, reflections, and leakage, and / or according to explicit user settings. For example, in high dynamic range (HDR) content, adjusting the reference white point may reduce the range of luminance levels dedicated to highlights in the high dynamic range content (so-called “headroom”). The resulting modified display data from step 720 may then be encoded (step 725) according to a transfer function associated with the common composite space.

[0086] Referring now to FIG. 8 , another embodiment of a process 800 for performing display adjustments based on the dynamic system OOTF is illustrated in flowchart form. In particular, FIG. 8 details one exemplary process for obtaining content from a common composite space and adapting that content based on current viewing conditions around a display device. First, continuing from step 725 of FIG. 7 , in step 805, process 800 may begin by relinearizing encoded data representing a first content item according to an inverse transfer function associated with the common composite space, i.e., to perform linear processing in the common composite space. Next, process 800 may map the relinearized data representing the first content item from a second color space gamut associated with the common composite space to a third color space gamut associated with the display device (step 810). Next, process 800 may apply a chromatic adaptation operation to the mapped, relinearized data representing the first content item based on the measured white point of the current viewing conditions around the display device (step 815). For example, a chromatic adaptation operation may be employed to move the white point from a nominal white point value (e.g., D65) to a white point that matches the actual current viewing conditions. Finally, if necessary, process 800 may perform a simultaneous contrast adaptation operation on the first content item based on a third difference between the second intended viewing conditions associated with the common composite space and the current viewing conditions around the display device (step 820).

[0087] In some embodiments, rather than performing a simultaneous contrast adaptation operation based on current viewing conditions around the display device in step 820, a "reference preset" may instead be used, e.g., as selected by the viewer or the system. In some cases, for example, when an extended dynamic range pixel buffer format (e.g., EDR) is used, brightness control mapping may be applied to map a reference white value to a desired reference white luminance value before display. Finally, one or more additional ambient adaptation corrections may be applied to the display data, such as, for example, adapting the display black point, if necessary, based on the viewer's predicted adaptation level when content is displayed.

[0088] Exemplary Electronic Devices

[0089] 9 , a simplified functional block diagram of an exemplary electronic device for processing a display, according to some embodiments, is shown. Electronic device 900 can be, for example, a mobile phone, a personal media device, an HMD, a portable camera, or a tablet, notebook, or desktop computer system. As shown, electronic device 900 can include a processor 905, a display 910, a user interface 915, graphics hardware 920, device sensors 925 (e.g., proximity sensor / ambient light sensor, accelerometer, and / or gyroscope), a microphone 930, audio codec(s) 935, speaker(s) 940, communications circuitry 945, image sensor / camera circuitry 950, which may include, for example, multiple camera units / optical sensors with different characteristics (as well as a camera unit housed external to, but in electronic communication with, device 900), video codec(s) 955, memory 960, storage 965, and a communications bus 970.

[0090] The processor 905 may execute instructions necessary to perform or control the operation of the many functions performed by the device 900 (e.g., generating and / or processing signals according to various embodiments described herein, etc.). The processor 905 may, for example, drive the display 910 and receive user input from a user interface 915. The user interface 915 may take a variety of forms, such as buttons, a keypad, a dial, a click wheel, a keyboard, a display screen, and / or a touch screen. The user interface 915 may, for example, be a conduit through which a user can view captured images or video streams and / or indicate a particular frame or frames that the user wants to, for example, play / pause, etc., or have particular adjustments applied (e.g., by clicking a physical or virtual button the moment the desired frame appears on the device's display screen).

[0091] In one embodiment, display 910 may display the video stream as it is captured, while processor 905 and / or graphics hardware 920 evaluate an ambient condition model to determine modifications to the display's transfer function or gamma boost, and optionally store the video stream in memory 960 and / or storage 965. Processor 905 may be a system-on-chip, such as found in mobile devices, and may include one or more dedicated graphics processing units (GPUs). Processor 905 may be based on a reduced instruction-set computer (RISC) or complex instruction-set computer (CISC) architecture, or any other suitable architecture, and may include one or more processing cores. Graphics hardware 920 may be dedicated computing hardware for processing graphics and / or assisting processor 905 in performing computational tasks. In one embodiment, graphics hardware 920 may include one or more programmable graphics processing units (GPUs).

[0092] Image sensor / camera circuitry 950 may comprise one or more camera units configured to capture images, e.g., images indicative of ambient lighting conditions in a viewing environment and that may affect the output of an ambient condition model, in accordance with the present disclosure. Output from image sensor / camera circuitry 950 may be processed, at least in part, by video codec(s) 955 and / or processor 905 and / or graphics hardware 920, and / or a dedicated image processing unit incorporated within circuitry 950. Images so captured may be stored in memory 960 and / or storage device 965. Memory 960 may include one or more different types of media used by processor 905, graphics hardware 920, and image sensor / camera circuitry 950 to perform the functions of the device. For example, memory 960 may include a memory cache, read-only memory (ROM), and / or random access memory (RAM). Storage device 965 can store media (e.g., audio, image, and video files), computer program instructions or software, preference information, device profile information, and any other suitable data. Storage device 965 can include one or more non-transitory storage media, including, for example, magnetic disks and tapes (fixed, floppy, and removable), optical media such as compact disc (CD)-ROMs and digital video disks (DVDs), and semiconductor memory devices such as electrically programmable read-only memories (EPROMs) and electrically erasable programmable read-only memories (EEPROMs). Memory 960 and storage device 965 can be used to hold computer program instructions or code organized into one or more modules and written in any desired computer programming language.For example, such computer program code when executed by processor 905 may perform one or more of the methods described herein. Power supply 975 may comprise a rechargeable battery (e.g., a lithium ion battery, etc.) or other electrical connection to a power source, such as a mains power source, used to manage and / or provide power to the electronic components and associated circuitry of electronic device 900.

[0093] The foregoing description of preferred and other embodiments is not intended to limit or restrict the scope or applicability of the inventive concepts contemplated by applicants. In exchange for disclosing the inventive concepts contained herein, applicants desire all patent rights afforded by the appended claims. It is therefore intended that the appended claims fully cover all modifications and variations that come within the scope of the following claims or equivalents thereof.

Claims

1. 1. A method for displaying content on a display device, comprising: receiving data indicative of a first content item; linearizing the data indicative of the first content item according to an inverse transfer function associated with the first content item; mapping the linearized data representing the first content item from a first color space gamut associated with the first content item to a second color space gamut associated with a common composite space; the mapped linearized data representing the first content item; (a) a first difference between a first intended viewing condition associated with the first content item and a second intended viewing condition associated with the common composition space; and (b) a second difference between a first intended viewer adaptation level and a first predicted viewer adaptation level associated with the first content item; and modifying the image based on at least one of encoding the modified data representing the first content item according to a transfer function associated with the common synthesis space; A method comprising:

2. re-linearizing the encoded data representing the first content item according to an inverse transfer function associated with the common synthesis space; mapping the relinearized data representing the first content item from the second color space gamut associated with the common composite space to a third color space gamut associated with the display device; applying a chromatic adaptation operation to the mapped and relinearized data representing the first content item based on a measured white point of current viewing conditions around the display device; and performing simultaneous contrast adaptation for the first content item based on a third difference between the second intended viewing condition associated with the common composite space and the current viewing conditions around the display device; and displaying the first content item on the display device; The method of claim 1 further comprising:

3. receiving data indicative of a second content item; linearizing the data indicative of the second content item according to an inverse transfer function associated with the second content item; mapping the linearized data representing the second content item from a fourth color space gamut associated with the second content item to the second color space gamut associated with the common composite space; the mapped linearized data representing the second content item; (c) a fourth difference between a third intended viewing condition associated with the second content item and the second intended viewing condition associated with the common composite space; and (d) a fifth difference between a second intended viewer adaptation level and a second predicted viewer adaptation level associated with the second content item; and modifying the image based on at least one of encoding the modified data indicative of the second content item according to the transfer function associated with the common synthesis space; The method of claim 2 further comprising:

4. re-linearizing the encoded data representing the second content item according to the inverse transfer function associated with the common synthesis space; mapping the relinearized data representing the second content item from the second color space gamut associated with the common composite space to the third color space gamut associated with the display device; applying the chromatic adaptation operation to the mapped and relinearized data representing the second content item based on the measured white point of the current viewing conditions around the display device; and performing the simultaneous contrast adaptation for the second content item based on the third difference between the second intended viewing condition associated with the common composite space and the current viewing conditions around the display device; and displaying the second content item on the display device; and The method of claim 3 further comprising:

5. receiving data indicative of ambient light conditions around the display device, wherein the current viewing conditions around the display device are based at least in part on the received data indicative of the ambient light conditions. The method of claim 2.

6. Displaying the first content item on the display device includes:

3. The method of claim 2, further comprising determining an adjustment to a black point, a white point, a system gamma, or a combination thereof, of the display device based at least in part on the first predicted viewer adaptation level.

7. The method of claim 4 , wherein the first content item and the second content item are of different media types.

8. A non-transitory program storage device having instructions stored thereon, the instructions causing one or more processors to: receiving data indicative of a first content item; linearizing the data indicative of the first content item according to an inverse transfer function associated with the first content item; mapping the linearized data representing the first content item from a first color space gamut associated with the first content item to a second color space gamut associated with a common composite space; the mapped linearized data representing the first content item; (a) a first difference between a first intended viewing condition associated with the first content item and a second intended viewing condition associated with the common composition space; and (b) a second difference between a first intended viewer adaptation level and a first predicted viewer adaptation level associated with the first content item; and modifying the image based on at least one of encoding the modified data representing the first content item according to a transfer function associated with the common composite space; Non-transitory program storage device.

9. instructions stored thereon that cause the one or more processors to: re-linearizing the encoded data representing the first content item according to an inverse transfer function associated with the common synthesis space; mapping the relinearized data representing the first content item from the second color space gamut associated with the common composite space to a third color space gamut associated with the display device; applying a chromatic adaptation operation to the mapped and relinearized data representing the first content item based on a measured white point of current viewing conditions around the display device; performing simultaneous contrast adaptation for the first content item based on a third difference between the second intended viewing condition associated with the common composite space and the current viewing conditions around the display device; displaying the first content item on the display device; The non-transitory program storage device of claim 8 further comprising instructions.

10. instructions stored thereon that cause the one or more processors to: receiving data indicative of a second content item; linearizing the data indicative of the second content item according to an inverse transfer function associated with the second content item; mapping the linearized data representing the second content item from a fourth color space gamut associated with the second content item to the second color space gamut associated with the common composite space; the mapped linearized data representing the second content item; (c) a fourth difference between a third intended viewing condition associated with the second content item and the second intended viewing condition associated with the common composite space; and (d) a fifth difference between a second intended viewer adaptation level and a second predicted viewer adaptation level associated with the second content item; and modifying the image based on at least one of encoding the modified data representing the second content item according to the transfer function associated with the common composite space; 10. The non-transitory program storage device of claim 9, further comprising instructions.

11. instructions stored thereon that cause the one or more processors to: re-linearizing the encoded data representing the second content item according to the inverse transfer function associated with the common synthesis space; mapping the relinearized data representing the second content item from the second color space gamut associated with the common composite space to the third color space gamut associated with the display device; applying the chromatic adaptation operation to the mapped and relinearized data representing the second content item based on the measured white point of the current viewing conditions around the display device; performing the simultaneous contrast adaptation for the second content item based on the third difference between the second intended viewing condition associated with the common composite space and the current viewing conditions around the display device; displaying the second content item on the display device; 11. The non-transitory program storage device of claim 10, further comprising instructions.

12. instructions stored thereon that cause the one or more processors to:

10. The non-transitory program storage device of claim 9, further comprising instructions to: receive data indicative of ambient light conditions around the display device, wherein the current viewing conditions around the display device are based at least in part on the received data indicative of the ambient light conditions.

13. The instructions for displaying the first content item on the display device may include instructions that cause the one or more processors to:

10. The non-transitory program storage device of claim 9, further comprising instructions for determining an adjustment to a black point, a white point, a system gamma, or a combination thereof, of the display device based at least in part on the first predicted viewer adaptation level.

14. A device, Memory and A display device; one or more processors operably coupled to the memory, wherein the one or more processors: receiving data indicative of a first content item; linearizing the data indicative of the first content item according to an inverse transfer function associated with the first content item; mapping the linearized data representing the first content item from a first color space gamut associated with the first content item to a second color space gamut associated with a common composite space; the mapped linearized data representing the first content item; (a) a first difference between a first intended viewing condition associated with the first content item and a second intended viewing condition associated with the common composition space; and (b) a second difference between a first intended viewer adaptation level and a first predicted viewer adaptation level associated with the first content item; and modifying the image based on at least one of and configured to execute instructions to cause the modified data representing the first content item to be encoded according to a transfer function associated with the common composite space. device.

15. Instructions stored in the memory, the instructions causing the one or more processors to: re-linearizing the encoded data representing the first content item according to an inverse transfer function associated with the common synthesis space; mapping the relinearized data representing the first content item from the second color space gamut associated with the common composite space to a third color space gamut associated with the display device; applying a chromatic adaptation operation to the mapped and relinearized data representing the first content item based on a measured white point of current viewing conditions around the display device; performing simultaneous contrast adaptation for the first content item based on a third difference between the second intended viewing condition associated with the common composite space and the current viewing conditions around the display device; displaying the first content item on the display device; The device of claim 14 further comprising instructions.

16. Instructions stored in the memory, the instructions causing the one or more processors to: receiving data indicative of a second content item; linearizing the data indicative of the second content item according to an inverse transfer function associated with the second content item; mapping the linearized data representing the second content item from a fourth color space gamut associated with the first content item to the second color space gamut associated with the common composite space; the mapped linearized data representing the second content item; (c) a fourth difference between a third intended viewing condition associated with the second content item and the second intended viewing condition associated with the common composite space; and (d) a fifth difference between a second intended viewer adaptation level and a second predicted viewer adaptation level associated with the second content item; and modifying the image based on at least one of encoding the modified data indicative of the second content item according to the transfer function associated with the common composite space; The device of claim 15 further comprising instructions.

17. Instructions stored in the memory, the instructions causing the one or more processors to: re-linearizing the encoded data representing the second content item according to the inverse transfer function associated with the common synthesis space; mapping the relinearized data representing the second content item from the second color space gamut associated with the common composite space to the third color space gamut associated with the display device; applying the chromatic adaptation operation to the mapped and relinearized data representing the second content item based on the measured white point of the current viewing conditions around the display device; performing the simultaneous contrast adaptation for the second content item based on the third difference between the second intended viewing condition associated with the common composite space and the current viewing conditions around the display device; displaying the second content item on the display device; The device of claim 16 further comprising instructions.

18. Instructions stored in the memory, the instructions causing the one or more processors to: receiving data indicative of ambient light conditions around the display device, wherein the current viewing conditions around the display device are based at least in part on the received data indicative of the ambient light conditions. The device of claim 15 further comprising instructions.

19. The instructions for displaying the first content item on the display device may include instructions that cause the one or more processors to: determining an adjustment to the black point, the white point, the system gamma, or a combination thereof, of the display device based at least in part on the first predicted viewer adaptation level; The device of claim 15 further comprising instructions.

20. The device of claim 17 , wherein the first content item and the second content item are of different media types.

Images